Page 1 of 2 12 LastLast
Results 1 to 10 of 16

Thread: CWAD (AKA: Compressed, Where's All the Data)

  1. #1

    CWAD (AKA: Compressed, Where's All the Data)

    WARNING: Long winded prelude to the actual content...
    Scroll to the second post for the code segment.


    What is a CWAD?
    C = Compressed
    WAD = "Where's All the Data" (Normally uncompressed RAW data used by a CORE program. Popularized by DOOM.)

    Actually, a WAD is like a ZIP or RAR file. It is one file that contains many other files. With one small exception, the data is structured with an intent. Images are a block, sounds are a block, animation is a block, levels are a block, objects are a block, monsters are a block, weapons are a block, scripts are a block... etc...

    This is stolen from an OLD OLD OLD concept that we normally call a "Database". (Part of the reason ZIP and RAR had a hard time getting patents for the "Stolen Concept". They were limited to "Specific format", and "Processing of data", in the patents.)

    Any-who...

    I have seen a lot of places where this TYPE/STYLE of structure could play a great role in overall development here with ThinBASIC. Much of the data we use, demands us to organize it, index it, and attempt to ensure it is all unique.

    This has led me to come-up with this general structure and concept of the CWAD. (The compressed version, which has obvious gains in this power-of-two expanding world.)

    I will use "Graphics" as the first example...

    Games which once used only a hand-full of 256x256 graphics, now use thousands of 1024x1024+ graphics, each year was getting exponentially larger. However, we have reached what is called a "Creative edge". Millions of images can not be created by one entity, so now we work in reverse. We are heading back towards the 256x256 graphics, which allows the quantity to grow as we grow. EG, What would have been one large image creating the entire side of a ship. Is now hundreds of images, which we originally used to create the rendered image of the side of the ship. The painted metal, the rusty-holes, the rivet-lines, the stained drip-marks, the dirty scuff-marks of wear, the barnacle patches.

    Why are we doing this? Because computers which were once limited to X surfaces (Triangles), are now able to handle 100 times more than they once could. Plus, new "Tricks" have been developed, which reduces the number of actual X surfaces (Triangles), we actually see. This gives us the advantage of using many smaller images to create larger images, as desired. (Like a mini-rendering tool inside of the game.)

    For the second example I will use animation...

    What was once a simple, push, pull, move, and slide, is now a complex chain of stacked events. When you type on the keyboard, your finger moves up and down and left and right. That is added to the motion of the other fingers, and your wrist. That is added to the motion of your arm as you reach for the far keys. That is added to your eye-motion as you look from screen to keys. That is added to your facial expressions as you squint your eyes, and raise your brow to insight. That is added to the pulsing vein that raises under your skin as you think. That is added to the chest rise and fall as you breath. That is added to the head-tilt which you do, as you contemplate how to finish your paragraph.

    Imagine all that done as one long line of code... Now you have to do the next animation, where you get-up for coffee. That could take another day to program, and you never even finished the paragraph.

    This is why we use editors. They "Script" animation, as independent blocks of data. That data can be compounded to form one split-second animation, which reacts to each other animations results. You program "Scratch nose", which is an animation of, "Bend finger", and "Raise arm", and "Bend wrist", and "Wiggle finger"... Followed by nothing (Which returns your arm/finger back to the last position, or leads it into a new animation.)

    Those individual animations are saved as a script, and those script-animations are saved as a script. And so-on, and so-on... Script on script on script... That is a "Generic" form of manual compression, but the scripts are still long, and can gain from further program compression. One or two scripts is only a couple bytes, but compounded millions of scripts and motions can easily be into the MB reach.

    Here is where it becomes an advantage for us.

    The program doesn't care that "Bone45" is called, "Right index finger", nor does the viewer. You have to create all these long unique names, which is a task in a task. Or, you could just throw them away, and save hours of wasted time. Since that is what the game does after you wasted hours creating all of them. (Not to mention, you have to creatively craft many conflicts, which throws many standards out the door, or makes them a total non-advantage.)

    Now picture this...

    You don't code every door and window and wall by hand. You don't track "Floor_Tile_Green_Dirty_Chinese_01.bmp". You don't even know what that image is, until you look at it! (If you had four tiles, it would be possible, but most games have more than four floor styles, each with four floor tile variations, and possibly separate sizes of each.)

    What you do, is this. You create "Ship", the name and object is irrelevant. That "Ship" has 45 images, (paint01, paint02, paint03, paint04, deck1, deck2, door1, door2, door3, door4, window1, window2, window3...)

    Once you create it, it gets added to the the WAD as, ObjID#7 (I#432,I#434,I#120,I#435,I#436...)

    You notice the third image... "paint03" was not a new image, but one that already exists in the WAD, and it was used, as opposed to creating another new image-block. When programming the game, if not using a script, you open the WAD, and see that ObjID#7 is the object you want, or you can actually associate a unique name with that. "Ship", which would return the ID# for use in the game, which is what the game code expects, a model ID#.

    Here is where it can get real interesting.

    ObjID#7 can be ObjID#2's Model, with ObjID#2's, ObjID#4's, ObjID#7's Images.

    Ok, hard to visualize... Model 7 was the same model ship as Model 2, but you just changed images, some which belonged to the WAD model 2 and 4, plus some new images. So, you just created a "Variation" of Model 2, which is ObjectID#7 in the game.

    EG, To change the tires on a car should not be a complex in-game task, but this still allows that. You just have to know how many modelID's there are in the WAD, and all your "Custom" styles would be that number + 1, 2, 3, 4. Instant "Unique" control, without funky stacked names, which the game doesn't care about anyways.

    You need a crowd with 100 people, all doing separate animations, and having unique bodies, clothing styles, heads, motions... You could set them all up in the WAD, if you want instant random crowd creation. But you would be better using the code to manage those objects. Using the object-parts inside the WAD as your source.

    Another advantage is... If you replace/update images or the object or the models... you instantly get notified of a name-conflict, (Which is still irrelevant if you don't use names), and all associated images, objects and models become updated. (Add animations, or user-data to that as well.)

    Now... For simple control, and to keep this first release down to a minimum. I am only going to deal with the two item/sections that will possibly be used the most. Since there are only a few "Standard" formats involved, this should be rather easy to launch.

    I do expect versions to change, but that should only impact the data-storage, and available output. I do not expect this to be used directly inside of code, as RAW control... Though, it can be, if you reduce it to only the parts you need for your program. (It is my hope, that this eventually turns into a MODULE. For now, it will operate as an INCLUDE or as a separate program.)

  2. #2

    Re: CWAD (AKA: Compressed, Where's All the Data)

    Here is what I have setup for a generic format, for version 1

    Flow goes something like this... (Similar to a folder-crawl)

    CWAD_Header
    CWAD_Model_Index
    - Model_File 1
    - - Model_Header 1
    - - - Model_RAW 1
    - Model_File 2
    - - Model_Header 2
    - - - Model_RAW 2
    CWAD_Image_Index
    - Image_File 1
    - - Image_Header 1
    - - - Image_RAW 1
    - Image_File 2
    - - Image_Header 2
    - - - Image_RAW 2

    But the actual data is stored like this...

    CWAD_Header
    CWAD_Model_Index
    CWAD_Image_Index
    - Model_File 1
    - Model_File 2
    - Image_File 1
    - Image_File 2
    - - Model_Header 1
    - - - Model_RAW 1
    - - Model_Header 2
    - - - Model_RAW 2
    - - Image_Header 1
    - - - Image_RAW 1
    - - Image_Header 2
    - - - Image_RAW 2

    Code removed, pending format change. (More flexible code.)

  3. #3

    Re: CWAD (AKA: Compressed, Where's All the Data)

    The data is able to be constructed into two parts, if desired.
    DATA-Header: This is user created, formulated from the file, extracted from the file, or just RAW-UNCOMPRESSED data.
    RAW-DATA: This should be the original data, which will be compressed. But this can be a formulated portion of the data.

    The (Header) will always be uncompressed, but is not required to hold any information.
    The (RAW) will always be compressed, but is not required to hold any information.
    One or the other MUST be present, or the "File" does not exist, and will be removed from that pack.

    The (Header) can contain two parts. The first part must be "FIXED-LENGTH" data. The second part must be "FIXED-SIZE-VALUES". One value in the first part, must indicate the quantity of "FIXED-SIZE-VALUES", and another value in the first part must be named "StepOver", and hold the value of total "Bytes" which must be stepped over, in addition to the header, to get to the beginning of the compressed data. (That will make more sense later.)

    For a simple example...

    (CWAD Head) Has 4 OBJ
    (CWAD Model_Index) "House" = #1, "Tree" = #2, {Each at byte position X, and is Y bytes long.}
    {"House" @ X to Y}
    - (Model_Head #1) Ver:1006, Vert:[3] {"Lawn"}
    - - (Model_RAW #1:1) X,Y,Z,...blah blah...
    - - (Model_RAW #1:2) X,Y,Z,...blah blah...
    - - (Model_RAW #1:3) X,Y,Z,...blah blah...
    {"Tree" @ X to Y}
    - (Model_Head #2) Ver:1006, Vert:[2] {"Tree"}
    - - (Model_RAW #2:1) X,Y,Z,...blah blah...
    - - (Model_RAW #2:2) X,Y,Z,...blah blah...


    For a more complex example...
    (CWAD Head) Has 4 OBJ
    (CWAD Model_Index) "House" = #1, "Tree" = #2, {Each at byte position X, and is Y bytes long.}
    {"House" @ X to Y}
    - (Model_Head #1) Ver:1006, Vert:[3] {"Lawn"}
    - - (Model_RAW #1:1) X,Y,Z,...blah blah...
    - - (Model_RAW #1:2) X,Y,Z,...blah blah...
    - - (Model_RAW #1:3) X,Y,Z,...blah blah...
    {"Tree" @ X to Y}
    - (Model_Head #2) Ver:1006, Vert:[2] {"Tree"}
    - - (Model_RAW #2:1) X,Y,Z,...blah blah...
    - - (Model_RAW #2:2) X,Y,Z,...blah blah...
    (CWAD Image_Index) "Roof" = #1, "Leaves" = #2, {Each at byte position X, and is Y bytes long.}
    {"Roof" @ X to Y}
    - (IMG Index #1:1) RAW-ICON BMP 32x32
    - - (IMG Index #1:2) FULL RAW FULL IMAGE DATA
    {"Leaves" @ X to Y}
    - (IMG Index #2:1) RAW-ICON BMP 32x32
    - - (IMG Index #2:2) FULL RAW FULL IMAGE DATA
    Attached Images Attached Images

  4. #4

    Re: CWAD (AKA: Compressed, Where's All the Data)

    How am I expecting this to function?

    Similar to this...

    You want your "Tree" loaded... (This is an object)
    1. You call the function to loadCWADobj("Tree") or loadCWADobj(2).
    2. It returns a value related to the NEW object SlotID where the object was loaded.
    3. You can modify that object, by saying swapIMG(SlotID,"Leaves","Grass").
    4. It returns (0) if it works. The SlotID will now hold the modified object.

    The above actions need to be controlled by the CWAD, due to the fact that the data is in the CWAD. However, individual data can be pulled from the CWAD, and fully managed by you.

    You want to create your "Tree" by yourself...
    1. You call the function to getCWADobj("Tree") or getCWADobj(2).
    2. It returns the DataID and ImageID for further manipulation. DataID is the RAW-M15, ImageID is a CSV image-list.
    3. You can alter the image-list to use FILES, and alter the M15 to load as an altered FILE. (Both written to the HD for now.)
    4. Alternately, you can manipulate the M15 and save as a FILE, but pass the IMAGE ID's back, or altered, along with the FILE LOCATION... and have the CWAD load the altered DATA.

    loadCWADobjFILE(MyFile) {CWAD will expect images contained in the WAD, but will try to load files if no ID#'s are used. That allows you to Mix CWAD with CUSTOM images. Where you may want to impose a screen-shot, or game-altered graphic.}

    There would not be a (MyFile,MyImages), because that is the native function of TBGL_M15LoadModel, and has no dependant use of CWAD data.

    This is not a game-manager, only a CWAD manager. The only functions "Built-in" would be specific to the types that are "Built-in". You have the ability to manipulate any data inside, but be aware that certain manipulations related to replacement and removal would be permanent to your data.

    The "Custom" area can use your own format, except for the "Header", and "Name" area. You can use the Name area optionally, so you may have it contain any data, or no data. It will always be Qed by the Name commands. You can technically throw CWADs inside CWADs... But you will loose performance until they are all expanded. One reason you MAY want to do that, would be for "Quality Control", or to get more "Custom Areas", or use them for "Game Progression", to speed-up loading from single large CWADs.

    Let me know what you think... The image portion is almost complete, and nearly ready for use. (Working on the M15 will be a little more of a challenge.)

  5. #5

    Re: CWAD (AKA: Compressed, Where's All the Data)

    I forgot to add that each area is modular, and can be included or removed, and potentially have separate versions related to them. (Where custom creation is desired.)

    The OBJ area may later have OBJ2, or you can obviously add your own as OBJx. (That should not be an issue for updated versions, as you would be required to "Update" as well. If Version1.0.0.0 did not have OBJ2, but you added OBJ2, that would not conflict with Version1.0.0.1 if it DID have OBJ2, because the data-version would not match. Your updated code would have to be called something new, but if called FROOBJ, it would still function the same. It would know to look for OBJ2 with an older version, as the "Owned" custom data. Just as the NEW OBJ2 would not attempt to load older version data that it was not designed to read.)

    EG, You should ignore any data that is not EXACT, or IN-RANGE for the custom code you create. (Otherwise you will be reading other data that may NOT be yours. Non expected data.)

    Other proposed areas of development may include... (Most of these are script data.)
    - Animation-Scripts (For multi-segmented animations and linear flow and forced motion.)
    - Camera-Tracks (Variation of an animation-script with non-linear flow and non-forced motion.)
    - AI-Paths (For those who use branching and thinking AI on non-linear paths.)
    - Scenes (Containing many objects, but not an object itself.)
    - Rooms (Large segmented objects, acting as one object.)
    - Light-Grids (Segmented lighting control, for deep games, to reduce overhead programming.)
    - Sounds (Um, for sounds in the game.)
    - Effects (Um, for visual effects in the game.)

    Many are specific to games, again, but the uses are endless... Network-logging, Backup-tracking, Update-tracking, Project-development, Presentation-display...

  6. #6
    thinBasic author ErosOlmi's Avatar
    Join Date
    Sep 2004
    Location
    Milan - Italy
    Age
    57
    Posts
    8,777
    Rep Power
    10

    Re: CWAD (AKA: Compressed, Where's All the Data)

    Jason,

    your idea is very interesting.
    There were some discussions few months ago about the possibility to store data needed by TBGL into a ZIP like file but at that time there was no real need.

    Maybe argument for a specific thinBasic module.

    Ciao
    Eros
    www.thinbasic.com | www.thinbasic.com/community/ | help.thinbasic.com
    Windows 10 Pro for Workstations 64bit - 32 GB - Intel(R) Xeon(R) W-10855M CPU @ 2.80GHz - NVIDIA Quadro RTX 3000

  7. #7

    Re: CWAD (AKA: Compressed, Where's All the Data)

    Well...

    I am thinking for large-scale ability. Since TBGL can handle it.

    I just finished reformatting the code for a more modular and custom arrangement. (The format above was sort-of locked-in at the first header.)

    Now, it acts like a file-system, at the core. Expansion only demands that the initial "Header" of any "Pack", exist and contain the critical elements. The individual "File-headers" are a free-format, as is the "File-Raw-Data". The only limitation for a header, is that if it is NOT fixed-length... It must include the "Step-Over" type-value, which gets added to the header-length, so it knows where the compressed segment begins. (You don't have to use both. Data can be in one or the other or both, as long as the correct values are set.)

    Now all I have to make is a quick GUI to interface the code, and display the results. (Adding items to, and removing them, and displaying the contents of the "Packed CWAD". As well as a demonstration of how to use the pack in an actual program, as the read-only, and using some internal functions.)

    I still imagine the greatest need for this, will come with larger format creations. For simple use, with only a hand-full of items, images, and scripts, this will offer no interest. This is why I am allowing such a wide range of potential values.

    However, at the moment, the largest single CWAD size can only be about 2GB. This is due to the use of the LoadFile() command. But that should be within everyone's reach.

    Personally, I just want it for image-swapping of models. The pre-rendered 32x32 icons make a nice touch to the ImageThumb portion of the CWAD. This also fixes the issue with the SketchUp OBJ conversion issue, where colors and images export as layers, and not image-maps.

    Ok, I need some sleep now... I am seeing cross-eyed!

  8. #8
    Super Moderator Petr Schreiber's Avatar
    Join Date
    Aug 2005
    Location
    Brno - Czech Republic
    Posts
    7,128
    Rep Power
    732

    Re: CWAD (AKA: Compressed, Where's All the Data)

    Hi Jason,

    thanks for your input.
    I will be able to read it in depth during weekend ( that is not bad joke related to textlength, just school crunch time now )


    Thanks,
    Petr
    Learn 3D graphics with ThinBASIC, learn TBGL!
    Windows 10 64bit - Intel Core i5-3350P @ 3.1GHz - 16 GB RAM - NVIDIA GeForce GTX 1050 Ti 4GB

  9. #9

    Re: CWAD (AKA: Compressed, Where's All the Data)

    Quote Originally Posted by Petr Schreiber
    I will be able to read it in depth during weekend
    Sure... LOL... If I had a Quarter... :P

    Just do me a favor... and don't change the format of the M15 for a while... ;D

    Though, an A15 or B15 file would go hand in hand with the M15. (Animation and Bones)

    I saw there was a way to "Rig" the M15 format. But when you get the collision module completed, this could help link all of them into one complete system. (With the Animation and Bones not being part of the GL collection, because there might eventually be a DX module that could use those same formats, but have no use for the M15. Plus, many 2D items use bones and animation also.)

    Ok, back to work, before I drift-off again.

  10. #10
    thinBasic MVPs kryton9's Avatar
    Join Date
    Nov 2006
    Location
    Naples, Florida & Duluth, Georgia
    Age
    67
    Posts
    3,869
    Rep Power
    404

    Re: CWAD (AKA: Compressed, Where's All the Data)

    Jason's explanation is really something and in reading it I kept thinking of working with the Unreal 2004 editor and it seemed to go hand in hand.
    I think if any of you play with any of the modern game editors this sort of paradigm fits right in. It is easier to see and use than to write about and read about.

    Jason you could even add custom scripts as you do graphics and models, for behaviors and physics as they do in unreal.
    Acer Notebook: Win 10 Home 64 Bit, Core i7-4702MQ @ 2.2Ghz, 12 GB RAM, nVidia GTX 760M and Intel HD 4600
    Raspberry Pi 3: Raspbian OS use for Home Samba Server and Test HTTP Server

Page 1 of 2 12 LastLast

Similar Threads

  1. USB Datapens and Compressed bundles
    By Michael Clease in forum thinBundle bugs report
    Replies: 4
    Last Post: 30-09-2009, 17:19

Members who have read this thread: 0

There are no members to list at the moment.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •