How do the games handle loading from MIX?

[Blade]

I'm curious as to how the game actually treats MIX files when it loads from them, does it just load the indexes into memory and add pointers to them in an array for when it iterates through looking for a file and treat the mix as a virtual file systems, loading individual files they contain off disk or does it load the entire thing into memory first and then just address chunks of it as it needs the file the chunk represents? My guess would be something more like a vfs, expecially for larger files like the vqas. I gather deciding if a file is actually in a mix is done via binary search given the headers must be sorted by id number.

[Nyerguds]

Uhh... in which game(s), specifically?

 

As far as I can see, there are several methods for searching for files; some that only search in the mix files, and some that also check the game folder. As for actual reading, though, I think the Westwood file class simply doesn't make a difference between a file pointer of a file on disk, and a file pointer of a file in a mix file. It just reads the contents from disk from the given pointer, knowing the beginning and end of that "file".

 

How files are actually handled in terms of loading into memory obviously depends on the file type. VQA videos are streamed, meaning small blocks are loaded into memory and played, and then replaced by the next read block. The mouse cursor is loaded on game start, and so is the strings file, and they stay in memory during the whole game. I think most of the ingame SHP files, of units/structures/trees/icons/etc, are loaded into memory when loading the theater, just before the mission starts, and stay in memory as long as the mission runs, only removed as they are replaced by the next theater load. Ini files are loaded into memory too, because lots of things need to be read from them.

 

None of that is in any way related to how files in mix files are treated in general, though. The actual system to access files inside mix files and treat them as file objects has no reasons whatsoever to go caching files in memory. That's simply not its function. It just gives a read pointer.

 

As for the specifics, I know each mix load in C&C1 is two functions: a "Retrieve" and an "Offset". So I assume the first is the loading, and the second is the indexing of its contents. Though, the "ss*.mix" file-removing system works by only loading a file, but skipping the "Offset" call. Since the files list inside the mix is clearly already read in this case (otherwise it wouldn't disable the files), this means that the original "Retrieve" clearly already reads the files table. I assume "Offset" might actually create some list of premade file pointers for the game to actually read these files.

 

To access a file, a "Finder" function is called, which, I presume, reads the loaded files tables, and retrieves the corresponding pointer information. (Which is not generated in case of these ss*.mix files)

 

I'm mostly just guessing here, though. CCHyper probably knows more of the details of this. I remember he once told me that ss*.mix probably has quite a different function, but is simply malfunctioning in C&C1. Not my problem, though, since it gives me a handy tool for more customized patching

[Blade]

I was interested in TD and RA mainly. So basically the difference between the mix and a normal dir is abstracted for most of the game functions and files are loaded when the game requires them?

[Nyerguds]

Yeah. Hyper just said on the cncnet developers irc that C&C does in fact have caching functions, but in C&C95 (the only game whose internals I'm familiar with, really) it's inside the Offset one. Here's the chat log:

 

CCHyper:

That's pretty much it, but there is a Cache function, that does not appear in C&C95 I think because of the way C&C95 was compiled.

C&C95 does have it, but you won't have it named, because C&C95 was compiled with some stupid switch that makes it produce crazy code.

So Cache is a part of Offset rather than separate, IIRC.

The game allocates an instance per mix file and then reads the head and uses a generic linker node/list system to store all the files referenced in the header.

When searching for files (Finder->List) it does a loop through the loaded node lists and then compares the filename CRC with the CRC generated by the Finder function. if it matches, then return that file pointer.

MixFileClass is not really a traditional file class, but more of a "how-to" alternative for CCFileClass to find stuff

RawFileClass is the base of all files classes (that works), it handles all file handling and when used itself only handles raw data/data in the game folder or operating system file structure.

CCFileClass is based on RawFileClass and also searchs mix files. as above

CDFileClass is based on CCFileClass and adds handling for the CD switching and CD path searching. -CD logic you see in the previous games uses this and allows the game to search a for files in a list of paths, -CD accepts multipule entrys (-CDpath;gdi,nod;test)

BufferIOFileClass does what the name says, used along with CCFileClass as it buffers MixFile stuff for fast retrieve

RAMFileClass creates the file (from raw/hardcoded data mostly) in the RAM/memory, very useful (see the Init_Keys() logic)

MixFileClass is for handling opened mix files

You use RawFileClass to load a mix file from the directory, or CCFileClass to load a mix file within a mix file or from the directory (as CCFileClass is based on RawFileClass)

So all in all, its best to always use CCFileClass, as the game will find your file no matter where it is.

 

Nyerguds:

What is the difference in loading between files you can and can't load from game folder? Like, .aud music and .vqa files work from the game folder, but .SHPs don't. What is it using for these shp files then?

 

CCHyper:

That's actually a interesting thing, as I fixed the code in TS to load VQA's from the raw directory, as the game referenced a pre loaded file, rather than a raw file. Might be the case in C&C, RA too.

TS basic searched the mix files first for the VQA, and if it didnt find it, it return null, so the VQA File Handler parsed the file loaded from the mix files or the null pointer. I made it reload the file and check if its available from the raw directory first before using the preloaded instance.

[Blade]

I'm surprised to hear that it uses a linked list when searching the header for a file present, when I discovered that the games all need the index sorted by file ID I had guessed that it would use binary search to determine if a file was present or not. I guess it just searches until it finds an ID higher than the one its looking for or reaches the end of the list before moving on to the next file. Once the game has loaded a mix, does it keep the file handle open to service all access to that file or does it just open it when its required? Or as before does it depend on what the mix contains?

[CCHyper]

I could be wrong in some areas as I have only recently looked into the file structure of the C&C games.

 

The game just loops through the loaded file data (header information for each file) and matches the ID to the ID created from the input filename. I have not seen anything like a binary search...

[Blade]

I'm just going by the empirical evidence I got from making a tool to create mix files. Until I added a sort to the index, XCC would read from the generated files fine, but the game would act like most of the assets in the mix were missing so I figured it must rely on the order to improve searching performance rather than data structures like map which is what my tool uses and I believe XCC does too.

[Nyerguds]

Ah yes, I noticed that in TS too, when I wrote a tool to fix the names of the intro videos in the TFD TS movies mix files... they needed to be sorted by ID.