Blade

Anyone know how the map data is stored in the bin file for sole survivor? I take it its not a one to one mapping of two bytes to every cell like C&C maps since the files are all different sizes?

What kind of math does the engine use to get the final index to use for a pixel when using these tables? I've been doing a bit of reading on old school 8bit blitting engines and how they tried to avoid using ifs to decide if a pixel was transparent and instead used bitwise operations and lookup tables which is how I'm imagining these work.

If the game reads them for the chrono votex, won't the exe have strings for their file names?

Does the game ever use them? Any idea what the file names are?

Anyone know what the 48 files or so are that XCC marks as unknown in local.mix? Are they all the same as they appear to be and are they for anything?

I'm just going by the empirical evidence I got from making a tool to create mix files. Until I added a sort to the index, XCC would read from the generated files fine, but the game would act like most of the assets in the mix were missing so I figured it must rely on the order to improve searching performance rather than data structures like map which is what my tool uses and I believe XCC does too.

I'm surprised to hear that it uses a linked list when searching the header for a file present, when I discovered that the games all need the index sorted by file ID I had guessed that it would use binary search to determine if a file was present or not. I guess it just searches until it finds an ID higher than the one its looking for or reaches the end of the list before moving on to the next file. Once the game has loaded a mix, does it keep the file handle open to service all access to that file or does it just open it when its required? Or as before does it depend on what the mix contains?

I was interested in TD and RA mainly. So basically the difference between the mix and a normal dir is abstracted for most of the game functions and files are loaded when the game requires them?

I'm curious as to how the game actually treats MIX files when it loads from them, does it just load the indexes into memory and add pointers to them in an array for when it iterates through looking for a file and treat the mix as a virtual file systems, loading individual files they contain off disk or does it load the entire thing into memory first and then just address chunks of it as it needs the file the chunk represents? My guess would be something more like a vfs, expecially for larger files like the vqas. I gather deciding if a file is actually in a mix is done via binary search given the headers must be sorted by id number.

If anyone knows the names of any other files that XCC doesn't know I can add them too, just send them to me in a text file one per line in the format "filename.ext,description" without the quotes.

Here is an updated global mix database.dat file that has many of the sole survivor file names XCC normally doesn't know. global_mix_database.dat.zip

I don't know really, without knowing what the algorithm is its hard to say what information we would need to construct a working encryption system. I have a gut feeling it must be something like RSA with a Westwood twist, but who knows. It also seems that the game checks the recovered key somehow since the game won't read mix files encrypted with a randomly generated 80byte block but XCC will implying there is some signature or pattern in the key that the game wants.

For my own curiosity, have you documented the file format for .mrf somewhere?

Those aren't the blowfish keys, those are the keys that encrypt and decrypt the block within the mix file header to the actual blowfish key (which seems to have been generated on a per mix basis). Blowfish isn't a public key crypto, its symetric so the key that encrypts also decrypts. Basically the blowfish key itself is encrypted with an unknown public key crypto scheme (which key.ini has the keys for) and then stored in the mix file just before the encrypted header itself. XCC mixer has the implementation for decryption but I can hardly fathom its big integer math. I doubt others can either because I've yet to see an implementation that isn't pretty much just a copy of the XCC one. The tools that write encrypted mixes get round not being able to generate new blowfish keys by basically dumping the whole 80byte block that is the encrypted form of the key and stick it on a mix that has been blowfish encrypted with the key that the block decrypts to. I hope that all made some kind of sense.

If anyone is still interested in a command line tool, I've attached a new windows build that needs testing. In theory it provides the following features: Can extract and create mix files for any of the main C&C games, including encrypted using the bundled key.source. Can handle mix files where the filenames aren't known when extracting and then give them the correct id when rebuilding (names files with the format [id]ABCDF987 where the characters after [id] are treated as hex). Can add and remove single files while preserving the attributes of the file being added to. Allows optional use of the local mix database.dat extension for XCC compatibility and to allow easier recovery of filenames if originals are lost. Don't try add and remove on mix files you aren't willing to loose. Although I've ran some tests and it seems to work okay, I can't guarentee it won't corrupt or overwrite data. ccmix_win32.zip

Blowfish is symmetric, so the implementation both encrypts and decrypts with a given key and we have a working implementation so it doesn't matter all that much if its custom or not, but it does affect if the public key crypto could be a standard one or not I guess. The checksum appears to be standard sha1 though. The blowfish key is what is encrypted with the unknown public key crypto with BlowfishKeyProvider being the decryption algo for the public key crypto and is different for every westwood created encrypted mix. We have to make do with reusing an encrypted key from an existing encrypted mix file as the game won't accept a random key made from creating a random 80 bytes for the encrypted key block (Olaf's docs call this block a "key source") and "decrypting" it. Using an existing key source is how XCC and ccmix create encrypted mixes that the game will acceptv as the game doesn't know that we have reused a key, it just knows its a proper westwood key. Without knowing 1. The criteria the game uses to verify the key and how to generate one and 2. The algorithm to encrypt that key into a key source, we cannot make uniquely encrypted mix files. From a user POV, I guess it doesn't really matter as they can create working encrypted mix files, but for curiosities sake I would like to know.

I haven't seen any tool that reads the encrypted mixes that doesn't use that decryption code and it wouldn't surprise me if it was a slightly tidied up decompilation of the games own functions for it. Its obviously some kind of pubic key crypto given that we have the public and private keys for it, but what algo it is I couldn't say. What algo's were available around 1995 and were used with blowfish for message encryption?

I've attached a windows build of the tool which requires more testing but should be able to create mixes in any of the 3 formats (td, ra or ts) and do encryption, checksums and local mix databases for those that support them. Adding and removing random files is not supported yet, and neither is converting unencrypted to encrypted directly or vice versa. Rough use instructions are on the github page, --checksum when creating will add a checksum to the new mix and --add --checksum with an existing mix will add one, not that the game seems to care. edit: updated exe to one that works extracting all files again. ccmix.zip

Well, I took a break from looking at the encryption to look at the checksum function and after very little effort identified it as a SHA1 digest of the body of the mix, which is good since it was the only hash function that was around when RA was written that generated 20 byte digests that came up in google when I searched.

It can't be just one key, because each encrypted mix seems to have a different key source included which is the blowfish symmetric key encrypted with some public key crypto (which we have the public key for thanks to key.ini). Olaf has reverse engineered the decryption algo as XCC and tools based on it (such as openra's code, my code and such) can recover the blowfix key and then decrypt the header. XCC Mix Editor doesn't generate new Blowfish keys and encrypt them with the private key (which we also have from key.ini) as far as I can tell though since it requires you to load a key from an existing mix file first. My program (https://github.com/OmniBlade/ccmix if you are interested) can generate working encrypted mix files in the same way, by using a key_source from an existing mix, I just would like to be able to generate new ones as well. Just randomly generating a 80byte block as I naively tried either breaks the decryption algo in the game or the "key" it recovers isn't considered a valid blowfish key as far as the game is concerned.

FunkyFr3sh, Sonarpulse's work doesn't really help me because 1, it's written in a language I have no experience of and 2, it doesn't appear to implement encrypted mix reading or writing anyway, it looks like it just has stubs. OpenRA can read encrypted mix files, but that doesn't help me because I want to write them. Only XCC Mix Editor does and even that seems to just grab a key_source from an existing mix which I can do, but it would be nice to be able to generate new ones. If I better understood the encryption code or how the code retrieves a key from the keyblock I might have some insight into why the game has problems with random key blocks, but just looking at the code with all the bit shifting gives me a head ache. Thanks Iran, shame it doesn't give more insight into the reason for the crash

I'm trying to write a command line tool to write encrypted mix files and to some extent I have succeeded by using a key source (as described in the XCC documents) derived from an existing encrypted mix file. I would prefer that the tool generate its own key source, but generating a random one causes Red Alert (Irans portablera to be exact) to crash. XCC and my tool can read mixes encrypted with a random key fine however. Any insight how I could generate one that passes the games scrutiny would be welcome if it is possible. I've attached the crash dump incase it helps. ra95crash.zip

This is all fine, I get that the dev team wanted to make their own thing rather than reimplement something else accurately and you've done a good job at that, but evangelists complaining that those who like the nostaligic experience should "get with the times" are not doing themselves or openra as a whole any favours by making out like it still is a recreation of the old engines when it isn't. Personally I find it dissapointing that none of the real reimplementations ever got close to being accurate, but trying to do things the old way I guess isn't as exciting. I disagree that it's the right choice for the users because had an accurate clone got to the point it was playable, the playerbase may well have formed around those too, but history is written by the winners.

Problem is that openra isn't a successful reimplementation, its a completely different engine that does things in completely different ways so comparing it to OpenTTD, which on the face of it appears to have initially reimplemented the game as it was, is disingenuous. If openra had been developed along the lines of initially being a drop in replacement that was configured by the existing ini files and used algorithms reverse engineered from C&C and RA and was then expanded with additional optional functionality then I could see your confusion. That isn't the case however and even if someone went to the trouble of making a mod of openra to make it act like ra or c&c, that still wouldn't make all the mods that currently exist for these games compatible would it? That to me would be one test of a correct reimplementation.

I'm sure this gets asked periodically, but with all the reverse engineering going on I thought I'd likely get an answer here. Are there any projects trying to accomplish something similar to what OpenDune has for Dune 2 which is a faithful open source recreation of the CnC (and or RA) game engine? I can find various derivitives of FreeCnC such as OpenRedAlert but I couldn't say how close what is implemented there is to the original. All these hacks to keep it going and fix the odd bug are great, but it isn't very portable even with Wine and such.