Encrypted mix file reading

[Blade]

I'm trying to write a command line tool to write encrypted mix files and to some extent I have succeeded by using a key source (as described in the XCC documents) derived from an existing encrypted mix file. I would prefer that the tool generate its own key source, but generating a random one causes Red Alert (Irans portablera to be exact) to crash. XCC and my tool can read mixes encrypted with a random key fine however. Any insight how I could generate one that passes the games scrutiny would be welcome if it is possible. I've attached the crash dump incase it helps.

ra95crash.zip

[FunkyFr3sh]

Sonarpulse tried to write such a tool too, but im not sure if its done/working

 

maybe it helps you in some way:

https://github.com/cnc-patch/cncmix

 

i think openra can read mix files too, maybe you could get some info there as well

[Iran]

The crashdump doesn't say anything other than the issue being related to memory, the memory error handler is called and it crashes while trying to display an ingame WWmessagebox...probably because fonts aren't loaded yet. It crashes inside the Bootstrap() function which loads MIX files at startup.

[Blade]

FunkyFr3sh, Sonarpulse's work doesn't really help me because 1, it's written in a language I have no experience of and 2, it doesn't appear to implement encrypted mix reading or writing anyway, it looks like it just has stubs. OpenRA can read encrypted mix files, but that doesn't help me because I want to write them. Only XCC Mix Editor does and even that seems to just grab a key_source from an existing mix which I can do, but it would be nice to be able to generate new ones. If I better understood the encryption code or how the code retrieves a key from the keyblock I might have some insight into why the game has problems with random key blocks, but just looking at the code with all the bit shifting gives me a head ache.

 

Thanks Iran, shame it doesn't give more insight into the reason for the crash 

[Nyerguds]

I suppose RA expects one specific key the game knows for the encryption, so it can decrypt it?

 

The XCC Mix Editor has encryption as option. Just use that?

[Iran]

The XCC Mix Editor supports creating mix files with encryption, the source for it is probably there.

[Blade]

It can't be just one key, because each encrypted mix seems to have a different key source included which is the blowfish symmetric key encrypted with some public key crypto (which we have the public key for thanks to key.ini). Olaf has reverse engineered the decryption algo as XCC and tools based on it (such as openra's code, my code and such) can recover the blowfix key and then decrypt the header. XCC Mix Editor doesn't generate new Blowfish keys and encrypt them with the private key (which we also have from key.ini) as far as I can tell though since it requires you to load a key from an existing mix file first. My program (https://github.com/OmniBlade/ccmix if you are interested) can generate working encrypted mix files in the same way, by using a key_source from an existing mix, I just would like to be able to generate new ones as well. Just randomly generating a 80byte block as I naively tried either breaks the decryption algo in the game or the "key" it recovers isn't considered a valid blowfish key as far as the game is concerned.

[Blade]

Well, I took a break from looking at the encryption to look at the checksum function and after very little effort identified it as a SHA1 digest of the body of the mix, which is good since it was the only hash function that was around when RA was written that generated 20 byte digests that came up in google when I searched.

[Blade]

I've attached a windows build of the tool which requires more testing but should be able to create mixes in any of the 3 formats (td, ra or ts) and do encryption, checksums and local mix databases for those that support them. Adding and removing random files is not supported yet, and neither is converting unencrypted to encrypted directly or vice versa. Rough use instructions are on the github page, --checksum when creating will add a checksum to the new mix and --add --checksum with an existing mix will add one, not that the game seems to care.

 

edit: updated exe to one that works extracting all files again.

ccmix.zip

[Matt]

Someone who refactors https://github.com/OpenRA/OpenRA/blob/bleed/OpenRA.Game/FileFormats/BlowfishKeyProvider.cs into actually readable code gets a :beer:

[Nyerguds]

Isn't that in the xcc code though?

[Matt]

I believe that is a direct C port from the XCC code.

[Blade]

I haven't seen any tool that reads the encrypted mixes that doesn't use that decryption code and it wouldn't surprise me if it was a slightly tidied up decompilation of the games own functions for it. Its obviously some kind of pubic key crypto given that we have the public and private keys for it, but what algo it is I couldn't say. What algo's were available around 1995 and were used with blowfish for message encryption?

[Nyerguds]

Ehh. You forget Westwood made their own audio and video compression formats (VQA and AUD), sprite formats (SHP, TMP) and archive formats (pak and later mix). It's entirely possible the encryption algorithm is all their own as well.

 

Doesn't TS have a blowfish.dll, though? I mean, maybe you could just... use that, somehow

[Matt]

I was told it is a custom and incompatible blowfish. Otherwise we would be using a pre-fabricated Open Source library for that.

[Blade]

Blowfish is symmetric, so the implementation both encrypts and decrypts with a given key and we have a working implementation so it doesn't matter all that much if its custom or not, but it does affect if the public key crypto could be a standard one or not I guess. The checksum appears to be standard sha1 though.

 

The blowfish key is what is encrypted with the unknown public key crypto with BlowfishKeyProvider being the decryption algo for the public key crypto and is different for every westwood created encrypted mix. We have to make do with reusing an encrypted key from an existing encrypted mix file as the game won't accept a random key made from creating a random 80 bytes for the encrypted key block (Olaf's docs call this block a "key source") and "decrypting" it. Using an existing key source is how XCC and ccmix create encrypted mixes that the game will acceptv as the game doesn't know that we have reused a key, it just knows its a proper westwood key. Without knowing 1. The criteria the game uses to verify the key and how to generate one and 2. The algorithm to encrypt that key into a key source, we cannot make uniquely encrypted mix files. From a user POV, I guess it doesn't really matter as they can create working encrypted mix files, but for curiosities sake I would like to know.

[XTF]

Does anything actually require encrypted MIXs?

[Allen262]

RAED must have any mix-es it used to be encrypted.

[Nyerguds]

I was hoping our hackers would finally kick that out of RAED.

 

I mean, how hard can it be to make it read the header directly instead of always attempting to decrypt it? The Encrypted status is a field in the header, after all...

[XTF]

Isn't there a better editor than RAED?

[Allen262]

Nope. It is the only one for making single player missions for RA1.

[Chad1233]

Isn't there a better editor than RAED?

 

RAED is the best editor for RA1 ever and beats the editors for C&C95 by a mile imo. it's pretty advanced with a few annoyances and it can make skirmish maps you just have to make a .mpr map with edwin or with the RAED launcher (Which you gotta use to open .mpr directly because RAED wont find any) funky has then you can build the map plus you can tweak it's mapsize in the built-in .mpr or .ini editor!

[Blade]

If anyone is still interested in a command line tool, I've attached a new windows build that needs testing. In theory it provides the following features:

Can extract and create mix files for any of the main C&C games, including encrypted using the bundled key.source.

Can handle mix files where the filenames aren't known when extracting and then give them the correct id when rebuilding (names files with the format [id]ABCDF987 where the characters after [id] are treated as hex).

Can add and remove single files while preserving the attributes of the file being added to.

Allows optional use of the local mix database.dat extension for XCC compatibility and to allow easier recovery of filenames if originals are lost.

 

Don't try add and remove on mix files you aren't willing to loose. Although I've ran some tests and it seems to work okay, I can't guarentee it won't corrupt or overwrite data.

ccmix_win32.zip

[CCHyper]

Blowfish is symmetric, so the implementation both encrypts and decrypts with a given key and we have a working implementation so it doesn't matter all that much if its custom or not, but it does affect if the public key crypto could be a standard one or not I guess. The checksum appears to be standard sha1 though.

 

The blowfish key is what is encrypted with the unknown public key crypto with BlowfishKeyProvider being the decryption algo for the public key crypto and is different for every Westwood created encrypted mix. We have to make do with reusing an encrypted key from an existing encrypted mix file as the game won't accept a random key made from creating a random 80 bytes for the encrypted key block (Olaf's docs call this block a "key source") and "decrypting" it. Using an existing key source is how XCC and ccmix create encrypted mixes that the game will acceptv as the game doesn't know that we have reused a key, it just knows its a proper Westwood key. Without knowing 1. The criteria the game uses to verify the key and how to generate one and 2. The algorithm to encrypt that key into a key source, we cannot make uniquely encrypted mix files. From a user POV, I guess it doesn't really matter as they can create working encrypted mix files, but for curiosities sake I would like to know.

 

I might be tired and unable to read correctly this morning, but KEY.INI that is located in some of the MIX files contains both WW Public and Private blowfish keys, (known as a FastKey's internally). The game hard codes this though as it would not be able to open the MIX file to load KEY.INI and I'm sure WW did not want to pub the INI in the root directory.

 

RA, RA2 and YR also uses the same Public and Private FastKeys's as TS.

[Blade]

Those aren't the blowfish keys, those are the keys that encrypt and decrypt the block within the mix file header to the actual blowfish key (which seems to have been generated on a per mix basis). Blowfish isn't a public key crypto, its symetric so the key that encrypts also decrypts. Basically the blowfish key itself is encrypted with an unknown public key crypto scheme (which key.ini has the keys for) and then stored in the mix file just before the encrypted header itself. XCC mixer has the implementation for decryption but I can hardly fathom its big integer math. I doubt others can either because I've yet to see an implementation that isn't pretty much just a copy of the XCC one. The tools that write encrypted mixes get round not being able to generate new blowfish keys by basically dumping the whole 80byte block that is the encrypted form of the key and stick it on a mix that has been blowfish encrypted with the key that the block decrypts to.

 

I hope that all made some kind of sense.