Working code for efficient LCW and XORDelta compression.

[Blade]

Find attached C++ code for efficiently compressing and encoding data in LCW(format80) and XORDelta(format40) formats. This is cleaned up and rewritten in parts from the code used in ShapeSet and WSA set and so provides the same or better compression as the original code used by WestWood for these formats.

 

Ideally any tools that write data in these formats should be updated to use this code (for C/ C++ programs) or else port the code to the relevant language unless their code already matches the performance of the WW originals.

 

This code is released under the GPL v2 or later. If you want to use it in a program using a different license I'm open to discuss it.

 

If anyone needs assistance with the algorithm that is used in the RA/TD shp files to decide what format a frame should have or any other questions related to these compression formats, please ask away.

encoders.zip

[Nyerguds]

Oh, nice

[SiRaLeX]

Please excuse my ignorance, but, what is this good/useful for?!    Binary delta compression and Lempel-Ziv-Welch?

 

On a second note, Blade obviously knows how to write in C and use static_cast.  :roll:

 

If you just compiled the code as C (which it is) instead of C++ you wouldn't even need to cast from "void *" to "uint8 *" in the first place.  :laugh:

 

[Blade]

Its lifted from a code base that is compiled as C++ so its written in that style, I'm well aware of this subtle difference between C++ and C treatment of void* and didn't feel like cleaning it up further to compile as straight C, its provided as is that it might prove useful to those writing tools to write the C&C graphics formats.

 

LCW (not LZW) is a proprietary compression format used in many WW games. Most implementations don't compress as well as the WW original, this one does. XORDelta is similarly a binary delta format used in the WW animation formats, the only other working implementation I know of is the Red Horizons Java version. Its required to implement a WSA format writer and can greatly improve the compression of the shp format if implemented as WW did.

 

You might see these formats referred to as format80 and format40 respectively, LCW and XORDelta are the original names used by Westwood.

 

 

[SiRaLeX]

Thank you, Blade for claryfing! But what are we supposed to do with these algorithms without the corresponding decompression functions? Or is this rather just for modding purposes because the game obviously has those decompression functions and there's better compression algorithms out there for general use?

 

Either way, XORDelta surely can be pretty handy for general use sometimes (file formats with a lot of overhead). 

 

Cool cakes! 

[Blade]

There were already plenty of examples of the decompression algorithm available in open source tools the community has made. https://github.com/OpenDUNE/OpenDUNE/tree/master/src/codec has good examples in C that are close to the original implementations, though the format80 implementation doesn't handle the extended algo for data over 64K in size. You can find an implementation that does at https://github.com/madmoose/replicant/blob/master/src/decompress.cpp, though the differences are minimal.

[hamster]

Good post...

[Nyerguds]

I'm working on a C# version... both for encode and decode