Blade Posted June 24, 2015 Share Posted June 24, 2015 As some of you may or may not know, support for the TD shp format used by TD, RA and SS generally sucks in all tools that implement it. Shp files support 2 different kinds of compression for each of their frames, LCW (more widely but incorrectly known as format80) and XORDelta (format40). Most tools take advantage of the fact that the game will quite happily use shp files that just use LCW to avoid implementing an XORDelta encoder leading to shp files that are much larger than they need to be. Worse still is that the LCW encoders are generally don't achieve the same compression as the WW algorithm did. XCC suffers from this and worse the OpenRA utilities don't even implement the full LCW compression scheme leading to the worst case "compression" on all frames (generally around 2x as large as the original pixel data). To remedy this situation I've written ShapeSet, this implements LCW the same as the WW algorithm and implements XORDelta which is slightly better than the WW original. It is command line only and currently only operates on a sequence of PCX files as might be generated by XCC, but if used with the -k switch it will do full LCW+XORDelta compression resulting in shp files at least as small if not smaller than the original WW encoded ones. Basically this is the tool you would want to use to generate your final shp files you ship to users. But wait, there is more. Since very few programs implement a working XORDelta encoder, there are few tools that can create wsa files. Since the format is similar to shp I've also thrown in WSASet, a program to create wsa files that are again slightly smaller than the WW encoded equivalents. Test away and let me know of any issues that crop up. I'm taking about bugs that break the encoders or that stop the generated files working correctly, not the fact that it won't do your palette conversions for you or doesn't take your favorite image format. shapeset.zip Link to comment Share on other sites More sharing options...
Nyerguds Posted June 24, 2015 Share Posted June 24, 2015 Oh, wow... What about Dune II's WSAs? The existing encoder (of the Red horizons toolset) had trouble with the remapping colours in these files; probably a result of certain decoding techniques not going on to the remapping algorithm while others did. [edit] Test away and let me know of any issues that crop up. I'm taking about bugs that break the encoders or that stop the generated files working correctly, not the fact that it won't do your palette conversions for you or doesn't take your favorite image format. I know you said that, but could you add png as input type, though? PCX is annoying since it needs an actual editor just to preview it in Windows. (By the way, I edited your post for readability. Using double lines between paragraphs is a very easy fix for "wall of text" ) Link to comment Share on other sites More sharing options...
Blade Posted June 24, 2015 Author Share Posted June 24, 2015 I can't believe you altered the artistic integrity of my post! Dune2 is pretty much the same format as far as I can tell except it doesn't have the X and Y pos to draw stored in the file so writing the files shouldn't be difficult. The OpenDUNE code suggests that the file just gets decoded to a screen buffer, pretty much exactly the same as how the RA code works, there is nothing regarding remapping in the file as read from the disk other than what is in the pixel data. The RedHorizon code handles the Delta field in the header incorrectly though, its treated as something related to frame rate when in fact its the size of the largest XORDelta encoded frame plus 33 (37 in RA). Its also handled as a dword when in fact its only a word and the word that follows it is a flag to determine if the file contains a palette and if the offsets should all be calculated +0x300 to compensate. The problems with remapping could be related to this somehow? Regarding PNG support. There is a small lib that I might look into for adding support in the future. These tools are side projects to a larger project with CCHyper as you may be aware so adding support for formats not immediately related to that project is low priority. In the mean time, I would suggest using XCC and an intermediate shp/png file system when preparing graphics and then exporting to pcx sequence and processing with this tool as the final step. I realise its not ideal and involves an intermediate tool and extra steps though. Link to comment Share on other sites More sharing options...
Nyerguds Posted June 24, 2015 Share Posted June 24, 2015 Well, it's a minor annoyance anyway; back in the old days we did everything with pcx Well, looks like you guys did some serious research into this! Nice work Link to comment Share on other sites More sharing options...
Matt Posted July 10, 2015 Share Posted July 10, 2015 the OpenRA utilities don't even implement the full LCW compression scheme leading to the worst case "compression" on all frames (generally around 2x as large as the original pixel data). I am not sure if that is true anymore. Last year a skilled person contributed Format80 compression. https://github.com/OpenRA/OpenRA/pull/5781 Link to comment Share on other sites More sharing options...
Blade Posted July 10, 2015 Author Share Posted July 10, 2015 True, the implementation of the RLE command will have greatly improved compression over just the copy from source command, but its still very incomplete and IMO shouldn't be recommended in guides over XCC's compression as it currently stands. Here are the rules that the WW LCW compressor appears to follow when deciding which compression function to use: 1. Runs of the same colour should only use 4 byte RLE command if longer than 64 bytes. 2. Runs of less than 2 should just be stored as is with the one byte fill command. 3. Absolute runs greater than 10 or if the relative offset is greater than 4095 use an absolute copy. Less than 64 bytes uses 3 byte command, else it uses the 5 byte command. 4. If Absolute rule isn't met then copy from a relative offset with 2 byte command. Link to comment Share on other sites More sharing options...
Nyerguds Posted July 15, 2015 Share Posted July 15, 2015 Neat. Does your toolset use that logic, then? Link to comment Share on other sites More sharing options...
Blade Posted July 31, 2015 Author Share Posted July 31, 2015 Yeah, the LCW compression code is currently some cleaned up and slightly rewritten pseudo generated from the assembly compressor used to write the mappack sections in the game itself. I would like to do a full rewrite to clean it up further before releasing it. Link to comment Share on other sites More sharing options...
Kilkakon Posted October 31, 2015 Share Posted October 31, 2015 Props for this Having some palette issues (XCC's fault) but this was able to get me a working WSA when other tools did not Thank you Link to comment Share on other sites More sharing options...
Blade Posted October 31, 2015 Author Share Posted October 31, 2015 Glad its proving to be of some use, its probably the only tool currently that can generate consistently working WSA files, but I wasn't sure if anyone actually wanted to replace them. Link to comment Share on other sites More sharing options...
Kilkakon Posted November 3, 2015 Share Posted November 3, 2015 Posting this here. Not sure it's relevant to your tool or it's my mod that's at fault so if it's in the wrong place, sorry! For Blade and Nyerguds' eyes primarily: I managed to import the stretch palette, however it seems to not have worked 100%? I'm getting a lot of pink instead of green here: I also get this crash the second time I open the map in the same execution of C&C: EDIT: Just to throw a spanner in the works, just managed to get it to crash on the third time and not the second with references 004d8295 and 01018000 respectively. It's quite possible that it's something I'm doing wrong. I can verify I have a custom palette in my pcx that doesn't seem to be recognised by the game, despite me having the -p switch on. Yes I have -l switch on as well. Not sure what files you need, so attaching the pcx frame shown there, the wsa and the stretch pal. hearth_a.zip Link to comment Share on other sites More sharing options...
Blade Posted November 3, 2015 Author Share Posted November 3, 2015 Looks like the stretch table still isn't right to me. BTW, just in case you didn't realise, the palette has to be the same across all the pcx files and it will only include the palette from the first file if -p is specified as a result. Regarding the error, I think we might need Nyerguds to look up what function it happens in to be able to guess what went wrong. Link to comment Share on other sites More sharing options...
Nyerguds Posted November 3, 2015 Share Posted November 3, 2015 I do have a tool for making these stretch tables... http://nyerguds.arsaneus-design.com/project_stuff/2012/wsapalgen/wsapalgenerator_src.rar Compiled binary is included in the pack. When using it, I suggest making sure it ignores fades to colour 0. Link to comment Share on other sites More sharing options...
Nyerguds Posted November 3, 2015 Share Posted November 3, 2015 All I can tell you is that it's in a function that's called in a function that's called in something which I called "WSAAnimClass_GetFrame", though looking at the stuff inside it seems like it doesn't get the frame, but rather paints it on the graphics surface. As for what goes wrong, no clue without debugging... Blegh. Debugging. Link to comment Share on other sites More sharing options...
Nyerguds Posted November 3, 2015 Share Posted November 3, 2015 Kilkakon, that's a hi-color image... unless you take screenshots with cnc-ddraw, I can't see what's going on inside them. Link to comment Share on other sites More sharing options...
CCHyper Posted November 3, 2015 Share Posted November 3, 2015 It's crashing inside "_Apply_XOR_Delta_To_Page_Or_Viewport". Blade: Address (0x005D5C45) In our RA Database. An interesting thing is that in RA, the line is "xor [offset], al", in C&C, its "mov [offset], al"... Link to comment Share on other sites More sharing options...
Nyerguds Posted November 3, 2015 Share Posted November 3, 2015 The one from the error message box, "004D81F6" is "xor [edi], al", though Link to comment Share on other sites More sharing options...
CCHyper Posted November 3, 2015 Share Posted November 3, 2015 Disregard what I mentioned, I had no idea what I was looking at!... Link to comment Share on other sites More sharing options...
Kilkakon Posted November 3, 2015 Share Posted November 3, 2015 I tried getting a screenshot with ddraw but it would not work. Must be an ingame only feature? Or the transition blocking keyboard presses. RE: Nyer's palette generator. This is what I used but I am not sure if I hit ignores fades to 0. I can certainly try this tonight. Thanks for looking into this everyone. Link to comment Share on other sites More sharing options...
Nyerguds Posted November 4, 2015 Share Posted November 4, 2015 I tried getting a screenshot with ddraw but it would not work. Must be an ingame only feature? Or the transition blocking keyboard presses. No, I've made screenshots of WSAs with it. If you're on an OS higher than XP make sure your game runs in compatibility mode for Win95, and press ctrl+s ingame to make the screenshot. It doesn't copy to clipboard; it saves a png. Link to comment Share on other sites More sharing options...
Kilkakon Posted November 4, 2015 Share Posted November 4, 2015 Yeah I definitely did that... that's the method I use to take my normal screenshots and nothing seemed to appear in the directory. I'll double check that when I get home to make sure I'm not doing something stupid. My C&C machine is XP incidentally. I did attach the source files on the previous post, although yeah I can see why having the ingame palette would be probably the most helpful thing. Link to comment Share on other sites More sharing options...
Nyerguds Posted November 4, 2015 Share Posted November 4, 2015 although yeah I can see why having the ingame palette would be probably the most helpful thing. See, the palette of the stuff you gave in that zip (both the pcx and the WSA) doesn't contain purple... Link to comment Share on other sites More sharing options...
Kilkakon Posted November 4, 2015 Share Posted November 4, 2015 Notably, the original palette for that WSA does. My dark green is pink in the original file, index 41. That's the XCC Copy to clipboard of the original file. I can also confirm I can't take a screenshot with Ctrl+S during the cinematic and I'm running ddraw. Just screenshotted some normal gameplay to be sure XD Link to comment Share on other sites More sharing options...
Blade Posted November 4, 2015 Author Share Posted November 4, 2015 Game still somehow using the old palette then? Guess its not loading it from the WSA or if it is, it isn't setting that as the current palette? Link to comment Share on other sites More sharing options...
Nyerguds Posted November 4, 2015 Share Posted November 4, 2015 Hmm. An external palette? I don't think it has any such thing for those WSAs... Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now