The MASM Forum

CD - Compress-Decompress Utility using the Microsoft Compression API

CDScreenshot.png CDScreenshotText.png

CD uses the Microsoft Compression API to compress or decompress data using one of the four supported compression algorithms: XPRESS, XPRESS with Huffman encoding, MSZIP or LZMS.

The files compressed by CD using those compression algorithms also store a signature DWORD value as the header at the start of the file. This is so that the appropriate compression algorithm can be used for the decompression.

CD also makes use of the Microsoft Compression API to store bitmap resources as LZMS compressed data. There are two ways in which CD uses that compressed bitmap data:

• In the about box, by uncompressing the bitmap data before creating the bitmap in memory. The LZMS compressed bitmap data is stored as static hex bytes in the CD128x128x4.bmp.asm file.

• Adding LZMS compressed bitmap files (*.lzms) as RC_DATA resources which are compiled into CD.exe. These resources are loaded into memory, and then uncompressed before creating the bitmaps in memory.

CD also allows an optional output of compressed data bytes as a masm .asm file with 'db' data defines, similar to bin2dbex.

Github Repository: https://github.com/mrfearless/CD (https://github.com/mrfearless/CD)

v1.0.0.2

• Optimized assembler output to speed up process overall and to fix hangs (buffer overflows) for larger files (100K+)
• Added checkbox option to specify .DATA segment or .CONST segment for masm output
• Added Compress Text button and accelerators for it.
• Increased the max text size in the input edit control in the compress text dialog to 256KB
• Uploaded x64 version of CD

v1.0.0.1

• Added compress text dialog box
• Added more accelerators for some buttons on the main dialog and in the compress text dialog
• Added CDCompressMem function
• Changed CDDecompressMem to allocate 4 additional nulled bytes, for resources like ansi/unicode strings that required null bytes
• Added compressed string example to About dialog box in an edit control using the MSZIP compressed text stored in infotext.asm

Note: the minimum version of windows to use for the Microsoft Compression API and thus to use the CD - Compress-Decompress Utility is Windows 8 (https://learn.microsoft.com/en-us/windows/win32/api/compressapi/nf-compressapi-compress#requirements):

Hi fearless
Very clever idea and good stuff (as always). :thumbsup:

The interface is clean and does what it should.  :cool:
I have tested the functionality a bit and need to integrate the code to handle decompression in my applications.

Chances are good that I will immediately add this tool to my tool set.
What I would like to see is a CL interface to call CD.exe from the tool chain.

Thanks for sharing!

Biterider

Thanks.

I had considered doing a console version similar to my snappy console application, with support for wildcards and switches for compress/decompress. I would need to add in switches/options for algorithm used and also the optional output to asm file. 

Probably:

/c | -c   Compress (using some default algo)
/d | -d   Decompress
/x | -x   Compress with XPRESS
/h | -h   Compress with XPRESS with Huffman
/m | -m   Compress with MSZIP 
/l | -l   Compress with LZMS
/a | -a   Asm output
/o | -o   Alternate asm output instead if not using /a | -a as a switch
/r | -r   Add .raw ext to decompressed file

Also have to consider overwriting existing files. So maybe the /r or -r for .raw extension instead for safety when decompressing.

So could look like:

CD -l -a *.bmp c:\temp
which would translate to: compress using lzms and output asm file for all bitmaps in current directory to c:\temp folder

Quote from: fearless on October 22, 2023, 04:37:40 AMI would need to add in switches/options for algorithm used and also the optional output to asm file. 

That would be really good.

Biterider

@fearless
on the beginning of your asm module there is a statement
.XMM
https://github.com/mrfearless/CD/blob/ff2ef6236bad279c6cd92fa21643064fa4c4dab9/CD.asm#L102 (https://github.com/mrfearless/CD/blob/ff2ef6236bad279c6cd92fa21643064fa4c4dab9/CD.asm#L102)

have you tried the higher level, does MASM have .YMM, .ZMM?

Hi fearless
I would like to give you feedback on my short journey using your code. 
I quickly managed to compress the bitmap I normally use as application background using various algos from your application (CD.exe).

In my particular case, I got the best result using the Huffman compression.

I replaced the reference to the compressed file in my .rc file and copy/pasted the CDBitmapCreateFromMem, CDDecompressMem and CDBitmapCreateFromCompressedRes procedures as well as some definitions and voilá... it worked out of the box.

I forgot to mention that I checked which include files were needed and luckily I had the compressapi.inc from a previous translation of "Windows Kits 10.0.22621.0".

I am very pleased with this development. Congratulations.  :thumbsup:
I will check what is necessary to compile for x64 as well. I am sure there are only minimal adjustments.  :cool:

Biterider

Nice. Glad to hear it all worked well. 

Yes some different algo's work better with different images. I opted for lzms for the resources in CD.exe mainly due to comparing the results of the output of all the files in one folder vs other folders for the other formats, found that the lzms slightly better for saving space for those images i used. But in most cases a mixture of the different algos for particular files would bring the best space savings.

I am in the middle of also adding an additional dialog to CD to allow compression of text strings to masm format, either as a block or each line separately, as it occurred to me that string resources could benefit in some cases from this. Also have a companion to the CDDecompressMem, a CDCompressMem function, which is used in the text string compression.

Also working on the console version of CD as well. 

I will probably add a few of the functions in CD to ModernUI at some point.

Cheers

Hi fearless
Also multi-image icons also tend to be large and can benefit from this technique.

Biterider

Really nice example fearless. Code is also clear  :thup:

Hi fearless,

your program works fine :thumbsup:

What I miss is a programmatical solution to, for example, create a zip archive for \Masm32\examples\exampl01\qikpad, i.e. something like
invoke TheZipper, chr$("test.zip"), chr$("\Masm32\examples\exampl01\qikpad")

Hi fearless,

Nice work. Notice that CD is a reserved keyword of the command prompt so you would like to rename the console version of your tool to avoid conflicts.

Keep up the good work :thumbsup:

Quote from: Vortex on October 27, 2023, 04:17:53 AMHi fearless,

Nice work. Notice that CD is a reserved keyword of the command prompt so you would like to rename the console version of your tool to avoid conflicts.

Keep up the good work :thumbsup:

Maybe CoDe - Compress-Decompress Utility

It fails under Win 7 32 => "The ordinal 43 could not be located in the dynamic link library Cabinet.dll." 
The version of cabinet.dll is 6.1.7601.1751.

Hi coaster, unfortunately the minimum version of windows to use for those functions is Windows 8 (https://learn.microsoft.com/en-us/windows/win32/api/compressapi/nf-compressapi-compress#requirements):

Minimum supported client	Windows 8 [desktop apps | UWP apps]
Minimum supported server	Windows Server 2012 [desktop apps | UWP apps]
Target Platform	Windows
Header	compressapi.h
Library	Cabinet.lib
DLL	Cabinet.dll

Updated first post with v1.0.0.1

Hi
A small contribution. 
Attached is the translation of the compressapi.h file.

Biterider

Hi fearless
I have a small suggestion: do not specify the segment in the generated asm file. 
Let the user specify the target segment before the assembler file is included. 
This way you have the choice between .data and .const.

Biterider

Something like adding a checkbox option to use .DATA or .CONST, like this?

CDScreenshot.png

Hi fearless
Yes, something like that or let the user write

.data
include .\Res\MyCompressedData.asm

That is only a small detail.
Of course, you can edit the file after it has been created, but if you want to automate the tool-chain, that could be a bit tedious.

Biterider

Updated first post with v1.0.0.2

• Optimized assembler output to speed up process overall and to fix hangs (buffer overflows) for larger files (100K+)
• Added checkbox option to specify .DATA segment or .CONST segment for masm output
• Added Compress Text button and accelerators for it.
• Increased the max text size in the input edit control in the compress text dialog to 256KB
• Uploaded x64 version of CD

(Reached max attachments, so only uploading the two source zips in the first post and just the binary only zips here)

Win7-64:
x86: Unable to find ordinal 43 in the dynamic link library Cabinet.dll
x64: Unable to find ordinal 40 in the dynamic link library Cabinet.dll

Quote from: jj2007 on October 29, 2023, 05:27:07 AMWin7-64:
x86: Unable to find ordinal 43 in the dynamic link library Cabinet.dll
x64: Unable to find ordinal 40 in the dynamic link library Cabinet.dll

https://masm32.com/board/index.php?topic=11371.msg124773#msg124773 (https://masm32.com/board/index.php?topic=11371.msg124773#msg124773)

I see - bad luck. I tried to assemble cd.asm but got some errors:
AboutDlg.asm(145) : Error A2275: Constant value too large: 54686973206973206D65616E7420746Fh
 AboutDlg.asm(145): Included by
  Tmp_File.asm(95): Main line code
AboutDlg.asm(145) : Error A2147: Too few arguments to INVOKE: SetWindowTextA
 AboutDlg.asm(145): Included by
  Tmp_File.asm(95): Main line code
CDText.asm(365) : Error A2275: Constant value too large: 546F6F6C746970735F636C6173733332h
 CDText.asm(365): Included by
  Tmp_File.asm(96): Main line code
CDText.asm(365) : Error A2147: Too few arguments to INVOKE: CreateWindowExA
 CDText.asm(365): Included by
  Tmp_File.asm(96): Main line code
CDText.asm(426) : Error A2275: Constant value too large: 436F7572696572204E6577h
 CDText.asm(426): Included by
  Tmp_File.asm(96): Main line code
CDText.asm(426) : Error A2147: Too few arguments to INVOKE: lstrcpyA
 CDText.asm(426): Included by
  Tmp_File.asm(96): Main line code
Tmp_File.asm(405) : Error A2275: Constant value too large: 546F6F6C746970735F636C6173733332h
Tmp_File.asm(405) : Error A2147: Too few arguments to INVOKE: CreateWindowExA
Tmp_File.asm(1097) : Error A2143: Symbol redefinition: CDCompressMem
Tmp_File.asm(1098) : Error A2092: PROC, MACRO or macro loop directive must precede LOCAL
Tmp_File.asm(1099) : Error A2092: PROC, MACRO or macro loop directive must precede LOCAL
Tmp_File.asm(1100) : Error A2092: PROC, MACRO or macro loop directive must precede LOCAL
Tmp_File.asm(1101) : Error A2092: PROC, MACRO or macro loop directive must precede LOCAL
Tmp_File.asm(1102) : Error A2092: PROC, MACRO or macro loop directive must precede LOCAL
Tmp_File.asm(1103) : Error A2092: PROC, MACRO or macro loop directive must precede LOCAL
Tmp_File.asm(1264) : Error A2142: Unmatched block nesting: CDCompressMem
Tmp_File.asm(1274) : Error A2143: Symbol redefinition: CDDecompressMem
Tmp_File.asm(1275) : Error A2092: PROC, MACRO or macro loop directive must precede LOCAL
Tmp_File.asm(1276) : Error A2092: PROC, MACRO or macro loop directive must precede LOCAL
Tmp_File.asm(1277) : Error A2092: PROC, MACRO or macro loop directive must precede LOCAL
Tmp_File.asm(1278) : Error A2092: PROC, MACRO or macro loop directive must precede LOCAL
Tmp_File.asm(1279) : Error A2092: PROC, MACRO or macro loop directive must precede LOCAL
Tmp_File.asm(1280) : Error A2092: PROC, MACRO or macro loop directive must precede LOCAL
Tmp_File.asm(1281) : Error A2092: PROC, MACRO or macro loop directive must precede LOCAL
Tmp_File.asm(1423) : Error A2142: Unmatched block nesting: CDDecompressMem
Tmp_File.asm(1957) : Error A2143: Symbol redefinition: CDBitmapCreateFromMem
Tmp_File.asm(1958) : Error A2092: PROC, MACRO or macro loop directive must precede LOCAL
Tmp_File.asm(1959) : Error A2092: PROC, MACRO or macro loop directive must precede LOCAL
Tmp_File.asm(1975) : Error A2142: Unmatched block nesting: CDBitmapCreateFromMem
Tmp_File.asm: 2075 lines, 1 passes, 93 ms, 0 warnings, 29 errors

Looks like an issue with the CTEXT macro in those places. For the x64 version I use Uasm64 and have a ctext macro in the windows.inc file (from the WinInc package).

At the end i have a ctext defined as:

CTEXT MACRO quoted_text:VARARG
    LOCAL local_text
    .data
    align 8
    local_text db quoted_text,0
    .code
    align 8
    EXITM <offset local_text>
ENDM


If I insert your CTEXT macro behind the include macros.asm line as follows...
include \masm32\macros\macros.asm
CTEXT MACRO quoted_text:VARARG
    LOCAL local_text
    .data
    align 8
    local_text db quoted_text,0
    .code
    align 8
    EXITM <offset local_text>
ENDM
... I get these errors:
*** Assemble using UAsm64  ***

***********
ASCII build
***********

Tmp_File.asm(1106) : Error A2143: Symbol redefinition: CDCompressMem
Tmp_File.asm(1107) : Error A2092: PROC, MACRO or macro loop directive must precede LOCAL
Tmp_File.asm(1108) : Error A2092: PROC, MACRO or macro loop directive must precede LOCAL
Tmp_File.asm(1109) : Error A2092: PROC, MACRO or macro loop directive must precede LOCAL
Tmp_File.asm(1110) : Error A2092: PROC, MACRO or macro loop directive must precede LOCAL
Tmp_File.asm(1111) : Error A2092: PROC, MACRO or macro loop directive must precede LOCAL
Tmp_File.asm(1112) : Error A2092: PROC, MACRO or macro loop directive must precede LOCAL
Tmp_File.asm(1273) : Error A2142: Unmatched block nesting: CDCompressMem
Tmp_File.asm(1283) : Error A2143: Symbol redefinition: CDDecompressMem
Tmp_File.asm(1284) : Error A2092: PROC, MACRO or macro loop directive must precede LOCAL
Tmp_File.asm(1285) : Error A2092: PROC, MACRO or macro loop directive must precede LOCAL
Tmp_File.asm(1286) : Error A2092: PROC, MACRO or macro loop directive must precede LOCAL
Tmp_File.asm(1287) : Error A2092: PROC, MACRO or macro loop directive must precede LOCAL
Tmp_File.asm(1288) : Error A2092: PROC, MACRO or macro loop directive must precede LOCAL
Tmp_File.asm(1289) : Error A2092: PROC, MACRO or macro loop directive must precede LOCAL
Tmp_File.asm(1290) : Error A2092: PROC, MACRO or macro loop directive must precede LOCAL
Tmp_File.asm(1432) : Error A2142: Unmatched block nesting: CDDecompressMem
Tmp_File.asm(1966) : Error A2143: Symbol redefinition: CDBitmapCreateFromMem
Tmp_File.asm(1967) : Error A2092: PROC, MACRO or macro loop directive must precede LOCAL
Tmp_File.asm(1968) : Error A2092: PROC, MACRO or macro loop directive must precede LOCAL
Tmp_File.asm(1984) : Error A2142: Unmatched block nesting: CDBitmapCreateFromMem
Tmp_File.asm: 2084 lines, 1 passes, 87 ms, 0 warnings, 21 errors

Quote from: fearless on October 29, 2023, 07:27:37 AMI use Uasm64 and have a ctext macro in the windows.inc (https://windows.inc/) file (from the WinInc package)

Does that mean you use the UAsm version of Windows.inc? I see include windows.inc in CD.inc - no path supplied, so it relies on the user's PATH variable. OTOH, there is include \masm32\macros\macros.asm in CD.asm, so you are clearly also using (parts of) Masm32 SDK.

Sorry, I probably mixing up the x86 and x64 versions

Yes i use masm32 sdk with the x86 version: ml, linker etc. For the x64 version i use uasm64, wininc.

ML.EXE /c /coff /Cp /nologo /I"M:\masm32\include" cd.asm
And i use the macros.asm in the x86 one including the CTEXT one:

        CTEXT macro Text
            local szText
            .data
            szText byte Text, 0
            .code
            exitm <offset szText> 
        endm
For the x64 this seems to work from the command line:

UASM64.exe /c -win64 -Zp8 /win64 /D_WIN64 /Cp /nologo /W2 /I" M:\UASM\include" cd.asm
And the CTEXT macro is defined in my WinInc windows.inc as:

CTEXT MACRO quoted_text:VARARG
    LOCAL local_text
    .data
    align 8
    local_text db quoted_text,0
    .code
    align 8
    EXITM <offset local_text>
ENDM
Im not sure why the CTEXT macro is not working. Only place close to having a conflicting definition is the following structs, which have szText1 - cant remember if its like that by default or if i modified it perhaps?

NMTTDISPINFOA STRUCT
  hdr              NMHDR  <>
  lpszText          DWORD  ?
  szText1          BYTE 80 DUP (?)
  union
    hInst          DWORD  ?
    hinst          DWORD  ?
  ends
  uFlags            DWORD  ?
  lParam            DWORD  ?
NMTTDISPINFOA ENDS

NMTTDISPINFOW STRUCT
  hdr              NMHDR  <>
  lpszText          DWORD  ?
  szText1          BYTE 80 DUP (?)
  union
    hInst          DWORD  ?
    hinst          DWORD  ?
  ends
  uFlags            DWORD  ?
  lParam            DWORD  ?
NMTTDISPINFOW ENDS


Interestingly when compiling the x64 version with Uasm64 i did get conflicting definition errors, so in that version i took out the EXTERN lines, which seemed to solve it. I imagine it would be the same if using Uasm to compile the x86 version.

I could build the x86 version with the attached batch file, plus this modification of line 78:

option casemap:none
include \masm32\macros\macros.asm
include \Masm32\include\debug.inc

Hi fearless
I tested both binaries successfully on Win10.  :thumbsup:

Biterider

Hi
I did some programming around the functionality of the CD.exe application to decompress bitmaps as shown in the CD source code, but also to create icons from compressed .ico files stored as RCDATA in the resources section.
In my test case, 408kB of resources are compressed into a 152kB binary file. 
Not all resources can be compressed, such as the application's main icon, which needs to remain accessible to the operating system so that the applications icon displays properly in File Explorer, among other things. In my test application, this icon consumes another 23kB from the 152kB, which makes compression a really excellent feature.

The clue to extract icons from a file structure is to replicate what the resource compiler does. It is explained here: 
https://devblogs.microsoft.com/oldnewthing/20231025-00/?p=108925 (https://devblogs.microsoft.com/oldnewthing/20231025-00/?p=108925)

Attached the source and the demo binary.

Biterider

Nicely done.

Now on, all code for Windows 10/11 should be mentioned.
Programming world goes further, so have to clearly tell, for what Windows OS code is for.
I write this message from Windows 7 PC, as i have a very little usage for Windows 10 PC, even it had two monitors.

Thanks Timo, I have updated the first post to indicate that, as it has been mentioned in this thread a few times, and it is a relevant point.