News:

Masm32 SDK description, downloads and other helpful links
Message to All Guests
NB: Posting URL's See here: Posted URL Change

Main Menu

How difficult is it to build a shrinking/deflating routine?

Started by frktons, December 15, 2012, 10:58:49 AM

Previous topic - Next topic

frktons

Quote from: qWord on December 17, 2012, 08:29:37 AM
gcc - you get what you pay for  :icon_mrgreen:

Sorry qWord, I didn't get it. Do you mean gcc, that is free, is better than vs or
the contrary?   ::)
There are only two days a year when you can't do anything: one is called yesterday, the other is called tomorrow, so today is the right day to love, believe, do and, above all, live.

Dalai Lama

qWord

nice ... you need to use the good documented linker option "-disable-long-section-names" (or for gcc "-Xlinker -disable-long-section-names") to get a standard conform PE.
:icon_rolleyes:
MREAL macros - when you need floating point arithmetic while assembling!

jj2007

Quote from: dedndave on December 17, 2012, 08:28:50 AM
http://wjradburn.com/software/

Merci :t

In the meantime, I found out that the part that I managed to compile (i.e. the LZMA library itself) is fully sufficient to write an archiver. Attached the first beta (Disclaimer: YOU HAVE BEEN WARNED!). It compresses around 16% better than WinZip with the non-portable maximum option.

Usage:
1. Compression: Drag a file over LzmaJJ.exe
2. Decompression: Drag a *.lzma file over LzmaJJ.exe

Limitations: 1. Only one file, 2. You need Pelles C to assemble the source (see OPT_DebugL in source code).

MichaelW

I have successfully compiled ARITH-N.C and the supporting code from the second edition of the Data Compression Book, using the VC Toolkit 2003 compiler with no optimization specified. The code is older than I remembered, using old-style declarations and recommending a LARGE memory model. It compiled as a Win32 console app, but with /W4 the compiler issued a bunch of warnings.

This is my preliminary result, compressing the current MASM32 windows.inc using order 3 arithmetic coding, manually timed at ~10 seconds:

Compressing \masm32\include\windows.inc to winc.z
Using Adaptive order n model with arithmetic coding
................................................................................
............
Input bytes:        977412
Output bytes:       188323
Compression ratio:  81%


Now to see if it will expand...

Looks OK, and FC /B found no differences, but the time was ~9 seconds?

With /O2 /G6 optimization compession and expansion each took ~~3 seconds.

To give you an idea of the code complexity, ARITH-N.C is 1102 lines with ~~25% comments.
Well Microsoft, here's another nice mess you've gotten us into.

frktons

Well Michael, considering that you have used a C compiled code, the
result is not bad at all, both in speed and compression ratio.
Probably hand written optimized assembly, with new SSE2/3  opcodes could produce
much better results on the speed side at least.
:t
There are only two days a year when you can't do anything: one is called yesterday, the other is called tomorrow, so today is the right day to love, believe, do and, above all, live.

Dalai Lama

frktons

Quote from: jj2007 on December 17, 2012, 10:03:30 AM

In the meantime, I found out that the part that I managed to compile (i.e. the LZMA library itself) is fully sufficient to write an archiver. Attached the first beta (Disclaimer: YOU HAVE BEEN WARNED!). It compresses around 16% better than WinZip with the non-portable maximum option.

Usage:
1. Compression: Drag a file over LzmaJJ.exe
2. Decompression: Drag a *.lzma file over LzmaJJ.exe

Limitations: 1. Only one file, 2. You need Pelles C to assemble the source (see OPT_DebugL in source code).

It seems to work fine, Jochen :t

One thing puzzles me. Your implementatio of RtlCompress function produces
a zip file 280k long, but if I do select --> sent to --> compressed folder it
produces a 190k zip file. Shouldn't them have the same compression rate?
Isn't it the same function Windows is using?  ::)
There are only two days a year when you can't do anything: one is called yesterday, the other is called tomorrow, so today is the right day to love, believe, do and, above all, live.

Dalai Lama

mineiro

The most easy paper that I have found to understand data compression is below. It's nice because the author speaks about the logic, instead of throw mathematical hierogliphs. Good introduction.

http://www.fadden.com/techmisc/hdc/index.htm

frktons

Quote from: mineiro on December 17, 2012, 11:49:05 PM
The most easy paper that I have found to understand data compression is below. It's nice because the author speaks about the logic, instead of throw mathematical hierogliphs. Good introduction.

http://www.fadden.com/techmisc/hdc/index.htm


Thanks a lot mineiro, this will be certainly useful along the
way. :t
There are only two days a year when you can't do anything: one is called yesterday, the other is called tomorrow, so today is the right day to love, believe, do and, above all, live.

Dalai Lama

jj2007

Quote from: frktons on December 17, 2012, 11:45:46 PM
It seems to work fine, Jochen :t
Thanks. One minor correction for the source posted in reply #77:
/LIBPATH:"%ProgramFiles%\PellesC\Lib"
Quotes are needed for those working with an English Windows version.

QuoteOne thing puzzles me. Your implementatio of RtlCompress function produces
a zip file 280k long, but if I do select --> sent to --> compressed folder it
produces a 190k zip file. Shouldn't them have the same compression rate?
Isn't it the same function Windows is using?  ::)

That's a known issue. Microsoft uses different algos for the two functions.

dedndave

you can use the windows zipper by shell'ing to rundll32, i think (probably via WMI, as well)
but, it only works on files - not buffers
i suppose you could make a temp file and do it that way

i guess windows 8 has a new method - re: CreateCompressor, et al
seems kinda useless

MichaelW

I could not understand how the code could work correctly but have a lower expansion rate than the statistics in the book listed, but I now see that I somehow misread the statistics. For order 3 arithmetic compression and text files the listed compression/expansion rates were 1206 and 1173. Order 3 produced the highest compression/expansion rates and the highest compression ratio for text and executables, but order 2 produced a slightly higher compression ratio for graphics.
Well Microsoft, here's another nice mess you've gotten us into.