Third Party Libraries

[fearless]

Lizard: https://github.com/inikep/lizard

Lizard (formerly LZ5) is an efficient compressor with very fast decompression. It achieves compression ratio that is comparable to zip/zlib and zstd/brotli

Lizard library is based on frequently used LZ4 library by Yann Collet but the Lizard compression format is not compatible with LZ4.

[jj2007]

Lots of good stuff, fearless

One problem I see is that most if not all of these projects just care for the raw compression, not for creating e.g. a zip archive that is readable by other archivers.

[fearless]

Thanks.

Yes most are raw compression. One I came across that might do what you want it Miniz: https://github.com/richgel999/miniz

The version (2.1.0) I have is older (about 4 years ago) than the current release of 3.0.2. I haven't done any testing on it or the older one tbh, so don't know if there are any issues.

I managed to compile the newer ones and the include files for the older version might be workable (Miniz_302_Libs.zip contains the newer v3.02 of the Miniz for x86 and x64). Anyhow attached is the v2.1.0 of Miniz for x86 and x64 as well.

[TimoVJL]

TLWHViewRE had reduced MiniZ code for unpacking help files.

[jj2007]

A very interesting recent article about compression in general and zstandard in particular

[fearless]

LZFSE: https://github.com/lzfse/lzfse

LZFSE by Apple is a Lempel-Ziv style data compression algorithm using Finite State Entropy coding. It targets similar compression rates at higher compression and decompression speed compared to deflate using zlib.

[fearless]

LZAV: https://github.com/avaneev/lzav

Fast In-Memory Data Compression Algorithm

LZAV is a fast general-purpose in-memory data compression algorithm based on now-classic LZ77 lossless data compression method. LZAV holds a good position on the Pareto landscape of factors, among many similar in-memory (non-streaming) compression algorithms.

[jj2007]

LZAV

POLINK: fatal error: File not found: 'LIBCMT.lib'

When I copy the two DLLs required to the current folder, it assembles and runs, but...

Code Select Expand

  Let esi=FileRead$("\Masm32\include\Windows.inc")
  mov eax, LastFileSize
  lea ecx, [4*eax] ; let's be very generous
  Let edi=New$(ecx)
  invoke lzav_compress_default, esi, edi, ecx, LastFileSize
... returns zero.

Usage Information

Code Select Expand

To compress data:

#include "lzav.h"

int max_len = lzav_compress_bound( src_len );
void* comp_buf = malloc( max_len ); // Or similar.
int comp_len = lzav_compress_default( src_buf, comp_buf, src_len, max_len );

if( comp_len == 0 && src_len != 0 )
{
    // Error handling.
}

[fearless]

Attached is a radasm test project for LZAV, I add 128 to the compressbound value before allocating the mem just in case. I dont get that linker error. Im using the microsoft linker tho, and the vcruntime and ucrt libs are from the visual studio 2022 installation i think.

[NoCforMe]

Code Select Expand

  Let esi=FileRead$("\Masm32\include\Windows.inc")
  mov eax, LastFileSize
  lea ecx, [4*eax]    ; let's be very generous
  Let edi=New$(ecx)
  invoke lzav_compress_default, esi, edi, ecx, LastFileSize

Don't you have those last two parameters reversed?

Code Select Expand

int comp_len = lzav_compress_default( src_buf, comp_buf, src_len, max_len );


Not sure that would make any difference ...

[jj2007]

Don't you have those last two parameters reversed?

You are absolutely right - my bad

And it works now, like a charm!

Code Select Expand

include \masm32\MasmBasic\MasmBasic.inc
include LZAV_x86.inc
includelib LZAV_x86.lib
  Init
  Let esi=FileRead$("\Masm32\include\Windows.inc")
  imul ecx, LastFileSize, 17
  sar ecx, 4
  Let edi=New$(ecx)    ; filesize*17/16
  push ecx
  invoke lzav_compress_default, esi, edi, LastFileSize, ecx
  pop ecx
  deb 4, "compress  ", eax
  .if eax
    push LastFileSize
    FileWrite "test.lza", edi, eax
    Let esi=FileRead$("test.lza")
    pop eax
    invoke lzav_decompress, esi, edi, LastFileSize, eax
    deb 4, "decompress", eax
    .if signed eax>0
        FileWrite "test.txt", edi, eax
    .endif
  .endif
EndOfCode
The docs are a bit cryptic:

* @param[in] src Source (uncompressed) data pointer, can be 0 if srcl==0.
 * Address alignment is unimportant.
 * @param[out] dst Destination (compressed data) buffer pointer. The allocated
 * size should be at least lzav_compress_bound() bytes large. Address
 * alignment is unimportant. Should be different to src.
 * @param srcl Source data length, in bytes, can be 0: in this case the
 * compressed length is assumed to be 0 as well.
 * @param dstl Destination buffer's capacity, in bytes.
 * @param ext_buf External buffer to use for hash-table, set to 0 for the
 * function to manage memory itself (via std malloc). Supplying a
 * pre-allocated buffer is useful if compression is performed during
 * application's operation often: this reduces memory allocation overhead and
 * fragmentation. Note that the access to the supplied buffer is not
 * implicitly thread-safe.
 * @param ext_bufl The capacity of the ext_buf, in bytes, should be a
 * power-of-2 value. Set to 0 if ext_buf is 0. The capacity should not be
 * lesser than 4*srcl, not lesser than 256, but not greater than 1
 * MiB. Same ext_bufl value can be used for any smaller sources.
 * @return The length of compressed data, in bytes. Returns 0 if srcl<=0, or
 * if dstl is too small, or if not enough memory.

I couldn't see any difference between the default option and one that does supply that buffer.

Code Select Expand

  if 1
    Let ebx=New$(524288)
    invoke lzav_compress, esi, edi, LastFileSize, ecx, ebx, 524288
    deb 4, "compress A", eax
  else
    invoke lzav_compress_default, esi, edi, LastFileSize, ecx
    deb 4, "compress B", eax
  endif

[fearless]

SQLite: https://www.sqlite.org

SQLite is a C-language library that implements a small, fast, self-contained, high-reliability, full-featured, SQL database engine

(Note: too large to attach to post, but they are uploaded to my github libraries repo)

https://github.com/mrfearless/libraries/blob/master/releases/sqlite3_x86.zip?raw=true

https://github.com/mrfearless/libraries/blob/master/releases/sqlite3_x64.zip?raw=true

[fearless]

Zlib 1.3.1: https://zlib.net/ - https://github.com/madler/zlib

A massively spiffy yet delicately unobtrusive compression library

[fearless]

GIFLIB v1.0 (revision 2): https://www.madwizard.org/programming/toolarchive

GIFLIB is an easy to use library that enables you to use the GIF file format in your programs. It comes as a static library which can be linked to your program. GIFLIB is very small.

https://www.madwizard.org/download/GIFLIB.zip

[fearless]

LMDB 0.9.70 - https://www.symas.com/lmdb

Symas LMDB is an extraordinarily fast, memory-efficient database we developed for the OpenLDAP Project. With memory-mapped files, LMDB has the read performance of a pure in-memory database while retaining the persistence of standard disk-based databases.

https://github.com/LMDB/lmdb
http://www.lmdb.tech/doc/group__mdb.html
https://blogs.kolabnow.com/2018/06/07/a-short-guide-to-lmdb