News:

Masm32 SDK description, downloads and other helpful links
Message to All Guests
NB: Posting URL's See here: Posted URL Change

Main Menu

Third Party Libraries

Started by fearless, November 03, 2023, 09:13:39 PM

Previous topic - Next topic

fearless

Lizard: https://github.com/inikep/lizard

QuoteLizard (formerly LZ5) is an efficient compressor with very fast decompression. It achieves compression ratio that is comparable to zip/zlib and zstd/brotli

Lizard library is based on frequently used LZ4 library by Yann Collet but the Lizard compression format is not compatible with LZ4.

jj2007

Lots of good stuff, fearless :thumbsup:

One problem I see is that most if not all of these projects just care for the raw compression, not for creating e.g. a zip archive that is readable by other archivers.

fearless

Thanks.

Yes most are raw compression. One I came across that might do what you want it Miniz: https://github.com/richgel999/miniz

The version (2.1.0) I have is older (about 4 years ago) than the current release of 3.0.2. I haven't done any testing on it or the older one tbh, so don't know if there are any issues.

I managed to compile the newer ones and the include files for the older version might be workable (Miniz_302_Libs.zip contains the newer v3.02 of the Miniz for x86 and x64). Anyhow attached is the v2.1.0 of Miniz for x86 and x64 as well.

TimoVJL

TLWHViewRE had reduced MiniZ code for unpacking help files.
May the source be with you


fearless

LZFSE: https://github.com/lzfse/lzfse

QuoteLZFSE by Apple is a Lempel-Ziv style data compression algorithm using Finite State Entropy coding. It targets similar compression rates at higher compression and decompression speed compared to deflate using zlib.

fearless

LZAV: https://github.com/avaneev/lzav

QuoteFast In-Memory Data Compression Algorithm

LZAV is a fast general-purpose in-memory data compression algorithm based on now-classic LZ77 lossless data compression method. LZAV holds a good position on the Pareto landscape of factors, among many similar in-memory (non-streaming) compression algorithms.

jj2007

Quote from: fearless on January 02, 2024, 07:27:22 AMLZAV

POLINK: fatal error: File not found: 'LIBCMT.lib'

When I copy the two DLLs required to the current folder, it assembles and runs, but...
  Let esi=FileRead$("\Masm32\include\Windows.inc")
  mov eax, LastFileSize
  lea ecx, [4*eax] ; let's be very generous
  Let edi=New$(ecx)
  invoke lzav_compress_default, esi, edi, ecx, LastFileSize

... returns zero.

Usage Information
To compress data:

#include "lzav.h"

int max_len = lzav_compress_bound( src_len );
void* comp_buf = malloc( max_len ); // Or similar.
int comp_len = lzav_compress_default( src_buf, comp_buf, src_len, max_len );

if( comp_len == 0 && src_len != 0 )
{
    // Error handling.
}

fearless

Attached is a radasm test project for LZAV, I add 128 to the compressbound value before allocating the mem just in case. I dont get that linker error. Im using the microsoft linker tho, and the vcruntime and ucrt libs are from the visual studio 2022 installation i think.

NoCforMe

Quote from: jj2007 on January 02, 2024, 08:51:22 AM  Let esi=FileRead$("\Masm32\include\Windows.inc")
  mov eax, LastFileSize
  lea ecx, [4*eax]    ; let's be very generous
  Let edi=New$(ecx)
  invoke lzav_compress_default, esi, edi, ecx, LastFileSize
Don't you have those last two parameters reversed?
Quoteint comp_len = lzav_compress_default( src_buf, comp_buf, src_len, max_len );
Not sure that would make any difference ...
Assembly language programming should be fun. That's why I do it.

jj2007

Quote from: NoCforMe on January 02, 2024, 03:51:01 PMDon't you have those last two parameters reversed?

You are absolutely right - my bad :sad:

And it works now, like a charm!

include \masm32\MasmBasic\MasmBasic.inc
include LZAV_x86.inc
includelib LZAV_x86.lib
  Init
  Let esi=FileRead$("\Masm32\include\Windows.inc")
  imul ecx, LastFileSize, 17
  sar ecx, 4
  Let edi=New$(ecx)    ; filesize*17/16
  push ecx
  invoke lzav_compress_default, esi, edi, LastFileSize, ecx
  pop ecx
  deb 4, "compress  ", eax
  .if eax
    push LastFileSize
    FileWrite "test.lza", edi, eax
    Let esi=FileRead$("test.lza")
    pop eax
    invoke lzav_decompress, esi, edi, LastFileSize, eax
    deb 4, "decompress", eax
    .if signed eax>0
        FileWrite "test.txt", edi, eax
    .endif
  .endif
EndOfCode

The docs are a bit cryptic:
Quote* @param[in] src Source (uncompressed) data pointer, can be 0 if srcl==0.
 * Address alignment is unimportant.
 * @param[out] dst Destination (compressed data) buffer pointer. The allocated
 * size should be at least lzav_compress_bound() bytes large. Address
 * alignment is unimportant. Should be different to src.
 * @param srcl Source data length, in bytes, can be 0: in this case the
 * compressed length is assumed to be 0 as well.
 * @param dstl Destination buffer's capacity, in bytes.
 * @param ext_buf External buffer to use for hash-table, set to 0 for the
 * function to manage memory itself (via std malloc). Supplying a
 * pre-allocated buffer is useful if compression is performed during
 * application's operation often: this reduces memory allocation overhead and
 * fragmentation. Note that the access to the supplied buffer is not
 * implicitly thread-safe.
 * @param ext_bufl The capacity of the ext_buf, in bytes, should be a
 * power-of-2 value. Set to 0 if ext_buf is 0. The capacity should not be
 * lesser than 4*srcl, not lesser than 256, but not greater than 1
 * MiB. Same ext_bufl value can be used for any smaller sources.
 * @return The length of compressed data, in bytes. Returns 0 if srcl<=0, or
 * if dstl is too small, or if not enough memory.

I couldn't see any difference between the default option and one that does supply that buffer.
  if 1
    Let ebx=New$(524288)
    invoke lzav_compress, esi, edi, LastFileSize, ecx, ebx, 524288
    deb 4, "compress A", eax
  else
    invoke lzav_compress_default, esi, edi, LastFileSize, ecx
    deb 4, "compress B", eax
  endif

fearless

SQLite: https://www.sqlite.org


QuoteSQLite is a C-language library that implements a small, fast, self-contained, high-reliability, full-featured, SQL database engine

(Note: too large to attach to post, but they are uploaded to my github libraries repo)

https://github.com/mrfearless/libraries/blob/master/releases/sqlite3_x86.zip?raw=true

https://github.com/mrfearless/libraries/blob/master/releases/sqlite3_x64.zip?raw=true

fearless

Zlib 1.3.1: https://zlib.net/ - https://github.com/madler/zlib

QuoteA massively spiffy yet delicately unobtrusive compression library

fearless

GIFLIB v1.0 (revision 2): https://www.madwizard.org/programming/toolarchive

QuoteGIFLIB is an easy to use library that enables you to use the GIF file format in your programs. It comes as a static library which can be linked to your program. GIFLIB is very small.

https://www.madwizard.org/download/GIFLIB.zip

fearless

LMDB 0.9.70 - https://www.symas.com/lmdb

QuoteSymas LMDB is an extraordinarily fast, memory-efficient database we developed for the OpenLDAP Project. With memory-mapped files, LMDB has the read performance of a pure in-memory database while retaining the persistence of standard disk-based databases.

https://github.com/LMDB/lmdb
http://www.lmdb.tech/doc/group__mdb.html
https://blogs.kolabnow.com/2018/06/07/a-short-guide-to-lmdb