Diff \ patch tools for binary files

[Vortex]

Diff \ patch utilities for binary files are useful to save storage space and reduce transfer time over the network.

A new toolset in GitHub, qbdiff :

https://github.com/kspalaiologos/qbdiff

Quick Binary Diff. An improved and modernised version of bsdiff. An overview:

Faster than bsdiff, can take advantage of multiple CPU cores.
Generally smaller patch files than bsdiff, supports only LZMA compression which performs excellently on sparse binary data.
Requires less memory than bsdiff.
Based on Colin Percival's algorithm used in bsdiff.
Integrity checking using BLAKE2b.
Can be used as a library.

I managed to build the toolset with Msys2 :

Building the project :

autoreconf -i
./configure
make
make install

The executables and dependencies :

Code Select Expand

libgcc_s_seh-1.dll
libgomp-1.dll
liblzma-5.dll
libqbdiff-0.dll
libwinpthread-1.dll
qbdiff64.exe
qbpatch64.exe

[Vortex]

Another project, HDiffPatch :

https://github.com/sisong/HDiffPatch

a C\C++ library and command-line tools for Diff & Patch between binary files or directories(folder); cross-platform; runs fast; create small delta/differential; support large files and limit memory requires when diff & patch.

[jj2007]

Hi Erol,

So what do these tools do? If I understand it correctly, you supply two folders, "old" and "new", and the tool checks the differences and stores them in an archive.

You send the archive to a machine that has the "old" folder, and it creates the "new" folder without having to copy gigabytes around.

Is that correct? If yes, why does Microsoft updates that require gigabyte downloads?

[Vortex]

Hi Jochen,

We can practically say that every Windows setup is unique as device drivers, event log files, temporary files and especially registry files are different on every Windows computer. It's impossible to find two computers having exactly the same Windows folder content so the diff \ patch method is not suitable to maintain Windows based devices.

A lot of diff \ patch utilities like bsdiff are memory intensive applications, I can also introduce the rdiff tool which is more modest in terms of memory usage :

https://man.archlinux.org/man/rdiff.1.en

https://github.com/librsync/librsync/blob/master/doc/rdiff.md

32-bit version of rdiff :

https://www.timedicer.co.uk/rdiff.zip

https://www.timedicer.co.uk/rdiff.php

Here is a case how to use the rdiff tool. Let's assume that you and your friend have a copy of the Windows 10 installation iso. Your friend is thinking to setup another localised version of Windows, for example the French version. The iso files and their respective sizes :

Code Select Expand

Win10_22H2_English_x64.iso   6.115.186.688 bytes
Win10_22H2_French_x64.iso    6.147.790.848  bytes
How to send the French version to your friend? The size difference of both of the ISOs is not considerable, they should deliver nearly the same content. We can try to create a signature file based on the English release.

Code Select Expand

rdiff.exe signature Win10_22H2_English_x64.iso Win10_22H2_English_x64.iso.rdiffsign
Creating the delta file, the file including all the binary difference :

Code Select Expand

rdiff.exe delta Win10_22H2_French_x64.iso.rdiffsign Win10_22H2_English_x64.iso Win10_22H2_English_x64.iso.rdiffdelta
As you can see, once you create the signature file, you can use it to make other delta files. ( or patch files ) For example a delta for the Italian release of Windows and another one for the German version. Other tools like bsdiff and qbdiff would require you to generate the deltas based on the English iso since they don't produce signature files. Consider the signatures like the hash of the original files.

File sizes and the names :

Code Select Expand

6.115.186.688   Win10_22H2_English_x64.iso
107.493.528   Win10_22H2_English_x64.iso.rdiffsign
6.147.790.848   Win10_22H2_French_x64.iso
867.790.953   Win10_22H2_French_x64.iso.rdiffdelta
108.066.648   Win10_22H2_French_x64.iso.rdiffsign

The delta file is nearly the 1/7 of the original French file. Your friend receives this delta \ patch file and rebuilds the French iso :

Code Select Expand

rdiff.exe patch Win10_22H2_English_x64.iso Win10_22H2_French_x64.iso.rdiffdelta Win10_22H2_French_x64.iso
You can search the net to find the SHA1 value of the iso files to verify the integritiy of your setup files.

Cygwin and Msys2 are providing the 64-bit version of rdiff, I can help you if you wish to receive a copy.

The English delta file based on the French iso :

Code Select Expand

833.256.487   Win10_22H2_English_x64.iso.rdiffdelta