Zipping files without a third party library

[jj2007]

Inspired by fearless, I added a ZipFiles macro to the latest MasmBasic version:

include \masm32\MasmBasic\MasmBasic.inc
  Init
  GfNoRecurse=1    ; current level only
  GetFolders "\Masm32\examples\exampl01\qikpad"
  AddFiles "\Masm32\examples\exampl01\qikpad\*.asm|*.inc|*.rc|*.ico|*.bmp"
  ZipFiles "MyQikpad.zip"
  ShEx "MyQikpad.zip"    ; have a look at the result... it's ShellExecute under the hood
EndOfCode

ZipFiles takes exactly one argument: the name of the archive. Use GetFiles & friends (GetFolders, AddFiles) to define which files and folders are being added. As shown above, you can limit the files to be included by extension.

Note GfNoRecurse=1 is needed here because by default GetFolders and GetFiles include all folders/files below the start directory; and the built-in Windows has the bad habit to do the same, i.e. it zips sub-folders, too - so you would end up with duplicates, no good ;-)

Recall "MyFiles.lst", Files$() is another, much simpler option to define what to zip.

Demo source & exe attached (builds with MasmBasic version 27.10.2023).

[NoCforMe]

Nice.

Can we non-MasmBasic users get a peek behind the scenes at how exactly that mechanism works, or would that be proprietary info? As the person suggested, it would be nice to be able to programmatically zip up files.

Are you simply EXECing the zip program, or are you doing the zipping yourself? (I suspect the former.)

[Vortex]

Hi Jochen,

I am not sure but does the ZipFiles macro require FreeArc? Thanks.

ZipFiles                        ; requires a FreeArc installation
    GetFiles \masm32\m32lib\*.asm        ; fill the Files$() array with the desired files
    ZipFiles "The Masm32 lib"        ; the minimum: just the name of the *.arc archive
    ; with archive name, file count, show, Launch console mode:
    ZipFiles "The_Masm32_lib", 10, SW_MINIMIZE, CREATE_NEW_CONSOLE    ; use first 10 files and a minimised new console
    ZipFiles "The_Masm32_lib", 0, SW_MINIMIZE, CREATE_NEW_CONSOLE    ; use all files in Files$() and a minimised new console
Rem    - requires FreeArc in its default location (e.g. C:\Program Files\FreeArc\bin\Arc.exe)
    - creates archive in *.arc format
    - return value: see Launch
    - use MbZipLog = 1 to see the command lines in ZipLog.txt

https://www.jj2007.eu/MasmBasicQuickReference.htm#Mb1073

[jj2007]

Nice.

Can we non-MasmBasic users get a peek behind the scenes at how exactly that mechanism works, or would that be proprietary info? As the person suggested, it would be nice to be able to programmatically zip up files.

See attachment.

Are you simply EXECing the zip program

No.

I am not sure but does the ZipFiles macro require FreeArc?

Hi Erol,

I have not updated yet the online manual. Indeed, the FreeArc thing still exists, but I've renamed it ArcFiles (the first time ever that I rename a macro ).

Note that FreeArc is a much better archiver, but, well, it means that the end user must install another program. The new ZipFiles "myfiles.zip" works out of the box on every recent Windows version. I've just tested it in my WinXP VM, and it works - very, very slow but the archive is ok.

[jj2007]

P.S.: Online manual updated.

Another usage example:

include \masm32\MasmBasic\MasmBasic.inc
$Data \Masm64\Examples\Simple
$Data \Masm64\Examples\Advanced
  Init
  Read Files$()
  ZipFiles "Masm64_examples.zip"
  ShEx "Masm64_examples.zip"    ; have a look at the result (it's ShellExecute)
EndOfCode

This one requires the text file Masm64_examples.lst with folder entries as above (see Recall):

include \masm32\MasmBasic\MasmBasic.inc
  Init
  Recall "Masm64_examples.lst", Files$()
  ZipFiles "Masm64_examples.zip"
EndOfCode

[adeyblue]

It might be buried in one of the macros but it looks like you forgot to release this interface
invoke CoCreateInstance, addr CLSID_Shell, NULL, CLSCTX_INPROC_SERVER, addr IID_IShellDispatch, edi

[jj2007]

Hi adeyblue,
Thanks for mentioning this - I am quite insecure how to handle this. So far I've done the following:

Code Select Expand

  invoke CoCreateInstance, addr CLSID_Shell, NULL, CLSCTX_INPROC_SERVER, addr IID_IShellDispatch, edi    ; addr pISD
  .if !eax
    ...
    invoke SysFreeString, vDest.bstrVal
    CoInvoke pFolder, IShellDispatch.Release    ; <<<<<< ok?
    ; deb 4, "folder released", eax
  .endif
  invoke CoUninitialize
  mov eax, fileCtOk
  ret
MbZip endp
The IShellDispatch.Release should be a Folder.Release, but that doesn't really matter because the code is identical. Still, I'll correct that in the source.

It seems to work fine, I can't see any leaked handles in Task Manager even if I call it a hundred times in a loop. Should it rather be like this?

Code Select Expand

CoInvoke pFolder, Folder.Release
CoInvoke pInterface, IShellDispatch.Release
Are both needed, or would the Folder.Release be enough, given that CoUnitialize follows suit? I've made a test with a hundred calls: there is clearly a leak when I forget the Folder.Release, but omitting IShellDispatch.Release has no consequences.

[jj2007]

It might be buried in one of the macros but it looks like you forgot to release this interface
invoke CoCreateInstance, addr CLSID_Shell, NULL, CLSCTX_INPROC_SERVER, addr IID_IShellDispatch, edi

Found this by guru Raymond Chen:

You need to take care even with local objects. Consider:

void Sample()
{
  if (SUCCEEDED(CoInitialize(NULL))) {
    CComPtr<IXMLDOMDocument> p;
    if (SUCCEEDED(p.CoCreateInstance(CLSID_IXMLDOMDocument))) {
    ...
    }
    CoUninitialize();
  }
}
Easy as pie. And there's a bug here.

When does the destructor for that smart-pointer run?

Answer: When the object goes out of scope

So the question is here, will the destructor be called in CoUninitialize?

I've made a test with a hundred calls: there is clearly a leak when I forget the Folder.Release, but omitting IShellDispatch.Release has no consequences

[adeyblue]

Then your test was incomplete, and it's pretty simple to prove so

Do
    CoInit
    CoCreate
    Print pInterface
    pInterface.Release
    CoUnint
Loop
Do that 2000 times, pInterface is always the same address or one of a very few - so the same few blocks of memory are continuously recycled

Do
    CoInit
    CoCreate
    Print pInterface
    CoUnint
Loop
Do that 2000 times, pInterface is now always a different address.

If you're feeling particularly spicy, put a HeapWalk loop after those loops and count how many times it, er, loops. Put the Release loop first so you can count the difference.

[jj2007]

Thanks for looking into this, adeyblue.

Do that 2000 times, pInterface is now always a different address.

It is, indeed. However, that is the case with or without the IShellDispatch.Release - here are the differences between the old and the new interface:

Code Select Expand

    mov eax, pInterface
    push eax
    sub eax, DiffInterface
    pop DiffInterface
    fdeb 4, "out", eax

without release:

Code Select Expand

out    eax            528
out    eax            440
out    eax            176
out    eax            264
out    eax            792
out    eax            1320
out    eax            -1672
out    eax            1760
out    eax            -2376
out    eax            1496
out    eax            -1848
out    eax            704
out    eax            1408
out    eax            -1936
out    eax            1408
out    eax            1232
out    eax            -2992
out    eax            1056
with release:

Code Select Expand

out    eax            528
out    eax            440
out    eax            176
out    eax            264
out    eax            792
out    eax            1320
out    eax            -1672
out    eax            1760
out    eax            -2376
out    eax            1496
out    eax            -1848
out    eax            704
out    eax            1408
out    eax            -1936
out    eax            1408
out    eax            1232
out    eax            -2992
out    eax            1056
No errors in both cases, both pInterface.Release (if used) and CoUninitialize return S_OK, i.e. eax==0.

I tried it with a thousand iterations. Occasional big jumps but the overall pattern is somebody grabbing memory from the heap. The working set stays put at 10MB, with or without pInterface.Release

Note I do release the folder interface in both runs; the difference is that in one of them I release pISD, i.e. the pointer to IShellDispatch, in the other run I don't. My best guess is that CoUninitialize does decrement the counter and releases it automagically. What is your explanation? You seem to know much more about COM than I do...

[TimoVJL]

Perhaps only LCOM gives same address every call.
LCOM: lightweight component object model
COM is dynamic and LCOM is often static

[jj2007]

It's the first time I hear about LCOM, Timo. It exists but apparently not on Windows: a search for "LCOM" lightweight component object model yields 25,000 hits but no official M$ docs on top.

A search for "LCOM" "lightweight component object model" yields only one hit here, and it's dead. Where did you find LCOM mentioned?

[TimoVJL]

It is actually client side feature, but used many ways.
I can't remember nor find any good in from it either.
Some COM interfaces in html and richedit use it.

Why MS don't mentioned it :
https://learn.microsoft.com/en-us/windows/win32/com/the-lightweight-client-side-handler

Basically in server-side C++ isn't good for it, only languages like C and asm.

[jj2007]

Interesting, thanks Timo

I did some more testing, and while omitting CoInvoke pInterface, IShellDispatch.Release has no impact at all, doing the same with CoInvoke pFolder, Folder.Release is not a good idea: if it's disabled, there is a 100k leak per call.

What bothers me here is that it's so badly documented. Of course, I could add the IShellDispatch.Release, apparently it does no harm, but I'd like to know what's happening under the hood: is it needed, yes or no? If yes, why?

[TimoVJL]

Globally shared memory is always problem for that kind of interfaces.
Memory is released, when usage count become zero, so release is very important.
If C++ compiler fail, code is just bad.