Using 32-bit code in 64-bit land

[jj2007]

This is a proof of concept: A 64-bit window asks a 32-bit console application for assistance, using shared memory (MMF).
The relevant code is in the WM_KEYDOWN handler; press g to see the action (before clicking into the edit control!).
I am curious if it works everywhere

Source and executables attached, standard Masm32 plus MasmBasic required to build it (and nothing else - no other libraries or headers needed). To test it, extract to a folder and launch GetStringsFrom32BitLib.exe; to rebuild, open the sources in RichMasm and hit F6.

[hutch--]

I have not checked your apps yet but the technique works just as well the other way around, having a server in 64 bit with access to large amounts of memory that a 32 bit app can access in smaller blocks via a memory mapped file. A 64 memory extender for 32 bit apps is viable and may add some life to 32 bit apps that are running out of memory.

[jj2007]

a server in 64 bit with access to large amounts of memory that a 32 bit app can access in smaller blocks via a memory mapped file

That sounds interesting, too :t

Just stumbled over a good article: What are the pros and cons of the different IPC methods?

WM_COPYDATA works fine between 32- and 64-bit applications. I wonder if MMFs are much faster, if at all.

[hutch--]

I think this says it all.

Memory Mapped files

MMFs don't necessarily need to be associated with a disk file, so you can use them for IPC. Since MMFs are the underlying implementation technique for most of the other IPC methods, you won't get much better performance if a shared memory technique fits your requirement

[jj2007]

> I wonder if MMFs are much faster

I suspect that WM_COPYDATA (easier to use) is just a thin wrapper of MMFs. As an old friend of mine always says, "if in doubt, time it"

[adeyblue]

WM_COPYDATA involves at least two separate memory allocations and copies of whatever is sent. One in kernel mode when it places the message into the other apps queue (see below), and then one in user mode when it receives the message so the app can access the data.

Code Select Expand


// from private\ntos\w32\ntuser\kernel\sendmsg.c
if (cbCapture &&
                (psms->pvCapture = UserAllocPoolWithQuota(cbCapture, TAG_SMS_CAPTURE)) != NULL) {

            lParamSave = lParam;

            /*
             * now actually copy memory from lpCapture to psms->pvCapture
             * and fixup any references to the indirect data to point to
             * psms->pvCapture.
             */
            switch (message) {
            case WM_COPYDATA:     // fnCOPYDATA
                {
                    PCOPYDATASTRUCT pcdsNew = (PCOPYDATASTRUCT)psms->pvCapture;
                    lParam = (LPARAM)pcdsNew;
                    RtlCopyMemory(pcdsNew, pcds, sizeof(COPYDATASTRUCT));
                    if (pcds->lpData) {
                        pcdsNew->lpData = (PVOID)((PBYTE)pcdsNew + sizeof(COPYDATASTRUCT));
                        RtlCopyMemory(pcdsNew->lpData, pcds->lpData, pcds->cbData);
                    }
                }
                break;


Not sure about the claim MMF's are the underlying mechanism of the rest. As it mentions mailslots, pipes and sockets can go inter-machine so sharing memory pages doesn't make sense and I kinda doubt they went to the trouble of coding two separate implementations for each. WM_COPYDATA doesn't. Shared sections do. I've no idea how DDE works.

[jj2007]

WM_COPYDATA involves
...
Not sure about the claim MMF's are the underlying mechanism of the rest. As it mentions mailslots, pipes and sockets can go inter-machine so sharing memory pages doesn't make sense and I kinda doubt they went to the trouble of coding two separate implementations for each. WM_COPYDATA doesn't. Shared sections do. I've no idea how DDE works.

WM_COPYDATA doesn't ... what? I have no doubt that you know much more about these things than I do - if you know documentation, please post a link. I am genuinely curious. Btw from my tests it seems that COPYDATASTRUCT.lpData points to HeapAlloc'ed memory; GlobalFree and VirtualFree fail but HeapFree works.

Afaik DDE uses the same mechanism as the clipboard.

From what I understand, memory mapped "files" are basically identical areas of physical memory "mapped" into the address spaces of different processes; which explains why they are fast: both processes read and write from/to the same physical memory. Is that correct?

[adeyblue]

Doesn't use shared memory. There aren't any official docs about it, you have to look at the leaked source and/or trace the kernel mode execution.
If the copied data is over 2048 bytes then in the receiving app it's in pages allocated by VirtualAlloc, otherwise it's on the thread stack or a heap. In 32-bit proceses on x64, it's on the 64-bit thread stack, not the 32-bit one.

Yeah, that's how sections work.

[qWord]

Doesn't use shared memory. There aren't any official docs about it

Both, file mappings and shared data sections are documented. Sharing memory between 32- and 64 bit processes seems also to be allowed.
I guess you mean something different?

[adeyblue]

Yeah, I meant there's no MS documentation about how WM_COPYDATA works internally, which I think JJ was asking for.
So those things mentioned may be true now, but as it's not documented, they're not guaranteed to remain so.

[jj2007]

Thanks. Interprocess Communication Between 32-bit and 64-bit Applications is interesting, but note the confusion:
Microsoft:

sign-extend the handle (when passing it from 32-bit to 64-bit)

msalters (comment):

The proper way to share such handles across process boundaries is by zero-extending 32 bits handles to 64 bits

[hutch--]

From what I have seen the documentation for memory mapped files has not changed since the early days of Win32 and the same reference material is still used for Win64 so I would imagine that there is no restriction placed on using this technique between 32 and 64 bit apps. SendMessage/PostMessage using the handle HWND_BROADCAST works in both so both messaging and data transfer gives you any communication your need between two apps.

The obvious would apply here in terms of size limit that a 32 bit app should not have data pointed at it that is larger than its address range and as long as you stay under what win32 can allocate, just under 2 gig, you are safe.

[jj2007]

there is no restriction placed on using this technique between 32 and 64 bit apps. SendMessage/PostMessage using the handle HWND_BROADCAST works

Using HWND_BROADCAST is kind of an anti-social act, but FindWindows works just fine.

Using CreateFileMapping, INVALID_HANDLE_VALUE will work between 32 and 64 bit apps because
a) there is no reason why it shouldn't - it just means your memory manager gives you a pointer to physical memory (and that is the same act for all kinds of processes and languages)
b) thousands of software developers need fast IPC.

Backwards compatibility is important for Micros**t:
 

Code Select Expand

jinvoke WinExec, Chr$("Notepad.exe \Masm32\include\Windows.inc"), SW_MAXIMIZE

Works just fine in a 64-bit app. Imagine what would happen if Micros**t declared next month "from now on, INVALID_HANDLE_VALUE will be -3 instead of -1. Please install the new headers from MSDN"

How many times did they try to discourage people from using DDE ("Atoms and Shared Memory Objects")? It still works with Micros**t Office...

EDIT: Apparently, DDE does no longer work with Excel 2016:

Anonymous commented  ·  January 12, 2017

I agree with anonymous October 6, 2016. Mail merge is now a nightmare!

Anonymous commented  ·  October 06, 2016

Numeric data from an Excel data source that contains formatted numbers like percentages and currency values is not transferring correctly ($55,000.00 turns into 55000, or 6.0% turns into 0.060000) for Mail Merge. There is no longer the DDE option, so what do we use?

[hutch--]

 

> Using HWND_BROADCAST is kind of an anti-social act

This is probably why it word so well. If you think of it, it does a fast scan of what is running, does a quick spray of the message to everything that is going, most of which reject it and only the app that traps that private message gets it.

[jj2007]

Hmmm.... invoke SendMessage, HWND_BROADCAST, WM_CLOSE, 0, 0 works just fine but IMHO it's not really "private" ;-)

Seriously: There are proper uses of HWND_BROADCAST, but an app that asks a server to send data should know the name and class of the server, and thus can restrict the request to what FindWindow returns; besides, if FindWindow returns 0, the app can tell the user with certainty that the server is not running, or even launch it automatically.