This is a proof of concept: A 64-bit window asks a 32-bit console application for assistance, using shared memory (MMF).
The relevant code is in the WM_KEYDOWN handler; press g to see the action (before clicking into the edit control!).
I am curious if it works everywhere :biggrin:
Source and executables attached, standard Masm32 plus MasmBasic required (http://masm32.com/board/index.php?topic=94.0) to build it (and nothing else - no other libraries or headers needed). To test it, extract to a folder and launch GetStringsFrom32BitLib.exe; to rebuild, open the sources in RichMasm (http://masm32.com/board/index.php?topic=5314.0) and hit F6.
I have not checked your apps yet but the technique works just as well the other way around, having a server in 64 bit with access to large amounts of memory that a 32 bit app can access in smaller blocks via a memory mapped file. A 64 memory extender for 32 bit apps is viable and may add some life to 32 bit apps that are running out of memory.
Quote from: hutch-- on March 10, 2017, 04:42:44 PMa server in 64 bit with access to large amounts of memory that a 32 bit app can access in smaller blocks via a memory mapped file
That sounds interesting, too :t
Just stumbled over a good article: What are the pros and cons of the different IPC methods? (http://bobmoore.mvps.org/win32/w32tip9.htm)
WM_COPYDATA works fine between 32- and 64-bit applications. I wonder if MMFs are much faster, if at all.
I think this says it all.
Quote
Memory Mapped files
MMFs don't necessarily need to be associated with a disk file, so you can use them for IPC. Since MMFs are the underlying implementation technique for most of the other IPC methods, you won't get much better performance if a shared memory technique fits your requirement
> I wonder if MMFs are much faster
I suspect that WM_COPYDATA (easier to use) is just a thin wrapper of MMFs. As an old friend of mine always says, "if in doubt, time it" :bgrin:
WM_COPYDATA involves at least two separate memory allocations and copies of whatever is sent. One in kernel mode when it places the message into the other apps queue (see below), and then one in user mode when it receives the message so the app can access the data.
// from private\ntos\w32\ntuser\kernel\sendmsg.c
if (cbCapture &&
(psms->pvCapture = UserAllocPoolWithQuota(cbCapture, TAG_SMS_CAPTURE)) != NULL) {
lParamSave = lParam;
/*
* now actually copy memory from lpCapture to psms->pvCapture
* and fixup any references to the indirect data to point to
* psms->pvCapture.
*/
switch (message) {
case WM_COPYDATA: // fnCOPYDATA
{
PCOPYDATASTRUCT pcdsNew = (PCOPYDATASTRUCT)psms->pvCapture;
lParam = (LPARAM)pcdsNew;
RtlCopyMemory(pcdsNew, pcds, sizeof(COPYDATASTRUCT));
if (pcds->lpData) {
pcdsNew->lpData = (PVOID)((PBYTE)pcdsNew + sizeof(COPYDATASTRUCT));
RtlCopyMemory(pcdsNew->lpData, pcds->lpData, pcds->cbData);
}
}
break;
Not sure about the claim MMF's are the underlying mechanism of the rest. As it mentions mailslots, pipes and sockets can go inter-machine so sharing memory pages doesn't make sense and I kinda doubt they went to the trouble of coding two separate implementations for each. WM_COPYDATA doesn't. Shared sections do. I've no idea how DDE works.
Quote from: adeyblue on March 12, 2017, 08:07:01 AM
WM_COPYDATA involves
...
Not sure about the claim MMF's are the underlying mechanism of the rest. As it mentions mailslots, pipes and sockets can go inter-machine so sharing memory pages doesn't make sense and I kinda doubt they went to the trouble of coding two separate implementations for each. WM_COPYDATA doesn't. Shared sections do. I've no idea how DDE works.
WM_COPYDATA doesn't ... what? I have no doubt that you know much more about these things than I do - if you know documentation, please post a link. I am genuinely curious. Btw from my tests it seems that COPYDATASTRUCT.lpData points to HeapAlloc'ed memory; GlobalFree and VirtualFree fail but HeapFree works.
Afaik DDE uses the same mechanism as the clipboard.
From what I understand, memory mapped "files" are basically identical areas of physical memory "mapped" into the address spaces of different processes; which explains why they are fast: both processes read and write from/to the same physical memory. Is that correct?
Doesn't use shared memory. There aren't any official docs about it, you have to look at the leaked source and/or trace the kernel mode execution.
If the copied data is over 2048 bytes then in the receiving app it's in pages allocated by VirtualAlloc, otherwise it's on the thread stack or a heap. In 32-bit proceses on x64, it's on the 64-bit thread stack, not the 32-bit one.
Yeah, that's how sections work.
Quote from: adeyblue on March 13, 2017, 04:33:37 AM
Doesn't use shared memory. There aren't any official docs about it
Both, file mappings and shared data sections are documented. Sharing memory between 32- and 64 bit processes seems also to be allowed (https://msdn.microsoft.com/en-us/library/windows/desktop/aa384203%28v=vs.85%29.aspx).
I guess you mean something different?
Yeah, I meant there's no MS documentation about how WM_COPYDATA works internally, which I think JJ was asking for.
So those things mentioned may be true now, but as it's not documented, they're not guaranteed to remain so.
Thanks. Interprocess Communication Between 32-bit and 64-bit Applications (https://msdn.microsoft.com/en-us/library/windows/desktop/aa384203%28v=vs.85%29.aspx) is interesting, but note the confusion:
Microsoft:
Quotesign-extend the handle (when passing it from 32-bit to 64-bit)
msalters (comment):
QuoteThe proper way to share such handles across process boundaries is by zero-extending 32 bits handles to 64 bits
From what I have seen the documentation for memory mapped files has not changed since the early days of Win32 and the same reference material is still used for Win64 so I would imagine that there is no restriction placed on using this technique between 32 and 64 bit apps. SendMessage/PostMessage using the handle HWND_BROADCAST works in both so both messaging and data transfer gives you any communication your need between two apps.
The obvious would apply here in terms of size limit that a 32 bit app should not have data pointed at it that is larger than its address range and as long as you stay under what win32 can allocate, just under 2 gig, you are safe.
Quote from: hutch-- on March 13, 2017, 12:13:15 PMthere is no restriction placed on using this technique between 32 and 64 bit apps. SendMessage/PostMessage using the handle HWND_BROADCAST works
Using HWND_BROADCAST is kind of an anti-social act, but FindWindows works just fine.
Using
CreateFileMapping, INVALID_HANDLE_VALUE will work between 32 and 64 bit apps because
a) there is no reason why it shouldn't - it just means your memory manager gives you a pointer to physical memory (and that is the same act for all kinds of processes and languages)
b) thousands of software developers need fast IPC.
Backwards compatibility is important for Micros**t:
jinvoke WinExec, Chr$("Notepad.exe \Masm32\include\Windows.inc"), SW_MAXIMIZEWorks just fine in a 64-bit app. Imagine what would happen if Micros**t declared next month "from now on, INVALID_HANDLE_VALUE will be -3 instead of -1. Please install the new headers from MSDN" :greenclp:
How many times did they try to discourage people from using DDE ("Atoms and Shared Memory Objects" (https://msdn.microsoft.com/en-us/library/windows/desktop/ms648774(v=vs.85).aspx#_win32_Atoms_and_Shared_Memory_Objects))? It still works with Micros**t Office...
EDIT: Apparently, DDE does no longer work with Excel 2016: (https://excel.uservoice.com/forums/304921-excel-for-windows-desktop-application/suggestions/15890263-why-dde-is-not-longer-supportet)
QuoteAnonymous commented · January 12, 2017
I agree with anonymous October 6, 2016. Mail merge is now a nightmare!
Anonymous commented · October 06, 2016
Numeric data from an Excel data source that contains formatted numbers like percentages and currency values is not transferring correctly ($55,000.00 turns into 55000, or 6.0% turns into 0.060000) for Mail Merge. There is no longer the DDE option, so what do we use?
:biggrin:
> Using HWND_BROADCAST is kind of an anti-social act
This is probably why it word so well. If you think of it, it does a fast scan of what is running, does a quick spray of the message to everything that is going, most of which reject it and only the app that traps that private message gets it.
Hmmm.... invoke SendMessage, HWND_BROADCAST, WM_CLOSE, 0, 0 works just fine but IMHO it's not really "private" ;-)
Seriously: There are proper uses of HWND_BROADCAST, but an app that asks a server to send data should know the name and class of the server, and thus can restrict the request to what FindWindow returns; besides, if FindWindow returns 0, the app can tell the user with certainty that the server is not running, or even launch it automatically.
For secure inter app communication you use a private message that is unique to your app and the other you are communicating with.
invoke PostMessage,HWND_BROADCAST,PM_YOURMESSAGE,0,0
On topic: Does anybody have experience with ZwExtendSection? Apparently, it's the only way to implement a realloc on a memory-mapped file, but it's kind of undocumented...
With a bit of messing around you could do this with the existing capacity without having to go to risky undocumented functions. Use messaging to tell both apps to shut down the existing MMF, send another message with the new size then send a message telling both to create a new MMF. Written properly it would work OK and probably be easily fast enough.
Complicated. Here is a shorter option:
mov rax, 8000000000
jinvoke CreateFileMapping, INVALID_HANDLE_VALUE, NULL, PAGE_READWRITE, 0, rax, Chr$("~TmpObject")
Works fine and returns immediately.
Quote from: jj2007 on March 14, 2017, 01:10:05 AM
On topic: Does anybody have experience with ZwExtendSection? Apparently, it's the only way to implement a realloc on a memory-mapped file, but it's kind of undocumented...
It basically does this part of the CreateFileMapping documentation, but on demand.
Quote
If an application specifies a size for the file mapping object that is larger than the size of the actual named file on disk, the file on disk is increased to match the specified size of the file mapping object
It is only for mapped disk files that you've opened for writing. It'll fail if you instead pass NULL as the file handle to CreateFileMapping. Here's an example.
#define WIN32_LEAN_AND_MEAN
#include <windows.h>
#include <stdio.h>
extern "C"
DECLSPEC_IMPORT
long
WINAPI
NtExtendSection(HANDLE hMap, PLARGE_INTEGER pNewSize);
int main()
{
HANDLE hFile = CreateFileW(L"file.txt", GENERIC_ALL, 0, NULL, OPEN_ALWAYS, FILE_ATTRIBUTE_TEMPORARY | FILE_FLAG_DELETE_ON_CLOSE, NULL);
if(hFile == INVALID_HANDLE_VALUE) {return printf("CreateFile fail - error %lu!\n", GetLastError());}
SetFilePointer(hFile, 1, NULL, FILE_BEGIN); // You can't map a 0 length file
SetEndOfFile(hFile);
DWORD size = GetFileSize(hFile, NULL);
printf("Original file size is %lu\n", size);
HANDLE hMapping = CreateFileMapping(hFile, NULL, PAGE_READWRITE, 0, 8192, NULL);
if(hMapping == NULL) {return printf("CreateFileMapping fail - error %lu!\n", GetLastError());}
size = GetFileSize(hFile, NULL);
printf("After CreateFileMapping, file size is %lu\n", size);
HANDLE hAllPermHandle = NULL;
// docs lie, CreateFileMapping doesn't create an all access handle
// it doesn't have SECTION_EXTEND_SIZE right which is required for NtExtendSection
// without this, NtExtendSection will fail with 0xc0000022 (STATUS_ACCESS_DENIED)
DuplicateHandle(GetCurrentProcess(), hMapping, GetCurrentProcess(), &hAllPermHandle, SECTION_ALL_ACCESS, FALSE, DUPLICATE_CLOSE_SOURCE);
if(hAllPermHandle == NULL) {return printf("DuplicateHandle fail - error %lu!\n", GetLastError());}
hMapping = hAllPermHandle;
PVOID pMapping = MapViewOfFile(hMapping, FILE_MAP_WRITE, 0, 0, 0);
if(pMapping == NULL) {return printf("MapFileMapping fail - error %lu!\n", GetLastError());}
MEMORY_BASIC_INFORMATION mbi = {0};
VirtualQuery(pMapping, &mbi, sizeof(mbi));
printf("Mapped size is %Iu\n", mbi.RegionSize);
LARGE_INTEGER newSize = {16384};
long stat = NtExtendSection(hMapping, &newSize);
if(stat < 0) {return printf("NtExtendSection fail - err %#x\n", stat);}
size = GetFileSize(hFile, NULL);
printf("After NtExtendSection file size is %lu\n", size);
VirtualQuery(pMapping, &mbi, sizeof(mbi));
// It doesn't update the mapped memory size though, you have to map any new bits yourself
printf("Mapped size after NtExtendSection is %Iu\n", mbi.RegionSize);
}
Quote from: adeyblue on March 14, 2017, 10:37:48 AMIt'll fail if you instead pass NULL as the file handle to CreateFileMapping.
You mean -1, INVALID_HANDLE_VALUE?
Just for fun, attached a 64-bit application that calls a 32-bit server which translates Windows.inc into a string array using Recall (http://www.webalice.it/jj2006/MasmBasicQuickReference.htm#Mb1172). The 64-bit program prints the first n strings of the array to the console.
To test it, extract all files to a folder and start GetStringsFrom32BitLib.exe; afterwards, press O (like open) to see the first 39 strings and timings. For me (Core i5), it's around 100 ticks with printing 33 lines, and around 8 ticks when printing only one line (see source, line 93, .Until ct>32). Task manager looks OK, even if I keep the "O" pressed.
Every time you "open" the file, a 800MB MMF gets created (and closed):
mov rax, 800000000
jinvoke CreateFileMapping, INVALID_HANDLE_VALUE, NULL, PAGE_READWRITE, 0, rax, Chr$("~TmpObject")
Quote from: jj2007 on March 14, 2017, 11:16:43 AM
Quote from: adeyblue on March 14, 2017, 10:37:48 AMIt'll fail if you instead pass NULL as the file handle to CreateFileMapping.
You mean -1, INVALID_HANDLE_VALUE?
NULL is accepted as -1 (or rather -1 is converted to NULL), so they're equivalent for CreateFileMapping.
Quote from: adeyblue on March 15, 2017, 04:57:48 AMNULL is accepted as -1 (or rather -1 is converted to NULL)
You have hacked the CreateFileMapping source :eusa_naughty:
Under the hood, 000007FEFD3F19D2:
000007FEFD3F19C4 | 4C 8D 4C 24 40 | lea r9, qword ptr ss:[rsp+40] |
000007FEFD3F19C9 | 89 44 24 40 | mov dword ptr ss:[rsp+40], eax |
000007FEFD3F19CD | 44 89 6C 24 44 | mov dword ptr ss:[rsp+44], r13d |
000007FEFD3F19D2 | 48 83 FD FF | cmp rbp, FFFFFFFFFFFFFFFF |
000007FEFD3F19D6 | 0F 84 C4 00 00 00 | je 7FEFD3F1AA0 |
@@:
000007FEFD3F19DC | 83 FE FF | cmp esi, FFFFFFFF |
000007FEFD3F19DF | 0F 85 23 D5 00 00 | jne 7FEFD3FEF08 |
...
000007FEFD3F1AA0 | 4D 85 C9 | test r9, r9 |
000007FEFD3F1AA3 | 0F 84 E5 D3 00 00 | je 7FEFD3FEE8E |
000007FEFD3F1AA9 | 33 ED | xor ebp, ebp |
000007FEFD3F1AAB | E9 2C FF FF FF | jmp @B ;7FEFD3F19DC |So -1 and 0 work - and no other value :t
Slightly odd the test r9, r9 at AA0; there is no such test if you pass handle zero, and with the lea r9, [rsp+40], it can never be zero ::)
Btw this is the limit on my 4GB RAM machine - slightly under one GB:
mov rax, 1020000000
jinvoke CreateFileMapping, -1, NULL, PAGE_READWRITE, 0, rax, Chr$("~TmpObject")
Quote from: jj2007 on March 13, 2017, 08:32:59 PMHow many times did they try to discourage people from using DDE ("Atoms and Shared Memory Objects" (https://msdn.microsoft.com/en-us/library/windows/desktop/ms648774(v=vs.85).aspx#_win32_Atoms_and_Shared_Memory_Objects))? It still works with Micros**t Office...
EDIT: Apparently, DDE does no longer work with Excel 2016: (https://excel.uservoice.com/forums/304921-excel-for-windows-desktop-application/suggestions/15890263-why-dde-is-not-longer-supportet)
QuoteAnonymous commented · January 12, 2017
I agree with anonymous October 6, 2016. Mail merge is now a nightmare!
Users trying to blackmail Micros**t ("If it is not included we never update our office Package to a newer Version in the whole company"): 47 comments (https://excel.uservoice.com/forums/304921-excel-for-windows-desktop-application/suggestions/18533818-we-need-dynamic-data-exchange-dde-in-combinatio?tracking_code=a40c73d4c8449041b0bc64a27fe74cd3) (and M$ regularly deletes older comments). Yeah, long live DDE (http://www.webalice.it/jj2006/MasmBasicQuickReference.htm#Mb1085) :t
Is mov rax, [ebx] possible? Test yourself...
include \Masm32\MasmBasic\Res\JBasic.inc
Init ; OPT_64 1 ; put 0 for 32 bit, 1 for 64 bit assembly
push rax ; keep align 16
push 11111111h
mov rbx, rsp
mov eax, [ebx]
Print Str$("eax=%x\n", eax) ; result: 11111111h
add rsp, SIZE_P*2
push rax ; keep align 16
push 22222222h
mov rbx, rsp
mov rax, [ebx] ; <<<<<<<<<<<<<<
Inkey Str$("rax=%x\n", rax) ; result: 22222222h
add rsp, SIZE_P*2
EndOfCode
Don't know. Seems like that Init and others expand to a lot of code...You know, i'm just guessing... :idea:
Quote from: felipe on June 06, 2018, 04:37:16 AMDon't know. ... just guessing
Why do you post if you don't know anything, and are just guessing? To increase your post count? You won't get a banana if you reach your 1,000 posts. Concentrate on your code, and eliminate your bugs, Felipe.
Quote from: jj2007 on June 06, 2018, 05:46:04 AM
Why get a banana if you reach 1,000. Concentrate on your bugs, Felipe.
You are right. :icon14:
> Is mov rax, [ebx] possible?
Even mov rax,[bx] is possible, as long as bx points to valid memory (hint: it doesn't in most modern OSes).
Quote from: sinsi on June 06, 2018, 09:08:01 AM(hint: it doesn't in most modern OSes).
Yep, that's the point. In another forum, somebody posted a school exercise with
mov eax, [bx] (probably a typo), and to my surprise, it assembled well with my default assembler, so I tried to convince Windows to accept addresses below 10000h but no cigar... so I wrote it in 64-bit code, and voilà,
mov rax, [ebx] works just fine ;)