The MASM Forum

I bothered to have a quick play with h2incx and while the output is a tangled mess, it may be useful for folks who want to use the Watcom forks if someone wanted to do enough work on them. If you can identify the structures, the data types appear to have been converted correctly and it would not take all that much work to convert them to intelligible assembler structures that can be read. Here is one as a test piece.

  IMAGE_OPTIONAL_HEADER64 STRUCT QWORD
    Magic                           dw ?
    MajorLinkerVersion              db ?
    MinorLinkerVersion              db ?
    SizeOfCode                      dd ?
    SizeOfInitializedData           dd ?
    SizeOfUninitializedData         dd ?
    AddressOfEntryPoint             dd ?
    BaseOfCode                      dd ?
    ImageBase                       dq ?
    SectionAlignment                dd ?
    FileAlignment                   dd ?
    MajorOperatingSystemVersion     dw ?
    MinorOperatingSystemVersion     dw ?
    MajorImageVersion               dw ?
    MinorImageVersion               dw ?
    MajorSubsystemVersion           dw ?
    MinorSubsystemVersion           dw ?
    Win32VersionValue               dd ?
    SizeOfImage                     dd ?
    SizeOfHeaders                   dd ?
    CheckSum                        dd ?
    Subsystem                       dw ?
    DllCharacteristics              dw ?
    SizeOfStackReserve              dq ?
    SizeOfStackCommit               dq ?
    SizeOfHeapReserve               dq ?
    SizeOfHeapCommit                dq ?
    LoaderFlags                     dd ?
    NumberOfRvaAndSizes             dd ?
    DataDirectory                   IMAGE_DATA_DIRECTORY IMAGE_NUMBEROF_DIRECTORY_ENTRIES dup <>
  IMAGE_OPTIONAL_HEADER64 ENDS

This structure is DWORD align not QWORD
extract from Winnt.sdk

Quote
;---- DEFALIGNMASM = DWORD ----------
IMAGE_OPTIONAL_HEADER64   STRUCT DEFALIGNMASM
   Magic WORD ?
   MajorLinkerVersion BYTE ?
   MinorLinkerVersion BYTE ?
   SizeOfCode DWORD ?
   SizeOfInitializedData DWORD ?
   SizeOfUninitializedData DWORD ?
   AddressOfEntryPoint DWORD ?
   BaseOfCode DWORD ?
   ImageBase REAL8 ?
   SectionAlignment DWORD ?
   FileAlignment DWORD ?
   MajorOperatingSystemVersion WORD ?
   MinorOperatingSystemVersion WORD ?
   MajorImageVersion WORD ?
   MinorImageVersion WORD ?
   MajorSubsystemVersion WORD ?
   MinorSubsystemVersion WORD ?
   Win32VersionValue DWORD ?
   SizeOfImage DWORD ?
   SizeOfHeaders DWORD ?
   CheckSum DWORD ?
   Subsystem WORD ?
   DllCharacteristics WORD ?
   SizeOfStackReserve REAL8 ?
   SizeOfStackCommit REAL8 ?
   SizeOfHeapReserve REAL8 ?
   SizeOfHeapCommit REAL8 ?
   LoaderFlags DWORD ?
   NumberOfRvaAndSizes DWORD ?
   DataDirectory IMAGE_DATA_DIRECTORY IMAGE_NUMBEROF_DIRECTORY_ENTRIES dup (<>)
IMAGE_OPTIONAL_HEADER64      ENDS

Example was byte aligned as it is stucture in program image, there is no C alignment to handle.

How ml64 struct align actually works?
From here (http://www.ntcore.com/files/vista_x64.htm)

QuoteThe first thing to notice scrolling this code is the structure:

MSG struct     
  hwnd              dq      ?
  message           dd      ?
  padding1          dd      ?      ; padding
  wParam            dq      ?
  lParam            dq      ?
  time              dd      ?
  pt                POINT   <>
  padding2          dd      ?      ; padding
MSG ends

It requires two paddings which the x86 declaration of the same structure didn't. The reason, in a few words, is that qword members should be aligned to qword boundaries (this for the first padding). The additional padding at the end of the structure follows the rule that: every structure should be aligned to its largest member. So, being its largest member a qword, the structure should be aligned to an 8-byte boundary.

deleted

Interesting, nidud :t

Another problem:MSG2 struct QWORD
  hwnd              dq      ? ; OxPT_DebugA -Zp8 ; same effect
  byte1             db      ?
  message           dd      ?
  ; byte2             db      ? ; when active, increases size by 8
  wParam            dq      ?
  lParam            dq      ?
  time              dd      ?
  pt                POINT   <>
MSG2 ends

wParam should be aligned 8, of course, so both versions should yield the same total size:
byte1+message=5  ; less than 8
byte1+message+byte2=6  ; less than 8

But it doesn't. What's the logic here?

deleted

Quote from: nidud on September 10, 2016, 03:15:59 AM
Think the logic here is that "message" is aligned 4

Right, that makes sense :t

Actually if you do not want to have problems with structures in WIN64 with ml64, align the structures to 16.

There is a part that I don't understand: option -Zp8 says structure alignment is 8 bytes, a QWORD. But even with this option set, a SMALL_RECT is 8 bytes long, i.e. none of the Top, Right, Bottom elements is QWORD-aligned:
SMALL_RECT STRUCT
  Left      WORD      ?
  Top       WORD      ?
  Right     WORD      ?
  Bottom    WORD      ?
SMALL_RECT ENDS

So what exactly does -Zp8?  ::)

Btw with ML64, the option seems to make it hang. I have to kill the process. HJWasm and AsmC work fine.

deleted

Yes. I found a clearer description in MSDN's Structure Alignment, Visual Studio 6.0 (https://msdn.microsoft.com/en-us/library/aa296569(v=vs.60).aspx) (my highlighting):

QuoteFor example, /Zp2 permits up to 1 byte of padding, /Zp4 permits up to 3 bytes of padding, and so on. The default pack size for Windows 3.x is 2, whereas the default for Win32 is 8. As a consequence:

    If you have specified a packing limit with /Zp or #pragma pack, you may not get the proper alignment (the default value) for Win32.

Question is why padding should be required in TWell's example if the default alignment is QWORD anyway*) :

Quote from: TWell on September 09, 2016, 06:42:37 PM

Code Select Expand

MSG struct     
  hwnd              dq      ?
  message           dd      ?
  padding1          dd      ?      ; padding  <<<<<< really needed? With align 8, wParam should start at a QWORD offset...
  wParam            dq      ?
  lParam            dq      ?
  time              dd      ?
  pt                POINT   <>
  padding2          dd      ?      ; padding
MSG ends

*) Tests confirm that ML64 uses DWORD alignment by default.

Microsoft
---------

typedef struct tagMSG {
  HWND   hwnd;
  UINT   message;
  WPARAM wParam;
  LPARAM lParam;
  DWORD  time;
  POINT  pt;
} MSG, *PMSG, *LPMSG;

Vasily
------

IFDEF WIN32
   STRUCT_ALIGN   equ 4
ELSE
   STRUCT_ALIGN   equ 8
ENDIF

MSG STRUCT STRUCT_ALIGN
   hwnd      HWND ?
   message   DWORD ?
   wParam   WPARAM ?
   lParam   LPARAM ?
   time      DWORD ?
   pt      POINT<>
MSG ENDS

My own structure conversion tool.

  MSG STRUCT QWORD
    hwnd dq ?
    message dd ?
    wParam dq ?
    lParam dq ?
    time dd ?
    pt POINT <>
  MSG ENDS

My comments is that QWORD aligned structures for Win64 are safe and work well.

Quote from: hutch-- on September 10, 2016, 06:49:18 AMMy comments is that QWORD aligned structures for Win64 are safe and work well.

One would indeed have to construct/find a case where -Zp8 would lead to wrong 32-bit code: M$ says default Win32 alignment is QWORD.

_WIN64 is standard for windows x64 headers, so why not use it?
ifdef rax
_WIN64 equ= 1
else
_WIN32 equ= 1
endif

ifdef _WIN64
echo _WIN64
else
echo _WIN32
endif
end

I agree with Tim here, 8 byte alignment is the spec for Win 64, it works correctly with ML64, if anyone wants to try it the wrong way and fail, have PHUN !  :biggrin:

I started the topic to see if anyone wanted to use Japheth's h2incx to try and construct a working readable set of includes for the Watcom forks, I wonder why it changed to a debate about structures.

Quote from: hutch-- on September 10, 2016, 04:28:39 PMI started the topic to see if anyone wanted to use Japheth's h2incx to try and construct a working readable set of includes for the Watcom forks, I wonder why it changed to a debate about structures.

Quote from: jj2007 on September 10, 2016, 05:02:57 AMMSDN Structure Alignment, Visual Studio 6.0 (https://msdn.microsoft.com/en-us/library/aa296569(v=vs.60).aspx) (my highlighting):

QuoteThe default pack size for Windows 3.x is 2, whereas the default for Win32 is 8.

Hutch,

This is the trickiest single question with regard to the include files. I've checked Windows.inc (the 32-bit version) for structures containing QWORDs - there are about 17 only. And so far, luckily, there is no problem with ALIGN 4:
JIT_DEBUG_INFO STRUCT
   dwSize                 DWORD ?
   dwProcessorArchitecture DWORD ?
   dwThreadID             DWORD ?
   dwReserved0            DWORD ?
   lpExceptionAddress     QWORD ?
   lpExceptionRecord      QWORD ?
   lpContextRecord        QWORD ?
JIT_DEBUG_INFO ENDS

The dwReserved0 element provides the ALIGN 8 for lpExceptionAddress 8)

Did you sneak that in, or was it like that in the original M$ headers? Has anybody ever used this one?
SHQUERYRBINFO STRUCT
cbSize         dd ?
i64Size        qword ?
i64NumItems    qword ?
SHQUERYRBINFO ENDS

Visual Studio 6
Porting 16-Bit Code to 32-Bit Windows
1998

jj, are you on a nostalgia trip?

No, sinsi, but that was the only document where they clearly explained how structure alignment works.

> jj, are you on a nostalgia trip?  :P

https://msdn.microsoft.com/en-au/library/83ythb65.aspx?f=255&MSPPError=-2147217396 (https://msdn.microsoft.com/en-au/library/83ythb65.aspx?f=255&MSPPError=-2147217396)

Quote from: align (C++) Visual Studio 2015Without __declspec(align(#)), Visual C++ generally aligns data on natural boundaries based on the target processor and the size of the data, up to 4-byte boundaries on 32-bit processors, and 8-byte boundaries on 64-bit processors.

Bingo  :t

h2incx have problem with DECLSPEC_ALIGN(16)typedef struct DECLSPEC_ALIGN(16) _M128A {
    ULONGLONG Low;
    LONGLONG High;
} M128A, *PM128A;
typedef struct __declspec(align(16)) _M128A
{
ULONGLONG Low;
LONGLONG High;
} M128A, *PM128A;


Quote from: sinsi on September 10, 2016, 07:53:42 PM
https://msdn.microsoft.com/en-au/library/83ythb65.aspx?f=255&MSPPError=-2147217396 (https://msdn.microsoft.com/en-au/library/83ythb65.aspx?f=255&MSPPError=-2147217396)

Quote from: align (C++) Visual Studio 2015Without __declspec(align(#)), Visual C++ generally aligns data on natural boundaries based on the target processor and the size of the data, up to 4-byte boundaries on 32-bit processors, and 8-byte boundaries on 64-bit processors.

Sinsi, that is a) either Chinese or Australian and b) microsoft.com/en-au, so it's not valid for the rest of the World 8)

(besides, I have a 64-bit processor - or did they mean 64-bit OS? it's Redmond speaking, so I have my doubts ::))

Quote from: jj2007 on September 10, 2016, 05:02:57 AMMSDN Structure Alignment, Visual Studio 6.0 (https://msdn.microsoft.com/en-us/library/aa296569(v=vs.60).aspx) (my highlighting):

QuoteThe default pack size for Windows 3.x is 2, whereas the default for Win32 is 8.

Unless they recently lowered the alignment from 8 to 4 ::)

Quote from: hutch-- on September 10, 2016, 07:27:38 PM
> jj, are you on a nostalgia trip?  :P

No, I won't join the push push call fraction. I am a fan of 21st century invoke & .Repeat ... .Until ecx>=123 etc :P

#pragma pack(show)

>bin\cl -c TestPacking.c
Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86
Copyright (C) Microsoft Corporation.  All rights reserved.

TestPacking.c
TestPacking.c(1) : warning C4810: value of pragma pack(show) == 8

>bin\amd64\cl -c TestPacking.c
Microsoft (R) C/C++ Optimizing Compiler Version 14.00.40904 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

TestPacking.c
TestPacking.c(1) : warning C4810: value of pragma pack(show) == 16



> No, I won't join the push push call fraction. I am a fan of 21st century invoke & .Repeat ... .Until ecx>=123 etc

I am glad to see you are among the modern in technology, its now the mov mov mov call (do domething) ret generation. Call automation is a convenience, not a syntax.  :biggrin:

 :biggrin:

I gather from the response that no-one should hold their breath waiting for anyone else to help John and Branislav in the support system for HJWASM with main include files, prototypes and help files. I wonder what has happened to the sense of collaboration that is supposed to exist in the "open sauce" community ? These guys are actually doing good development work and deserve support, not just people sucking the life out of them for whatever they can get.

Hutch,

You have done an impressing job, first with Masm32, now with Masm64. While we are all happy with Masm32, very few people seem to consider working with the CrippleWare^TM Microsoft MASM64 assembler. You have put all your bets on this one - your choice.

I would like to help with alternatives, but Real Life^TM interferes a lot with my programming. Yet, I have managed to put up a modest 64-bit system, as shown in the screenshot below. Those who manage to install both Masm32 and MasmBasic (http://masm32.com/board/index.php?topic=94.0) can use it immediately, without bothering to install the Microsoft behemoths VS and/or SDK, without any additional libraries. It allows assembling as 64-bit or 32-bit code (see attached project: open the *.asc in RichMasm (http://masm32.com/board/index.php?topic=5314.0), hit F6), and it includes a powerful debug (http://www.webalice.it/jj2006/MasmBasicQuickReference.htm#Mb1019) macro.

> very few people seem to consider working with the CrippleWareTM Microsoft MASM64 assembler.

Strange as it is the most widely distributed assembler in the world ALA being part of VC. While it already had a full set of libraries and include files, the rest are "vapourware[TM]" in that with years of development, they still don't have a viable support system. What you cannot get with a "hand holder" is the knowledge of how Win64 works where ML64 left you with no choice on having to learn just to get it going but it is up, production capable and ready to rock 'n roll. Over 50 library modules, over 100 macros and much more to come.

It will bite the hand that feeds it but for anyone who has written low level assembler over the years, this is nothing new, having written a lot of inline assembler in my time, ml64 is a luxury in comparison, seems to have no problems at all with 256 bit AVX and when I get far enough in front I will have a look at AVX2 in 512 bit.

I have attached a development toy as I need UI code to further develop library modules and test code. It was spat out of a code generator in a few seconds. I tweaked a few bits so you could recognise it as a demo for you.

Quote from: hutch-- on September 13, 2016, 12:10:51 AMIt was spat out of a code generator in a few seconds. I tweaked a few bits so you could recognise it as a demo for you.

I feel honoured, Hutch :biggrin:

Quote from: hutch-- on September 10, 2016, 04:28:39 PM
I agree with Tim here, 8 byte alignment is the spec for Win 64, it works correctly with ML64

I've settled for this.. a while back  :biggrin:

It is important to understand that Windows couldn't care less what your assembler or C++ compiler decides.
The structures that you are passing to a Windows function are being used in a DLL, and in front of that DLL is a sign saying "compilers, stay outta here!".

So if you pass a structure with your favourite alignment, and Windows expects another one, you are doomed.

Unfortunately, the MSDN documentation is either old and false (VS 6.0 (https://msdn.microsoft.com/en-us/library/aa296569(v=vs.60).aspx): the default for Win32 is 8 bytes, VS2003 (https://msdn.microsoft.com/en-us/library/aa290049(v=vs.71).aspx): "The default for this switch is /Zp8"), or new and misleading (VS 2015 (https://msdn.microsoft.com/en-au/library/83ythb65.aspx): "Visual C++ generally aligns data on natural boundaries based on the target processor and the size of the data, up to 4-byte boundaries on 32-bit processors, and 8-byte boundaries on 64-bit processors.")

More recently, Under "Using the Windows Headers", MSDN states:

QuoteThe header files for the Windows API enable you to create 32- and 64-bit applications.
...
Controlling Structure Packing
Projects should be compiled to use the default structure packing, which is currently 8 bytes

OKAY.... that was 8 bytes in 2003, 4 bytes "on 32-bit processors" in VS2015, and 8 bytes again for Win 8.1 (https://msdn.microsoft.com/en-us/library/windows/desktop/aa383745(v=vs.85).aspx#controlling_structure_packing) ::)

So if you are a little bit uncertain, go the empirical route! Here is a test (32-bit and 64-bit executables attached) with a nasty structure:SHQUERYRBINFO STRUCT
cbSize         dd ?
i64Size        qword ?
i64NumItems    qword ?
SHQUERYRBINFO ENDS


include \Masm32\MasmBasic\Res\JBasic.inc      ; ## OPT_64 0 ; put 0 for 32 bit, 1 for 64 bit assembly ##
.data
qrb SHQUERYRBINFO <SHQUERYRBINFO, 0, 0>
Init
  PrintLine Chr$("This code was assembled with ", @AsmUsed$(1), " in ", jbit$, "-bit format")
  lea rax, qrb.i64Size
  lea rdx, qrb.cbSize
  sub rax, rdx
  deb 4, "offset i64Size - cbSize", rax
  .if rv(SHQueryRecycleBin, Chr$("C:"), addr qrb)
      Inkey Err$()
  .else
      Print Str$(" \nThere are %i elements in the recycle", dword ptr qrb.i64NumItems)
      Inkey Str$(" bin, and their total size is %i bytes\n\n", dword ptr qrb.i64Size)
  .endif
EndOfCode

Output:This code was assembled with HJWasm32 in 64-bit format
offset i64Size - cbSize rax     8

There are 2 elements in the recycle bin, and their total size is 4640 bytes

This code was assembled with HJWasm32 in 32-bit format
offset i64Size - cbSize rax     4

There are 2 elements in the recycle bin, and their total size is 4640 bytes

So it turns out that the correct alignment for 32-bit code on my 64-bit processor is 4 bytes, not 8. You can test it with -Zp8 for 32-bit code: SHQueryRecycleBin does not report an error, but you'll get 0 elements with 0 bytes.

Now the good news: By sheer coincidence, all other 17 or so structures in Windows.inc+WinExtra.inc that contain QWORDs have an even number of DWORDs before the QWORDs, so it doesn't matter whether you use -Zp4 or Zp8. Isn't that cute?

P.S.: One more good advice from Redmond, in /Zp (Struct Member Alignment), Visual Studio 2015 (https://msdn.microsoft.com/en-us/library/xh3e3fd0.aspx):  "You should not use this option" :greensml:

> Now the good news: By sheer coincidence, all other 17 or so structures in Windows.inc+WinExtra.inc that contain QWORDs have an even number of DWORDs before the QWORDs, so it doesn't matter whether you use -Zp4 or Zp8. Isn't that cute?

If you have a look at a WNDCLASSEX structure in Win64, you will see its done the same way, 8 byte aligned and doubled DWORD values to keep it that way.

PS: I should have added, the code generator is written in that "CrippleWare[tm]" Microsoft assembler as well. It really terrible to be crippled to only being able to build 64 bit EXE, DLL, OBJ in either console or UI applications.  :biggrin:

Quote from: hutch-- on September 13, 2016, 08:44:23 AMPS: I should have added, the code generator is written in that "CrippleWare[tm]" Microsoft assembler as well. It really terrible to be crippled to only being able to build 64 bit EXE, DLL, OBJ in either console or UI applications.  :biggrin:

I didn't expect you to write in HJWasm, Hutch. You are an excellent programmer, you are able to code without proper HLL elements and without a valid invoke (one that validates parameters). The question is whether it's a solution for the not-so-brilliant coder who is tempted by going X64. By the way, will you port \Masm32\examples\exampl07\slickhuh\slickhuh.asm to \Masm32\examples64?

Quote from: hutch-- on September 13, 2016, 08:44:23 AMIf you have a look at a WNDCLASSEX structure in Win64, you will see its done the same way, 8 byte aligned and doubled DWORD values to keep it that way.

\Masm32\MasmBasic\Res\DualWin.inc:
WNDCLASSA STRUCT
  style             DWORD       ?   ; oops, Redmond forgot the dwReserved ;-)
  lpfnWndProc       SIZE_P      ?
  cbClsExtra        DWORD       ?
  cbWndExtra        DWORD       ?
  hInstance         SIZE_P      ?
  hIcon             SIZE_P      ?
  hCursor           SIZE_P      ?
  hbrBackground     SIZE_P      ?
  lpszMenuName      SIZE_P      ?
  lpszClassName     SIZE_P      ?
WNDCLASSA ENDS

Fails with an exception in Win64 - unless you use option -Zp8, of course (which RichMasm kindly takes care of :biggrin:).

Thi should work as it is in 64 bit.

  WNDCLASSEXA STRUCT QWORD
    cbSize dd ?
    style dd ?
    lpfnWndProc dq ?
    cbClsExtra dd ?
    cbWndExtra dd ?
    hInstance dq ?
    hIcon dq ?
    hCursor dq ?
    hbrBackground dq ?
    lpszMenuName dq ?
    lpszClassName dq ?
    hIconSm dq ?
  WNDCLASSEXA ENDS

Quote from: hutch-- on September 13, 2016, 08:17:26 PM
Thi should work as it is in 64 bit.

Sure it does. This version works in 64- and 32-bit code (\Masm32\MasmBasic\Res\DualWin.inc):WNDCLASSEXA STRUCT
  cbSize            DWORD      ?
  style             DWORD      ?
  lpfnWndProc       SIZE_P      ?
  cbClsExtra        DWORD      ?
  cbWndExtra        DWORD      ?
  hInstance         SIZE_P      ?
  hIcon             SIZE_P      ?
  hCursor           SIZE_P      ?
  hbrBackground     SIZE_P      ?
  lpszMenuName      SIZE_P      ?
  lpszClassName     SIZE_P      ?
  hIconSm           SIZE_P      ?
WNDCLASSEXA ENDS

But the point was a different one: In the example attached above, I used the older WNDCLASS (no EX) structure, which doesn't have the even number of DWORDs:WNDCLASSA STRUCT
  style             DWORD      ?
  lpfnWndProc       SIZE_P      ?
..
WNDCLASSA ENDS

And that one works fine if you build it with Zp4 for 32-bit and Zp8 for 64-bit code, but it crashes for X64 and Zp4.

Which means that the default structure alignment of the Windows API is DWORD in 32-bit code and QWORD in 64-bit code. Both on a 64-bit processor, of course (see Reply #30); Redmond should take their documentation a bit more seriously 8)

The humour here is the old WNDCLASS is just a macro for the WNDCLASSEX for folks who want to use the old Win3.? format, it has not been a valid structure since the end of the 16 bit Windows days. Be thankful if you never had to write 16 bit Windows.  :biggrin:

Quote from: hutch-- on September 13, 2016, 11:24:59 PM
The humour here is the old WNDCLASS is just a macro for the WNDCLASSEX for folks who want to use the old Win3.? format, it has not been a valid structure since the end of the 16 bit Windows days.

Interesting that it still works on Win7-64, see Reply #32.

Btw: "macro"? See \masm32\include\Windows.inc:
IFDEF __UNICODE__
    WNDCLASS  equ  <WNDCLASSW>
ELSE
    WNDCLASS  equ  <WNDCLASSA>
ENDIF

WNDCLASSA STRUCT
  style             DWORD      ?
  lpfnWndProc       DWORD      ?
..
WNDCLASSA ENDS