Thanks for an excellent/ detailed report!
Let's work through the issues one by one.
The first one is this annoyance around intptr_t in main.c
If we use long and forget about intptr_t.. (how it used to be once in the source) then the binary version of hjwasm works on Win7 but gives a general failure when executed on Win10...
If we use intptr_t .. it doesn't compile for some compilers like GCC ..
#ifdef __UNIX__
#if defined( intptr_t )
intptr_t fh; //fixed by ToutEnMasm intptr_t instead of long
#else
long fh; // more compatible type (linux builds etc) when intptr_t isn't valid.
#endif
#else
intptr_t fh;
#endif
This block needs to be modified to something that works with EVERY C compiler then.. suggestions welcome? :)
ALIGN3.ASO: missing warning
We intentionally removed this warning, because you would have it using align 32 as well, which was very annoying when trying to align AVX structs / data.
For this particular case I would say we need to amend the regression, as a user I would expect to align things however I see fit. [open for discussion for values > 32]?
working bottom up..
HJWASM
00000124 C5F877 vzeroall
vs
ASMC/JWASM
00000124 C5FC77 vzeroall
0000000000402004 C5 FC 77 vzeroall
0000000000402007 C5 F8 77 vzeroupper
This is what the encoding should be.. so this is a BUG.
HJWASM
000000D6 C4E37D04C101 vpermilps ymm0, ymm1, 1
vs
ASMC/JWASM
000000D6 C4E37D0CC101 vpermilps ymm0, ymm1, 1
In this case HJWASM encoding is correct and confirmed against FASM and Intel manual..
FASM = C4 E3 7D 04 C1 01
Intel Manual: VEX.256.66.0F3A.W0 04 /r ib [c4 e3 7d]= the vex portion which should be followed by 0x04.
vpsllq xmm0, xmm1, xmm2
This one is a BUG, the src/dst registers are being encoded the wrong way around.
C4E2FD190500000000 vbroadcastsd ymm0, m64
The W bit of the 3 byte vex encoding SHOULD be 0 yielding 7D not FD as the 3rd byte... bug
mov eax,HIGHWORD 80000000h appears to be leaving the top half of eax signed with 0xffff... this is wrong.
shld eax,ecx,1,2 <--- this isn't a valid instruction: SHLD eax,ecx,1 is and works ?