OPTION STACKBASE fails in x86

[hutch--]

This is a very good idea john, get the core guts of the assembler stable so you don't have to keep messing with it trying to do the impossible. Once you have the stable core you can then at your leisure keep adding features without messing up the very complex stuff.

The suggestion I made before was to have a simple and safe option so that everything works reliably which would be the default. Then have the options to do things differently, no stack frame at all, SSE call, AVX/2 call, with or without ESP, once you have the core, adding flexibility comes at the risk of the user's coding ability getting it right.

Old 32 bit and earlier MASM had a default stack frame with the optional USES modifier which was safe and reliable but you could always turn it off for short fast code. I imagine there were folks who wrote their own prologue / epilogue code but there were not many. As long as you maintain the custom prologue / epilogue capacity, the brave or foolish can cook up their own if they really want to.

What I did in 64 bit MASM was to write a custom prologue / epilogue and a matching "invoke" style notation macro that handles the routine API and procedure calls, I also wrote a simple reg only call macro for up to the first 4 args that would not accept over 4 args and that did not write to shadow space. One major difference was testing many things with the 64 bit ABI and one thing was no stack address alteration with arguments and this was done by not using PUSH POP at all in the calling technique. This meant that you can pass any size argument up to 64 bit (byte, word, dword and qword) and they all fit into the same stack addresses.

The macro that provides the "invoke" notation writes 64 bit arguments directly to the sequence of stack locations which with MASM turn up at the right locations in a written procedure from arg 5 upwards. The macro auto-fills up to the first four reg arguments into shadow space so with a written "proc" you just use the passed arguments in a normal high level way.

Writing your own assembler would give you many more options than what can be done with macros and this in conjunction with your own inbuilt macro capacity should give you all of the extra calling conventions you are after.