FASTCALL calling convention and MACROS

[bigbadbob]

I wanted to know which MACROS are doing this?

This is a portion of the source code

Code Select Expand


; ¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤

TestMsgBox proc hndl:QWORD,text:QWORD,titl:QWORD,mbstyle:QWORD

    LOCAL myHndl:QWORD
    LOCAL myText:QWORD
    LOCAL myTitle:QWORD
    LOCAL myMBStyle:QWORD

    mov rax, hndl
    mov myHndl, rax
    mov rax, text
    mov myText, rax
    mov rax, titl
    mov myTitle, rax
    mov rax, mbstyle
    mov myMBStyle, rax

    fn MessageBox,myHndl,myText,myTitle,myMBStyle
  ; ----------------------
  ; return value is in RAX
  ; ----------------------
    ret

TestMsgBox endp

; ¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤

    end


Listing file:

Code Select Expand


00000051 TestMsgBox proc hndl:QWORD,text:QWORD,titl:QWORD,mbstyle:QWORD

    LOCAL myHndl:QWORD
    LOCAL myText:QWORD
    LOCAL myTitle:QWORD
    LOCAL myMBStyle:QWORD

00000051  C8 0080 00      1       ??0047 = stackframe_default 
00000055  48/ 81 EC      1       ??0048 = ??0048 + ??0047           
   00000080
0000005C  48/ 89 4D 10      1       IF ??004A GT 0             
00000060  48/ 89 55 18      1       IF ??004A GT 1
00000064  4C/ 89 45 20      1       IF ??004A GT 2
00000068  4C/ 89 4D 28      1       IF ??004A GT 3
0000006C  48/ 8B 45 10     mov rax, hndl
00000070  48/ 89 45 98     mov myHndl, rax
00000074  48/ 8B 45 18     mov rax, text
00000078  48/ 89 45 90     mov myText, rax
0000007C  48/ 8B 45 20     mov rax, titl
00000080  48/ 89 45 88     mov myTitle, rax
00000084  48/ 8B 45 28     mov rax, mbstyle
00000088  48/ 89 45 80     mov myMBStyle, rax

    fn MessageBox,myHndl,myText,myTitle,myMBStyle
0000008C  48/ 8B 4D 98      3         ELSEIF svar EQ 8
00000090  48/ 8B 55 90      3         ELSEIF svar EQ 4
00000094  4C/ 8B 45 88      3         ELSEIF svar EQ 2
00000098  4C/ 8B 4D 80      3         ELSEIF svar EQ 1
0000009C  FF 15 00000000 E  2
  ; ----------------------
  ; return value is in RAX
  ; ----------------------
    ret
000000A2  C9      1       ENDIF
000000A3  C3      1

000000A4 TestMsgBox endp


Disassembly from NDISASM

Code Select Expand


00000451  C8800000          enter 0x80,0x0
00000455  4881EC80000000    sub rsp,0x80
0000045C  48894D10          mov [rbp+0x10],rcx
00000460  48895518          mov [rbp+0x18],rdx
00000464  4C894520          mov [rbp+0x20],r8
00000468  4C894D28          mov [rbp+0x28],r9
0000046C  488B4510          mov rax,[rbp+0x10]
00000470  48894598          mov [rbp-0x68],rax
00000474  488B4518          mov rax,[rbp+0x18]
00000478  48894590          mov [rbp-0x70],rax
0000047C  488B4520          mov rax,[rbp+0x20]
00000480  48894588          mov [rbp-0x78],rax
00000484  488B4528          mov rax,[rbp+0x28]
00000488  48894580          mov [rbp-0x80],rax
0000048C  488B4D98          mov rcx,[rbp-0x68]
00000490  488B5590          mov rdx,[rbp-0x70]
00000494  4C8B4588          mov r8,[rbp-0x78]
00000498  4C8B4D80          mov r9,[rbp-0x80]
0000049C  FF155E0F0000      call qword [rel 0x1400]
000004A2  C9                leave
000004A3  C3                ret


I know that invoke and fn are identical MACROS.
What I noticed in my PROC is that the shadow space is saved right after entry.
Then I added some instructions to copy the parameters to local variables.
Then I called fn which came automatically generated for me.  I just changed the parameters.
I noticed that fn is only writing to the 4 registers.

[bigbadbob]

In the Helpfile masm64.chm we have the following:

Invoke style procedure call automation

Arguments are passed left to right with the first 4 arguments being written to 4 specific registers, RCX, RDX, R8 and R9, the rest are writtten to a sequence of specific stack locations. With the first four arguments, if you use any of the registers that the first four arguments are written to you risk overwriting an argument. For example if your third argument is RDX, it will be overwritten by the value written to the second register RDX. You can avoid the problem by either using other available registers or writing memory operands instead of registers.

This does not mention using the shadow space.  I believe that it is skipped.

In this help file MasmHelp.exe.

    The "invoke" series of macros write the first 4 args to shadow
    space as well as the 4 specified registers then writes additional
    args to the correct stack locations. The argument count limit is
    24 arguments in total.

So the 2 help files do not even match.
My disassembly and list file show that although the shadow space is being saved "invoke" and "fn" are not the MACROS that are saving to the shadow space.

I was wondering if the STACKFRAME and the NOSTACKFRAME macros are in control of that.
I think that using STACKFRAME hooks into the PROC somehow, but I have not determined how yet.  If it is enabled the shadow space is copied to.

[hutch--]

The macros for both the stackframe and the procedure call that other wrappers call are by no means easy to read, they are large and complex macros but the example I posted for you in the other thread makes it clear that invoke write to both shadow space and the registers, that is why I XORRED the 4 registers to show that the names in the proc head are used which means they come from shadow space. This is why I posted this code in the other thread.

; ¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤

    include \masm32\include64\masm64rt.inc

    .code

; ¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤

entry_point proc

    invoke testproc,150,300,450,600

    waitkey

    .exit

entry_point endp

; ¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤

testproc proc arg1:QWORD,arg2:QWORD,arg3:QWORD,arg4:QWORD

  ; clear the 4 registers

    xor rcx, rcx
    xor rdx, rdx
    xor r8, r8
    xor r9, r9

  ; display the values from shadow space

    conout str$(arg1),lf
    conout str$(arg2),lf
    conout str$(arg3),lf
    conout str$(arg4),lf

    ret

testproc endp

; ¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤

    end


The piece that you are missing is what MASM does with the input data and where it ends up at the head of the proc.

The first section that you quoted from the CHM help file actually addresses another problem inherent in the FASTCALL convention, the risks of trying to load registers in the first 4 arguments where you risk overwriting a register that is used in the first 4 args.

[hutch--]

Here is an example for you, the identical algorithm with both a stack frame and without. The comments in the disassembly are mine.

; ¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤

    include \masm32\include64\masm64rt.inc

    .code

; ¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤

entry_point proc

    LOCAL retval    :QWORD

    invoke slen1,"1234567890"
    mov retval, rv(slen1,"1234567890")
    conout str$(retval)," invoke call",lf

    .data
      numbers db "1234567890",0
      pnum dq numbers
    .code

    mov retval, rvcall(slen2,pnum)
    conout str$(retval)," register call",lf


    waitkey

    invoke ExitProcess,0

    ret

entry_point endp

; ¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤

slen1 proc psrc:QWORD

    mov rax, psrc
    sub rax, 1
  @@:
    add rax, 1
    cmp BYTE PTR [rax], 0
    jnz @B

    sub rax, psrc

    ret

slen1 endp

; ¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤

NOSTACKFRAME

slen2 proc

    mov rax, rcx
    sub rax, 1
  @@:
    add rax, 1
    cmp BYTE PTR [rax], 0
    jnz @B

    sub rax, rcx

    ret

slen2 endp

STACKFRAME

; ¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤

    end

comment #

; --------------------------------------------------------------------------
; sub_1400010d8
; --------------------------------------------------------------------------
sub_1400010d8   proc
.text:00000001400010d8 C8800000                   enter 0x80, 0x0                   ; set up the stack frame
.text:00000001400010dc 4883EC60                   sub rsp, 0x60                     ; set up stack space
.text:00000001400010e0 48894D10                   mov qword ptr [rbp+0x10], rcx     ; write register to shadow space
.text:00000001400010e4 488B4510                   mov rax, qword ptr [rbp+0x10]     ; shadow space variable to register
.text:00000001400010e8 4883E801                   sub rax, 0x1
.text:00000001400010ec
.text:00000001400010ec 0x1400010ec:
.text:00000001400010ec 4883C001                   add rax, 0x1
.text:00000001400010f0 803800                     cmp byte ptr [rax], 0x0
.text:00000001400010f3 75F7                       jne 0x1400010ec
.text:00000001400010f3
.text:00000001400010f5 482B4510                   sub rax, qword ptr [rbp+0x10]     ; sub address in shadow space from rax
.text:00000001400010f9 C9                         leave                             ; clean up after stack frame
.text:00000001400010fa C3                         ret
sub_1400010d8   endp

; --------------------------------------------------------------------------
; sub_1400010fb
; --------------------------------------------------------------------------
sub_1400010fb   proc
.text:00000001400010fb 488BC1                     mov rax, rcx
.text:00000001400010fe 4883E801                   sub rax, 0x1
.text:0000000140001102
.text:0000000140001102 0x140001102:
.text:0000000140001102 4883C001                   add rax, 0x1
.text:0000000140001106 803800                     cmp byte ptr [rax], 0x0
.text:0000000140001109 75F7                       jne 0x140001102
.text:0000000140001109
.text:000000014000110b 482BC1                     sub rax, rcx
.text:000000014000110e C3                         ret
sub_1400010fb   endp

.text:000000014000110f CC                         int3
; --------------------------------------------------------------------------

#

[bigbadbob]

I believe that MASM is copying from register parameters to the shadow space.
It is just not appear to be case that invoke is doing it.
For the two procs that are functionally the same, in the first one (slen1) the shadow space is saved, but there is no invoke within the proc.  So I think that shadow space is saved without invoke.

[hutch--]

Bob,

Its not guesswork, the difference between the two is the stackframe. With it you get the written shadow space, without it you don't. The procedure call macro that gives you "invoke" just writes the arguments in the right order and in the right places. The "rcall" and "rvcall" macros write directly to registers, from none to 4, after that it generates an error as it is only designed to handle direct register calls.

The two macro forms are presented as working which they do as most have difficulties writing MASM macros. For any who want to know the detail of how to write them, beware it is not for the faint of heart, the MASM pre-processor is a bad mannered old pig with appalling documentation and serious bugs. You write them by test piece until they test up correctly and that is after years of practice.

Both macro type fully conform to the Microsoft 64 bit ABI so you can just use them unless you want to write your own.