Stack problem

[minor28]

I am writing my first slightly larger 64 bit program to transfer videodata from four ip cameras over the internet. So far it has gone well and it works.

Now I was going to use a subroutine in a function as I have done several times in 32 bit programs. But I became aware that retn does not work for 64 bit. The solution was that instead of retn I use jmp as follows.

Code Select Expand


LoginProcess proc pCam:qword
...
push rdi
mov rdi,pCam
...
call @SendRequest
...
pop rdi
mov rax,1
ret

@SendRequest:
mov r8,HEADER_SIZE
lea rdx,hd
mov rcx,qword ptr [rdi].CAMERADATA.SendBuf.buf
call RtlMoveMemory

sub rsp,32
mov rcx,qword ptr [rdi].CAMERADATA.sfd
mov rdx,qword ptr [rdi].CAMERADATA.SendBuf.buf
mov r8D,dword ptr [rdi].CAMERADATA.SendBuf.lngth
mov r9,0
call send
add rsp,32
.if eax == SOCKET_ERROR
invoke MessageBox,hMainWnd,qword ptr [rdi].CAMERADATA.SendBuf.buf, 0, MB_ICONERROR or MB_OK
mov rcx,rdi
call FreeBuf
xor rax,rax
ret
.endif
pop rax
jmp rax

LoginProcess endp


The first attempt was without sub and add rsp,32 and I found that the stack became corrupt.

"It is the caller's responsibility to allocate 32 bytes of "shadow space" (for storing RCX, RDX, R8, and R9 if needed) before calling the function."

If needed! In my subroutine this was needed for the send function but not for RtlMoveMemory. The rax now holds the return address to jump to.

I am using masm64rt.inc.

I have also noticed that the stack is corrupt when I restore rdi. How do I know if shadow space is needed?

I would be grateful if someone could enlighten me on how to handle the stack. For example is it necessary for all include or call to both own and external functions.

[Mikl__]

Hi, minor28!
In 64-bit program your RSP-register contents must be a multiple of 16. Here either you need

• to push the contents of a pair of registers to the stack

Code Select Expand

push RDI
push RDI
call @SendRequest
pop RDI
pop RDI2. or use local variables to save the contents of registers

Code Select Expand

mov old_rdi,RDI
call @SendRequest
mov RDI,old_rdi

[mineiro]

Code Select Expand

LoginProcess proc pCam:qword            ;rsp=???8   not ok to call something
...
push rdi                            ;rsp=???0   ok to call something
mov rdi,pCam
...
call @SendRequest                   ;rsp=???0   call change rsp, so rsp ended with 8 at entry point of called procedure
...
pop rdi
mov rax,1
ret

@SendRequest:                           ;rsp=???8   from previous aligned stack minus call return address
mov r8,HEADER_SIZE
lea rdx,hd
mov rcx,qword ptr [rdi].CAMERADATA.SendBuf.buf
call RtlMoveMemory                  ;<---- not ok to call something, rsp=???8h, should ended with 0

sub rsp,32                          ;sub rsp,20h, stack continues unaligned, rsp=???8h
mov rcx,qword ptr [rdi].CAMERADATA.sfd
mov rdx,qword ptr [rdi].CAMERADATA.SendBuf.buf
mov r8D,dword ptr [rdi].CAMERADATA.SendBuf.lngth
mov r9,0
call send                           ;<--- not ok to call something
add rsp,32                          ;continues not ok to call something
.if eax == SOCKET_ERROR
invoke MessageBox,hMainWnd,qword ptr [rdi].CAMERADATA.SendBuf.buf, 0, MB_ICONERROR or MB_OK     ;<-- not ok
mov rcx,rdi
call FreeBuf                    ;<--- not ok
xor rax,rax
ret
.endif
pop rax
jmp rax

LoginProcess endp


At entry point of your program, rsp=???8h, so you should sub 32 (20h) to shadow space + 8 to align stack pointer.
So consecutive calls can be done because rsp ends with ???0h.
After stack aligned and you do a call to a function, rsp turns back to ???8 at entry of that procedure code, so inside a procedure you should deal with rsp again (sub rsp,8)to be rsp=???0. At end of procedure, before ret instruction, add rsp,8.
If function have 5 parameters, rcx,rdx,r8,r9 + push (sub rsp,8) as an example, you can insert a foo push to parameters be even instead of odd; so stack gets aligned (rsp=???0) and you're able to call a function. So, count stack parameters (pairs) and subtract(in this case, add)  that after function being called.
Exception happens when a function being called does not return values (does not have a instruction call inside)(leaf functions), so you can call that function with rsp=???8 or rsp=???0; but good habits say to align stack before call instruction.

If you're creating a stack frame to local variables, remember that a push rbp is done. So, a function like:
somefunction proc uses r8 r9 r10 r11
local one:qword
local two:qword

will be aligned, because will save r8 r9 r10 r11 + rbp, or in other words, 5*8. So, rsp starts at rsp=???8h + 5*8 = ???0h
Take same concept to local variables, pairs.

1- align stack at entry point of your code, so you can call some function
2- function have odd numbers of parameters when > 4 (rcx,rdx,r8,r9) parameters!?, turn then even
3- have N local variables inside that procedure, turn N local variables even.

Not sure if helps.

[hutch--]

You are probably getting problems with procedure alignment and I would advise against manually using push/pop in 64 bit as its really easy to make a mess of it. I have attached a zip file with the current help file and latest macro file.  In the help file, look at the section "Stackframes" and these are designed to use the normal "proc" form and you have enough options for different circumstances.

[minor28]

Thank you all for your answers. I now understand that I have to be vigilant with the stack. I have to go through everything I have written so far and correct the stack. I'm going to read through the help file.

Thank you.