Debugging: how to recognise a no stack frame proc

[jj2007]

Where is the bug? The proc is 200+ lines, here are the 10% that caused a bug that was well hidden:

Code Select Expand

xor ecx, ecx
.if edx==STILL_ACTIVE
push chr$("'still active'")
.if [ebx.BuildVars.IsConsole] ; a Windows app is always still active...
dec ecx ; set the flag
.endif
.else
dec ecx
push str$(edx)
.endif
lea edi, buffer
jecxz @F
push eax
invoke lstrcpy, edi, Chr$("** RichMasm warning:", 13, 10, "exit code ")
push edi
call lstrcat
invoke lstrcat, edi, chr$(" **", 13, 10)
SetWin$ hStatus=edi
@@:

I caught it by comparing the stack at start and end of the proc with a dedicated macro that replaced the ret before the endp. Over 100 rets in the 21,000 lines source are ok, this one misbehaved.

The nastiest aspect here is that often you will not notice the problem, because the leave or mov esp, ebp makes sure you return to the correct address. But you are returning there with incorrectly restored esi, edi and ebx... and that may or may not cause a crash, and it may happen when you least suspect it. One of those bugs that will cause you sleepless nights

So far the routine works fine, and once it will be well tested and stable enough, I will post macro and *.lib file.

But I have a question: Does anybody have an idea how to safely recognise the start and end of a proc that has no stack frame?

[jj2007]

In the meantime, I found at least a method to recognise that a proc has no stack frame.

The idea is very simple: in a large source, replace every ret that is followed by somecode endp with @MbRet. It takes an optional argument that should be the name of the proc; if there is no argument, the source line will be shown instead (as in the CodeNoArgs example below).

If the source has stack problems, they will be ## flagged to the console ##, even if the exe was built as a Windows application.

There are basically three cases:
1. too many push instructions: @MbRet will tell you about it, and correct the stack so that at least the uses esi edi ebx works properly.

2. too many pop instructions: @MbRet will tell you about it, and correct the stack but one or more of the regs in the uses esi edi ebx list will be trashed anyway.

3. stack corruption: the return address is somewhere in nirwana land. @MbRet will inform you about it, but the program will crash anyway.

I've tested it with a 20k lines source, and so far I've found one bug with it, see first post above. @MbRet tests runtime behaviour, so its messages will pop up only when you are really using a proc. So for rarely used exotic functions it may take a while until you discover the bug.

Once you have verified that there are no problems, put @MbRet off below the include line, so that only an ordinary ret will be generated.

Here is a little demo with some procs that treat the stack badly (tested ok with WinXP, Win7-64 and Win10):

Code Select Expand

include \masm32\include\masm32rt.inc ; purest Masm32 SDK code
include MbRet.inc ; must be in the exe's folder
; @MbRet off ; switch it off once tests are OK

.code
TwoArgsNoLocals proc uses esi edi ebx mode, crashArg
  mov ebx, mode ; no Locals, but has args, and therefore a stack frame
  print "2 args: ebx="
  print str$(ebx), 13, 10
  .while sdword ptr ebx>=0
push eax ; stack imbalances will be flagged
dec ebx
  .Endw
  pop eax ; stack is balanced
  pop ecx ; for mode=3
  pop edx
  .if crashArg==99
mov dword ptr [ebp+4], 12345678h ; bad news for the return address ;-)
  .endif
  @MbRet 2ArgsNoLocals ; instead of the ret
TwoArgsNoLocals endp

ThreeArgsLocal123 proc uses ecx mode, arg2, arg3
Local buffer[123]:BYTE
  push eax ; stack not balanced, will be flagged
  @MbRet ThreeArgsLocal123
ThreeArgsLocal123 endp

CodeNoArgs proc uses esi edi ebx ecx ebp
  print "CodeNoArgs - cannot work with @MbRet: "
  ; push eax ; activate to produce a crash
  @MbRet ; no label -> line number 32; @MbRet will say "no frame found"
CodeNoArgs endp

start:
  cls
  m2m esi, 3
  mov ebx, 12345678h
  .Repeat
invoke TwoArgsNoLocals, esi, 123
print hex$(ebx), ": ebx after the invoke", 13, 10, 10
dec esi
  .Until Zero?
  invoke CodeNoArgs
  invoke ThreeArgsLocal123, 123, 456, 789
  inkey chr$(10, "hit any key for the stack corruption test:", 13, 10)
  invoke TwoArgsNoLocals, 2, 99
  MsgBox 0, "You won't see this box", "Hi", MB_OK
  invoke ExitProcess, 0
end start

The procs have little problems with a sloppy use of the push and pop instructions. Here is the output:

Code Select Expand

2 args: ebx=3
### MbRet 2ArgsNoLocals: stack is -4 bytes off, 1 push(es) too many ###
12345678: ebx after the invoke

2 args: ebx=2
12345678: ebx after the invoke

2 args: ebx=1
### MbRet 2ArgsNoLocals: stack is 4 bytes off, 1 pop(s) too many - check uses ###
0040107A: ebx after the invoke

CodeNoArgs - cannot work with @MbRet: ### MbRet 31: no stack frame found ###
### MbRet ThreeArgsLocal123: stack is -4 bytes off, 1 push(es) too many ###

hit any key for the stack corruption test:

2 args: ebx=2
### stack corruption? 2ArgsNoLocals: retaddr 12345678h is above the code segment ###

After the stack corruption test, it will produce an exception.

[HSE]

Very interesting JJ   

When there is no stack perhaps you can make an @MbTestStack.

Text output is a little confusing, perhaps in a line " MbRet name  [file , line]" , details in following lines and finally an empty line.

Of course a summary: N_stack_procs, N_non_stack_procs (yellow if > 0?), unbalances detected (green when 0, red some case)   

[hutch--]

It may help a little if you looked for the alignment padding after a RET but fundamentally a no stack frame procedure is hard to determine because you don't have an immediate technique to find the start and end of the proc. If it is being found with a disassembler, the output generally shows the start label (address) of the proc but if you have to do it in bare mnemonics you have a hard task in front of you.

With a disassembler it checks the address that a CALL mnemonic addresses which gives you the start address but there is no easy way to get the procedure end, possible the last RET before the next known CALL label and if it has alignment padding between the RET and the next label.

[jj2007]

The current version can determine if it is a no-frame procedure, and where its end is. What it cannot do so easily is find the start of the proc. So at present it will just issue a line saying "can't check a no frame proc".

Is anybody aware of a big asm source in the public domain? I need a test case that looks different from my own stuff.

[HSE]

What it cannot do so easily is find the start of the proc. So at present it will just issue a line saying "can't check a no frame proc".

iI you bother to write @MbRet at procedures ends, cost you just a cent to write @MbTestStack at its beginnings. 

[Biterider]

What about writing custom prologues and epilogues?
Biterider

[jj2007]

If you bother to write @MbRet at procedures ends, cost you just a cent to write @MbTestStack at its beginnings. 

Correct, but replacing a hundred ret..whatever endp pairs in a big source is done in a single Find & Replace command. Inserting a macro on top, after the LOCALS, is a bit more involved.

What about writing custom prologues and epilogues?

Sure, that's the best option, but you risk breaking old code. The current @MbRet does not require such efforts, and it also doesn't generate any extra code once you switch if off with a single statement. It is an absolutely harmless add-on.

Custom prologues have much bigger implications. For example, you must manage 'by hand' all those cases where you restore prologuedef because a proc required option prologue:none. That's a can of worms.

[HSE]

Inserting a macro on top, after the LOCALS, is a bit more involved.

Perhaps a little parser.

[daydreamer]

if you make a test.asm piece that is made up with calling every proc,mixed with INT3's and run in debugger and it would be possible to find the faulty PROC

[jj2007]

if you make a test.asm piece that is made up with calling every proc,mixed with INT3's and run in debugger and it would be possible to find the faulty PROC

Great advice, Magnus

Can you post one of your major sources, so that we can test it?

[hutch--]

What I am not sure about is what format you are working with, if its source code, you would list every CALL mnemonic which includes any variant of invoke and list the targets by their line number once you had the list. I gather you already have a technique to find the end of the procedure, even if it has multiple RET instructions in it. If you are dealing with binary format mnemonics you would basically be designing a dis-assembler.

[jj2007]

Hutch,
It's not source code, it's the executable:

Code Select Expand

MyStuff proc uses esi edi bla1 bla2
...
ret
MyStuff endp

Code Select Expand

MyStuff proc uses esi edi bla1 bla2
...
@MbRet  ; <<<<<<<<<<<<< it it's OFF, just a ret; if it's ON, call TESTWHATEVER, then ret
MyStuff endp

[daydreamer]

if you make a test.asm piece that is made up with calling every proc,mixed with INT3's and run in debugger and it would be possible to find the faulty PROC

Great advice, Magnus

Can you post one of your major sources, so that we can test it?

maybe if I put together all PROC's I wrote in one file converted to MACROs?

[jj2007]

Yes, go ahead, do that and post it here!