The len() macro crashes

[jj2007]

Just for fun, a little demo showing the virtues of len() in combination with VirtualAlloc

Code Select Expand

include \masm32\include\masm32rt.inc

.code
start:
  invoke VirtualAlloc, 0, 4096, MEM_COMMIT, PAGE_READWRITE
  xchg eax, edi
  mov esi, InputFile("\Masm32\include\Windows.inc")
  invoke RtlMoveMemory, edi, esi , 4095    ; we leave a nullbyte at the end of the committed memory
  print str$(len(esi)), " bytes in esi", 13, 10
  print str$(len(edi)), " bytes in edi", 13, 10
  invoke RtlMoveMemory, edi, esi , 4096    ; no nullbyte...
  print str$(len(edi)), " bytes in edi", 13, 10

  MsgBox 0, "q.e.d.", "Test of len():", MB_OK
  exit

end start

[zedd151]

977412 bytes in esi
4095 bytes in edi
4096 bytes in edi


???

Code Select Expand

include \masm32\include\masm32rt.inc

.code
start:
  invoke GlobalAlloc, GPTR, 4097
  xchg eax, edi
  mov esi, InputFile("\Masm32\include\Windows.inc")
  invoke RtlMoveMemory, edi, esi , 4095 ; we leave a nullbyte at the end of the committed memory
  print str$(len(esi)), " bytes in esi", 13, 10
  print str$(len(edi)), " bytes in edi", 13, 10
  invoke RtlMoveMemory, edi, esi , 4096 ; no nullbyte...
  print str$(len(edi)), " bytes in edi", 13, 10
  invoke GlobalFree, edi
  MsgBox 0, "q.e.d.", "Test of len():", MB_OK
  exit

end startUsed GlobalAlloc, instead. 

edit= forgot to free the global allocated memory. 

Also, and more importantly the buffer needs to be big enough to be able to read the desired size plus one character for zero termination, else it will give erroneous results.
I changed the buffer size by one byte to accomodate the zero termination. Not a len() bug, imo.

[jj2007]

Why do you think I used VirtualAlloc instead of GlobalAlloc? See the Instring thread.

len() in combination with VirtualAlloc

Also, I didn't write it was a len() bug. I wrote that len() crashes under certain circumstances.

[zedd151]

Also, I didn't write it was a len() bug. I wrote that len() crashes under certain circumstances.

Seemed to have been inferred. Apologies.

I rarely use anything other than GlobalAlloc btw. Hasn't failed me. (as long as the buffer is big enough for what I need it for)

[jj2007]

The point is that GlobalAlloc and HeapAlloc are tolerant: there are some trailing bytes, typically 8*AB, that you (or the len macro) can read ("DWO" is the end of the 4096 bytes):

Code Select Expand

Address   Hex dump                                         ASCII
0052D508  20 20 20 20|20 74 79 70|65 64 65 66|20 44 57 4F|      typedef DWO
0052D518  AB AB AB AB|AB AB AB AB|00 00 00 00|00 00 00 00| ««««««««
0052D528  77 12 37 0C|B1 D6 00 00|C4 00 4F 00|00 61 4F 00| w␒7␌±Ö  Ä O  aO
0052D538  EE FE EE FE|EE FE EE FE|EE FE EE FE|EE FE EE FE| îþîþîþîþîþîþîþîþ
VirtualAlloc has no such tolerance, so len() crashes when trying to examine byte #4097

[Biterider]

Hi
It could be a canary to detect buffer overflows. 
Certainly an undocumented feature that can change at any time.

Biterider

[jj2007]

It could be a canary to detect buffer overflows.
Certainly an undocumented feature that can change at any time.

Attention, this is The Campus - please don't confuse n00bs who consult this area (this is neither a feature, nor can it be used to detect buffer overflows).

The len() macro crashes with the example code posted above because VirtualAlloc returns memory organised in 4096 byte pages. So if code (e.g. len) tries to see if byte #4097 is a nullbyte, i.e. the end of a string, it "hits the wall" and raises an exception.

This will not happen with strings allocated with GlobalAlloc or HeapAlloc, because (as explained above) they return always more bytes than requested.

[Biterider]

Hi JJ
Please only refer to the official API documentation, in this particular case heapapi. In the comments, it clearly says "at least", which means that it is possible that exactly the requested amount of memory will be allocated. 
Everything else is pure speculation (which will confuse newcomers). Instead, encourage people to read the documentation carefully.

Biterider

[jj2007]

it is possible that exactly the requested amount of memory will be allocated.

It is not only possible, it is normal. However, you can read approx. 30-40 bytes beyond the HeapAlloc'ed memory, otherwise tons of software would miserably fail.

Code Select Expand

include \masm32\include\masm32rt.inc

.code
start:
  cls
  if 0
    invoke GlobalAlloc, 0, 4089
  else
    invoke VirtualAlloc, 0, 4096, MEM_COMMIT, PAGE_READWRITE
  endif
  xchg eax, edi
  mov esi, InputFile("\Masm32\include\Windows.inc")
  lea eax, [edi+4096-16]
  ; int 3        ; have a look in the debugger
  invoke RtlMoveMemory, edi, esi , 4095    ; we leave a nullbyte at the end of the committed memory
  invoke lstrlen, esi
  print str$(eax), " lstrlen bytes in esi (the full Windows.inc)", 13, 10
  invoke lstrlen, edi
  print str$(eax), " lstrlen bytes in edi", 13, 10
  invoke szLen, edi
  print str$(eax), " szLen  bytes in edi", 13, 10
  invoke RtlMoveMemory, edi, esi , 4096    ; no nullbyte...
  print "trying lstrlen:", 13, 10
  invoke lstrlen, edi
  print str$(eax), " lstrlen bytes in edi (0 is wrong, should be 4096)", 13, 10
  print "trying szLen:", 13, 10
  invoke szLen, edi
  print str$(eax), " szLen  bytes in edi", 13, 10
  MsgBox 0, "q.e.d.", "Test of len():", MB_OK
  exit

end start
Output:

Code Select Expand

977412 lstrlen bytes in esi (the full Windows.inc)
4095 lstrlen bytes in edi
4095 szLen  bytes in edi
trying lstrlen:
0 lstrlen bytes in edi (0 is wrong, should be 4096)
trying szLen:
Both lstrlen and szLen yield wrong results with VirtualAlloc (but not with GlobalAlloc: it always adds some bytes extra...); szLen crashes with an exception, just as the len() macro; same for invoke crt_strlen, edi

[HSE]

And the reason to use VirtualAlloc for strings is ... ¿?

[mineiro]

In linux (wine) your program stops at with access violation exactly in that part that compare bytes to zero:
trying lstrlen:

Why do you overwrite end of string?

.data
binary db "string"
string db "string",0

[jj2007]

And the reason to use VirtualAlloc for strings is ... ¿?

Yep, that's the whole point: Biterider says the fast pcmpistri algo is unsafe because it goes beyond the allocated memory. So I made some tests, and it turns out that
a) he is right, if it's VirtualAlloc'ed memory, and
b) that applies also to szLen, crt_strlen and probably other functions that need to read until the last byte and beyond.

I made some tests with randomly HeapAlloc'ed memory, 80,000 allocations between 3 and 50k bytes, and you can read approximately 30 bytes beyond the allocated zone without risking an exception. If you write a single byte beyond, you are in trouble, though. Since functions like strlen() and instr() don't have to write, they are safe for HeapAlloc'ed strings.

Biterider is right, of course, that this is not documented. However, lstrlen is an official Windows function, and fails for VirtualAlloc'ed strings if there are no nullbytes in the last allocated page. If it would fail also for HeapAlloc'ed strings, Windows would be in deep trouble. Therefore I consider pcmpistri to be safe for strings, as long as they are HeapAlloc'ed.

[mineiro]

If the GlobalAlloc function succeeds, it allocates at least the amount of memory requested.
If the actual amount allocated is greater than the amount requested, the process can use the entire amount.

Code Select Expand

invoke GlobalAlloc, 0, 4089

850039 lstrlen bytes in esi (the full Windows.inc)
4095 lstrlen bytes in edi
4095 szLen  bytes in edi
trying lstrlen:
4099 lstrlen bytes in edi (0 is wrong, should be 4096)
trying szLen:
4099 szLen  bytes in edi
message box


Code Select Expand

invoke GlobalAlloc, 0, 0

850039 lstrlen bytes in esi (the full Windows.inc)
4095 lstrlen bytes in edi
4095 szLen  bytes in edi
trying lstrlen:
4096 lstrlen bytes in edi (0 is wrong, should be 4096)
trying szLen:
4096 szLen  bytes in edi
wine: Unhandled page fault on write access to 3D2D3D31 at address 7BC2A1A2 (thread 06b4), starting debugger...
06b4:err:seh:NtRaiseException Unhandled exception code c0000005 flags 0 addr 0x7bc2a1a2
no message box


[TimoVJL]

Windows OS gives zeroed heap to program, so there will be zeroes in first use.

[jj2007]

If the GlobalAlloc function succeeds, it allocates at least the amount of memory requested.
If the actual amount allocated is greater than the amount requested, the process can use the entire amount.

...
wine: Unhandled page fault on write access

Unfortunately, the docs are not very clear on that: truth is that with HeapAlloc, you can "use" more than you requested, but only reading the memory.

Windows OS gives zeroed heap to program, so there will be zeroes in first use.

If you used the HEAP_ZERO_MEMORY flag. If not, you may have zeroes but there is no guarantee.