News:

Masm32 SDK description, downloads and other helpful links
Message to All Guests
NB: Posting URL's See here: Posted URL Change

Main Menu

The len() macro crashes

Started by jj2007, August 08, 2023, 09:46:04 PM

Previous topic - Next topic

jj2007

Just for fun, a little demo showing the virtues of len() in combination with VirtualAlloc :biggrin:

include \masm32\include\masm32rt.inc

.code
start:
  invoke VirtualAlloc, 0, 4096, MEM_COMMIT, PAGE_READWRITE
  xchg eax, edi
  mov esi, InputFile("\Masm32\include\Windows.inc")
  invoke RtlMoveMemory, edi, esi , 4095    ; we leave a nullbyte at the end of the committed memory
  print str$(len(esi)), " bytes in esi", 13, 10
  print str$(len(edi)), " bytes in edi", 13, 10
  invoke RtlMoveMemory, edi, esi , 4096    ; no nullbyte...
  print str$(len(edi)), " bytes in edi", 13, 10

  MsgBox 0, "q.e.d.", "Test of len():", MB_OK
  exit

end start

zedd151

977412 bytes in esi
4095 bytes in edi
4096 bytes in edi


???


include \masm32\include\masm32rt.inc

.code
start:
  invoke GlobalAlloc, GPTR, 4097
  xchg eax, edi
  mov esi, InputFile("\Masm32\include\Windows.inc")
  invoke RtlMoveMemory, edi, esi , 4095 ; we leave a nullbyte at the end of the committed memory
  print str$(len(esi)), " bytes in esi", 13, 10
  print str$(len(edi)), " bytes in edi", 13, 10
  invoke RtlMoveMemory, edi, esi , 4096 ; no nullbyte...
  print str$(len(edi)), " bytes in edi", 13, 10
  invoke GlobalFree, edi
  MsgBox 0, "q.e.d.", "Test of len():", MB_OK
  exit

end start
Used GlobalAlloc, instead.  :tongue:

edit= forgot to free the global allocated memory.  :biggrin:

Also, and more importantly the buffer needs to be big enough to be able to read the desired size plus one character for zero termination, else it will give erroneous results.
I changed the buffer size by one byte to accomodate the zero termination. Not a len() bug, imo.

jj2007

Why do you think I used VirtualAlloc instead of GlobalAlloc? See the Instring thread.

Quote from: jj2007 on August 08, 2023, 09:46:04 PMlen() in combination with VirtualAlloc

Also, I didn't write it was a len() bug. I wrote that len() crashes under certain circumstances.

zedd151

Quote from: jj2007 on August 09, 2023, 12:30:48 AMAlso, I didn't write it was a len() bug. I wrote that len() crashes under certain circumstances.
Seemed to have been inferred. Apologies.

I rarely use anything other than GlobalAlloc btw. Hasn't failed me. (as long as the buffer is big enough for what I need it for)

jj2007

The point is that GlobalAlloc and HeapAlloc are tolerant: there are some trailing bytes, typically 8*AB, that you (or the len macro) can read ("DWO" is the end of the 4096 bytes):

Address   Hex dump                                         ASCII
0052D508  20 20 20 20|20 74 79 70|65 64 65 66|20 44 57 4F|      typedef DWO
0052D518  AB AB AB AB|AB AB AB AB|00 00 00 00|00 00 00 00| ««««««««
0052D528  77 12 37 0C|B1 D6 00 00|C4 00 4F 00|00 61 4F 00| w␒7␌±Ö  Ä O  aO
0052D538  EE FE EE FE|EE FE EE FE|EE FE EE FE|EE FE EE FE| îþîþîþîþîþîþîþîþ

VirtualAlloc has no such tolerance, so len() crashes when trying to examine byte #4097 :sad:

Biterider

Hi
It could be a canary to detect buffer overflows. 
Certainly an undocumented feature that can change at any time.

Biterider

jj2007

Quote from: Biterider on August 09, 2023, 02:20:10 AMIt could be a canary to detect buffer overflows.
Certainly an undocumented feature that can change at any time.

Attention, this is The Campus - please don't confuse n00bs who consult this area (this is neither a feature, nor can it be used to detect buffer overflows).

The len() macro crashes with the example code posted above because VirtualAlloc returns memory organised in 4096 byte pages. So if code (e.g. len) tries to see if byte #4097 is a nullbyte, i.e. the end of a string, it "hits the wall" and raises an exception.

This will not happen with strings allocated with GlobalAlloc or HeapAlloc, because (as explained above) they return always more bytes than requested.

Biterider

Hi JJ
Please only refer to the official API documentation, in this particular case heapapi. In the comments, it clearly says "at least", which means that it is possible that exactly the requested amount of memory will be allocated. 
Everything else is pure speculation (which will confuse newcomers). Instead, encourage people to read the documentation carefully.

Biterider

jj2007

#8
Quote from: Biterider on August 09, 2023, 06:17:23 AMit is possible that exactly the requested amount of memory will be allocated.

It is not only possible, it is normal. However, you can read approx. 30-40 bytes beyond the HeapAlloc'ed memory, otherwise tons of software would miserably fail.

include \masm32\include\masm32rt.inc

.code
start:
  cls
  if 0
    invoke GlobalAlloc, 0, 4089
  else
    invoke VirtualAlloc, 0, 4096, MEM_COMMIT, PAGE_READWRITE
  endif
  xchg eax, edi
  mov esi, InputFile("\Masm32\include\Windows.inc")
  lea eax, [edi+4096-16]
  ; int 3        ; have a look in the debugger
  invoke RtlMoveMemory, edi, esi , 4095    ; we leave a nullbyte at the end of the committed memory
  invoke lstrlen, esi
  print str$(eax), " lstrlen bytes in esi (the full Windows.inc)", 13, 10
  invoke lstrlen, edi
  print str$(eax), " lstrlen bytes in edi", 13, 10
  invoke szLen, edi
  print str$(eax), " szLen  bytes in edi", 13, 10
  invoke RtlMoveMemory, edi, esi , 4096    ; no nullbyte...
  print "trying lstrlen:", 13, 10
  invoke lstrlen, edi
  print str$(eax), " lstrlen bytes in edi (0 is wrong, should be 4096)", 13, 10
  print "trying szLen:", 13, 10
  invoke szLen, edi
  print str$(eax), " szLen  bytes in edi", 13, 10
  MsgBox 0, "q.e.d.", "Test of len():", MB_OK
  exit

end start

Output:
977412 lstrlen bytes in esi (the full Windows.inc)
4095 lstrlen bytes in edi
4095 szLen  bytes in edi
trying lstrlen:
0 lstrlen bytes in edi (0 is wrong, should be 4096)
trying szLen:

Both lstrlen and szLen yield wrong results with VirtualAlloc (but not with GlobalAlloc: it always adds some bytes extra...); szLen crashes with an exception, just as the len() macro; same for invoke crt_strlen, edi

HSE

And the reason to use VirtualAlloc for strings is ... ¿?
Equations in Assembly: SmplMath

mineiro

In linux (wine) your program stops at with access violation exactly in that part that compare bytes to zero:
trying lstrlen:

Why do you overwrite end of string?

.data
binary db "string"
string db "string",0
I'd rather be this ambulant metamorphosis than to have that old opinion about everything

jj2007

Quote from: HSE on August 09, 2023, 09:03:14 AMAnd the reason to use VirtualAlloc for strings is ... ¿?

Yep, that's the whole point: Biterider says the fast pcmpistri algo is unsafe because it goes beyond the allocated memory. So I made some tests, and it turns out that
a) he is right, if it's VirtualAlloc'ed memory, and
b) that applies also to szLen, crt_strlen and probably other functions that need to read until the last byte and beyond.

I made some tests with randomly HeapAlloc'ed memory, 80,000 allocations between 3 and 50k bytes, and you can read approximately 30 bytes beyond the allocated zone without risking an exception. If you write a single byte beyond, you are in trouble, though. Since functions like strlen() and instr() don't have to write, they are safe for HeapAlloc'ed strings.

Biterider is right, of course, that this is not documented. However, lstrlen is an official Windows function, and fails for VirtualAlloc'ed strings if there are no nullbytes in the last allocated page. If it would fail also for HeapAlloc'ed strings, Windows would be in deep trouble. Therefore I consider pcmpistri to be safe for strings, as long as they are HeapAlloc'ed.

mineiro

QuoteIf the GlobalAlloc function succeeds, it allocates at least the amount of memory requested.
If the actual amount allocated is greater than the amount requested, the process can use the entire amount.

invoke GlobalAlloc, 0, 4089

850039 lstrlen bytes in esi (the full Windows.inc)
4095 lstrlen bytes in edi
4095 szLen  bytes in edi
trying lstrlen:
4099 lstrlen bytes in edi (0 is wrong, should be 4096)
trying szLen:
4099 szLen  bytes in edi
message box


invoke GlobalAlloc, 0, 0

850039 lstrlen bytes in esi (the full Windows.inc)
4095 lstrlen bytes in edi
4095 szLen  bytes in edi
trying lstrlen:
4096 lstrlen bytes in edi (0 is wrong, should be 4096)
trying szLen:
4096 szLen  bytes in edi
wine: Unhandled page fault on write access to 3D2D3D31 at address 7BC2A1A2 (thread 06b4), starting debugger...
06b4:err:seh:NtRaiseException Unhandled exception code c0000005 flags 0 addr 0x7bc2a1a2
no message box
I'd rather be this ambulant metamorphosis than to have that old opinion about everything

TimoVJL

Windows OS gives zeroed heap to program, so there will be zeroes in first use.
May the source be with you

jj2007

Quote from: mineiro on August 09, 2023, 11:57:21 AM
QuoteIf the GlobalAlloc function succeeds, it allocates at least the amount of memory requested.
If the actual amount allocated is greater than the amount requested, the process can use the entire amount.
...
wine: Unhandled page fault on write access

Unfortunately, the docs are not very clear on that: truth is that with HeapAlloc, you can "use" more than you requested, but only reading the memory.

Quote from: TimoVJL on August 09, 2023, 04:36:42 PMWindows OS gives zeroed heap to program, so there will be zeroes in first use.

If you used the HEAP_ZERO_MEMORY flag. If not, you may have zeroes but there is no guarantee.