The len() macro crashes

[jj2007]

I made a testbed demonstrating that you can read dword ptr [eax+28], where eax points to the first byte after a heapalloc'ed buffer. The "tolerance" is the same for Win XP, Win 7-64 and Win 10, but grateful for tests on other machines.

As noted in previous posts, there is zero tolerance for VirtualAlloc'ed memory, and writing beyond the HeapAlloc'ed buffer is also not allowed by the OS. But reading is ;-)

Code Select Expand

  mov edi, offset handles
  push edi
  push loops-1  ; 80,000 tests
  .Repeat
    lea eax, [1+Rand(50000)]
    ;and eax, 0FFFFFFF0h
    stosd
    add total, eax
    invoke HeapAlloc, MbProHeap, flags, eax
    stosd
    Print Str$("%i MB", total/1048576), Cr$
    dec stack
  .Until Sign?
  pop edx
  pop esi
  Print Str$("Allocated %3f GB - only extreme sizes are shown below\n", total/40000000h)
  push loops-1
  .Repeat
    lodsd
    mov elsize, eax
    lodsd
    xchg eax, edi
    invoke HeapSize, MbProHeap, flags, edi
    .if eax<=10 || eax>49990
        fdeb 4, "size", eax   ; only extreme sizes
    .endif
    lea ecx, [eax+edi]
    mov edx, [eax+edi+28]    ; read the ABABABAB behind the allocated memory; +28 is ok for Win XP, Win7-64 and Win 10
    sub eax, elsize
    .if dword ptr pcTable[-4]
    ; ifidni @Environ(oDebug), <1>    ; OPT_Debug    1    ; 1=use \Masm32\MasmBasic\Res\DebugHeap.exe
        echo debugging is ON
        .if dword ptr [ecx+4]!=0ABABABABh    ; the ABABs are there when a) Olly is launched or b) debugheap.exe is running
            fdeb 1, "No ABAB found", x:edx, eax
            ; mov dword ptr [ecx], Mirror$("Ciao")
        .endif
    ;else
    ;    echo debugging is OFF
    .endif
    .if eax
        Print Str$("allocated-requested=%i\n", eax)
    .endif
    invoke HeapFree, MbProHeap, flags, edi
    ; fdeb 7, "HF", $Err$()
    dec stack
  .Until Sign?
  pop edx
  Inkey Str$("Released %3f GB\n", total/40000000h)

Code Select Expand

Allocated 1.86 GB - only extreme sizes are shown below
size    eax             2
size    eax             49994
size    eax             7
size    eax             50000
size    eax             1
size    eax             6
size    eax             49999
size    eax             3
size    eax             4
size    eax             49991
size    eax             2
size    eax             2
size    eax             49998
size    eax             49997
size    eax             10
size    eax             3
size    eax             49997
size    eax             3
size    eax             7
size    eax             1
size    eax             49999
size    eax             6
size    eax             6
size    eax             2
size    eax             49997
size    eax             3
size    eax             10
size    eax             49993
size    eax             49998
size    eax             49997
size    eax             49996
size    eax             49999
size    eax             9
Released 1.86 GB

[Biterider]

Yep, that's the whole point: Biterider says the fast pcmpistri algo is unsafe because it goes beyond the allocated memory

No, that's not the point. I clearly wrote that if you try to access an uncommitted page, the processor will raise a fault.
This is by hardware!

It is not only possible, it is normal. However, you can read approx. 30-40 bytes beyond the HeapAlloc'ed memory, otherwise tons of software would miserably fail.

It's not "normal", just a possible case. The allocator gives the user what better fits its allocation granularity. There may be some space before and after, a control block, or other structures to protect the heap, but this is an operating system responsibility and its implementation is subject to change. The only thing that is certain is what the documentation dictates. It's like some kind of contract.
The history of software development is full of examples where this rule has been broken and the catastrophic consequences it has had. If the documentation says you can do something, then do it. All other statements based on experiments may work for this or another OS version, but there is absolutely no guarantee that it will continue to work in the future.

It's not so relevant for MasmBasic, because strings and buffers are all HeapAlloc'ed with a little reserve of 24 bytes that helps to avoid this scenario.

That may be true in the closed ecosystem you mention if there is always enough additional committed memory in the trail. Note that an overhead of 64 bytes will be required once the ISA AVX512 standard is established and appropriate instructions are available.

See Peter Cordes' very detailed description of pcmpistri & pcmpestri performance at SOF

Apart from the bottleneck discussion, they hit the nail on the head here
https://stackoverflow.com/questions/58901232/

Unfortunately, the docs are not very clear on that: truth is that with HeapAlloc, you can "use" more than you requested, but only reading the memory.

The documentation is very clear. The allocator returns a piece of memory, sometimes even slightly larger than requested (see above). How much can be determined with the function HeapSize https://learn.microsoft.com/en-us/windows/win32/api/heapapi/nf-heapapi-heapsize)
Can you show where you got the information you can only read this memory (official sources only, e.g. MSDN, Raymond Chen, etc.). On x86 systems, the smallest unit of memory protection and management is at the page level. This means that memory permissions such as read, write and execute are managed at the this level rather than for individual bytes or smaller units.
This means that this "read only" memory must be located on the next memory page... ???

I have made myself clear. If you want to continue, that should be fine for you, but think of all this misinformation that is being spread. As an experienced programmer, you have a duty to encourage newcomers to follow good practices. One of these is to read the documentation carefully. It's annoying, but there are good reasons why it's there.

Biterider

[zedd151]

Come on guys, this isn't the Colosseum. 

The Workshop is where general purpose questions and answers are posted. Any assembler programming topic is welcome and discussion is encouraged as long as its friendly.

[jj2007]

think of all this misinformation that is being spread

HeapGames.zìp
24.52 KB
downloaded 0 times

You don't even bother to look at the testbed, compliments.