Help with VirtualMemory Allocation

[guga]

Hi guys

i need a bit help on virtual alloc functions. I´m looking for a routine or library that can be able to handle huge virtual memory, because the ones i´m implementing from RosAsm are crashing badly due to fragmented memory blocks, i guess.

The problem is for the newer pdbdumper i´m making. When it loads a file with let´s say 6 Mb, the virtualloc is ok even if i double the size (or mul by 4) with:

Code Select Expand


    mov eax D@FileSize | shl eax 2 | Align_On MEM_ALIGNMENT eax
    call 'RosMem.VMemAlloc' D@pOutput, eax


This is ok, since internally, VMemAlloc uses VirtualAlloc to reserve and commit the necessary pages.

But...when i load a file pdb with 22 Mb, the app simply crashes, because VMemAlloc allocated the necessary size, but as far i saw, the allocated memory is not entirely commited, causing the parser for pdb crash when it tries to access a certain address in virtualmemory.

If needed, i can post here, the functions used internally in VMemAlloc, but i would like to know 1st, if it do exists a library or routines that can handle the necessary memory for huge apps (Allocate, free and reallocate huge memory blocks). Something like what hexworkshop does, i mean, you can open a file with whatever size and it will still load it without crash.

Btw, i don´t need a growable memory. All i was thinking in something simple. I mean, allocate XXX Mb of memory at once and use it according and later free it when it is no longer needed.

[TouEnMasm]

The virtual alloc is a bad choice to use memory.
global or heap memory are surely a good soluce.

[guga]

Tks...but, is there a example of using heap memory to allocate huge files, without having to make it growable ? I mean, simply 1 function at the start of the app like "createheap" or allocateheapmem (or something..) from where i can allocate enough memory to handle the whole file, and then, once i finish to use the memory, free it.

I don´t use heapmemory very often, so i´m not really use to it. If there is already a function/routines that can handle huge files (With heapalloc/heapcreate etc), it would be better then doing it from scratch.

[jj2007]

No problem with this one, allocating 20 x 200 MB with HeapAlloc (and writing to it...):

include \masm32\MasmBasic\MasmBasic.inc      ; download
  Init
  msize=200000000
  invoke SetLastError, 0
  REPEAT 20
      Let esi=New$(msize)      ; uses HeapAlloc
      PrintLine "Last error = ", Err$()
      mov edi, esi
      mov al, "x"
      mov ecx, msize
      rep stosb
  ENDM
  Inkey "ok?"
  Exit
end start

Even after a REPEAT 200, it just shows me "ok?".

[guga]

Many tks JJ. I´ll give a try.

I should have masmbasic here in another HD to test.

Btw....to free the memory i presume you did something like

Delete$(mem)

right?

[jj2007]

Btw....to free the memory i presume you did something like

Delete$(mem)

right?

The Let esi=... takes care of that inside the loop (Let checks in a table whether esi was already allocated, and if yes, deletes the previous one). One might use Clr$ esi after the ENDM, although Exit aka ExitProcess takes care of freeing everything.

Other example:

include \masm32\MasmBasic\MasmBasic.inc      ; download
  Init
  xor ecx, ecx
  ; xor esi, esi not needed, Init zeroes esi edi ebx
  .Repeat
      Let esi=Str$(ecx)+" "+esi+" "+Chr$(ecx+"a")
      inc ecx
  .Until ecx>9
  PrintLine esi
  Clr$ esi
  Inkey "[", esi, "]"
  Exit
end start

Output:
9 8 7 6 5 4 3 2 1 0  a b c d e f g h i j
[]

My point was to show that HeapAlloc works, even for very large sizes. Your problem seems to be elsewhere...

[guga]

JJ many many thanks :)

You gave an increadible and simple solution. the heapalloc function i already made on rosmem.dll, but i forgot where i was using it. After testing the heapalloc/heapfree, pdbdumper could completelly load a 22Mb file without crashing.

I´ll use heapalloc for this app and wil try to see why Virtualmem is crashing when loading huge files. If i couldn´t find a viable solution, i´ll recreate the memory allocation routines on rosasm (and rosmem.dll) only with heapalloc instead virtualalloc.

It seems pretty stable.

[MichaelW]

For VirtualAlloc 22MB should be a very small allocation.

Code Select Expand


;===============================================================================
include \masm32\include\masm32rt.inc
.686
.xmm
;===============================================================================
.data
    sysinfo SYSTEM_INFO <>
.code
;===============================================================================
start:
;===============================================================================

    invoke GetSystemInfo, ADDR sysinfo
   
    mov     ebx, sysinfo.dwPageSize   
   
    printf("Page Size       : %d\n", ebx)
   
  @@:
 
    printf("Allocation Size : %u\n", ebx)
   
    invoke VirtualAlloc, NULL, ebx, MEM_COMMIT or MEM_RESERVE, PAGE_READWRITE
   
    mov     esi, eax
   
    .IF esi == 0
        printf("%s\n", LastError$())
        jmp     @F
    .ENDIF

    mov     eax, 0ffh
    mov     edi, esi
    mov     ecx, ebx
    rep     stosb

    invoke VirtualFree, esi, 0, MEM_RELEASE
   
    shl     ebx, 1
   
    jmp     @B
   
  @@:   

    inkey
    exit
   
end start


Result on a system with 8GB of physical memory:

Code Select Expand


Page Size       : 4096
Allocation Size : 4096
Allocation Size : 8192
Allocation Size : 16384
Allocation Size : 32768
Allocation Size : 65536
Allocation Size : 131072
Allocation Size : 262144
Allocation Size : 524288
Allocation Size : 1048576
Allocation Size : 2097152
Allocation Size : 4194304
Allocation Size : 8388608
Allocation Size : 16777216
Allocation Size : 33554432
Allocation Size : 67108864
Allocation Size : 134217728
Allocation Size : 268435456
Allocation Size : 536870912
Allocation Size : 1073741824
Allocation Size : 2147483648
Not enough storage is available to process this command.


[guga]

Hi Michael

indeed...I tested it again, and after a reboot, the crash stopped. I don´t understand what was happening.

[dedndave]

sounds like a memory leak of some kind - commonly with GDI objects   :P