Author Topic: Help with VirtualMemory Allocation  (Read 6959 times)

guga

  • Moderator
  • Member
  • *****
  • Posts: 1196
  • Assembly is a state of art.
    • RosAsm
Help with VirtualMemory Allocation
« on: August 02, 2014, 02:45:32 AM »
Hi guys

i need a bit help on virtual alloc functions. I´m looking for a routine or library that can be able to handle huge virtual memory, because the ones i´m implementing from RosAsm are crashing badly due to fragmented memory blocks, i guess.

The problem is for the newer pdbdumper i´m making. When it loads a file with let´s say 6 Mb, the virtualloc is ok even if i double the size (or mul by 4) with:
Code: [Select]
    mov eax D@FileSize | shl eax 2 | Align_On MEM_ALIGNMENT eax
    call 'RosMem.VMemAlloc' D@pOutput, eax

This is ok, since internally, VMemAlloc uses VirtualAlloc to reserve and commit the necessary pages.

But...when i load a file pdb with 22 Mb, the app simply crashes, because VMemAlloc allocated the necessary size, but as far i saw, the allocated memory is not entirely commited, causing the parser for pdb crash when it tries to access a certain address in virtualmemory.

If needed, i can post here, the functions used internally in VMemAlloc, but i would like to know 1st, if it do exists a library or routines that can handle the necessary memory for huge apps (Allocate, free and reallocate huge memory blocks). Something like what hexworkshop does, i mean, you can open a file with whatever size and it will still load it without crash.

Btw, i don´t need a growable memory. All i was thinking in something simple. I mean, allocate XXX Mb of memory at once and use it according and later free it when it is no longer needed.
Coding in Assembly requires a mix of:
80% of brain, passion, intuition, creativity
10% of programming skills
10% of alcoholic levels in your blood.

My Code Sites:
http://rosasm.freeforums.org
http://winasm.tripod.com

TouEnMasm

  • Member
  • *****
  • Posts: 1283
    • EditMasm
Re: Help with VirtualMemory Allocation
« Reply #1 on: August 02, 2014, 04:15:37 AM »

The virtual alloc is a bad choice to use memory.
global or heap memory are surely a good soluce.
Fa is a musical note to play with CL

guga

  • Moderator
  • Member
  • *****
  • Posts: 1196
  • Assembly is a state of art.
    • RosAsm
Re: Help with VirtualMemory Allocation
« Reply #2 on: August 02, 2014, 04:52:19 AM »
Tks...but, is there a example of using heap memory to allocate huge files, without having to make it growable ? I mean, simply 1 function at the start of the app like "createheap" or allocateheapmem (or something..) from where i can allocate enough memory to handle the whole file, and then, once i finish to use the memory, free it.

I don´t use heapmemory very often, so i´m not really use to it. If there is already a function/routines that can handle huge files (With heapalloc/heapcreate etc), it would be better then doing it from scratch.
Coding in Assembly requires a mix of:
80% of brain, passion, intuition, creativity
10% of programming skills
10% of alcoholic levels in your blood.

My Code Sites:
http://rosasm.freeforums.org
http://winasm.tripod.com

jj2007

  • Member
  • *****
  • Posts: 10438
  • Assembler is fun ;-)
    • MasmBasic
Re: Help with VirtualMemory Allocation
« Reply #3 on: August 02, 2014, 06:56:27 AM »
No problem with this one, allocating 20 x 200 MB with HeapAlloc (and writing to it...):

include \masm32\MasmBasic\MasmBasic.inc      ; download
  Init
  msize=200000000
  invoke SetLastError, 0
  REPEAT 20
      Let esi=New$(msize)      ; uses HeapAlloc
      PrintLine "Last error = ", Err$()
      mov edi, esi
      mov al, "x"
      mov ecx, msize
      rep stosb
  ENDM
  Inkey "ok?"
  Exit
end start


Even after a REPEAT 200, it just shows me "ok?".

guga

  • Moderator
  • Member
  • *****
  • Posts: 1196
  • Assembly is a state of art.
    • RosAsm
Re: Help with VirtualMemory Allocation
« Reply #4 on: August 02, 2014, 07:06:09 AM »
Many tks JJ. I´ll give a try.

I should have masmbasic here in another HD to test.

Btw....to free the memory i presume you did something like

Delete$(mem)

right?
Coding in Assembly requires a mix of:
80% of brain, passion, intuition, creativity
10% of programming skills
10% of alcoholic levels in your blood.

My Code Sites:
http://rosasm.freeforums.org
http://winasm.tripod.com

jj2007

  • Member
  • *****
  • Posts: 10438
  • Assembler is fun ;-)
    • MasmBasic
Re: Help with VirtualMemory Allocation
« Reply #5 on: August 02, 2014, 08:16:03 AM »
Btw....to free the memory i presume you did something like

Delete$(mem)

right?

The Let esi=... takes care of that inside the loop (Let checks in a table whether esi was already allocated, and if yes, deletes the previous one). One might use Clr$ esi after the ENDM, although Exit aka ExitProcess takes care of freeing everything.

Other example:

include \masm32\MasmBasic\MasmBasic.inc      ; download
  Init
  xor ecx, ecx
  ; xor esi, esi not needed, Init zeroes esi edi ebx
  .Repeat
      Let esi=Str$(ecx)+" "+esi+" "+Chr$(ecx+"a")
      inc ecx
  .Until ecx>9
  PrintLine esi
  Clr$ esi
  Inkey "[", esi, "]"
  Exit
end start

Output:
9 8 7 6 5 4 3 2 1 0  a b c d e f g h i j
[]

My point was to show that HeapAlloc works, even for very large sizes. Your problem seems to be elsewhere...

guga

  • Moderator
  • Member
  • *****
  • Posts: 1196
  • Assembly is a state of art.
    • RosAsm
Re: Help with VirtualMemory Allocation
« Reply #6 on: August 02, 2014, 11:15:52 AM »
JJ many many thanks :)

You gave an increadible and simple solution. the heapalloc function i already made on rosmem.dll, but i forgot where i was using it. After testing the heapalloc/heapfree, pdbdumper could completelly load a 22Mb file without crashing.

I´ll use heapalloc for this app and wil try to see why Virtualmem is crashing when loading huge files. If i couldn´t find a viable solution, i´ll recreate the memory allocation routines on rosasm (and rosmem.dll) only with heapalloc instead virtualalloc.

It seems pretty stable.
Coding in Assembly requires a mix of:
80% of brain, passion, intuition, creativity
10% of programming skills
10% of alcoholic levels in your blood.

My Code Sites:
http://rosasm.freeforums.org
http://winasm.tripod.com

MichaelW

  • Global Moderator
  • Member
  • *****
  • Posts: 1209
Re: Help with VirtualMemory Allocation
« Reply #7 on: August 02, 2014, 04:57:17 PM »
For VirtualAlloc 22MB should be a very small allocation.
Code: [Select]
;===============================================================================
include \masm32\include\masm32rt.inc
.686
.xmm
;===============================================================================
.data
    sysinfo SYSTEM_INFO <>
.code
;===============================================================================
start:
;===============================================================================

    invoke GetSystemInfo, ADDR sysinfo
   
    mov     ebx, sysinfo.dwPageSize   
   
    printf("Page Size       : %d\n", ebx)
   
  @@:
 
    printf("Allocation Size : %u\n", ebx)
   
    invoke VirtualAlloc, NULL, ebx, MEM_COMMIT or MEM_RESERVE, PAGE_READWRITE
   
    mov     esi, eax
   
    .IF esi == 0
        printf("%s\n", LastError$())
        jmp     @F
    .ENDIF

    mov     eax, 0ffh
    mov     edi, esi
    mov     ecx, ebx
    rep     stosb

    invoke VirtualFree, esi, 0, MEM_RELEASE
   
    shl     ebx, 1
   
    jmp     @B
   
  @@:   

    inkey
    exit
   
end start

Result on a system with 8GB of physical memory:
Code: [Select]
Page Size       : 4096
Allocation Size : 4096
Allocation Size : 8192
Allocation Size : 16384
Allocation Size : 32768
Allocation Size : 65536
Allocation Size : 131072
Allocation Size : 262144
Allocation Size : 524288
Allocation Size : 1048576
Allocation Size : 2097152
Allocation Size : 4194304
Allocation Size : 8388608
Allocation Size : 16777216
Allocation Size : 33554432
Allocation Size : 67108864
Allocation Size : 134217728
Allocation Size : 268435456
Allocation Size : 536870912
Allocation Size : 1073741824
Allocation Size : 2147483648
Not enough storage is available to process this command.
« Last Edit: August 02, 2014, 06:02:26 PM by MichaelW »
Well Microsoft, here’s another nice mess you’ve gotten us into.

guga

  • Moderator
  • Member
  • *****
  • Posts: 1196
  • Assembly is a state of art.
    • RosAsm
Re: Help with VirtualMemory Allocation
« Reply #8 on: August 02, 2014, 07:52:49 PM »
Hi Michael

indeed...I tested it again, and after a reboot, the crash stopped. I don´t understand what was happening.
Coding in Assembly requires a mix of:
80% of brain, passion, intuition, creativity
10% of programming skills
10% of alcoholic levels in your blood.

My Code Sites:
http://rosasm.freeforums.org
http://winasm.tripod.com

dedndave

  • Member
  • *****
  • Posts: 8827
  • Still using Abacus 2.0
    • DednDave
Re: Help with VirtualMemory Allocation
« Reply #9 on: August 03, 2014, 12:10:34 AM »
sounds like a memory leak of some kind - commonly with GDI objects   :P