64 bit memory allocation

[AKRichard]

Hello all,

  I am fairly new to assembly and the board.

  I have a BigNumber library written mostly in c++/cli (windows 7 64 bit) with the algorithms written in assembly language.  I am using visual studio 2010 targeting .net 4.0.  The values of the library are held in an array of unsigned 64 bit numbers.  I have finally got to the point where I created the return value within the algorithm (as I had been passing in the return value before).  Everything works fine until the values in the array reach a few hundred entries long, at which point I start getting errors within the call to malloc itself sometimes.  I have been reading all I could find on malloc and memory allocation, but am not sure how to debug this issue.  The code calling malloc looks like this:

Code Select Expand


                mov     rax, qword ptr[r11]     ;size of all arrays are held at index 0 of the array
                cmp     rax, qword ptr[r12]     ;looking for larger of the two arrays
                cmovb   rax, qword ptr[r12]     ;if value in r12 larger move into rax
                inc rax         ;size of return value will either be same size or one larger
                push r11                 ;save pointers to variables
push r12
                push    rax                     ;save size of ret value
                inc     rax                     ;make room for the size of the array at index 0
mul _eight                 ;multiply by eight since using 64 bit numbers
mov rcx, rax                ;move value into rcx
sub rsp, 28h         ;make room on stack for call to malloc
clc         ;clear the carry flag
call calloc         ;call malloc
jc sGoOut         ;if carry flag set there was an error
mov r8, rax                 ;mov returned pointer into r8
add rsp, 28h ;reset stack to original position
pop r12                     ;reset the pointers to the values
pop r11
                pop     rax
                mov     qword ptr[r8], rax      ;move the size of the array into index 0


As mentioned above, there are no problems whatsoever till the arrays hold about 100 elements or more, then I start getting memory access violation errors (the exact error is "Access Violation writing location 0X0") in the call to malloc (not everytime but sometimes) it happens apparantly randomly with requesting different values from malloc, but when it does happen the value in rcx is allways about 800 or more. whenever the error does crop up the visual studio debugger brings up a disassembly screen which says "No source code available" and the remote debugger disconnects.

  I am not sure what other info would help so if I left anything out, just point it out to me and Ill fill it in.

Thanks in advance for any help.

[qWord]

;if carry flag set there was an error   

I've never heard of such an error handling mechanism in the WinAPI or CRT...
Do it the right way and test the return value in RAX: if it is zero, the allocation has failed, otherwise it is a valid pointer.

Code Select Expand


call calloc
test rax,rax
jz @error

[dedndave]

 :P
16-bit DOS used to signal errors with the carry flag
most DOS INT 21h functions are like that
as a result, many 16-bit libraries use the same method

[AKRichard]

I found on msdn that malloc sets the carry flag if the allocation failed, but it also said that rax would be 0 if it failed.  I am having a hard time figuring out WHY its failing though.  When stepping through the code it errors in the call to malloc and doesnt return to my code, it just brings up a blank disassembly screen and breaks.  At which point I can access other points in the code through the call stack, but all the variables are undefined at that point.  I think the heap is getting corrupted somehow, but I am not sure what I am doing wrong.

  Anyways, its easy enough for me to change the code from checking the carry flag to checking rax,  I am still having the problem of the code not returning from the call to malloc so that it can be checked.  I noticed you used calloc in your example, should I be using calloc instead of malloc?  Whats the difference?  I only learned enough native c++ for the managed code to be able to call the assembly code and my library is finally becoming comparable (speed wise) to microsofts.

Thanks again for the help

[qWord]

Hi,

I found on msdn that malloc sets the carry flag if the allocation failed, but it also said that rax would be 0 if it failed.

Can you show us a link to that information?

When stepping through the code it errors in the call to malloc and doesnt return to my code, it just brings up a blank disassembly screen and breaks.

Sounds like a stack corruption. You must take care that the stack is aligned to 16 before calling a function.

Overview of x64 Calling Conventions

[dedndave]

also - if malloc returns 0, use GetLastError to get a more in-depth description of the error

[AKRichard]

Can you show us a link to that information?

Ill look for the article I read it in, Ill probably come accross it again as I am learning more about memory allocation in windows then I ever wanted to know

Sounds like a stack corruption. You must take care that the stack is aligned to 16 before calling a function.

I thought the sub  rsp, 28h realigned the stack (32 bytes for 4 variables even though only one is used + 8 bytes because the call will push an 8 byte return address for a total of 48).
I was thinking I was screwing up the heap memory but now I am not so sure.  I finally managed to step into the malloc call, which looks like thus:

Code Select Expand



void * __cdecl _malloc_base (size_t size)
{
    void *res = NULL;

    //  validate size
    if (size <= _HEAP_MAXREQ) {
        for (;;) {

            //  allocate memory block
            res = _heap_alloc(size);

            //  if successful allocation, return pointer to memory
            //  if new handling turned off altogether, return NULL

            if (res != NULL)
            {
                break;
            }
            if (_newmode == 0)
            {
                errno = ENOMEM;
                break;
            }

            //  call installed new handler
            if (!_callnewh(size))
                break;

            //  new handler was successful -- try to allocate again
        }
    } else {
        _callnewh(size);
        errno = ENOMEM;
        return NULL;
    }

    RTCCALLBACK(_RTC_Allocate_hook, (res, size, 0));
    if (res == NULL)
    {
        errno = ENOMEM;
    }
    return res;
}


its making it in to the call with the correct value in the correct place, but the

Code Select Expand


res = _heap_alloc(size);


call returns a null pointer followed by RTCCALLBACK returning a null pointer.  The algorithms are not usually failing on the first pass through (my debugging program just generates random numbers of specific lengths until a wrong answer or an error crops up), it usually errors after a few times through. on a single pass with 2 variables of approximately 100 elements each my mod algorithm will create 2 temp variables and 1 return variable for a total of about 2400 bytes each pass.  I have a call to free for each call to malloc, could it be that the heap is becoming too fragmented, hits a call to malloc requesting a block bigger then is available (even though it may have enough free it allways returns a contiguous block right?), and errors out?  Isnt malloc supposed to increase the size of the heap?

Anyways, the reason I said I am not sure its not the stack now is because now that I am able to step into malloc, ive lost my call stack when it errors out.

also - if malloc returns 0, use GetLastError to get a more in-depth description of the error

Ive changed my code to check rax for zero and rerouting through a section of code trying to make it fail gracefully.  However, I havent started reading about error handling within assembly language yet.  I have error handling within my managed code, and a little bit within the native c++, but zip for the assembly, and of course the errors are not propagating to the managed code.

Should I revert to passing in the temp variables and return values until I learn error handling?  Because to be honest, I feel lost here.  Ive been banging my head for the last 4 days trying to figure out why it works fine for smaller numbers but once I hit around 100 elements I start having problems.  Or is it as easy as something like:

Code Select Expand



cmp   rax, 0
ja      GoodPointer
push   r11
push   r12
sub     rsp, 28h
call     GetLastError
add     rsp, 28h
pop     r12
pop     r11
cmp    rax,
...
...
...
GoodPointer:


Or something along those lines?  Also, where would I find the error codes so I could decipher what it means?

[AKRichard]

Thank you for the getlasterror info, its returning error code 8: ERROR_NOT_ENOUGH_MEMORY, which is funny since Im not even stressing my computer debugging.  Now off to read some more.

  I do appreciate it, I was getting upset that I couldnt find out what the problem is, at least now I have a direction to go.

[dedndave]

that's the idea
sometimes - you only need the error handling code during program debug (temporary)
it may be that you are requesting more than you think   :P
it may also be that you are allocating memory in other places, then not releasing it with Free

if you are trying to allocate a lot of memory, you can use something like VirtualAlloc
HeapAlloc will only allocate RAM that is available

[AKRichard]

Ive never had to mess with memory management before.  When I decided to learn c++ I started with managed c++, it was an easier jump from vb.  It wasnt until I wanted my library to be comparable in speed with microsofts version that starterd learning assembly.  Turns out the hardest thing in assembly has been debugging. my debugging code is easily twice the size as my release code, trying to catch errors before they halt my code.

And yes you were right about not freeing memory, I found that I wasnt releasing the return values, they propagate back to the managed side of my code and I wasnt freeing them there.  Ive still got a small memory leak somewhere, and am still reading about memory management in windows, but Im sure Ill figure it out.


  Thanks again for all the help,  Im sure Ill be back