Can you show us a link to that information?
Ill look for the article I read it in, Ill probably come accross it again as I am learning more about memory allocation in windows then I ever wanted to know
Sounds like a stack corruption. You must take care that the stack is aligned to 16 before calling a function.
I thought the sub rsp, 28h realigned the stack (32 bytes for 4 variables even though only one is used + 8 bytes because the call will push an 8 byte return address for a total of 48).
I was thinking I was screwing up the heap memory but now I am not so sure. I finally managed to step into the malloc call, which looks like thus:
void * __cdecl _malloc_base (size_t size)
{
void *res = NULL;
// validate size
if (size <= _HEAP_MAXREQ) {
for (;;) {
// allocate memory block
res = _heap_alloc(size);
// if successful allocation, return pointer to memory
// if new handling turned off altogether, return NULL
if (res != NULL)
{
break;
}
if (_newmode == 0)
{
errno = ENOMEM;
break;
}
// call installed new handler
if (!_callnewh(size))
break;
// new handler was successful -- try to allocate again
}
} else {
_callnewh(size);
errno = ENOMEM;
return NULL;
}
RTCCALLBACK(_RTC_Allocate_hook, (res, size, 0));
if (res == NULL)
{
errno = ENOMEM;
}
return res;
}
its making it in to the call with the correct value in the correct place, but the
res = _heap_alloc(size);
call returns a null pointer followed by RTCCALLBACK returning a null pointer. The algorithms are not usually failing on the first pass through (my debugging program just generates random numbers of specific lengths until a wrong answer or an error crops up), it usually errors after a few times through. on a single pass with 2 variables of approximately 100 elements each my mod algorithm will create 2 temp variables and 1 return variable for a total of about 2400 bytes each pass. I have a call to free for each call to malloc, could it be that the heap is becoming too fragmented, hits a call to malloc requesting a block bigger then is available (even though it may have enough free it allways returns a contiguous block right?), and errors out? Isnt malloc supposed to increase the size of the heap?
Anyways, the reason I said I am not sure its not the stack now is because now that I am able to step into malloc, ive lost my call stack when it errors out.
also - if malloc returns 0, use GetLastError to get a more in-depth description of the error
Ive changed my code to check rax for zero and rerouting through a section of code trying to make it fail gracefully. However, I havent started reading about error handling within assembly language yet. I have error handling within my managed code, and a little bit within the native c++, but zip for the assembly, and of course the errors are not propagating to the managed code.
Should I revert to passing in the temp variables and return values until I learn error handling? Because to be honest, I feel lost here. Ive been banging my head for the last 4 days trying to figure out why it works fine for smaller numbers but once I hit around 100 elements I start having problems. Or is it as easy as something like:
cmp rax, 0
ja GoodPointer
push r11
push r12
sub rsp, 28h
call GetLastError
add rsp, 28h
pop r12
pop r11
cmp rax,
...
...
...
GoodPointer:
Or something along those lines? Also, where would I find the error codes so I could decipher what it means?
