visual studio heap corruption(can someone help me plz)

[Troy Marchant]

Hi. I am starting to write a demo scene demo in direct 3d 12 using masm64 and am having some problems with heap corruption. I just cant seem to get it to work. I am using the HeapAlloc function and HeapReAlloc to alocate some dxgi objects(i know this would probably be better done in c++ but im intent on it being assembler). It seems to get through the dxgi routine. but then crashes on later code. if i comment out the DXGI code it runs fine. Although i cant fill the dialogs combo boxes without it working..
I'm gona zip up the entire solution and attach it.
Any help would be awesome.

[jj2007]

I clicked on ...\Troy\GRNMASM\GRNMASM.vcxproj and got this a few minutes later:

Code: [Select]

An exception has been encountered. This may be caused by an extension.

You can get more information by examining the file 'C:\Users\Jochen\AppData\Roaming\Microsoft\VisualStudio\14.0\ActivityLog.xml'.
The only error there is:

Code: [Select]

Extension will not be loaded because an extension with the same ID 'Microsoft.Windows.DevelopmentKit.Desktop' is already loaded at C:\PROGRAM FILES (X86)\COMMON FILES\MICROSOFT\EXTENSIONMANAGER\EXTENSIONS\MICROSOFT\WINDOWS KITS\8.0\DESKTOP SDK\... ::)

[Troy Marchant]

arg crap. My VS is 2017 community edition with the latest platform sdk. Ive probably added crap to the solution also which wont allow it to load. I can see I added jwasm to it which shouldn't be there either.
Ill try again to zip up something that can load.

[Troy Marchant]

cant seem to see what it could be. clicking the solution file GRNMASM.sln might work. It NEEDS windows 10 installed to execute too because that is the only version of windows that has dx12.
if you can right click on the project in VS you might be able to retarget it to a different version that you have installed.

[jj2007]

clicking the solution file GRNMASM.sln might work.

Indeed, no more exception :t

But:

Code: [Select]

------ Build started: Project: GRNMASM, Configuration: Debug x64 ------
========== Build: 0 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========
No additional info :(

Wouldn't it be easier to insert a few int 3's in strategic places, and to post the exe so that we can debug it?
Can you build a 32-bit version, and does it crash as well?

[Troy Marchant]

I'm looking into the int 3 thing cause i'm not familiar with it. but here is a normal debug exe. 32 bit versions probably wouldnt run dx12 though.

[jj2007]

"Can't find entry point CreateDXGIFactory2 in dxgi.dll" - bad luck. This is Win7-64.

What about your debugging tools? VC must have something, no? What I usually do is launch the heap debugger:

include \masm32\include\masm32rt.inc

.code
start:
  mov esi, alloc(100)   ; HeapAlloc a small buffer
  print "before doing illegal things to "
  print hex$(esi), 13, 10
  mov dword ptr [esi+200], "oaiC"            ; write Ciao to no man's land
  print " after doing illegal things", 13, 10
  free esi
  inkey "buffer was freed"
  exit

end start

OPT_Debug 1      ; use the heap debugger (requires RichMasm)

That gives me this output:

Code: [Select]

before doing illegal things to 004DD148
 after doing illegal things

## HEAP[HeapCorruption.exe]:
## HEAP: Free Heap block 4dd1c0 modified at 4dd210 after it was freed
buffer was freed
Which is usually enough to identify the culprit. I am sure Microsoft has something like that.

I attach the project. It's 32-bit, though.

[BugCatcher]

ZeroMemory PROC

  ; rcx = memory address
  ; rdx = byte count

    mov r11, rdi        ; preserve RDI

    xor rax, rax        ; zero fill RAX
    mov rdi, rcx        ; memory address in RDI
    mov rcx, rdx        ; byte count into RCX
    shr rcx, 3          ; int div by 8                       
    rep stosq           ; write byte data

    mov rcx, rdx        ; byte count into RCX
    and rcx, 7          ; calculate remainder
    rep stosb           ; write byte data

    mov rdi, r11        ; restore RDI

    ret

ZeroMemory ENDP

the format for rep stosq  =copy contents of the pointer rsi to contents of pointer rdi -- a qword type size.
then increment rsi and rdi by a qword and dec rcx. rax is doing nothing.
rsi=source indexing
rdi=destination indexing
rcx=counter

I believe zeromemory is a reserved word

[jj2007]

The routine (GRNMASM\ZEROFILL.ASM) looks correct. Where do you see a bug?

(attention, don't confuse stos* with movs*)

[BugCatcher]

Yes your right, I must be getting senile!

[Troy Marchant]

ok i found my problem. its got something to do with struct packing and alignment.
I tried this out.
   ;-------------------------------------------------------------------------------------------------
   ;Allocate an adapter struct and put it in the array at last pos
   ;-------------------------------------------------------------------------------------------------
   mov rcx,SIZEOF Adapter+100h
   call GAlloc
   test rax,rax
   jz EnumerateDXGIFailed
   mov rcx,QWORD PTR [pAdapterArray]
   mov rcx,QWORD PTR [rcx+AdapterArray.ppAdapterArray]
   mov rdx,QWORD PTR [qAdapterCount]
   mov QWORD PTR [rcx+rdx*8],rax
and it totally works. My DXGI_ADAPTER_DESC2 struct i made in the header file is obviously the wrong length when using sizeof on it. thanks for you guys help anyway.