Strange behavior with direct2d and ml64

[HSE]

Won't that create a memory leak?

Yes there is a problem.

[jj2007]

Sorry, folks, I had assumed that all coinvoke calls return S_OK if there is no error - see below, Microsoft docs.

Apparently, that's not the case for D2D1; for example, ID2D1RenderTarget,GetSize returns 0000000140003824 in rax, but x=887, y=499 in the sizeRenderTarget. Btw no matching picture for these values, which are clearly floats and look plausible but I can't see the jpg or png file that matches these dimensions

Error Handling in COM (COM)
Almost all COM functions and interface methods return a value of the type HRESULT. The HRESULT (the name can be read as "result handle") is a way of returning a success, warning, or error value. A HRESULT is actually not a handle (see Why does HRESULT begin with H when it's not a handle to anything?); it's just a value with several fields encoded into it. As per the COM specification, a result of zero indicates success, and a nonzero result indicates failure.

Dx64MacroHelpers.inc, coinvoke with error checking:

Code Select Expand

coinvoke MACRO pInterface:REQ, Interface:REQ, Function:REQ, arg1,arg2,arg3,arg4,arg5,arg6,arg7,arg8,arg9,arg10
LOCAL istatement, lbl
...
istatement CATSTR <call qword ptr >,<(Interface PTR[rax]).&Interface>,<_>,<&Function>
mov rax, qword ptr [rcx]
istatement
test rax, rax
je lbl
mov _rcx, rcx ; define in .DATA?
mov _rdx, rdx
conout "CoInvoke fails in line ", str$(@Line)
conout chr$(13, 10)
mov rcx, _rcx
mov rdx, _rdx
xor eax, eax
ENDM

That returns this (line numbers might be slightly different with your code):

Code Select Expand

CoInvoke fails in line 228
CoInvoke fails in line 339
CoInvoke fails in line 341
CoInvoke fails in line 266
CoInvoke fails in line 367
CoInvoke fails in line 369
CoInvoke fails in line 385
CoInvoke fails in line 367
CoInvoke fails in line 369
CoInvoke fails in line 385

So the error-checking coinvoke macro is more challenging. Instead of the test rax, rax, there should be a more individual test that checks if rax returns the expected value.

[Caché GB]

Hi BugCatcher

In 64bit when calling Win APIs and COM you need to make shadow SPACE always and sometimes stack "argument" SPACE when
there are greater than 4 parameters. Remember COM has one extra "hidden" parameter.

You also need to keep the stack aligned to 16 bytes before calling APIs or COM, i.e rsp's address must always end with a "0" .

If you do not add the stack "argument" SPACE, you will be writing over any data in your locals, if you have any, when you use
the invoke or coinvoke macros.

Just look in your coinvoke macro

IFNB <arg10>
  mov rax, arg10
  mov qword ptr [rsp+50h], rax ; <- This WILL do damage without stack space
ENDIF

Worst-case you will overwrite your proc's return address placed on the stack - then you crash and burn.

Code Select Expand


===============================>

main proc
LOCAL wc      :WNDCLASSEX
LOCAL lft     :QWORD
LOCAL top     :QWORD
LOCAL wid     :QWORD
LOCAL hgt     :QWORD

            add  rsp, -68h
            and  rsp, -10h

===============================>

msgloop proc
    LOCAL msg    :MSG
    LOCAL pmsg   :QWORD

            add  rsp, -28h
            and  rsp, -10h

===============================>

WndProc proc hWin:QWORD,uMsg:QWORD,wParam:QWORD,lParam:QWORD

            mov  hWin, rcx
            mov  uMsg, rdx
            mov  wParam, r8
            mov  lParam, r9

            add  rsp,-28h  ; <- This wind proc ALWAYS comes in unaligned.

===============================>

D2D1Init proc hWin:HWND

.......

.code

            add  rsp, -68h
            and  rsp, -10h

            mov  hWin, rcx

invoke D2D1CreateFactory,\

===============================>

LoadD2Resources proc ;; hWin:HWND  <-  leave this out.

......

.code
           push  rbp 
            mov  rbp, rsp 
            add  rsp, -68h
            and  rsp, -10h

    invoke LoadResourceBitmap,\

......
          leave
            ret

===============================>


LoadResourceBitmap proc pRenderTarget:ptr ID2D1RenderTarget,\
    pIWICFactory :ptr IWICImagingFactory,\
    resourceName:PCWSTR,\
    resourceType:PCWSTR,\
    destinationWidth:UINT,\
    destinationHeight:UINT,\
    ppBitmap:ptr ptr ID2D1Bitmap
..........

.code

            mov  pRenderTarget, rcx
            mov  pIWICFactory, rdx
            mov  resourceName, r8
            mov  resourceType, r9

            add  rsp, -68h
            and  rsp, -10h

     invoke FindResource,hInstance,resourceName,resourceType

===============================>

Render proc

....

.code

           push  rbp 
            mov  rbp, rsp 
            add  rsp, -68h
            and  rsp, -10h

       f2f dRect.left,FLT4(600.0)
......
          leave
            ret

===============================>