64-bit: Why Can't I get "CreateFileA" to Access a File or Device?

[Dan-TheStarman]

Hey Guys,

  First my apologies... This is NOT really a masm32 question, however, I know a number of people here are familiar with VS2019 Community and its x64 code assembly and linking, AND honestly, if I asked this at Microsoft, how many there would be interested in a 64-bit MASM code question??

Background: I had lots of fun assembling, linking and then using Oly to debug my 32-bit code for reading raw sectors from a Disk, BUT every time I try to access a disk or EVEN just a FILE using 64-bit code in VS2019, it always FAILS to do so with INVALID_HANDLE_VALUE (-1)!  It's NOT that there's an issue with VS2019 failing to assemble and link the code (I had to go through quite a lot of headaches to get past the many linker errors), BUT THAT using the same function that works fine under masm32 simply FAILs to do what it should (at least from everything I've looked at so far!).  In my mind, it's as if Windows 10 refuses to allow _CreateFileA_ to function as a 64-bit program! Which is ridiculous... well, unless there actually is something extra you need to do for a 64-bit executable before using this API Function?!  (Or, did I do something wrong?)

Since I had problems accessing Disks raw when I tried that, I decided to try the function to get a FILE handle instead, and can't do that either!  So where am I going wrong??  Here's the CODE I used for FILE example (remember, it's NOT that it can't find the file, BUT that it doesn't even get round to trying... it just comes right back with INVALID_HANDLE_VALUE which means it FAILS to function properly):

Code Select Expand


; This file is for testing a MASM 64-bit Code sample
; that refuses to function as expected in VS2019:

includelib kernel32.lib

; The following External Functions are all included with
; the kernel32.lib library:

EXTERN CreateFileA: proc ; from kernel32.lib
EXTERN CloseHandle: proc ; from kernel32.lib
EXTERN ExitProcess: proc ; from kernel32.lib

GENERIC_READ EQU 80000000h
GENERIC_WRITE EQU 40000000h
GENERIC_EXECUTE EQU 20000000h

FILE_ATTRIBUTE_NORMAL EQU 128
OPEN_EXISTING EQU 3
FILE_SHARE_WRITE EQU 2
FILE_SHARE_READ EQU 1

ERROR_FILE_NOT_FOUND EQU 2
INVALID_HANDLE_VALUE EQU -1

.data

szFileName db "C:\Users\[b]<whatever>[/b]\source\TESTFILE.bin", 0
; (Note: You must use your own valid PATH here!)

.data?

hFile dq ?

.code

main proc

; [[ Edited at a later date so as to be helpful to anyone reading this much later: ]]
;     =====================================================
;  The following instruction turned out to be in the WRONG place! Read notes further
;  below!  So I commented this out here!

;  sub rsp, 32 ; First, a required "Shadow Space" on the Stack of 32, 8-bit bytes MUST
;  ALWAYS be reserved (the size of FOUR 64-bit parameters) by
  ; 'subtracting' 32 (0x20) from the 64-bit Stack Pointer (RSP). A 64-bit MASM file may
  ; not even LINK without having this minimum "_Shadow_, _Home_ or _Spill_ Space" being
  ; reserved. The extra 8 bytes are for a RETurn Address for the CALL to function.


; ==============================
; Here we attempt to access an existing file:
; ==============================

; First we need to obtain a "handle" to an existing file in order to actually read
; data from it using:
; https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-createfilea
; If "CreateFileA" provides us with an "open handle" to the file, only then can Data
; actually be read from it!
;
; 1) Offset szFileName = Name of file to be opened (limited to MAX_PATH characters).
; 2) GENERIC_READ    = The requested access to the device is "GENERIC_READ" (see:
; https://docs.microsoft.com/en-us/windows/win32/secauthz/generic-access-rights).
; 3) Share Mode is: "FILE_SHARE_READ or FILE_SHARE_WRITE" = means anything else can
;      access the file for reading or writing!
; 4) Next is "lpSecurityAttributes" and we just set this to NULL (zero).
; 5) OPEN_EXISTING = Opens the file only if it exists. It's possible the location
; is wrong, in which case this error is generated:
; ERROR_FILE_NOT_FOUND (2).
;
; 6) The file "attributes flag" is set to FILE_ATTRIBUTE_NORMAL (128) for this example.
;
; 7) The last parameter is only for the creation of a new file so it's set to NULL (0).
;
; If CreateFile succeeds, the return value is an "open handle" to the file in RAX.
; If the function fails, return value is INVALID_HANDLE_VALUE ("-1").

; Note: This Windows API Function uses 7 parameters, so we need to reserve space for
; THREE more sets of 8 bytes (space for a Return Address having already been
; reserved previously), for a total of 24 more bytes
; [[ The note above is somewhat incorrect, because the 3 extra parameters ended up being
;     PUSHed onto the Stack, before any space needed to be reserved for them; see below!]]

; According to this MICROSOFT page (see URL below): "By default, the x64 calling convention
; passes the first four arguments to a function in registers. The registers used for these
; arguments depend on the position and type of the argument. _Remaining arguments get
; pushed on the stack in _right-to-left_ order._"
; https://docs.microsoft.com/en-us/cpp/build/x64-calling-convention

; [[ Edited at a later date so as to be helpful to anyone reading this much later: ]]
;     =====================================================
; NOTE: I had misunderstood the ambiguous (in my opinion) "rules" at Microsoft; which
;            made me think that parameters beyond the first four were supposed to be
;            pushed onto the Stack AFTER reserving the 32 bytes for those first four....
;            Well, WRONG, when using more than four, you must have those additional
;            parameters already pushed onto the Stack in reverse order from what you see
;            in the Windows API Function documentation. To put it differently, BEFORE ever
;            reserving the 32-byte "Shadow Space," you must push the 7th (in this case),
;            then 6th, then 5th parameters in that order, and only then do:  sub  rsp, 32 !

push 0 ; hTemplateFile (see 7 above).
push FILE_ATTRIBUTE_NORMAL ; dwFlagsAndAttributes (see 6 above).
push OPEN_EXISTING ; dwCreationDisposition ("3"; see 5 above).

; [[ Edited at a later date so as to be helpful to anyone reading this much later: ]]
;     =====================================================
;    Only now can we carry out the "Shadow Space" reservation for the first four params:

  sub rsp, 32 ;  Reserve the required "Shadow Space" on the Stack for 32 bytes
;  which must ALWAYS be reserved (the size of FOUR 64-bit
; parameters) by 'subtracting' 32 (0x20) from the 64-bit Stack Pointer (RSP)

; The order in which you handle the data in the R9, R8, RDX and RCX registers doesn't
; really matter, just that only RCX is used if there's only one, RCX and RDX for two, etc.

xor r9,r9 ; lpSecurityAttributes (see 4 above); set to 0.

mov r8, FILE_SHARE_WRITE OR FILE_SHARE_READ ; dwShareMode [2 OR 1 = 3](see 3 above).

mov rdx, GENERIC_READ ; _dwDesiredAccess_ is set to: 80000000h (see 2 above).
mov rcx, offset szFileName ; lpFileName = Pointer to filename (see 1 above).

call CreateFileA ; Attempt to access file: TESTFILE.bin for reading!

mov hFile, rax ; If successful, copy "file handle" in RAX to hFile.

; A couple examples for checking the RETURN VALUE from the API:

cmp hFile, INVALID_HANDLE_VALUE ; The Function FAILED.

cmp hFile, ERROR_FILE_NOT_FOUND ; The file could not be found.


; If anyone can get it to actually function correctly, then of course code for READING
; info from the FILE would go here!

mov rcx, hFile ; The _hObject_ for CloseHandle to act on.

call CloseHandle ; Close the open handle (hObject) to the disk drive.

; If CloseHandle is successful, it returns a non-zero value; or 0 if it fails.


call ExitProcess

main endp

end



   Likewise, when using the different parameter values required for attempting to access "\\.\PhysicalDrive0," I am always getting the INVALID_HANDLE_VALUE; which I didn't bother showing here, since you MUST run such a program with ADMIN PRIVILEGES and there's no point adding more problems for anyone willing to help me with this! ;-)   

Dan.

[Vortex]

Hi Dan,

You don't push the API parameters following rcx,rdx,r8 and r9 to the stack. You copy those values coming after r9 to the correct stack locations.

Reading the document \masm32\help64\calling.htm :

Calling a function

The first four (4) arguments are written to the RCX RDX R8 and R9 in any of the 4 data sizes supported by the calling convention and any following arguments are written to a stack relative location in memory without changing the stack pointer (RSP). Many function have 4 or less arguments and obtain the advantage of lower calling overhead by receiving the 4 or less arguments directly in the four specified registers.

When there are more than four arguments, arguments 5 and upwards are written to the stack and here there is another consideration that will become obvious at the receiving end of a function call, the first four locations on the stack are left empty so that the 4 registers can be stored at those locations if necessary. The first four stack addresses are [rsp], [rsp+8], [rsp+16] and [rsp+24]which are left empty. Argument 5 and upwards are written to the RSP relative address [rsp+32] and upwards with an increase in displacement of 8 bytes for each argument.

A typical function call with 6 arguments will look like this.

mov rcx, arg1
mov rdx, arg2
mov r8, arg3
mov r9, arg4
mov QWORD PTR [rsp+32], arg5
mov QWORD PTR [rsp+40], arg6
call FunctionName

It is worth noting that with the stack arguments, if they are either a register or an immediate operand they are written directly to the RSP relative stack address. If the argument is a memory operand, either LOCAL of GLOBAL, it will be written to a register first then the register is written to the stack address as x86 - 64 processors do not support direct memory to memory copy.

It will look like this.
mov rax, arg5
mov QWORD PTR [rsp+32], rax
mov rax, arg6
mov QWORD PTR [rsp+40], rax

The file attached here :

http://masm32.com/board/index.php?topic=8929.msg98109#msg98109

Code Select Expand

    invoke  CreateFile,ADDR szFileName,\
            GENERIC_READ,FILE_SHARE_READ or FILE_SHARE_WRITE,0,\
            OPEN_EXISTING,0,0
    mov     hFile,rax


Disassembling the object module :

Code Select Expand

        lea     rcx, [szFileName] 
        mov     rdx, 80000000H
        mov     r8, 3
        mov     r9, 0
        mov     qword ptr [rsp+20H], 3
        mov     qword ptr [rsp+28H], 0
        mov     qword ptr [rsp+30H], 0
        call    qword ptr [__imp_CreateFileA]
        mov     qword ptr [hFile], rax


[Vortex]

Hi Dan,

Here is the modified example :

Code Select Expand

;    invoke  CreateFile,ADDR szFileName,\
;            GENERIC_READ,FILE_SHARE_READ or FILE_SHARE_WRITE,0,\
;            OPEN_EXISTING,0,0

    mov     rcx,OFFSET szFileName
    mov     rdx,GENERIC_READ
    mov     r8,FILE_SHARE_READ or FILE_SHARE_WRITE
    xor     r9,r9
    mov     QWORD PTR [rsp+32],OPEN_EXISTING
    mov     QWORD PTR [rsp+40],0
    mov     QWORD PTR [rsp+48],0
    call    CreateFile

    mov     hFile,rax

[TouEnMasm]

HANDLE CreateFileA(
  LPCSTR                lpFileName,
  DWORD                 dwDesiredAccess,
  DWORD                 dwShareMode,
  LPSECURITY_ATTRIBUTES lpSecurityAttributes,
  DWORD                 dwCreationDisposition,
  DWORD                 dwFlagsAndAttributes,
  HANDLE                hTemplateFile
);

I see many QWORD where there must be DWORD

CreateFileA PROTO lpFileName:QWORD ,dwDesiredAccess:DWORD ,dwShareMode:DWORD ,lpSecurityAttributes:QWORD ,dwCreationDisposition:DWORD ,dwFlagsAndAttributes:DWORD ,hTemplateFile:HANDLE

[Vortex]

I see many QWORD where there must be DWORD

No need for that. Even if you prefer to use DWORDs, the upper 32-bit part of the QWORDs will be set to 0. You can verify this using a debugger :

Code Select Expand

    mov     rcx,OFFSET szFileName
    mov     rdx,-1
    mov     edx,GENERIC_READ
    mov     r8,-1
    mov     r8d,FILE_SHARE_READ or FILE_SHARE_WRITE


What happens to the HIDWORD of rdx after mov edx,GENERIC_READ ?

[Dan-TheStarman]

Hi Dan,
Here is the modified example :  ETC. ETC.

Unfortunately, I was not posting about the 64-bit Assembler/Linker that Hutch (and others?) has provided us with, and that you used for your 64-bit examples!

I'm talking about the Visual Studio 2019 Community Edition IDE; which is supposed to follow these rules (only some of the full "Microsoft 64 Calling Convention" rules are shown here):

1) "It is the caller's responsibility to allocate 32 bytes of "shadow space" (for storing RCX, RDX, R8, and R9 if needed) before calling the function."  Thus, my:

Code Select Expand

sub rsp, 32 instruction.
2) "RCX, RDX, R8, R9 are used for integer and pointer arguments in that order left to right."
3) "Additional arguments are pushed on the stack left to right."  Which is why I was using PUSH instructions.
[[ To help anyone reading this at a later date: The above "rule" is WRONG; it's a 'typo error'... it should read: right to left as the Microsoft documentation referenced below states!  It's also NOT very helpful, because it fails to state: THESE ARGUMENTS must be PUSHed onto the Stack BEFORE you reserve space for the first 4 !!!! ]]
4) "The stack is 16-byte aligned. The "call" instruction pushes an 8-byte return value, so the [sic] all non-leaf functions must adjust the stack by a value of the form 16n+8 when allocating stack space."
(Reference: https://software.intel.com/content/www/us/en/develop/articles/introduction-to-x64-assembly.html)

IMPORTANT NOTE: Look at #3) above...  It is in DIRECT CONFLICT with the rule I found at MICROSOFT at this URL:
https://docs.microsoft.com/en-us/cpp/build/x64-calling-convention

Where it states under "Parameter Passing" :

"By default, the x64 calling convention passes the first four arguments to a function in registers. The registers used for these arguments depend on the position and type of the argument. Remaining arguments get pushed on the stack in right-to-left order."

So, I tried it BOTH WAYS... But it doesn't matter which order I used, it still results in the INVALID error either way!!  Note: I'm pretty much convinced that the statement on the INTEL website is a 'typo error' where someone simply wrote the same phrase they had used in #2) above.


Lastly, for Vortex and everyone else here:
NOTE: It's quite easy to see WHY under Hutch's "masm64" (if you will) Assembler/Linker that you must NOT use PUSH: It's because that software is set up to handle the 16-byte alignment AND Stack space reservations ITSELF, so you CANNOT do anything which changes the initial RSP when calling such a Function!

So, is there anyone here who did a VS2019 Community INSTALL and can confirm that there is something wrong there(!), or with my code?


Dan.

[jj2007]

Dan,
As Vortex already pointed out, there is no pushing & popping in 64-bit code. The stack pointer gets adjusted once to its desired position, and then the arguments get moved into the stack, not pushed. Here is another example (it works fine):

Code Select Expand

  mov rbx, rv(CreateFile, Chr$("CreateFile.asm"), GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, 0)

Code Select Expand

48 83 64 24 30 00               | and qword ptr ss:[rsp+30],0                               | arg7: 0
48 C7 44 24 28 80 00 00 00      | mov qword ptr ss:[rsp+28],80                              | arg6: FILE_ATTRIBUTE_NORMAL
48 C7 44 24 20 03 00 00 00      | mov qword ptr ss:[rsp+20],3                               | arg5: OPEN_EXISTING
45 33 C9                        | xor r9d,r9d                                               | arg4: 0
41 B8 01 00 00 00               | mov r8d,1                                                 | arg3: FILE_SHARE_READ
BA 00 00 00 80                  | mov edx,80000000                                          | arg2: GENERIC_READ
48 8D 0D 17 02 00 00            | lea rcx,qword ptr ds:[<sub_140001268>]                    | arg1: "CreateFile.asm"
FF 15 09 05 00 00               | call qword ptr ds:[<&CreateFileA>]                        |

[TouEnMasm]

at this time,better is to post a sample with a batch,so we can see :
Wich system you have ?
Wich compiler You have ?
Wich libraries you used ?

I see that you want to use X64 at his primary state,without any compilers help, but that all.

[TimoVJL]

Simple old example:

Code Select Expand

int foo(int a, int b, int c, int d, int e, int f)
{
return 0;
}

int __cdecl main(void)
{
foo(1,2,3,4,5,6);
return 0;
}


Code Select Expand

foo:
00000000  B800000000               mov eax, 0h
00000005  C3                       ret
00000006  C3                       ret
main:
00000007  4883EC38                 sub rsp, 38h
0000000B  C744242806000000         mov dword ptr [rsp+28h], 6h
00000013  C744242005000000         mov dword ptr [rsp+20h], 5h
0000001B  B901000000               mov ecx, 1h
00000020  BA02000000               mov edx, 2h
00000025  41B803000000             mov r8d, 3h
0000002B  41B904000000             mov r9d, 4h
00000031  E8CAFFFFFF               call $-31h
00000036  B800000000               mov eax, 0h
0000003B  4883C438                 add rsp, 38h
0000003F  C3                       ret
When thinking RTL(C), have to think stack order
From Agner Fog

Parameter order on stack. The Pascal order means that the first parameter has the highest address on the stack and the last parameter has the lowest address, immediately above the return address. If parameters are put on the stack by push instructions then the first parameter is pushed first because the stack grows downwards. The C order is opposite: The first parameter has the lowest address, immediately above the return address, and the last parameter has the highest address. This method was introduced with the C language in order to make it possible to call a function with a variable number of parameters, such as printf. Each parameter must take a whole number of stack entries. If a parameter is smaller than the stack word size then the rest of that stack entry is unused. Likewise, if a parameter is transferred in a register that is too big, then the rest of that register is unused.If the type of a parameter is not specified explicitly because the function has no prototype or

[Vortex]

Hi Dan,

Lastly, for Vortex and everyone else here:
NOTE: It's quite easy to see WHY under Hutch's "masm64" (if you will) Assembler/Linker that you must NOT use PUSH: It's because that software is set up to handle the 16-byte alignment AND Stack space reservations ITSELF, so you CANNOT do anything which changes the initial RSP when calling such a Function!

If you are operating under Microsoft Windows, no matter which development tool you select for 64-bit coding, you are bound to follow the same Microsoft 64-bit calling convention :

https://software.intel.com/content/www/us/en/develop/articles/introduction-to-x64-assembly.html

The word "push" is a bit problematic here. As jj2007 stated, the extra arguments after r9 are moved into the stack.  You will get absolutely the same result with the assembler shipped with VS2019. The only condition is to respect the rules of the 64-bit calling convention.

[TouEnMasm]

How to get an answer by youself ?.
Begin by write your code with a compiler (uasm for example)
This one will say you you have forgotten one argument in createfile and more.
Then use dumpin /DISASM on the object file to get what you are trying to write

main:
   sub         rsp,48h
   ;-------------------------- end prologue
   lea         rcx,[szFileName]
   mov         edx,80000000h
   xor         r8d,r8d
   xor         r9d,r9d
   mov         dword ptr [rsp+20h],3
   mov         dword ptr [rsp+28h],80h
   mov         qword ptr [rsp+30h],0
   call        CreateFileA
   mov         qword ptr [hFile],rax
   cmp         qword ptr [hFile],0FFFFFFFFFFFFFFFFh
   jne         000000000000005A
      mov         r11,0
      lea         rcx,[??000C]
      call        printf
      jmp         0000000000000072
000000000000005A:
   lea         rcx,[??000D]
   call        printf
   mov         rcx,qword ptr [hFile]   
   call        CloseHandle
   ;------- start epilogue ------------------------
0000000000000072:
   add         rsp,48h
   ret

The sample will answer it failed,but it is only a bad use of createfile not a coding fault.

[hutch--]

I wonder at the point of contributing here in this topic, a number of members have tried to correct the invalid assumptions but we keep hearing that a failed system should work when it does not. In order Dan needs to do these things.

1. Write a matching prologue / epilogue pair of macros that balance the stack and provide shadow space.
2. Write a macro that writes up to the first 4 arguments to the specified registers then WRITES (not PUSHES) each further argument to an ascending 64 bit stack address.
3. ABSOLUTELY do not try and balance the stack manually.
4. Do not use the stack balancing argument after RET.
5. FORGET 32 bit STDCALL and/or any of its variants, the Microsoft x64 Application Binary Interface does not work that way.

[TimoVJL]

Opening a file or sector could needs admin rights.

[nidud]

deleted

[hutch--]

nidud,

> The problem here is that there are no assembler template for loading arguments in 64-bit given ML64 does not support INVOKE as it does in 32-bit.

You are out of date by about 4 years. masm64 project has had invoke and variation for a long time and it works according to the x64 ABI, that why it works perfectly.

It also has a number of different stack frame designs, all of which work perfectly.