64 bit debuggers for masm 64

[hutch--]

Mark,

JJ's comment is right, when you are building a binary file you are writing BYTE data to a disk file and it could be done with anything that can write BYTE Data to a file. There is no immediate reason to build compilers, linker, assembler and the like as 64 bit binaries as there is no real advantage in doing so and it would involve complete rewrites of these tools from scratch which is a massively complex task. Under a 64 bit version of Windows (win7 and up) you can get close to 3.5 gig of memory and you can build really big files that way so 64 bit does not provide any immediate advantage here.

What I do with Win 10 64 bit is download the enterprise version, get all of the BIN binaries from it along with the include files and libraries and then uninstall it as I don't want a ton of crap buried on my boot drive. For the earlier 64 bit OS versions, you do the same thing with their OS specific versions.

[markallyn]

Hutch and JJ-

Thanks for the clarification.

OK, here's where I am now.  I downloaded and installed all day yesterday VS 2015 Express.  Fortunately, I had to mow a great deal of lawn and so I passed the time away ...

Everything proceeded well.  I found the necessary binaries in the /VC/amd64/bin folder and thought I was in the clear.  But, not so!. Turns out that I needed to be using the "Developer Command Prompt for VS2015" in order to assemble and link properly.  I don't understand why the ordinary Command Prompt doesn't do the job, but so be it.  It can run ml64.exe, but not link.

Anyway, I now have a functioning cl.exe as a useful byproduct of the download/install.

Thanks, gentlemen, for your patience.

Now I have to tackle how to write leaf and frame functions.  Still tryimg to understand how to deal with rsp.

Regards,
Mark

[hutch--]

Mark,

Generally you leave the stack pointer alone, you have enough registers to not have to worry about it.

Look in the directory above bin to get the required "linkui.dll". It needs to be put in a directory called "1033" directly off the bin64 directory.

This is what I have in my Win10 version in the 1033 directory.

atlprovui.dll
bscmakeui.dll
clui.dll
cvtresui.dll
linkui.dll
LocalESPCui.dll
lst.txt
mspdbcmfui.dll
mspft140ui.dll
nmakeui.dll
pgort140ui.dll
pgoui.dll

Look around the folders in vc to find the ones you need.

[markallyn]

Hutch,

Thanks for the info about the 1033 directory.  I will investigate.

As for wiggling the rsp, you are most certainly right that the 4 registers are sufficient most of the time.  Unfortunately, I have this awful habit of investigating every nook and cranny, so not really understanding leaf and frame calls bothers me a lot.

I investigated your "invoke" macro to see how you were managing the rsp  since it looks like your invoke can tell from the function PROTO how to set rsp in the caller.  I couldn't follow the macro code. 

A couple of good examples of "non-invoked" calls to a leaf and the leaf code would be very helpful.  Same with frame functions.

Thanks very much for all of your tutoring, and others such as aw27 and JJ and Fearless among others.  Your generosity gives me hope for humanity.

Mark

Mark

[hutch--]

Mark,

The distinction between a stack frame as against no stack frame is the one you are looking for here. A leaf procedure that only uses mnemonics can usually be written without a stack frame whereas a procedure that calls other procedures or locals usually has a stack frame. The choice here is to either use a stack frame of not use one. There are a couple of macros that control this which are simple enough to use.

  NOSTACKFRAME    ; turn the stack frame off
  STACKFRAME      ; turn it back on again.

You put these before and after a procedure that does not have a stack frame.

Now with the integer registers, you can modify rax rcx rdx r8 r9 r10 r11, the rest need to be protected so that they are the same on procedure exit and with MASM, avoid using push/pop as the preferred method is to use local variables to save and restore registers.

64 bit MASM does not use prototypes which is a blessing in that you don't have to write them but it means you must get the argument count right.

Now with calling procedures, you can manually call simple procedures with up to 4 arguments by writing the arguments in the order of rcx rdx r8 & r9 and using the CALL mnemonic but with a higher argument count, you are better off to use the "invoke" technique as arguments 5 and up are written to the stack and can be messy and complicated to do manually. This is due to the structure of the Microsoft FASTCALL calling convention.

LATER : Something I should have added, a good starter debugger/disassembler is called ArkDasm http://www.arkdasm.com/ and it was very useful to me when I first started to work on 64 bit MASM. There are more complex and more powerful ones around but this one is simple and its dis-assembler will show you much of what you are looking for.

[markallyn]

Hutch,

Played with STACKFRAME.  Now I understand where the ENTER 80h,0 and SUB rsp, 60 come from.  Big Help.  Question: why 80h for enter and sub 60?    Seems kind of arbitrary, without knowing more than I do.

Yes, I've been using Arkdasm and also x64dbg.  Nice programs!

Still searching for how to manipulate rsp--either with or without frame.

Mark

[hutch--]

Mark,

Have a read of the masm64 help file, it explains how and why the stackframe when used is constructed in this manner. Alignment is important, mess that up and the app will not run and will not tell you why. You are dealing with the Microsoft FASTCALL calling convention which use 4 registers for the first 4 arguments and any more are written to the stack.

For reference on using ENTER, you will need to have a good read of the Intel manual for both ENTER and LEAVE. The question you have asked does not have a simple answer, FASTCALL in 64 bit is a tangled mess that you must get right and that is why it is presented as a macro to avoid the complexity and inherent mess of its design. The macro that "invoke" calls is just as messy and complicated as it has to provide the arguments in the right order and format.

As far as directly manipulating RSP, tread with caution here as the pitfalls are many and the uses are few. The problem with getting any alignment wrong is that of the app will not run and just disappears with no explanation of why.

[markallyn]

Hutch-

I looked in help64, but couldn't find a reference to stackframe. 

I downloaded the entire x86 IA32 4 volume set you referenced.  Should be very helpful in future.

Is that a sly reference to Woody Guthrie's old great song "Dark as a Dungeon" in the next to last line?  The Guthrie bit goes like this:  "It's dark as a dungeon and damp as the dew, where the dangers are many and the pleasures are few".  Can't be a coincidence.

Regards,
Mark

[hutch--]

Design Criteria

Calling Convention
How the Win 64 calling convention works.

Stack frame reference
How the MASM64 stackframe works.                    <<< This one.

Call automation
How to use invoke and similar notation.

The rcall macro
A low overhead alternative to invoke for 4 arguments or less.

[markallyn]

Hutch,

Thanks for the info.  This whole business is very messy, as you say.  I don't like messy.

Mark

[aw27]

I don't like messy.

I prefer Ronaldo to messi as well  :t

[HSE]

  I don't like messy.

I prefer Ronaldo to messi as well  :t

Don't worry! We are almost out of next Mundial   

[aw27]

Don't worry! We are almost out of next Mundial   

The World is not fair.  :shock: