Hello everyone,
Back on August 25, 2019 VORTEX submitted code that would create stdin, stdout, and stderr. I had been trying for several weeks to figure out how to do this because these streams are very useful. I finally saw V's post and recognized its brilliance as a solution to my failed efforts. So, I set out to try to take his solution a bit further and make it a static library function. In doing so I discovered that Pelles' POCRT does not contain __p_iob but msvcrt.lib has __iob_func which works similarly.
In the following submission I'm using FGETS and FPUTS simply as test functions. FGETS fails in this code and for the life of me I can't understand why. I have tried to annotate to help read the darned thing.
Quote
;trying to create subroutine to define stdin, stdout, and stderr. Subroutine myio is based on
;a submission by VORTEX on August 25, 2019 to MASM Forum, Miscellaneous, The Orphanage, as response to JC FULLER post
;August 24.
include myincludes64.inc ;Homemade .inc file with typedefs, structs, et. I havve found useful.
includelib c:\masm32\lib64\msvcrt.lib ;This MSVCRT contains __iob_func to access stdin, stdout, stderr.
__iob_func PROTO :QWORD
puts PROTO :QWORD
printf PROTO :QWORD
fgets PROTO :QWORD, :DWORD, :QWORD
fputs PROTO :QWORD, :DWORD, :QWORD
myio PROTO
.data
msg1 db "Enter a string:",13,10,0
msg2 db "The string you entered was:",13,10,0
.data?
stdin QWORD ?
stdout QWORD ?
stderr QWORD ?
buffer BYTE 256 dup (?)
.const
.code
main PROC
enter 0,0 ;this section creates a stack, aligns the stack, and
and rsp, -10h ;makes room for spill space
sub rsp, 20h
lea rcx, stdin
lea rdx, stdout
lea r8, stderr
call myio ;this call sets up stdin, stdout, stderr
;Myio successfully returns to here.
mov r11, rcx ;stdin should be in r11
mov r12, rdx ;stdout should be in r12
mov r13, r8 ;stderr should be in r13
lea rcx, msg1
call printf ;printf successfully prints message1
lea rcx, buffer
mov rdx, 80
mov r8, r11
call fgets ;FGETS fails! WHY??
lea rcx, msg2
call printf
lea rcx, buffer
mov r8, r12
call fputs
add rsp, 20h ;this section removes spill space and tears down stack
leave
ret
main ENDP
myio PROC ;This little PROC is based on VORTEX, August 25, 2019. Thanks.
enter 0,0
and rsp, -10h
sub rsp, 20h
push rcx
mov rcx, 0 ;this section sets up stdin and stdout
call __iob_func ;__iob_func does what VORTEX's crt___p__iob function did in V's version
pop rcx
mov qword ptr [rcx], rax
mov r9, SIZEOF (FILE)
add rax, r9
mov qword ptr [rdx], rax
add rax, r9
mov qword ptr [r8], rax
add rsp, 20h
leave
ret ;Myio successfully returns.
myio ENDP
END
Many thanks again to VORTEX for his original contribution.
Mark Allyn
Hi markallyn,
Did you take care of the structure alignment?
https://www.geeksforgeeks.org/structure-member-alignment-padding-and-data-packing/
My guess is that r11 gets clobbered by printf. R10 and r11 are not guaranteed to be call-preserved.
https://docs.microsoft.com/en-us/cpp/build/x64-calling-convention?view=msvc-160
deleted
Hello VORTEX, TENKEY, and NIDUD,
Owing to family responsibilities (my daughter is in deep pregnancy) I only had a chance to look over Vortex's reply. I read his attachment re structure alignment and found it very informative on a subject I didn't understand well. However, I need a bit more detail on why Vortex thought there might be a misalignment in my FILE Struct (as this is the only struct I know about in the program).
Tenkey: I will test your idea later this evening and report back.
NIDUD: You have spent a lot of time on this response and I really appreciate what you have done. I'm going to learn a lot. As I think you can tell from what I did I don't fully understand parameter passing using x64 ABI. That was part of my motivation for trying to do this little project. I notice that you used __acrt_iob_func!
Regards to each of you,
Mark
Tenkey,
Of course you're right about r11 not being preserved! I forgot. I tried changing r11 to r12 and r12 to r13. No luck, I'm sorry to report. Same problem with FGETS.
Thanks for the suggestion!
Mark
Good Morning, NIDUD,
I have been trying to replicate your code on my machine using visual studio 2017 x64 native tools command. I cannot link to libcmt.lib and __acrt_iob_func. I have found this latter function in windows\system32, however. __acrt_iob_func on my machine is located in libucrt.dll. But, there is no corresponding .lib or .inc in Windows \system32 or \Windows from which I can get the prototype.
Would you kindly tell me what is going on?
Regards,
Mark
deleted
Good afternoon, NIDUD,
I looked in ALMOST the same VS 2019 directory that you mention, but there is a change of version in my file. Instead of 14.28.29333\lib\x64 directory mine is 14.25.28610\lib\x64. In my version there is no __acrt_iob_func as far as I can tell.
You answered one question I had as a result of this. Namely, I can use the __iob_func as a substitute. But I gotta tell you that I would really like to use the __acrt_iob_func. Not finding it (so far) is highly unsatisfying.
I agree it could be useful to figure this out--couldn't agree more. I thought I might be able to use windows\system32 but no .lib to include. Would have to "roll my own" and I'm lazy.
Thanks again,
Mark
Hi markallyn,
You don't need any other specific library to assemble the code below, the Masm64 additions developed by Hutch is doing all the job. The largest element in the _iobuf structure, QWORD has a size of 8 bytes so they need to be aligned to 8 byte boundary. The DWORD members in the same structure will require attention, so we instruct the assembler to do it for us with _iobuf STRUCT 8. You can see that the first DWORD member will "break" the alignment :
_cnt DWORD ?
Here is a quick example for the Masm64 SDK :
include \masm32\include64\masm64rt.inc
_iobuf STRUCT 8 ; align to 8 byte boundary
_ptr QWORD ?
_cnt DWORD ?
_base QWORD ?
_flag DWORD ?
_file DWORD ?
_charbuf DWORD ?
_bufsiz DWORD ?
_tmpfname QWORD ?
_iobuf ENDS
_FILE TYPEDEF _iobuf
EXTERN __iob_func:PROC
.data?
_stdout dq ?
_stdin dq ?
_stderr dq ?
.data
msg db "Please type your name :",13,10,0
.data?
szName db 16 dup(?)
.code
start PROC
invoke __iob_func
mov _stdin,rax ; #define stdin (&__iob_func()[0])
add rax,SIZEOF(_FILE)
mov _stdout,rax ; #define stdout (&__iob_func()[1])
add rax,SIZEOF(_FILE)
mov _stderr,rax ; #define stderr (&__iob_func()[2])
invoke vc_fputs,ADDR msg,_stdout
invoke vc_fgets,ADDR szName,16,_stdin
invoke vc_printf,"Nice to meet you, %s",ADDR szName
invoke ExitProcess,0
start ENDP
END
markallyn,
You stored a file pointer with:
mov qword ptr [rcx], rax
So you need to retrieve the file pointer with:
mov r8, qword ptr [r11] ; you will need to change r11
or
mov r8, stdin ; not lea !!!
I got it to work by replacing both instances of r11 with r14 and adding qword ptr.
deleted
Good evening, tenkey, vortex, and nidud,
Tenkey: I made the change you suggested and the code worked perfectly. Thanks much for your continued interest.
Nidud and Vortex: You have both provided code that requires much closer study. I promise to reply to what you have given me.
All of you: I can't tell you how grateful I am for your continued interest in my rather trivial problem. I have learned a great deal and I am delighted by your generosity.
I wish there was a way to pay you back.
Mark
Good morning, Vortex,
In your _iobuf struct I make out the size to be 48 bytes including 4 bytes of padding in the first DWORD breakage. Confirmed with a little program I just wrote to count struct bytes.
That same program counts 80 bytes for my FILE struct as it is defined in my include file. Leads to a question about my original code. As follows:
Within my myio proc I set r9 to be sizeof(FILE). Obviously not the same as your _FILE typedef. One (ME!) would naively think that this would lead to a failure of my code NO MATTER WHAT ELSE might have been wrong. I would have expected that adding the incorrect size to rax would have incorrectly computed stdout.
But, after making the change that tenkey suggested, my orginal code in fact runs.
Can you tell me why?
BTW, your previous discussion and link has been very helpful to me in the matter of padding and alignment.
Thanks again,
Mark
Hi markallyn,
fputs takes only two arguments.
It fails when you add:
mov rdx, r12 ; set argument 2 to address of stdout "variable"
Like the third argument of fgets, the second argument of fputs should be set with:
mov rdx, qword ptr [r12] ; set argument 2 from stdout
Of course, that requires the correct value of sizeof(FILE).
You're just lucky that printf leaves rdx set to the stdout FILE pointer.
Hi Mark,
QuoteThat same program counts 80 bytes for my FILE struct as it is defined in my include file.
It looks like that there is something wrong with your FILE stucture. The size of 80 bytes is not correct. Kindly, could you post it here?
Hello tenkey:
Brilliant! In this case being "lucky" was actually a disservice I did to myself. Where I'm lucky is having you ferret out the "luck". I also note that I was dumb in not remembering that fputs and fgets are not symmetric.
Needless to say, I'll fix the code.
Mark
Good morning, Vortex,
Here is the FILE struct I have in myincludes64.inc:
Quote
FILE struct
mode DWORD ?
fh DWORD ?
buf DWORD ?
bufend DWORD ?
xptr DWORD ?
getend DWORD ?
putend DWORD ?
backptr DWORD ?
wbackptr DWORD ?
wbackbuf DWORD 2 dup (?)
getback DWORD ?
wgetend DWORD ?
wputend DWORD ?
wstate mbstate_t ?
tmpnam DWORD ?
backbuf BYTE 8 dup (?)
cbuf BYTE ?
locknum DWORD ?
FILE ends
LPFILE typedef ptr FILE
It is very different from _iobuf! The _iobuf resident in the same .inc file has the same members as yours, but whereas you use all QWORDs for your fields my _iobuf varies in byte-size in several instances.
Thanks again.
Mark
With msvcrt.dll FILE structure size is important.
I use sometimes this with msvcrt.dll
typedef struct FAKEFILE {
char *_ptrs[3];
int _ints[5];
} FILE;
so 3 pointers and 5 integers
for x64fakefile STRUCT 8
ptrs PTR 3 DUP(?)
ints DWORD 5 DUP(?)
fakefile ENDS
A test with ml64
option casemap:none
exit PROTO C :DWORD
printf PROTO C :PTR,:VARARG
INCLUDELIB msvcrt
fakefile STRUCT ;8
ptrs QWORD 3 DUP(?)
ints DWORD 5 DUP(?)
fill DWORD ?
fakefile ENDS
.data
fmt db "size: %d",10,0
.code
mainCRTStartup PROC C
; invoke printf, ADDR msg
; invoke printf, ADDR fmt, sizeof fakefile
; invoke exit,0
mov rdx, sizeof fakefile
mov rcx, offset fmt
call printf
call exit
mainCRTStartup ENDP
END ;mainCRTStartup
outputs size: 48
Hello TIMOVJL,
I sort of see what you did inasmuch as the number of each struct member type matches _iobuf. What I don't get is how the assembler knows to associate your two types (int, ptr) with the same type in the struct. The _iobuf members are not 3 ptrs in a row followed by 5 ints. So how does the assembler "know" that the ptr array in your struct needs to be distributed across the "real" _iobuf pointers. Likewise, the 5 integers.
But, I must say this is very elegant once you know what _iobuf struct looks like.
Thanks. I'll try it.
Mark
Quote from: markallyn on October 15, 2021, 02:40:19 AM
What I don't get is how the assembler knows to associate your two types (int, ptr) with the same type in the struct.
Hi markallyn,
The assembler doesn't know. You tell the assembler how the data is distributed. And you can lie about it. You could actually use:
fakefile STRUCT
filedata BYTE 48 dup(?)
fakefile ENDS
and it would work if the only info you need is the size of the struct.
as pointers are QWORD in x64, programmer sees, what makes difference to 32-bit version, but there are also those align things, you still can't use same struct for 32/64-bit.
Assembler don't know; so we need instruct assembler.
Generally types are listed in some header file, or include file.
If you like to enter in any project the first question is where are types? Whats their size? ... .
If you try to start translate C, Gtk, QT, windows, ..., you should start by finding types in header files or documentation.
In Linux they are described in ABI paper, I suppose in windows too. Both have "handles" that are "types to type" too. So the rest of documentation is in help files, other header files, ... .
A structure should be aligned to 8 in linux/windows x86-64. When I first meet that without understanding whats happening I adopt a technique to give enough space to not know strucures. So, when I see a structure but don't like to read manuals or translate that to asm I simply do like:
mystruct dq 8*N ;a 8 bytes multiple
An example of a structure being aligned to 8 and not aligned follow:
mystruct struct
one db ? ;1 byte 1
two dw ? ;2 bytes 3
three dd ? ;4 bytes 7
four dq ? ;8 bytes 15
mystruct ends
The sizeof mystruct without alignment is 1+2+4+8=15 bytes.
With 8 alignment that sizeof should be multiple of 8, so assembler create some "pad" elements inside that structure.
mystruct struct 8
one db ? ;1 byte 1
two dw ? ;2 bytes 3
three dd ? ;4 bytes 7
;next element is 8 bytes, the total sum for a while is 7 bytes.
;assembler inserts a "pad" element here to us to be 8 bytes aligned
pad db ? ;<----| 8
four dq ? ;8 bytes 16
mystruct ends
-----------------
Mytype equ db
.data
one Mytype ?
Good morning tenkey, TimoVJL, and Mineiro,
I see your point regarding structure member size. Also, Mineiro's discussion of padding was very useful. I think what was bothering me was partly related to padding. In TimoVJL's simplified _iobuf structure I could see how 3 pointers and 5 ints (3*8 bytes and 5 * 4 bytes) -- would work out to 48 bytes (divisible by 8) and so the assembler would just merrily populate the struct with bytes and the programmer -- as Mineiro says so well -- would know, because of documentation in headers, etc. -- how the bytes in the struct should be handled. But, what about the case where chars, short ints, were scattered in between 4 and 8 byte members? Now the programmer would have to insert padding to make the struct work out to something divisible by 8? Just as Mineiro's first example struct requires padding at certain points.
Aside: What makes this byte business about structs confusing to a beginner like me is that structs need 8-byte alignment and the stack needs to be 16 byte aligned. No doubt there is a very good reason why this is so (and it must be processor related I'll bet), but it is confusing.
Many thanks to all of you for a very informative discussion.
deleted
deleted
-> But, what about the case where chars, short ints, were scattered in between 4 and 8 byte members?
char equ db
mystruct struct
one char ? ;1
two ptr ? ;8
mystruct ends
mystruct struct
one char ? ;1 ;8-1=7, next member fits in 7 bytes (is dd,dw,db)?
;no, so pad that
pad db 7 dup (?) ;8
two ptr ? ;16
mystruct ends
-> Now the programmer would have to insert padding to make the struct work out to something divisible by 8?
If you do that by hands, answer is yes. And this gets more crazy when exist structures inside structures.
When our program starts, stack pointer ends with 8, or, rsp == ???????8h. To do any call to a function, rsp should be rsp=???????0h (16 bytes aligned means that a number ends with 0 in hexadecimal).
The only exception happens with leaf functions. Functions that do not call other functions internally is one example of leaf function. In this case we can call that function with rsp ending with 8 or 0 in hexadecimal. But, to be sure, it's better call that with rsp ending with 0h.
deleted
Thanks for talking about shadow space sir nidud. You're absolutelly right.
Good afternoon, Mineiro and Nidud,
First, when I said that the programmer would have to insert the padding in structs not divisible by 8, for clarity I should have said either the programmer OR the assembler would have to do this. Since asking this question, I subsequently came across an old post by NIDUD in which he showed how to insert padding "by hand" in a struct. What led me to this was a question I had about programs that could do this--of course, a silly question because that's exactly what assemblers like ml64 do.
NIDUD helpfully pointed out that SIMD on x64 processors (amd, intel) eats chunks of 16 bytes from memory. I suppose then, by extrapolation, 256 byte vector processors would want 32 byte alignment?