The MASM Forum

Miscellaneous => The Orphanage => Topic started by: markallyn on October 12, 2021, 03:07:59 AM

Title: Problem with subroutine to create stdin, stdout, and stderr
Post by: markallyn on October 12, 2021, 03:07:59 AM
Hello everyone,

Back on August 25, 2019 VORTEX submitted code that would create stdin, stdout, and stderr.  I had been trying for several weeks to figure out how to do this because these streams are very useful.  I finally saw V's post and recognized its brilliance as a solution to my failed efforts.  So, I set out to try to take his solution a bit further and make it a static library function.  In doing so I discovered that Pelles' POCRT does not contain __p_iob but msvcrt.lib has __iob_func which works similarly.

In the following submission I'm using FGETS and FPUTS simply as test functions.  FGETS fails in this code and for the life of me I can't understand why.  I have tried to annotate to help read the darned thing.

Quote
;trying to create subroutine to define stdin, stdout, and stderr.  Subroutine myio is based on
;a submission by VORTEX on August 25, 2019 to MASM Forum, Miscellaneous, The Orphanage, as response to JC FULLER post
;August 24.

include   myincludes64.inc    ;Homemade .inc file with typedefs, structs, et. I havve found useful.
includelib   c:\masm32\lib64\msvcrt.lib  ;This MSVCRT contains __iob_func to access stdin, stdout, stderr.

__iob_func PROTO :QWORD
puts       PROTO :QWORD
printf      PROTO :QWORD
fgets      PROTO :QWORD, :DWORD, :QWORD
fputs      PROTO :QWORD, :DWORD, :QWORD
myio      PROTO
.data
msg1   db   "Enter a string:",13,10,0
msg2   db   "The string you entered was:",13,10,0

.data?
stdin   QWORD    ?
stdout   QWORD   ?
stderr   QWORD   ?
buffer   BYTE   256 dup   (?)

.const

.code
main   PROC
enter    0,0               ;this section creates a stack, aligns the stack, and
and   rsp, -10h            ;makes room for spill space
sub   rsp,  20h

lea   rcx,   stdin
lea   rdx,   stdout
lea   r8,      stderr
call   myio            ;this call sets up stdin, stdout, stderr
                     ;Myio successfully returns to here.
mov   r11,   rcx            ;stdin should be in r11
mov   r12,   rdx            ;stdout should be in r12
mov   r13,   r8            ;stderr should be in r13
lea   rcx,   msg1         
call   printf            ;printf successfully prints message1
lea   rcx,    buffer
mov rdx,   80
mov   r8,      r11
call   fgets            ;FGETS fails!  WHY??      
lea rcx,   msg2
call    printf
lea   rcx,   buffer
mov   r8,     r12
call   fputs

add   rsp, 20h            ;this section removes spill space and tears down stack
leave
ret
main   ENDP

myio      PROC           ;This little PROC is based on VORTEX, August 25, 2019.  Thanks.
enter   0,0
and      rsp,   -10h
sub      rsp,   20h

push rcx
mov   rcx,   0            ;this section sets up stdin and stdout
call    __iob_func         ;__iob_func does what VORTEX's crt___p__iob function did in V's version
pop   rcx
mov   qword ptr [rcx],   rax
mov   r9,   SIZEOF (FILE)
add   rax,   r9
mov   qword ptr [rdx],   rax
add   rax,   r9
mov   qword ptr [r8],   rax

add   rsp,   20h
leave
ret                     ;Myio successfully returns.
myio   ENDP
END

Many thanks again to VORTEX for his original contribution.

Mark Allyn
Title: Re: Problem with subroutine to create stdin, stdout, and stderr
Post by: Vortex on October 12, 2021, 04:43:49 AM
Hi markallyn,

Did you take care of the structure alignment?

https://www.geeksforgeeks.org/structure-member-alignment-padding-and-data-packing/
Title: Re: Problem with subroutine to create stdin, stdout, and stderr
Post by: tenkey on October 12, 2021, 05:48:26 AM
My guess is that r11 gets clobbered by printf. R10 and r11 are not guaranteed to be call-preserved.
https://docs.microsoft.com/en-us/cpp/build/x64-calling-convention?view=msvc-160
Title: Re: Problem with subroutine to create stdin, stdout, and stderr
Post by: nidud on October 12, 2021, 08:08:33 AM
deleted
Title: Re: Problem with subroutine to create stdin, stdout, and stderr
Post by: markallyn on October 12, 2021, 09:05:02 AM
Hello VORTEX, TENKEY, and NIDUD,

Owing to family responsibilities (my daughter is in deep pregnancy) I only had a chance to look over Vortex's reply.  I read his attachment re structure alignment and found it very informative on a subject I didn't understand well.  However, I need a bit more detail on why Vortex thought there might be a misalignment in my FILE Struct (as this is the only struct I know about in the program).

Tenkey: I will test your idea later this evening and report back.

NIDUD:  You have spent a lot of time on this response and I really appreciate what you have done.  I'm going to learn a lot.  As I think you can tell from what I did I don't fully understand parameter passing using x64 ABI.  That was part of my motivation for trying to do this little project.  I notice that you used __acrt_iob_func!

Regards to each of you,

Mark
Title: Re: Problem with subroutine to create stdin, stdout, and stderr
Post by: markallyn on October 12, 2021, 09:42:12 AM
Tenkey,

Of course you're right about r11 not being preserved!  I forgot.  I tried changing r11 to r12 and r12 to r13.  No luck, I'm sorry to report.  Same problem with FGETS.

Thanks for the suggestion!

Mark
Title: Re: Problem with subroutine to create stdin, stdout, and stderr
Post by: markallyn on October 12, 2021, 11:25:16 PM
Good Morning, NIDUD,

I have been trying to replicate your code on my machine using visual studio 2017 x64 native tools command.  I cannot link to libcmt.lib and __acrt_iob_func.  I have found this latter function in windows\system32, however.  __acrt_iob_func on my machine is located in libucrt.dll.  But, there is no corresponding .lib or .inc in Windows \system32 or \Windows from which I can get the prototype.

Would you kindly tell me what is going on?

Regards,
Mark
Title: Re: Problem with subroutine to create stdin, stdout, and stderr
Post by: nidud on October 13, 2021, 12:15:57 AM
deleted
Title: Re: Problem with subroutine to create stdin, stdout, and stderr
Post by: markallyn on October 13, 2021, 04:59:06 AM
Good afternoon, NIDUD,

I looked in ALMOST the same VS 2019 directory that you mention, but there is a change of version in my file.  Instead of 14.28.29333\lib\x64 directory mine is 14.25.28610\lib\x64.  In my version there is no __acrt_iob_func as far as I can tell. 

You answered one question I had as a result of this.  Namely, I can use the __iob_func as a substitute.  But I gotta tell you that I would really like to use the __acrt_iob_func.  Not finding it (so far) is highly unsatisfying.

I agree it could be useful to figure this out--couldn't agree more.  I thought I might be able to use windows\system32 but no .lib to include.  Would have to "roll my own" and I'm lazy. 

Thanks again,
Mark

Title: Re: Problem with subroutine to create stdin, stdout, and stderr
Post by: Vortex on October 13, 2021, 05:39:48 AM
Hi markallyn,

You don't need any other specific library to assemble the code below, the Masm64 additions developed by Hutch is doing all the job. The largest element in the _iobuf structure, QWORD has a size of 8 bytes so they need to be aligned to 8 byte boundary. The DWORD members in the same structure will require attention, so we instruct the assembler to do it for us with _iobuf STRUCT 8. You can see that the first DWORD member will "break" the alignment :

Code: [Select]
    _cnt        DWORD   ?
Here is a quick example for the Masm64 SDK :

Code: [Select]
include \masm32\include64\masm64rt.inc

_iobuf STRUCT 8 ; align to 8 byte boundary

    _ptr        QWORD   ?
    _cnt        DWORD   ?
    _base       QWORD   ?
    _flag       DWORD   ?
    _file       DWORD   ?
    _charbuf    DWORD   ?
    _bufsiz     DWORD   ?
    _tmpfname   QWORD   ?
   
_iobuf ENDS

_FILE TYPEDEF _iobuf

EXTERN __iob_func:PROC

.data?

_stdout     dq ?
_stdin      dq ?
_stderr     dq ?

.data

msg db "Please type your name :",13,10,0

.data?

szName db 16 dup(?)

.code

start PROC

    invoke  __iob_func

    mov    _stdin,rax           ; #define stdin  (&__iob_func()[0])
    add    rax,SIZEOF(_FILE)
   
    mov    _stdout,rax          ; #define stdout (&__iob_func()[1])
   
    add    rax,SIZEOF(_FILE)
    mov    _stderr,rax          ; #define stderr (&__iob_func()[2])

    invoke  vc_fputs,ADDR msg,_stdout

    invoke  vc_fgets,ADDR szName,16,_stdin

    invoke  vc_printf,"Nice to meet you, %s",ADDR szName

    invoke  ExitProcess,0

start ENDP

END
Title: Re: Problem with subroutine to create stdin, stdout, and stderr
Post by: tenkey on October 13, 2021, 05:42:09 AM
markallyn,
You stored a file pointer with:

Code: [Select]
mov   qword ptr [rcx],   rax

So you need to retrieve the file pointer with:

Code: [Select]
mov   r8,      qword ptr [r11]   ; you will need to change r11

or

Code: [Select]
mov   r8,      stdin   ; not lea !!!

I got it to work by replacing both instances of r11 with r14 and adding qword ptr.
Title: Re: Problem with subroutine to create stdin, stdout, and stderr
Post by: nidud on October 13, 2021, 06:03:32 AM
deleted
Title: Re: Problem with subroutine to create stdin, stdout, and stderr
Post by: markallyn on October 13, 2021, 11:28:35 AM
Good evening, tenkey, vortex, and nidud,

Tenkey:  I made the change you suggested and the code worked perfectly.  Thanks much for your continued interest.

Nidud and Vortex:  You have both provided code that requires much closer study.  I promise to reply to what you have given me. 

All of you:  I can't tell you how grateful I am for your continued interest in my rather trivial problem.  I have learned a great deal and I am delighted by your generosity.

I wish there was a way to pay you back.

Mark
Title: Re: Problem with subroutine to create stdin, stdout, and stderr
Post by: markallyn on October 14, 2021, 02:58:31 AM
Good morning, Vortex,

In your _iobuf struct I make out the size to be 48 bytes including 4 bytes of padding in the first DWORD breakage.  Confirmed with a little program I just wrote to count struct bytes.

That same program counts 80 bytes for my FILE struct as it is defined in my include file.  Leads to a question about my original code.  As follows:

Within my myio proc I set r9 to be sizeof(FILE).  Obviously not the same as your _FILE typedef.  One (ME!) would naively think that this would lead to a failure of my code NO MATTER WHAT ELSE might have been wrong.  I would have expected that adding the incorrect size to rax would have incorrectly computed stdout.

But, after making the change that tenkey suggested, my orginal code in fact runs.

Can you tell me why?

BTW, your previous discussion and link has been very helpful to me in the matter of padding and alignment. 

Thanks again,

Mark
Title: Re: Problem with subroutine to create stdin, stdout, and stderr
Post by: tenkey on October 14, 2021, 04:25:48 AM
Hi markallyn,

fputs takes only two arguments.
It fails when you add:

Code: [Select]
mov rdx, r12    ; set argument 2 to address of stdout "variable"

Like the third argument of fgets, the second argument of fputs should be set with:

Code: [Select]
mov rdx, qword ptr [r12]    ; set argument 2 from stdout

Of course, that requires the correct value of sizeof(FILE).
You're just lucky that printf leaves rdx set to the stdout FILE pointer.
Title: Re: Problem with subroutine to create stdin, stdout, and stderr
Post by: Vortex on October 14, 2021, 04:38:08 AM
Hi Mark,

Quote
That same program counts 80 bytes for my FILE struct as it is defined in my include file.

It looks like that there is something wrong with your FILE stucture. The size of 80 bytes is not correct. Kindly, could you post it here?
Title: Re: Problem with subroutine to create stdin, stdout, and stderr
Post by: markallyn on October 14, 2021, 07:45:11 AM
Hello tenkey:

Brilliant!  In this case being "lucky" was actually a disservice I did to myself.  Where I'm lucky is having you ferret out the "luck".  I also note that I was dumb in not remembering that fputs and fgets are not symmetric.

Needless to say, I'll fix the code.

Mark
Title: Re: Problem with subroutine to create stdin, stdout, and stderr
Post by: markallyn on October 14, 2021, 10:02:40 PM
Good morning, Vortex,

Here is the FILE struct I have in myincludes64.inc:

Quote
FILE   struct
mode   DWORD   ?
fh   DWORD   ?
buf   DWORD   ?
bufend   DWORD   ?
xptr   DWORD   ?
getend   DWORD   ?
putend   DWORD   ?
backptr   DWORD   ?
wbackptr   DWORD   ?
wbackbuf   DWORD 2 dup (?)
getback   DWORD   ?
wgetend   DWORD   ?
wputend   DWORD   ?
wstate   mbstate_t   ?
tmpnam   DWORD   ?
backbuf   BYTE 8 dup (?)
cbuf   BYTE   ?
locknum   DWORD   ?
FILE   ends

LPFILE typedef ptr FILE

It is very different from _iobuf!  The _iobuf resident in the same .inc file has the same members as yours, but whereas you use all QWORDs for your fields my _iobuf varies in byte-size in several instances. 

Thanks again.

Mark
Title: Re: Problem with subroutine to create stdin, stdout, and stderr
Post by: TimoVJL on October 14, 2021, 11:22:33 PM
With msvcrt.dll FILE structure size is important.

I use sometimes this with msvcrt.dll
Code: [Select]
typedef struct FAKEFILE {
char *_ptrs[3];
int _ints[5];
} FILE;
so 3 pointers and 5 integers

for x64
Code: [Select]
fakefile STRUCT 8
ptrs PTR 3 DUP(?)
ints DWORD 5 DUP(?)
fakefile ENDS
A test with ml64
Code: [Select]
option casemap:none
exit PROTO C :DWORD
printf PROTO C :PTR,:VARARG
INCLUDELIB msvcrt

fakefile STRUCT ;8
ptrs QWORD 3 DUP(?)
ints DWORD 5 DUP(?)
fill DWORD ?
fakefile ENDS

.data
fmt db "size: %d",10,0
.code
mainCRTStartup PROC C
; invoke printf, ADDR msg
; invoke printf, ADDR fmt, sizeof fakefile
; invoke exit,0
mov rdx, sizeof fakefile
mov rcx, offset fmt
call printf
call exit
mainCRTStartup ENDP
END ;mainCRTStartup
outputs size: 48
Title: Re: Problem with subroutine to create stdin, stdout, and stderr
Post by: markallyn on October 15, 2021, 02:40:19 AM
Hello TIMOVJL,

I sort of see what you did inasmuch as the number of each struct member type matches _iobuf.  What I don't get is how the assembler knows to associate your two types (int, ptr) with the same type in the struct.  The _iobuf members are not 3 ptrs in a row followed by 5 ints.  So how does the assembler "know" that the ptr array in your struct needs to be distributed across the "real" _iobuf pointers.  Likewise, the 5 integers.

But, I must say this is very elegant once you know what _iobuf struct looks like.

Thanks.  I'll try it.

Mark
Title: Re: Problem with subroutine to create stdin, stdout, and stderr
Post by: tenkey on October 15, 2021, 03:04:49 PM
What I don't get is how the assembler knows to associate your two types (int, ptr) with the same type in the struct.

Hi markallyn,

The assembler doesn't know. You tell the assembler how the data is distributed. And you can lie about it. You could actually use:

Code: [Select]
fakefile STRUCT
filedata BYTE 48 dup(?)
fakefile ENDS

and it would work if the only info you need is the size of the struct.
Title: Re: Problem with subroutine to create stdin, stdout, and stderr
Post by: TimoVJL on October 15, 2021, 04:25:10 PM
as pointers are QWORD in x64, programmer sees, what makes difference to 32-bit version, but there are also those align things, you still can't use same struct for 32/64-bit.
Title: Re: Problem with subroutine to create stdin, stdout, and stderr
Post by: mineiro on October 15, 2021, 09:58:16 PM
Assembler don't know; so we need instruct assembler.
Generally types are listed in some header file, or include file.
If you like to enter in any project the first question is where are types? Whats their size? ... .
If you try to start translate C, Gtk, QT, windows, ..., you should start by finding types in header files or documentation.
In Linux they are described in ABI paper, I suppose in windows too. Both have "handles" that are "types to type" too. So the rest of documentation is in help files, other header files, ... .

A structure should be aligned to 8 in linux/windows x86-64. When I first meet that without understanding whats happening I adopt a technique to give enough space to not know strucures. So, when I see a structure but don't like to read manuals or translate that to asm I simply do like:

mystruct dq 8*N           ;a 8 bytes multiple

An example of a structure being aligned to 8 and not aligned follow:

mystruct struct
one db ?        ;1 byte     1
two dw ?        ;2 bytes    3
three dd ?      ;4 bytes    7
four dq ?       ;8 bytes    15
mystruct ends

The sizeof mystruct without alignment is 1+2+4+8=15 bytes.
With 8 alignment that sizeof should be multiple of 8, so assembler create some "pad" elements inside that structure.

mystruct struct 8
one db ?        ;1 byte     1
two dw ?        ;2 bytes    3
three dd ?      ;4 bytes    7
                ;next element is 8 bytes, the total sum for a while is 7 bytes.
                ;assembler inserts a "pad" element here to us to be 8 bytes aligned
pad db ?        ;<----|     8
four dq ?       ;8 bytes    16
mystruct ends

-----------------
Mytype equ db
.data
one Mytype ?
Title: Re: Problem with subroutine to create stdin, stdout, and stderr
Post by: markallyn on October 15, 2021, 10:33:39 PM
Good morning tenkey, TimoVJL, and Mineiro,

I see your point regarding structure member size.  Also, Mineiro's discussion of padding was very useful.  I think what was bothering me was partly related to padding.  In TimoVJL's simplified _iobuf structure I could see how 3 pointers and 5 ints (3*8 bytes and 5 * 4 bytes) -- would work out to 48 bytes (divisible by 8) and so the assembler would just merrily populate the struct with bytes and the programmer -- as Mineiro says so well -- would know, because of documentation in headers, etc. -- how the bytes in the struct should be handled.  But, what about the case where chars, short ints, were scattered in between 4 and 8 byte members?  Now the programmer would have to insert padding to make the struct work out to something divisible by 8?   Just as Mineiro's first example struct requires padding at certain points.

Aside:  What makes this byte business about structs confusing to a beginner like me is that structs need 8-byte alignment and the stack needs to be 16 byte aligned.  No doubt there is a very good reason why this is so (and it must be processor related I'll bet), but it is confusing.

Many thanks to all of you for a very informative discussion. 
Title: Re: Problem with subroutine to create stdin, stdout, and stderr
Post by: nidud on October 15, 2021, 11:00:46 PM
deleted
Title: Re: Problem with subroutine to create stdin, stdout, and stderr
Post by: nidud on October 15, 2021, 11:20:06 PM
deleted
Title: Re: Problem with subroutine to create stdin, stdout, and stderr
Post by: mineiro on October 16, 2021, 12:03:03 AM
-> But, what about the case where chars, short ints, were scattered in between 4 and 8 byte members?

char equ db
mystruct struct
one char ?      ;1
two ptr ?       ;8
mystruct ends

mystruct struct
one char ?          ;1     ;8-1=7, next member fits in 7 bytes (is dd,dw,db)?
                    ;no, so pad that
pad db 7 dup (?)    ;8
two ptr ?           ;16
mystruct ends

-> Now the programmer would have to insert padding to make the struct work out to something divisible by 8?
If you do that by hands, answer is yes. And this gets more crazy when exist structures inside structures.

When our program starts, stack pointer ends with 8, or, rsp == ???????8h. To do any call to a function, rsp should be rsp=???????0h (16 bytes aligned means that a number ends with 0 in hexadecimal).
The only exception happens with leaf functions. Functions that do not call other functions internally is one example of leaf function. In this case we can call that function with rsp ending with 8 or 0 in hexadecimal. But, to be sure, it's better call that with rsp ending with 0h.
Title: Re: Problem with subroutine to create stdin, stdout, and stderr
Post by: nidud on October 16, 2021, 12:56:13 AM
deleted
Title: Re: Problem with subroutine to create stdin, stdout, and stderr
Post by: mineiro on October 16, 2021, 03:40:08 AM
Thanks for talking about shadow space sir nidud. You're absolutelly right.
Title: Re: Problem with subroutine to create stdin, stdout, and stderr
Post by: markallyn on October 16, 2021, 06:38:01 AM
Good afternoon, Mineiro and Nidud,

First, when I said that the programmer would have to insert the padding in structs not divisible by 8, for clarity I should have said either the programmer OR the assembler would have to do this.  Since asking this question, I subsequently came across an old post by NIDUD in which he showed how to insert padding "by hand" in a struct.  What led me to this was a question I had about programs that could do this--of course, a silly question because that's exactly what assemblers like ml64 do.

NIDUD helpfully pointed out that SIMD on x64 processors (amd, intel) eats chunks of 16 bytes from memory.  I suppose then, by extrapolation, 256 byte vector processors would want 32 byte alignment?