returning pointer to SSL structure from SSL function

[cyrus]

Well I am a bit stumped for a couple days trying to figure out why it fails when calling SSL_CTX_new() and I figure the forum is so dead that I decided to give you guys something to answer by creating this thread 


In C++, I am basically calling

Code Select Expand

SSL_library_init();
const SSL_METHOD *meth = TLS_client_method();
SSL_CTX *ctx = SSL_CTX_new (meth);
SSL *ssl = SSL_new (ctx);

Which works and I continue on with the rest of my code but in assembly, the tricky thing here is that the pointer returned isn't going to just be 8 bytes. It's an entire struct.

It is a somewhat complex struct because it has members that are not only int, but some of these are function pointers and some that even return function pointers and some of them even have OTHER structs in there. This is the actual struct itself.

Code Select Expand

struct ssl_method_st {
    int version;
    unsigned flags;
    unsigned long mask;
    SSL *(*ssl_new) (SSL_CTX *ctx);
    void (*ssl_free) (SSL *s);
    int (*ssl_reset) (SSL *s);
    int (*ssl_init) (SSL *s);
    int (*ssl_clear) (SSL *s);
    void (*ssl_deinit) (SSL *s);
    int (*ssl_accept) (SSL *s);
    int (*ssl_connect) (SSL *s);
    int (*ssl_read) (SSL *s, void *buf, size_t len, size_t *readbytes);
    int (*ssl_peek) (SSL *s, void *buf, size_t len, size_t *readbytes);
    int (*ssl_write) (SSL *s, const void *buf, size_t len, size_t *written);
    int (*ssl_shutdown) (SSL *s);
    int (*ssl_renegotiate) (SSL *s);
    int (*ssl_renegotiate_check) (SSL *s, int);
    int (*ssl_read_bytes) (SSL *s, uint8_t type, uint8_t *recvd_type,
                          unsigned char *buf, size_t len, int peek,
                          size_t *readbytes);
    int (*ssl_write_bytes) (SSL *s, uint8_t type, const void *buf_, size_t len,
                            size_t *written);
    int (*ssl_dispatch_alert) (SSL *s);
    long (*ssl_ctrl) (SSL *s, int cmd, long larg, void *parg);
    long (*ssl_ctx_ctrl) (SSL_CTX *ctx, int cmd, long larg, void *parg);
    const SSL_CIPHER *(*get_cipher_by_char) (const unsigned char *ptr);
    int (*put_cipher_by_char) (const SSL_CIPHER *cipher, WPACKET *pkt,
                              size_t *len);
    size_t (*ssl_pending) (const SSL *s);
    int (*num_ciphers) (void);
    const SSL_CIPHER *(*get_cipher) (unsigned ncipher);
    OSSL_TIME (*get_timeout) (void);
    const struct ssl3_enc_method *ssl3_enc; /* Extra SSLv3/TLS stuff */
    int (*ssl_version) (void);
    long (*ssl_callback_ctrl) (SSL *s, int cb_id, void (*fp) (void));
    long (*ssl_ctx_callback_ctrl) (SSL_CTX *s, int cb_id, void (*fp) (void));
};
So to get the size (because I don't know of any other way), I have to look it up in ssl_local.h, copy it but create my own version of this struct otherwise I get an error that the struct is already defined which is true. And then I do a sizeof(my_ssl_method_st). It has things like OSSL_TIME I have to figure out what is, assumming 4 bytes so in the end I get 248 bytes. So then I declare the SSL_METHOD pointer in masm as such:

SSL_METHOD  DQ 31 DUP (0)

which is 31 x 8 = 248. So now I can return a pointer to this struct that will get correctly populated. So I now feed that into the next call but it fails. Here is basically a summary of the above C code.

Code Select Expand

sub rsp, 20h
call OPENSSL_init_ssl

sub rsp, 20h
call TLS_client_method
mov SSL_METHOD, rax

mov rcx, SSL_METHOD
sub rsp, 20h
call SSL_CTX_new
mov SSL_CTX, rax
In C, these functions succeed but fails with masm. In the debugger, it fails stating STATUS ACCESS VIOLATION. Environment variable not found.

LastError: The system could not find the environment option that was entered.

[NoCforMe]

Hmm; take my comments with a rather large grain of salt, as I'm not a 64-bit guy. But you're both overthinking and wrong-thinking some aspects here, methinks.

No. 1 problem, seems to me, is defining that structure in your (assembly-language) code. So do it. I'll get you started:

C++:

Code Select Expand

struct ssl_method_st {
    int version;
    unsigned flags;
    unsigned long mask;

    ...

translates to

Code Select Expand

my_ssl_method_st  STRUC
  version    DD ?
  flags      DD ?
  mmask      DD ?

  ...

You can handle that, right? Good. (I'm only showing snippets of the whole thing.) I'll leave converting the pointer stuff to you, or someone else here. Also notice that I gave it a unique name in case there's already a structure defined in one of your include files.

Fine. Now that youve got that structure defined, you can immediately get the size of it by simply coding

Code Select Expand

SIZEOF my_ssl_method_st

and to declare an instance of this structure, instead of faking it with your SSL_METHOD  DQ 31 DUP (0), simply declare it by

Code Select Expand

SSL_METHOD    my_ssl_method_st<>
which will make addressing individual members super-easy without any futzing around:

Code Select Expand

    MOV    EAX, SSL_METHOD.version

Regarding the function returning this structure, I can't say for sure, but I'm willing to bet that it doesn't "return" the entire structure (i.e., copying it somewhere), but instead returns a pointer to a structure, which is the typical method for functions that return structures. In which case you already know how to deal with that, right?

Let me know if I'm wrong here ...

[cyrus]

Yes I understand of course it returns a pointer to a structure. I come from a C background. What's going on here is that in assembly, you can't just return that to some pointer declared as pointer DQ ? because that pointer needs to point to a struct and that struct gets populated when you call TLS_client_method.

I can't just use the included function because there are no header files, just the library which has the struct inside of it. The only way to get that info is looking for it in ssl_local.h. I counted up the bytes for that structure which is 248 bytes so I don't really need to write it out in asm, I just need to declare a pointer of the right size. I do this for the STARTUP INFORMATION struct and PROCESS INFORMATION struct as well when I use CreateProcessA. However, those are small structs. Even if I declare an enormous amount for this struct, it doesn't work. The key here is that rax returns a pointer to a struct but that pointer must have enough size to hold that struct and that's why I declared 248 bytes for it so it isn't that I am faking it, I just have to declare that size initially.

I wanted to mention that I dont need to use any of the data in that struct. In the STARTUP INFORMATION struct for CreateProcessA, I just load it into a register and I know at what position to put data in there. like if i load it in r12, i just use [r12 + 4] and add what data i need there but here i am passing off this entire struct as an argument to SSL_CTX_new. That structure is enormous, over 800 bytes actually. Over 300 lines of code.

[NoCforMe]

I can't just use the included function because there are no header files, just the library which has the struct inside of it.

That's why you need to create your own "header" (include) file with a function description (PROTO) and a structure definition.

The only way to get that info is looking for it in ssl_local.h.

Yes. So use that info to translate the structure definition to assembly language. It won't take long and will save you a lot of headaches.

I counted up the bytes for that structure which is 248 bytes so I don't really need to write it out in asm, I just need to declare a pointer of the right size. I do this for the STARTUP INFORMATION struct and PROCESS INFORMATION struct as well when I use CreateProcessA. However, those are small structs. Even if I declare an enormous amount for this struct, it doesn't work. The key here is that rax returns a pointer to a struct but that pointer must have enough size to hold that struct and that's why I declared 248 bytes for it so it isn't that I am faking it, I just have to declare that size initially.

Correction, and maybe clear up some confusion: the pointers don't have "different sizes". They're all the same size, the size of a pointer in 64-bit ASM (quadword). What's different is what they point to. Remember, a pointer is just a piece of data that holds the address of something. (You already know this, just trying to reinforce it.)

So again, if you define the structure, you can declare it and it will be exactly the right size, no need to worry about that. Either declare it on the stack as a LOCAL, or as a global ("static" in C parlance) in your .data or .data? section.

Does this help any?

[cyrus]

all pointers are 8 bytes yes. when i use the STARTUP INFORMATION struct, I declare it as such:

Code Select Expand

sinfo DQ 14 DUP(0)  ; used for sinfo struct which is 108 bytes but 108/8 = 13.5 so 14 is needed (112) ;
If I just did

Code Select Expand

sinfo DQ ?
that would never work because a pointer (8 bytes) isn't enough storage to use if i loaded it into a register. i can declare those both like that but it wouldn't make sense to use the pointer. i guess I could hold the address of that sinfo there but then I could just lea my storage anyway.

later down when i call CreateProcessA, i do

lea r12, sinfo

The other way is to push bytes onto the stack which you've seen me do before. I simply would push 112 bytes on the stack in 8 byte pushes 14 times, then just

Code Select Expand

mov r12, rsp
mov QWORD PTR [r12 + 4], rcx
whatever rcx data i put goes into the 2nd member of that struct

now maybe I'm wrong because my sinfo isn't exactly a pointer, its just storage. a pointer to that isn't necessary unless I just declare another one and point that to my array, but that doesn't seem to benefit me. its possible that maybe i am not 100% on pointers to structs as that seems to be my issue in asm because that is new to me.

[jj2007]

I've tried to translate it automagically to MASM syntax, but it's tougher than I thought: the algo that works for most C structures fails miserably for this one. One wonderful thing about C/C++ is that thousands of enthusiastic coders are actively seeking to create new, innovative syntaxes to confuse those freaks who believe in simple standards.

Anyway, it's clear that much more than 248 bytes are there. SIZE_P is 4 bytes in 32-bit and 8 bytes in 64-bit code.

Code Select Expand

ssl_method_st    STRUCT   
 version    SDWORD ?
 flags        flags; <>
 mask        DWORD ?
 ctx)        SIZE_P ?
 s)        SIZE_P ?
 s)        SIZE_P ?
 s)        SIZE_P ?
 s)        SIZE_P ?
 s)        SIZE_P ?
 s)        SIZE_P ?
 s)        SIZE_P ?
 readbytes)    SIZE_P ?
 readbytes)    SIZE_P ?
 written)    SIZE_P ?
 s)        SIZE_P ?
 s)        SIZE_P ?
 int)        SIZE_P ?
 ????        SIZE_P ?
 ????        SIZE_P ?
 readbytes)    SIZE_P ?
 ????        SIZE_P ?
 written)    SIZE_P ?
 s)        SIZE_P ?
 parg)        SIZE_P ?
 parg)        SIZE_P ?
 ptr)        SIZE_P ?
 ????        SIZE_P ?
 len)        SIZE_P ?
 s)        SIZE_P ?
 void)        SIZE_P ?
 ncipher)    SIZE_P ?
 void)        SIZE_P ?
 ssl3_enc    SIZE_P ?
 void)        SIZE_P ?
 )        SIZE_P ?
 )        SIZE_P ?
ssl_method_st ENDS       
Attention, it seems that this structure combines often several pointers in one line, e.g.
int (*ssl_read) (SSL *s, void *buf, size_t len, size_t *readbytes);

So the readbytes SIZE_P translation is most probably wrong.

If I just did

Code Select Expand

Code Select Expand

sinfo DQ ?
that would never work because 8 bytes isn't enough storage to use if i loaded it into a register

The registers in 64-bit code (rax, rcx, r12 etc) are 8 bytes wide, so I can't see the problem here. Unless you talk about SSE* registers like xmm0 or ymm0, of course.

[NoCforMe]

You're confusing pointers with data storage areas. Lessee, how can I illustrate this?

OK, your example is sinfo DQ 14 DUP(0), a 112-byte area of storage. Fine. But it's not a pointer. A pointer would be

Code Select Expand

sinfo DQ 14 DUP(0)
sinfo_ptr DQ sinfo
the 2nd item becomes a pointer which points to sinfo, and that pointer is always going to be the native size of a pointer, in this case a QWORD.

For the structure I wrote about up there, the code would be

Code Select Expand

SSL_METHOD    my_ssl_method_st<>    ;in .data, .data? or as a LOCAL
and then you could pass a pointer to this declared structure to your function, which would fill it in. The pointer will always be the same size--8-bytes--only what it points to will change in size. You could create the pointer (in a register) by

Code Select Expand

    LEA R12, SSL_METHOD
or pass it to a function by

Code Select Expand

    ADDR SSL_METHOD
(or OFFSET SSL_METHOD if it's declared in .data or .data?).

[cyrus]

Ok so I guess I would have to actually create a skeleton of these structs just to use them? See I thought I could just do what I do with the PROCESS INFORMATION struct in CreateProcessA because that struct actually gets populated, I don't add data. A call populates it, then it is passed off to CreateProcessA. With the SSL functions, a pointer to SSL_METHOD *meth is simply returned from calling TLS_client_method(). But what is it pointing to? That storage has to exist somewhere. That's what I thought declaring it in .data would actually do.

[HSE]

Obviously SSL is an Object, not just an Structure.

SSL_CTX also is an Object.

[cyrus]

Yes but isn't an object just a large structure? I've looked at the structs themselves. At the end of the day though, isn't just simply declaring enough storage than passing that off enough to do what I need? What is required for me to get this to work?

[HSE]

Yes but isn't an object just a large structure?

No.

What is required for me to get this to work?

You need the main part: all the methods.

[cyrus]

Can you be specific about the main part? Do I need to create the structs as mentioned by NoCforMe? How does one declare an object in assembly? I guess that is what I am looking for. I can't find much of that online. And to be quite frank, I don't see how an object is any different than just a structure. They seem to label them that way but it's just a struct to me.

Just like their other contexts are just structs. In C, there isn't another data type anyway so there isn't any secret there. I learned C 8 years ago but not saying I am a true expert in it, but I was able to break down the entire PEM_read_PrivateKey_ex() down to about 12 lines of actual code, meaning all the functions needed to decrypt an SSL private key to the DER format with BytesToKey() and assembly routines, etc. And all that code was thousands of lines of it but I followed it all the way through. I mention that because there were tons of contexts they use passing it off to other functions and creating other contexts, basically a bunch of error handling and different conditions that I didn't need. I was able to bypass all they did without any contexts.

[jj2007]

I thought I could just do what I do with the PROCESS INFORMATION struct in CreateProcessA because that struct actually gets populated

Code Select Expand

[out] lpProcessInformation

A pointer to a PROCESS_INFORMATION structure that receives identification
information about the new process.
Yes indeed, it gets populated, i.e. you receive a pointer ("out") to a (filled) structure that was created by Windows somewhere in memory.

Is that the case for you? You are rather vague about the API you want to use, and which arguments the API call takes.

[NoCforMe]

OK, here's (hopefully) a little more help. I've converted that structure for you. Here it is. It took me all of 10 minutes to do it.

Code Select Expand

SSL_METHOD    STRUCT
  version            DQ ?
  flags            DQ ?
  mmask            DQ ?
  Pssl_new        DQ ?
  Pssl_free        DQ ?
  Pssl_reset        DQ ?
  Pssl_init        DQ ?
  Pssl_clear        DQ ?
  Pssl_deinit        DQ ?
  Pssl_accept        DQ ?
  Pssl_connect        DQ ?
  Pssl_read        DQ ?
  Pssl_peek        DQ ?
  Pssl_write        DQ ?
  Pssl_shutdown        DQ ?
  Pssl_renegotiate        DQ ?
  Pssl_renegotiate_check    DQ ?
  Pssl_read_bytes        DQ ?
  Pssl_write_bytes        DQ ?
  Pssl_dispatch_alert    DQ ?
  Pssl_ctrl        DQ ?
  Pssl_ctx_ctrl        DQ ?
  PPget_cipher_by_char    DQ ?
  Pput_cipher_by_char    DQ ?
  Pssl_pending        DQ ?
  Pnum_ciphers        DQ ?
  PPget_cipher        DQ ?
  Pget_timeout        DQ ?
  Pssl3_enc        DQ ?
  Pssl_version        DQ ?
  Pssl_callback_ctrl    DQ ?
  Pssl_ctx_callback_ctr    DQ ?
SSL_METHOD    ENDS

Members starting with "P" are pointers; those starting with "PP" are pointers to pointers.

This should at least get you started; I believe if you define this structure (put this code at the top of your program) AND declare and instance of this structure

Code Select Expand

MySSLmethod      SSL_METHOD <>

then at least the function call should succeed.

What the hell to do with the contents of the structure once you get it, I have no idea. You apparently have to use all those pointers to functions (or "methods" to use the fancy-schmancy C++ lingo) to call them, and I have no idea how to do that. Probably something like this, to take one member as an example:

Code Select Expand

    int (*ssl_clear) (SSL *s);

    INVOKE Pssl_clear, s        ;where "s" is a pointer to an SSL, whatever that is

; or if you don't use INVOKE, use good old "push-push-call":

    PUSH    s
    CALL    <struct name>.Pssl_clear


[cyrus]

I thought I could just do what I do with the PROCESS INFORMATION struct in CreateProcessA because that struct actually gets populated

Code Select Expand

[out] lpProcessInformation

A pointer to a PROCESS_INFORMATION structure that receives identification
information about the new process.
Yes indeed, it gets populated, i.e. you receive a pointer ("out") to a (filled) structure that was created by Windows somewhere in memory.

Is that the case for you? You are rather vague about the API you want to use, and which arguments the API call takes.

I'm just using OpenSSL. It does the same thing for SSL in Windows as well. In fact I am programming for a windows machine using masm64. The same SSL functions that work in Linux are the same ones in Windows. Using the same source code, I configured it in linux and in windows. In fact I first wrote the program in c on linux first, then converted it to cpp on windows, and now converting that to asm on windows.