EasyCode with NASM/YASM

[rsala]

Hi David,

I have this DLL 64 test code (attached) but I have never compiled it. However, you will need the MASM64 package.

Regards,

Ramon

[David BS]

Thanks Ramon, I have ML64.

I'll try something with him. I'm looking now the OBJ file created by Gunther from the original INTEL's YASM code.
I'm studing it and analyzing if I will convert it to MASM64 syntax or not.

Thank you.
Kindest regards.

[Gunther]

Hi David,

after reading the entire thread again, I came up with that ideas. Your system hangs, if you're calling the DLL from VB. The reason must not be the DLL code.

The Intel source is designed for a LIB. C and BASIC have very different calling conventions. BASIC passes parameters to procedures by reference (factory default) while C passes by value. BASIC passes from right to left, C passes from left to right. Furthermore, in C cleans the caller the stack, in BASIC has the callee to clean the stack.

That's all a bit tricky and you have to know the details. I'm not very familiar with VB, but I know PowerBASIC very well. You can declare in PB a procedure or function as CDECL and BASIC handles all things like a C compiler. I hope that VB has a similar feature.

Gunther

[qWord]

I'm trying now call ASM code from VB utilizing MARSHAL functions (it pass addresses instead values), but I'm getting error all the time.
I believe the main problem is the fact of the ASM source receive parameters in YASM syntax (directly from stack??) and all of them are VOID type, which is a little bit confuse to set in VB.

The problem for the DLL source from post #21 is that the DllMain follows Stdcall, which is invalid for x64 (fastcall is required). It should be something like:

Code: [Select]

global DllMain:function
DllMain:
mov eax,1
ret
Remakrs that the code only implement the core loop of SHA-512. With the above modification and the following amateurish VB code I was at least able to call sha512_sse4 (VS 2012, Win7 x64). I guess the result is not correct due to wrong preprocessing of the message...

Code: [Select]

Imports System.Runtime.InteropServices.Marshal

Module Module1

    Declare Sub sha512_sse4 Lib "\yasm\hash.dll" (ByVal msg As IntPtr, ByVal sha512 As IntPtr, ByVal N As IntPtr)

    Sub Main()

        Dim str As String = "Hello World"

        Dim zero(127) As Byte

        Dim start_hash As Long() = {&H6A09E667F3BCC908, &HBB67AE8584CAA73B, &H3C6EF372FE94F82B, &HA54FF53A5F1D36F1, &H510E527FADE682D1, &H9B05688C2B3E6C1F, &H1F83D9ABFB41BD6B, &H5BE0CD19137E2179}

        Dim cb_str_len As Integer = str.Length * 1                                  ' or str.Length * SystemDefaultCharSize

        Dim msg_length As Integer = cb_str_len + 1 + 16                             ' 1 + 128 bit
        Dim block_size As Integer = 128                                             ' SHA-512 block size
        Dim hash_size As Integer = 64
        Dim buffer_len As Integer = (msg_length + block_size - 1) And -block_size
        Dim msg_n_blocks As IntPtr = buffer_len / block_size
        Dim pMsg As IntPtr = StringToHGlobalAnsi(str)                               ' or StringToHGlobalAuto
        pMsg = ReAllocHGlobal(pMsg, buffer_len)                                     ' change buffer size

        ' extend message by one bit
        WriteByte(pMsg + cb_str_len, 1)

        ' zero further bytes
        If buffer_len - cb_str_len - 1 Then
            Copy(zero, 0, pMsg + cb_str_len + 1, buffer_len - cb_str_len - 1)
        End If

        ' append message size
        WriteInt32(pMsg + buffer_len - 4, cb_str_len * 8)

        ' create inital hash value
        Dim pSHA512 As IntPtr = AllocHGlobal(hash_size)
        Copy(start_hash, 0, pSHA512, start_hash.Length)

        ' ASM
        sha512_sse4(pMsg, pSHA512, msg_n_blocks)

        '' print message block(s)
        'System.Console.WriteLine("msg bytes: ")
        'Dim b(buffer_len - 1) As Byte
        'Copy(pMsg, b, 0, buffer_len)
        'For i = 0 To b.GetUpperBound(0)
        '    System.Console.Write("{0:X2}", b(i))
        'Next
        'System.Console.WriteLine("\\ end msg bytes \\")

        ' get hash value
        Dim sha512(63) As Byte
        Copy(pSHA512, sha512, 0, 64)

        System.Console.WriteLine("msg = " & str)
        System.Console.Write("hash = ")

        For i = 0 To sha512.GetUpperBound(0)
            System.Console.Write("{0:X2}", sha512(i))
        Next

        System.Console.WriteLine("")

        FreeHGlobal(pMsg)
        FreeHGlobal(pSHA512)

        System.Console.ReadKey()

    End Sub

End Module

[David BS]

The problem for the DLL source from post #21 is that the DllMain follows Stdcall, which is invalid for x64 (fastcall is required). It should be something like:

Hey qWord!! I had read it in some place but I really hadn't paid attention to that! Hmmm.... For sure, it may impact in the wrong results.... :icon_rolleyes:

The Intel source is designed for a LIB. C and BASIC have very different calling conventions. BASIC passes parameters to procedures by reference (factory default) while C passes by value. BASIC passes from right to left, C passes from left to right. Furthermore, in C cleans the caller the stack, in BASIC has the callee to clean the stack.
Gunther

Hmmm.... I didn't know about THESE languages particularities, mainly with the stack usage. I'm really not familiarized with C/C++ since I had never had interest on them. I had always choosen to know a high-level language AND assembly... I made that decision since Apple ][, joining BASIC (and its Peek/Pode instruction and others I don't remember anymore) with 6502 assembly to make my CP/M card to emulate a decent screen size (wow, 128 text columns on screen!!).  It remind me the 6 lines of 8088 asm code to break the 640kb limit of XT to get more memory to Clipper indexes... 

Thank YOU very much guys. I'll try the code you sent, and the tip you're referencing Gunther.
I really had appreciated the whole efforts of all of you. 

[Gunther]

Hi David,

you should have now enough to note. Let us know the results and if you need more help.

Gunther

[David BS]

 

Thanks to ALL of you for the precious tips and advices, BUT I conclude that the INTEL's code is really wrong OR it's made to inform wrong result in proposital way.

Let me explain:

1- The code is running CLEAR. I made a lot of modifications in ASM and VB2010 - all of them crashes the execution. Just ONE way can provide SOME right results (no crash at all).

2- Gunther, I can inform to VB2010 if I want to pass parameters as POINTER or VALUE. The INTEL code gets both: pointer to VOID* and value for INT. I tried to change the order from left to right in the parameters command-line and ALL tries made the program crash. The sent order is the SAME the source is planned to receive.

3- I have here 2 machines: one Windows 7 x64 (not SSE4x compatible) but another one Windows 8.1 x64 100% SSE4.x compatible. Both gave me the SAME results in the SHA call, so, I conclude that something is wrong. It couldn't happened...

4- I had tried also to pad the original message to SHA as required by FIPS140 (in BITS!! :icon_eek:). None effect.

5- In despite of to make changes in code (eg. pad or not the message), the most interesting thing is that the result are THE SAME, in VALID SIZE, and ramdomly acceptable (any change in the original message caused the SHA change its bytes entirely). But the results are NOT the expected ones if compared to valid SHA512 results (I had utilized two different sources: .NET library and an online SHA service).

6- The tips that all of you gave me were fundamental to compile the ASM and call from VB. Thank YOU all. I couldn't make it alone... It's the main success to me:  the fact of get a original LIB YASM source code and change it to a valid DLL file and call it from Visual Basic 2010/2012. Congratulations to all of you and your efforts were really, really appreciated by me.

Just to help SOMEONE with similar problem, basically the correct procedures to have INTEL code running here as DLL file were (considering it were compiled by YASM/NASM):

a) make both DLL entry and function PUBLIC in the source code:

    Global DllMain
    Global sha512_sse4


b) create the DLL entry in MS WINDOWS format within the source code (__FastCall syntax since it's a x64 code):

    ..start:
    global DllMain:function
    DllMain:
   mov eax,1
   ret


c)  create the .DEF file (Export)
   
     LIBRARY HASH
     EXPORTS sha512_sse4


d) compile it and link utilizing the following commands:

    vsyasm -f x64 -D WINABI sha512_sse4.asm
    link /entry:DllMain /MACHINE:X64 /NODEFAULTLIB /def:hash.def /DLL /export:sha512_sse4 /export:DllMain /subsystem:windows sha512_sse4.obj /out:"hash.dll"


These procedures CAN create a valid x64 DLL using YASM or a compatible compiler.

In the Visual Basic 2010 code I made the following steps:

a) I had declared a SUB call to the ASM function (it's NOT a function call since the ASM code does NOT return a value but changes a pointed entried variable):

    Declare Sub sha512_sse4 Lib "hash.dll" (ByVal sMensagem As IntPtr, ByVal ssha As IntPtr, ByVal stamanhomensagem As UInteger)


    Interesting to note that all parameters are sent BY VALUE since the next commands can get the Memory Pointers of each of them (see IntPtr above).

b) I build all parameters and after, transformed them to POINTERS to be viewed by an original C++ code - a way to make unmanaged code (C++/C) interface with a managed one (C# or VB):

        Dim VarSized64bits As UInteger = (novamensagem.Trim.Length Mod 64) + 1
        Dim Mess2Sha As IntPtr = Marshal.StringToHGlobalAuto(Original_String_Message)
        Dim ReturnSHA(64) As Byte      << an array of 64 bytes = 512 bits
       
        Dim MessFromSha As IntPtr = Marshal.AllocHGlobal(64)     ' pointer to ReturnSHA
        Marshal.Copy(ReturnSHA, 0, MessFromSha, 64)                 ' filled by ...

        sha512_sse4(Mess2Sha, MessFromSha, VarSized64bits)     ' call to the %@&^#% INTEL's code   

        Marshal.Copy(MessFromSha, ReturnSHA, 0, 64)                 ' Get the result and move it to array of bytes.

        ' Free allocated memory.

        Marshal.FreeHGlobal(Mess2Sha)                                       
        Marshal.FreeHGlobal(MessFromSha)



One day I will return to that code... I'm really tired since I'm facing on it around 2 or 3 weeks.
I understand the INTEL code HAS some problem. I can't believe the problem is about parameters order or so forth, since ANY change on the above crashes the IDE.

Again, all of you have been TOO MUCH kind with me. Thank YOU very much.

[qWord]

Some problems I see:
 - The block size for SHA-512 is 1024 bit => 128 byte
 - I guess the default text encoding in VB is Unicode thus the actual memory size in bytes is (at least) doubled
 - I can't see where you copy the initial hash value

You might break down your code and post a short, runnable example...

[Gunther]

David,

qWords points are important. He's a smart guy and you can learn a lot by attentive reading his posts. Moreover, I would suggest that: Does Intel provide a C code example with appropriate test values? If so, I think it's written for gcc, because that compiler is available in both worlds. I've a native Windows 7-64 and a native 64-bit Linux running here (both with the appropriate gcc). I could try to re-compile the entire code to keep the yasm code running. That would be the first step. If it works, we can see how to call it from a DLL.

Gunther

[David BS]

Dear guys,

INTEL does NOT provide ANY C sample to utilize with that ASM code. It recommends ONLY to utilize it as a LIB.

In my VB code I had tried ALSO to PAD the original message (any lenght) as required to SHA-512 (a lot of zero bits to make a 1024-mod size). The results didn't changed. I believe the ASM code does not PAD the original message as required, and because of it I perform it at my own.

I don't believe VB translate the string to UNICODE unless we require it explicitly. When I debug the string it appears with the ASCII length. I can check it since even appending a LOT of NULL, the string does NOT show it but the its size confirms it 100%.  So, if UNICODE, that lenght should be the double (I agree with qWord).

Yes, I appreciate your efforts in debug the code. I'll send it in attachment as Visual Studio project (ready to load and compile). Please, if you need another format (like pure ASCII for instance), please, let me know.

And once again, THANK YOU very much!

PS: if you have ANY problem to understand the code or the Portuguese variables names, please contact me and I try to change it to English and in a readable way. The main reason to do not make this now is because I'm involved with a hurry support to a client - I can't stop NOW.  But if you have any difficulty, please, let  me know.

[David BS]

Hi,

Follows in attachment the Visual Basic code IN ENGLISH format and few comments.

Thanks in advance....

[qWord]

After some time I finally got it run correct. The key points are:
 - the words^FIPS140 (actual QWORDs) of the returned hash are in Big-endian byte order.
 - the append bit must be add as a byte of the value 0x80 = 128
 - the 128 bit integer that holds the message size in bits, must also be stored in Big-endian byte order
 - the hash must be initialized with H⁽⁰⁾_0...7 (see FIPS140) before calling sha512_sse4

In the attachment you can find an example application. Those who want to build it need jWasm, Yasm, polink and polib - the last two are also included in the MASM32 SDK (otherwise see PellesC).

BTW: Currently I'm a bit confused about the name "sha512_sse4", because (AFAICS) the maximal feature level is SSSE3 due to PSHUFB instruction...


EDIT:
I did modified the attachment. Now it contains a DLL that exports the function sha512(pData,byte_count,pHash), which simplifies usage in VB:

Code: [Select]

Imports System.Runtime.InteropServices.Marshal

Module Module1

    Declare Sub sha512 Lib "hash.dll" (ByVal data As IntPtr, ByVal cb As IntPtr, ByVal hash As IntPtr)


    Sub Sha512Auto(ByRef msg As String, ByRef hash() As Byte)

        Dim raw_size As Integer = msg.Length * SystemDefaultCharSize

        Dim pBuffer As IntPtr
        If raw_size Then
            pBuffer = StringToHGlobalAuto(msg)
        End If

        Dim pHash As IntPtr = AllocHGlobal(64)

        ' call Dll function
        sha512(pBuffer, raw_size, pHash)

        ' get hash value
        Copy(pHash, hash, 0, 64)

        ' free memory
        FreeHGlobal(pHash)
        If raw_size Then
            FreeHGlobal(pBuffer)
        End If

    End Sub

    Sub Sha512Ansi(ByRef msg As String, ByRef hash() As Byte)

        Dim raw_size As Integer = msg.Length * 1

        Dim pBuffer As IntPtr
        If raw_size Then
            pBuffer = StringToHGlobalAnsi(msg)
        End If

        Dim pHash As IntPtr = AllocHGlobal(64)

        ' call Dll function
        sha512(pBuffer, raw_size, pHash)

        ' get hash value
        Copy(pHash, hash, 0, 64)

        ' free memory
        FreeHGlobal(pHash)
        If raw_size Then
            FreeHGlobal(pBuffer)
        End If

    End Sub

    Sub Main()

        Dim str As String = ""
        Dim sha512(63) As Byte


        Sha512Auto(str, sha512)
        Console.WriteLine("msg = '" & str & "'")
        Console.Write("hash = ")
        For i = 0 To sha512.GetUpperBound(0)
            Console.Write("{0:X2}", sha512(i))
        Next
        Console.WriteLine("")
        Console.WriteLine("")

        Console.WriteLine("auto string conversion: ")
        str = "Some Text foo nil ..."
        Sha512Auto(str, sha512)
        Console.WriteLine("msg = '" & str & "'")
        Console.Write("hash = ")
        For i = 0 To sha512.GetUpperBound(0)
            Console.Write("{0:X2}", sha512(i))
        Next
        Console.WriteLine("")
        Console.WriteLine("")

        Console.WriteLine("conversion to ANSI string: ")
        str = "Some Text foo nil ..."
        Sha512Ansi(str, sha512)
        Console.WriteLine("msg = '" & str & "'")
        Console.Write("hash = ")
        For i = 0 To sha512.GetUpperBound(0)
            Console.Write("{0:X2}", sha512(i))
        Next
        Console.WriteLine("")

        Console.ReadKey()

    End Sub

End Module

EDIT: replaced SSE3 with SSSE3

[David BS]

After some time I finally got it run correct. The key points are:
 - the words^FIPS140 (actual QWORDs) of the returned hash are in Big-endian byte order.
 - the append bit must be add as a byte of the value 0x80 = 128
 - the 128 bit integer that holds the message size in bits, must also be stored in Big-endian byte order
 - the hash must be initialized with H⁽⁰⁾_0...7 (see FIPS140) before calling sha512_sse4

Hi qWORD, nice to hear you.

WOW! When we are tired s*** happens!   

It's new to me that the format must be send (and return) in BigEndian syntax (is it the syntax of CPU access into memory, isn't it?). I was used to this in pure ASM code but really I'm not able to suppose that OS (or VB) shouldn't correct this after return from function.

I'll analyse your code (I believe in you, it's just to see with my own eyes).
And thank you very, very much for spend time and efforts in that analysis.
I'll be back.  See you very soon... :t

[Gunther]

Hi qWord,

good catch.  :t I've overlooked the Big Endian format.

Gunther

[David BS]

qWord, you made a GREAT job really!

Yes, I saw the code is working very nice.
Just to satisfy my curiosity: why are you utilizing H0 const in SHA512.ASM? Which its paper there? I see it is participating of the "initial hash" routine but WHY exactly those bytes (and not 00's for instance)?

I couldn't solve this alone... The need to make it all BIG-ENDIAN format was completely unknown to me: neither INTEL doesn't mention it, nor VB do not provide any internal mechanism to provide it before/after the call.

And yout interface SHA512 is really a joy! I see the procedures there should be really a hard-work to make in pure VB, I mean, in number of lines and CPU load.

THANK YOU VERY MUCH GUY. 
As Gunther said before (tks to you too, you helped me a lot!), you are really bad!!!
And I liked the jWASM syntax, it seems really objective and allows "macroASM" commands.

I have no words to thank all of you.
I hope can properly retribute your efforts sometime.
 
Thank you all.