How to fix a deadlock in LoadLibrary Api ?

[guga]

Hi Guys

I faced a very very weird behavior in Windows 10. Has someone found a deadlock on LoadLibrary Api ? I never saw that before.

While i was trying to understand how ffmpeg works and making a small app to convert mkv files to hls, i found a weird behaviour inside LoadLibrayExW function. (Also in LoadLibrary api, since it calls internally to LoadLibrayExW).

This is what happened.

I was assembling a api call (with rosasm) to ffmped dll like this:

Code Select Expand

call 'avformat-58.av_register_all'

It is a simple call to avformat-58 dll from ffmpeg library. The av_register_all function does not uses any parameter (it´s a void function)

At some point, while it is assembling this dll, rosasm entered on a deadlock inside avformat-58 dll.  During assemblement, in order to find the handle of certain dlls to get their import address and functions etc, Rosasm uses a call to LoadLibraryA Api.


Analyzing from my debugger it simply does this:

call 'kernel32.LoadLibraryA' { B$ "c:\guga\avformat-58.dll", 0}

The problem is that, once LoadLibrary is called, it never returns.  The debbuger shows me that, internally immediatelly before it exits the call to avformat-58, it don´t returns to the caller in "call 'kernel32.LoadLibraryA' " from RosAsm.exe.
Instead, somehow the esp seems to be exchanged and it ends on returning to a crypt.dll (or something).


I was a bit surprised of that behavior and gave another test. This time, i tested with a naked file containing only the call to LoadLibrary, rather then a full App. I created a simple file to test and it assembled ok. Like this:

Code Select Expand

Main:
         call 'kernel32.LoadLibraryA' { B$ "c:\guga\avformat-58.dll", 0}
ret


At 1st i thought it may be a problem somewhere inside RosAsm assemblement functions that maybe where causing a incorrect stack pointer, but then i gave another test.

This time, i tried (on the full app), a call to loadlibraryExA using different flags, such as: LOAD_LIBRARY_AS_DATAFILE, LOAD_LIBRARY_AS_DATAFILE_EXCLUSIVE, DONT_RESOLVE_DLL_REFERENCES etc.

Everytime the function entered on a deadlock, except when using DONT_RESOLVE_DLL_REFERENCES equate. (Which is the only one that fixes somehow the deadlock internally and exits the function to the proper caller)

So, i realised that it was not a problem inside RosAsm functions, because they uses the same amount of parameters to call this api. So it must be something happening internally LoadLibraryEx.

The same error happens when i simply use NULL and 0 as the parameters of loadlibraryEx (Which is the default/internal behaviour of LoadLibrary api). Ex:

Code Select Expand

     call 'KERNEL32.LoadLibraryExA' edi, &NULL, 0



To try to fix and chase the bug, i ended rebuilding the LoadLibrary and LoadLibraryEx Api, like this:

My version of LoadLibrary Api (in windos10. Should also work for other Windows versions as well). Btw, i didn´t tested it yet for twain_32.dll, but it may works, since this is exactly as how it is built in windows10)

Code Select Expand


Proc LoadLibraryA:
    Arguments @lpLibFileName
    Structure @TwainBufferString 260, @TwainBufferString.DataDis 0
    Uses ecx, edx, esi
   
    mov esi D@lpLibFileName
    ...If esi <> 0

        call stricmp esi, {B$ "twain_32.dll", 0}
        ..If eax = 0
            mov esi D@TwainBufferString
            call 'kernel32.GetWindowsDirectoryA' esi, 247
            .If_And eax > 0, eax < 247
                C_call 'msvcrt.strncat_s' esi, 260, {B$ "\twain_32.dll", 0}, 13
            .End_If
        ..End_If


    ...End_If
    call LoadLibraryExA esi, &NULL, 0

EndP


And LoadLibraryExA Api i rebuilt it like this:

Code Select Expand

;;

UNICODE_STRING structure

[UNICODE_STRING:
UNICODE_STRING.Lenght: W$ 0
UNICODE_STRING.MaximumLenght: W$ 0
UNICODE_STRING.Buffer: D$ 0] ; Pointer to a Unicode string

[ANSI_STRING:
ANSI_STRING.Lenght: W$ 0
ANSI_STRING.MaximumLenght: W$ 0
ANSI_STRING.Buffer: D$ 0] ; Pointer to a Ansi string

;;

Proc LoadLibraryExA:
    Arguments @lpLibFileName, @hFile, @dwFlags
    Structure @UnicodeString 8, @UnicodeString.LenghtDis 0, @UnicodeString.MaximumLenghtDis 2, @UnicodeString.BufferDis 4
    Uses ecx, edx, esi

    call Basep8BitStringToDynamicUnicodeString D@UnicodeString, D@lpLibFileName
    If eax <> 0
        call 'KERNEL32.LoadLibraryExW' D@UnicodeString.BufferDis, D@hFile, D@dwFlags
        mov esi eax
        call 'ntdll.RtlFreeUnicodeString' D@UnicodeString
        mov eax esi
    End_If

EndP

And the Basep8BitStringToDynamicUnicodeString function is like:

Code Select Expand

[UNICODE_STRING.LenghtDis 0
UNICODE_STRING.MaximumLenghtDis 2
UNICODE_STRING.BufferDis 4]

[Size_of_UNICODE_STRING 8]

; This function exists in Kernelbase.dll in windows 10

Proc Basep8BitStringToDynamicUnicodeString:
    Arguments @DestinationString, @SourceString
    Structure @AnsiString 8, @AnsiString.LenghtDis 0, @AnsiString.MaximumLenghtDis 2, @AnsiString.BufferDis 4
    Uses ecx, edx

    call RtlInitAnsiStringEx D@AnsiString, D@SourceString
    .If eax <> &STATUS_SUCCESS
        call 'ntdll.RtlSetLastWin32Error' &ERROR_FILENAME_EXCED_RANGE
        xor eax eax
    .Else
        call 'ntdll.RtlAnsiStringToUnicodeString' D@DestinationString, D@AnsiString, &TRUE
        .If eax = &STATUS_SUCCESS
            mov eax &TRUE
        .Else
            If eax = &STATUS_BUFFER_OVERFLOW
                call 'ntdll.RtlSetLastWin32Error' &ERROR_FILENAME_EXCED_RANGE
            Else
                call BaseSetLastNTError eax
            End_If
            xor eax eax
        .End_If
    .End_If

EndP

;;
       RtlInitAnsiStringEx   (NTDLL.@)

  Initializes a buffered ansi string.

Arguuments
    Source - Pointer to a Ansi string
    Target - Pointer to a UNICODE_STRING structure

  RETURNS
   An appropriate NTSTATUS value.

  NOTES
   Assigns source to target->Buffer. The length of source is assigned to
   target->Length and target->MaximumLength. If source is NULL the length of source is assumed to be 0.

reference:
https://stuff.mit.edu/afs/sipb/project/wine/src/wine-0.9.37/dlls/ntdll/rtlstr.c
;;

[ANSI_STRING.LenghtDis 0
ANSI_STRING.MaximumLenghtDis 2
ANSI_STRING.BufferDis 4]

[Size_of_ANSI_STRING 8]

Proc RtlInitAnsiStringEx:
    Arguments @Target, @Source
    Uses ecx, edx

    mov edx D@Target
    mov ecx D@Source

    mov W$edx+ANSI_STRING.LenghtDis 0
    mov W$edx+ANSI_STRING.MaximumLenghtDis 0
    mov D$edx+ANSI_STRING.BufferDis ecx

    ...If ecx = 0
        mov eax &STATUS_SUCCESS
    ...Else
        ;lea esi D$ecx+1
        call StrLenProc D@Source
        If eax <= 0FFFE
            mov W$edx+ANSI_STRING.LenghtDis ax
            inc eax
            mov W$edx+ANSI_STRING.MaximumLenghtDis ax
            mov eax &STATUS_SUCCESS
        Else
            mov eax &STATUS_NAME_TOO_LONG
        End_If

    ...End_If

EndP

Proc BaseSetLastNTError:
    Arguments @Status
    Uses esi, ecx, edx

    call 'ntdll.RtlNtStatusToDosError' D@Status
    mov esi eax
    call 'ntdll.RtlSetLastWin32Error' eax
    mov eax esi

EndP




So far so good. My version of LoadLibrary and LoadLibraryEx Api worked fine....untill....i made the same test on that damn avformat-58 library. And the deadlock showed up again.

Tracing my funcion on the debugger i found that the deadlock was being caused by LoadLibraryExW Api which is called inside my LoadLibraryExA or the default windows one.



How to fix that stuff ???? Someone ever foudn anything like that before ?  I would like to know if someone has any clue how to fix it, because i don´t want to rebuild completelly the LoadLibraryExW Api for something that maybe fixed more easily, and neither i know if it could also works even if i decided to rebuiild this api.



References:
I´m not sure, but a similar problem i found here
https://en.programqa.com/question/11933212/

[aw27]

LoadLibrary works fine, it is used billions of times every day by millions of people in this planet.

First, you can learn about its limitations by reading the documentation "The DLL entry-point function (aka DllMain) should perform only simple initialization or termination tasks. It must not call the LoadLibrary or LoadLibraryEx function (or a function that calls these functions), because this may create dependency loops in the DLL load order.
Second, you must also guarantee that all avformat-58 dependent DLLs can be loaded by avformat-58 itself.
Finally, you can use this confinement period to learn MASM because nobody really uses or is interested in using Rosasm. You can market it 24 hours a day like others do with their own pseudo MASM combos but results will not improve at all.

[guga]

LoadLibrary works fine, it is used billions of times every day by millions of people in this planet.

Yeah, i already thought that way, untill i faced this deadlock issue inside LoadLibrary. People using the api billions of times is something that really don´t bother me. What bothers me is when i face some bug like this. Btw, me and a lot of people who eventually face a problem like this.

About Dllmain, i have no idea what is inside the dllmain of avformat-58.dll. Since it was written with Mingw, most likely it do contains complex initialization data inside the dll entrypoint. It also may contains some apis used in CRYPT32.dll since this is where the LoadLibrary api is stuck while debugging. Therefore, i have no control on this ffmpeg dll. I can´t modify it or rewrite it.

I could, however use the executable version of that ffmpeg library which also contains exported functions that i´l test eventually, but the point is, if loadlibrary is not returning from whatever reason inside the avformat-58 dll, then there´s nothing to stop others dls to behave the same way.

So, it most likely a bug (a very rare one) inside windows loadlibrary api that i´m trying to find a way to fix without being forced to make a full rewrite of MS api or keeping using DONT_RESOLVE_DLL_REFERENCES in whatever dll i´m feeding in RosAsm. Specially because DONT_RESOLVE_DLL_REFERENCES is flawed (even M$ discourage it´s usage)
https://docs.microsoft.com/en-us/windows/win32/api/libloaderapi/nf-libloaderapi-loadlibraryexa
https://devblogs.microsoft.com/oldnewthing/20050214-00/?p=36463
https://stackoverflow.com/questions/16620982/load-dll-without-execute-dllmain-function

Second, you must also guarantee that all avformat-58 dependent DLLs can be loaded by avformat-58 itself.

I´m not sure what you meant, but all dlls dependent with avformat-58 are in the same directory as him. Also, it do works when i create a small file containing only one call to the api inside avformat-58 using loadlibrary api.  So, if it works for a tiny app there should be no reason why it wouldn´t work on a bigger one, specially because the file is not running while being assembled. All it is doing is call loadlibrary api to get the proper handle of the functions.

So, it works on a small file, but a bigger one, it fails miserably without any apparent reason.

Finally, you can use this confinement period to learn MASM because nobody really uses or is interested in using Rosasm. You can market it 24 hours a day like others do with their own pseudo MASM combos but results will not improve at all.

Well...sorry, but don´t. You really didn´t deserved an answer for that, but....well...ok then.

I don´t use masm on a regular daily basis although i do worked with it back in the 90´s when i met here excellent people from which i feel free to call them as friends despite never actually met, guys like Steve, Iczelion, Edgar, Jonchen, Gunther, DayDreamer, Betov, Randall, Seikmanski, Wayne, EWayne, Sven,  etc etc. And yes...i´m the co-author and main maintainer of RosAsm since ages.

No, RosAsm is not a pseudo Masm combo or style or what ever. It don´t even uses any masm files. The style is more similar to Nasm but it is a different assembler as Fasm, Tasm, Nasm, GoAsm.

No, i don´t have plans now to 'market" RosAsm in any extent. If i did i would certainly have done it almost 20 years ago, right ?

Anyway, i´m trying to be polite answering you, even if you don´t deserve it. In decades i have absolutely no problem with anyone around here or in the old forum, and it won´t be the 1st time either.  So, if you woke up out of the bed today, sorry, but i have nothing to do with that.

[aw27]

guga,

I don't care what your opinion about me is. I think you are totally wrong.

You appear to be hanging in time-space. When a product is dead, discontinue it, period.
I have done it with many products, even products featured in main stream magazines like PcMagazine. When a product is dead, assume it don't inject it by force. Something from the 90's with no public is to be buried don't keep it like a zombie!

[jj2007]

Anyway, i´m trying to be polite answering you, even if you don´t deserve it. In decades i have absolutely no problem with anyone around here or in the old forum, and it won´t be the 1st time either.  So, if you woke up out of the bed today, sorry, but i have nothing to do with that.

Guga, you are a true gentleman

[aw27]

Code Select Expand


include \masm32\include\masm32rt.inc


.data
avformat58dll db "avformat-58.dll",0
havformat58dll dd 0

.code

main proc
;int 3
invoke LoadLibrary, offset avformat58dll
mov havformat58dll, eax

ret
main endp

end


I download the ffmpeg stuff to test.
The above does not hang in my Windows 10, so it appears that you have some sort of a priori animosity against Microsoft.
Poor Microsoft is blamed for everyone's own faults.   

[guga]

As i said, a single call does not showed me the error. It happened when i was assembling this dll and others as well.




And this image below shows the same file assembling without any error (Only is ok, when i use DONT_RESOLVE_DLL_REFERENCES btw)

The above does not hang in my Windows 10, so it appears that you have some sort of a priori animosity against Microsoft.

At some extent, well....Who doesn´t ?

[hutch--]

Guga,

It may be useful to disassemble the DLL to see if it does something unusual at its entry point, the nominal DLLMAIN. The problems you are having seem unusual as DLLs are very well understood in 32 bit. Normally with LoadLibrary() you test its return value to see if you get a valid handle then you check is the the next call to GetProcAddress() and again test the return value to ensure the procedure exists.

[guga]

Hi Steve


I did that a couple of hours ago and the DllMain is huge. It has several calls to some windows Apis and other functions used by the avformat-58 and  seems to have also others that call to more dlls  dependent to it. I really don´t know what was the mess that was being done with that dll. All i can see is that somehow it breaks loadlibrary badly and this is what i´m trying to figure it out how to fix without needing to rebuild LoadLibraryEx completely from scratch  .

Normally with LoadLibrary() you test its return value to see if you get a valid handle then you check..

Yes    The problem is that LoadLibrary is not returning somehow. It is stuck inside another dll called by this damn ffmpeg library. While i was debugging loadlibrary i saw that before it returns, the value at eax had the proper handle, but once it reached the "ret" instruction (inside LoadLibrary), instead it returns to the caller (that is RosAsm.exe), it was going inside another dll (crypt or something) called anywhere in avformat-58.dll

I wonder if there is a way to fix such problematic dlls more easily. I´m clueless of what maybe causing the problem and a way to fix it . I never saw this weird behavior before.

[daydreamer]

I don´t use masm on a regular daily basis although i do worked with it back in the 90´s when i met here excellent people from which i feel free to call them as friends despite never actually met, guys like Steve, Iczelion, Edgar, Jonchen, Gunther, DayDreamer, Betov, Randall, Seikmanski, Wayne, EWayne, Sven,  etc etc. And yes...i´m the co-author and main maintainer of RosAsm since ages.

I totally agree Guga 

[hutch--]

Guga,

This does not answer your question but where I have used ffmpeg, I have written interfaces that interface with ffmpeg.exe via a simple batch file. The logic is simple, work out what you want ffmpeg to do, construct a command line that will do what you want then place it in a batch file which you then run. I do it for licencing reasons due to MASM not being compatible with anything GPL.

[aw27]

I don't think the problem is related to ffmpeg because, unlike pieces of code with no users, is something heavily scrutinized.
But to investigate ffmpeg I would go directly to the source code, instead of trying to disassemble the DLL.

[hutch--]

One other suggestion, ffmpeg documentation is truly lousy and you need to make sure you are passing the right number and type of arguments. This could account for the unusual problems you are having.

[guga]

One other suggestion, ffmpeg documentation is truly lousy and you need to make sure you are passing the right number and type of arguments. This could account for the unusual problems you are having.

Hi Steve, i´m aware that. The documentation is really pain but the error is showing when the assembler tries to parse a function that have no parameter whatsoever (a void function) such as av_register_all.

What RosAsm does is simply analyse the source code i´m building and parse the text before it tries to build the import function of dlls. The sequence is like that:

1 - Parse the text and create a table of a array of strings related to the dll name + dll function.
2 - Before the main import function, it then tries to load the imports to see if they do exists on the system we are currently working on. So, it will take from that list of strings the dll name and pass it on the LoadLibrary Api
3 - If a handle is found, it then simply stores the result (of the handle of the dll) on another table to check for duplication when it will later go onto the true routines that create the import section of the generated PE file.  But, if the handle is not found, it simply shows the proper error message and exit that  parsing routine.
4 - After finding the dll handles it then enters onto another routine to parse the list of api names related to those dlls. So, GetProcAddress is used only after the dll name verification is done. It do the same thing on the item 3 for the function names but this time to check for the api functions only
5 - while filling all the table and checking for duplication etc, it then releases the loaded library an perfom a loop to the next dllname+apiname of the list . After filling all the table it then exits this parsing routine
6 - Sort the Api tables created
7 - And finally create the import section.

What is going on here is that when it finds this weird dll (during item "2" and 3 above), and tries to import it with LoadLibrary, is that where somehow it hangs. So, LoadLibrary does not returns to the main caller routine to begin part 4, no matter which function name is used since, this part of the code does not use GetProcAddress yet.

Btw, did you used ffmpeg ?

Offtopic. Do you think you can help me trying to work with it ? I´m trying to create a hls file from a mkv one to be streamed. The generated hls file must contains multiple audio languages, multiple subtitles and multiple video resolutions to be exported on their own directories.

I found a tutorial but i got clueless because it wasn't made to work on windows, i presume. My goal is:

1 - Input a mkv containing a video (in mp4 format encoded with h264), 2 or more audio channels (mp3 files, in portuguese, spanish, english etc), 2 or more subtitles (srt files, in portuguese, spanish, english etc)
2 - Output a playlist (m3u8) and the needed hls (ts, aac and vtt files)
3 - Audio, video and subtitles must be converted to their own formats, but, split on chunks of 6 seconds and places on their own directories with the following condition:

a - Each video inside the mkv  (in mp4 coded with h264) must be converted to ts format and also must be decoded to different resolutions, so for 1280*960, 960*720, 640*480 etc. and place them on their proper directories with the corresponding m3u8 file.
b - Each audio inside the mkv  (in mp3 format ) file must also be converted to aac format (or whatever other format a server uses for this hls stuff) and place them on their proper directories with the corresponding m3u8 file.
c - Each subtitle inside the mkv  (in srt format ) file must also be converted to vtt format (webtt or whatever other format a server uses for this hls stuff) and place them on their proper directories with the corresponding m3u8 file.

4 -Finally create a general playlist in m3u8 format to the used by a html5 player to be streamed.

So, the mkv file (Ex: guga.mkv) will then transcoded to different resolutions in small chunks and place them on a main directory (Ex: gugamovie), which wil contains a playlist in m3u8 format and inside of it it will contain subdirectories related to the different file types. Ex: a subdir named gugaaudio containing the generated playlist to the proper subdirectories inside of it for each language. Ex. inside this dir will create a subdirectory called "portuguese' to store the pieces of aac file in portuguese language, other subdir labeled as "english" containing the english audio and so on.  The samething happens to the suubtitles, creating a directory named (gugasubtitle) and internally the proper m3u8 files ans long with their own subdirs containing the generated vtt files. And for the different video resolutions, the same thing is done.

The goal of what i´m trying to achieve is something like the result from this page, but using a mkv as input.
https://kipalog.com/posts/FFMPEG-HLS-STREAM-MULTIPLE-AUDIO-SUBTITLES

[guga]

Hi Guys


What are these flags that do exists internally LoadLibraryEx function in Windows10 ? Do it have any documentation about it ?

LOAD_LIBRARY_REQUIRE_SIGNED_TARGET ; value = 0x80h
LOAD_PACKAGED_LIBRARY ; value = 0x4h
LOAD_LIBRARY_OS_INTEGRITY_CONTINUITY ; value = 0x8000h

Those flags are used to convert the dwFlags to DllCharacteristics flags to be passed onto LdrLoadDll from ntdll.dll


I found those flags here
https://www.cyberforum.ru/blogs/172954/blog5934.html