RosAsm PDB Dumper v1.0

[guga]

A preview of the help file. I plan to make it in chm format and not pdf to make easier search inside.

Also, i´m currently finishing writing the help for the public symbols. I added 16 more or something and plan to add a few more i found here and there, including static float and double data, static guid, pch (precompile headers), IIDs, CLSIDs, Import descriptors.

For the rest of the public symbols, i´m also planning to add a routine to check if a undecorated symbol is really code or data (when the identification on the "normal" way failed)
The importance of the public symbol is also for disassembler purposes, since, it can be used as a guidance for correct code/data chunks identifications.

I would like to know what you guys think.... If the help file is readable, if it is easy to understand how it works etc.

[Gunther]

Hi Gustavo,

the PDF file looks good. I think you're explaining very detailed. That's good for interested coders.  :t

Gunther

[guga]

Hi Gunther. Many thanks. 

I`m struggling with the pdbparser, because i want it to be the more accurated as possible in order to allow anyone to build whatever tool is needed. Such as a pdb to inc, a better usage of a disassembler or debugger that loads pdb (RosAsm, IdaPro, Olly etc), write wrappers for classes on a automated and faster way (This is possible once i succeed to properly parse all classes info and build the related structure of it (or equate enumerations too) , or even it seems possible to write tiny lib files from some information gattered from the pdb. The possibilities of usage the information inside a pdb seems huge.

Too bad, M$ made such a mess on it. Although MsDia is a handy tool, it is a bit complicated to work because the enumeration of the data is not always that easy to retrieve on the proper manner.

Once i suceed to finish the public symbols on the necessary way, and finish the help file, i´ll build the dll. The next versions i´m planning to make a binary data of those tokens, because it seems that if i succeed to make it in binary, it would be even easier to grab more info and the export file may be much smaller. (Plus, with the binay data, i canfinally try finishing a Flirt system i´m planning for RosAsm since years)

[guga]

Ok, i´m delaying the release, because i´m fine tunning the results. The public symbol identification is the most painful thing ever. But, it provides valuable information. So far i achieved a margin of error of 3% of unidentified symbols  taking shell32.pdb as an example. From 24736 symbols existent (code, data, structures, linker info, delay load helpers routines, IAT data etc), it only missed 751 - in fact, it didn´t "missed", it simply couldn´t identify by it´s unmangled name neither the chunk size (Mainly the unidentified info is data), but i´m trying to achieve a higher rate closer to 99% of correct identification before release

Also, i enabled the hability to dl symbols from ms server based on the loaded executable/dll. For this, later i´ll add a match check for the symbol and executable and maybe force it to match even if they seems different (This is possible when the symbols and executable have different ages, but the same signature info)

I won´t upload the last version now, because i need to change one routine for the temporary dl of the pdb files.  For the executable demo, probably i´ll add a browse folder to save it when you try to parse and executable that contains the pdb info to be dl.


Btw...one question to better guarantee correctnes. SOmeone have a list of all possible Guid data existent ? (I mean, their names and values). I tried to apply Japeth´s comview tool here to create a text file that may help the parsing identification, but ComView is not opening here (probably because my OS is too heavy right now)

[guga]

Can someone, please test this to see if it is workingon other systems then WinXp ?

Added more Public Symbols structures and a progressbar computing the parsing position (It was necessary for huge pdbs.

Also....how to use multithreading ? I mean, i plan to place the main function inside a thread with createthead, but since it is using now a progressbar, does it means i have to put the progressbar function on another thread too ?


Also...how to pause/resume/stop the progressbar ?

[guga]

Damn, i hate threads ! :icon_mrgreen:

I´m testing another version but I can´t make it work. It keeps crashing all the time

What´s the point of threads afterall ? Does it release memory for oother applications ? If it is, then someone have a working example/tutorial on how to use it ? (Also for multithreads)

[TouEnMasm]

The dbghelp.dll offer a set of functions to browse the pdb.
Perhaps he could help,there is no need of thread.

[guga]

dbghelp is not the best way to retrieve the pdb info., unfortunatelly

It is mainly a kind of wrapper for MsDia (The older version on your system - v 1.09, i presume) and does not have all functions as MSDia.
This is why i´m putting so much efford in creating a parser using msdia 1.20 and not dbghelp

One question. If there´s no need of threads, how can i make the app be free to use while it is parsing ? I mean, try to parse a "huge" pdb (Example one with 12 Mb or 22 MB) and then try to move it´s window or the progressbar. You won´t be able to do it, since it is parsing the "huge" database. How can i handle this ? I mean, how to make the rest of the app available for usage while the pdb is being parsed, regardless how big the pdb is ?

[TouEnMasm]

I have the msdia100.dll provided by c++ express,version 10.00.40219.01
dumpbin show no exports functions for this dll,must be not easy to use it.
The dbghelp.dll is used by windbg and I don't see in what the msdia can be better.

If you are in trouble with thread,write your code without the thread and add it at he end.

[guga]

Hi

From what i read, msdia100 is buggy. They fixed some problems iin msdia120. DbgHelp is a wrapper mainly. I´m not sure if the last version of dbghelp (for VS 2013) uses msdia120. What i know is that dbghelp does not uses all functions in msdia. (At least the version that does not wrap msdia120). I´m not saying dbghelp is useless. It just don´t uses all functionality of msdia.

About threads, i tried putting at the end,but it still needs to wait the parser finishes and don´t release the app before it ends it´s job. For small pdbs this is not a problem, but for big files, you have to wait the parse finishes.

Try parsing the file "wmp_notestroot.pdb" (23 Mb - for windows 8 ) to you see what i mean. It have 76227 symbols. Maybe i´m placing the thread in the wrong place ? The main parsing engine is at GetPDBInfo

If i place the thread there, it won´t make any difference (also, it crashed whenever i tried. Or kept showing the error message in ScanHeaderFile, because the thread ended before it parsed)

Code Select Expand


Proc GetPDBInfo::
    Arguments @lpFileName, @Flags
    Local @InputFileLen, @hDiaLib
    Uses esi, edi, ecx, ebx, edx

    lea ebx D@hDiaLib | mov D$ebx 0
    call GetMSDiaLibrary D@lpFileName, ebx
    ..If eax = &TRUE
        call StartPDBProgressBar &NULL
        mov D$PdbInfoMem 0
        call CreatePDBinMemory D@lpFileName, PdbInfoMem
        .If eax <> 0
            mov D@InputFileLen eax

            mov esi D$PdbInfoMem
            Test_If D@Flags PDB_PARSE_ENUM
                call DumpAllEnums D$g_pDiaSymbol, esi | mov esi eax
            Test_End

            Test_If D@Flags PDB_PARSE_TYPEDEF
                call DumpAllTypeDefs D$g_pDiaSymbol, esi | mov esi eax
            Test_End

            Test_If D@Flags PDB_PARSE_UDT
                call DumpAllUDTs D$g_pDiaSymbol, esi | mov esi eax
            Test_End

            Test_If D@Flags PDB_PARSE_PUBLIC_SYMBOL
                call DumpAllPublicSymbols D$g_pDiaSymbol, esi | mov esi eax
            Test_End

            call ClosePDBProgressBar

            If D$g_pDiaSymbol <> 0
                call ReleaseInterface D$g_pDiaSymbol
                mov D$g_pDiaSymbol 0
            End_If

            If D$g_pDiaSession <> 0
                call ReleaseInterface D$g_pDiaSession
                mov D$g_pDiaSession 0
            End_If

            If D$g_pDiaSymbol <> 0
                call ReleaseInterface D$g_pDiaSymbol
                mov D$g_pDiaSymbol 0
            End_If

            mov ecx esi | sub ecx D$PdbInfoMem
            If ecx > D@InputFileLen
                Align_On MEM_ALIGNMENT ecx
                call ReAllocateMemory D$PdbInfoMem, ecx
                mov D$PdbInfoMem 0
            Else_If ecx = 0 ; Nothing was parsed
                call 'RosMem.Heap_Operator_Delete' D$PdbInfoMem
                xor eax eax
            Else
                mov eax D$PdbInfoMem
            End_If

        .End_If
    ..End_If

    mov esi eax

    ; release msdia library before exit
    If D@hDiaLib <> 0
        call 'KERNEL32.FreeLibrary' D@hDiaLib
    End_If

    mov eax esi

EndP


While the routine to call it, is at:

Code Select Expand


Proc ScanHeaderFile:
    Arguments @Adressee, @Message, @wParam, @lParam
    Local @hRtfEdit

     pushad

    ...If D@Message = &WM_COMMAND                  ; User action

        ..If D@wParam = IDC_EXIT
            call HeaderScanCleanUp D@Adressee
            call 'USER32.EndDialog' D@Adressee &NULL
        ..Else_If D@wParam = IDC_CLOSE

            call HeaderScanCleanUp D@Adressee

        ..Else_If D@wParam = IDC_OPEN

                mov B$HeaderSaveFilter 0
                move D$ofn.hwndOwner D@Adressee
                move D$ofn.hInstance D$hInstance
                mov D$ofn.lpstrFilter HeaderFileFilter
                call 'comdlg32.GetOpenFileNameW' ofn

            .If eax = &TRUE
                call HeaderScanCleanUp D@Adressee
                call GetPDBInfo HeaderSaveFilter, (PDB_PARSE_TYPEDEF+PDB_PARSE_ENUM+PDB_PARSE_UDT+PDB_PARSE_PUBLIC_SYMBOL);, PDBFlag ; 8 = 1st check only enum, UDT = 010, typedef = 4
                If eax <> 0
                    mov D$PdbMem eax
                    call 'user32.GetDlgItem' D@Adressee, IDC_TEXT | mov D@hRtfEdit eax
                    call 'USER32.SendMessageA' eax, &EM_SETLIMITTEXT, 0, 0 ; extends the text limit ot the edit control to 0-1
                    call 'USER32.SendMessageA' D@hRtfEdit, &EM_SETSEL, 0-2, 0-2

                    ;mov esi D$PdbMem Development routine to Test to see if everything went fine. Not needed anylonger
                    ;While B$esi <> 0
                     ;   inc esi
                    ;End_While

                    call 'USER32.SendMessageA' D@hRtfEdit, &EM_REPLACESEL, 0, D$PdbMem

                    ;call 'USER32.SetDlgItemTextA' D@Adressee, IDC_TEXT, D$PdbMem

                Else
                    call 'User32.MessageBoxA' &NULL, Sz_PDBError, {B$ "Error !!!", 0}, &MB_ICONERROR__&MB_ICONWARNING__&MB_TOPMOST
                End_If
            .End_If

        ..End_If


    ...Else_If D@Message = &WM_INITDIALOG

    ...Else_If D@Message = &WM_CLOSE

         call HeaderScanCleanUp D@Adressee
         call 'USER32.EndDialog' D@Adressee &NULL

    ...Else
        popad | mov eax &FALSE | ExitP

    ...End_If

L9: popad | mov eax &TRUE

EndP


Do i need to put a thread after in D@wParam = IDC_OPEN ?

[TouEnMasm]

To simplify your problem,you can choose the normal way to get a pointer on IDiaDataSource.
That is:
made a dynamic link on DllRegisterServer and DllUnregisterServer
There are just PROTO stdcall with no parameter.

init:  invoke DllRegisterServer
This load the clsid and the interfaces into the registry

Code Select Expand


invoke CoCreateInstance,addr CLSID_DiaSource, NULL,CLSCTX_INPROC_SERVER,addr IID_IDiaDataSource,addr ppvIDiaDataSource ;ppv out
;eax S_OK réussite
.if ppvIDiaDataSource != 0 ;
IDiaDataSource loadDataFromPdb,addr pdbfile ;OK
.if eax == S_OK
IDiaDataSource loadDataForExe,addr exefile,NULL,NULL ;catastrophic failure,need more experiment
.if eax != S_OK
;invoke GetLastError
invoke LireEr_Com,eax
.if eax != 0
invoke MessageBox,NULL,edx,ecx,MB_OK
.else
invoke MessageBox,NULL,TXT("IDiaDataSource loadDataForExe"),TXT("ERROR not FOUND"),MB_OK
.endif
.endif
IDiaDataSource Release
.endif
.endif



end: invoke DllUnregisterServer
Take care that some interfaces use a virtual stdcalll who need special features to avoid the crash

[guga]

Actually there is no need to use DllRegisterServer or CLSCTX_INPROC_SERVER

MSdia can be loaded "unregistered". The routine i used for this is:

Code Select Expand

Proc GetMSDiaLibrary:
    Arguments @SzOutput, @phDiaLib
    Uses esi, edi, ecx, ebx, edx

    call InitializeDiaInterface {U$ "msdia120.dll" 0}, CLSID_DiaSourceAlt, IID_IDiaDataSource, g_pDiaDataSource, D@phDiaLib
    If eax <> &S_OK
        xor eax eax
        ExitP
    End_If


    icall DIA_LOADDATA_FROMPDB D$g_pDiaDataSource, D@SzOutput
    .If eax <> &S_OK
        call TryLoadingSymbolForExe D@SzOutput
        On eax = &FALSE, EXitP
    .End_If

    icall DIA_OPENSESSION D$g_pDiaDataSource, g_pDiaSession
    If eax <> &S_OK
        call ReportWinError {'DiaDataSource: OpenSession' 0} ; show cause of failure
        xor eax eax
        ExitP
    End_If

    icall DIA_SESSION_GET_GLOBALSCOPE D$g_pDiaSession, g_pDiaSymbol
    If eax <> &S_OK
        call ReportWinError {'DiaSession: GlobalScope' 0} ; show cause of failure
        xor eax eax
    Else
        mov eax &TRUE
    End_If

EndP

Code Select Expand




; Initialize an object not registered
; http://msdn.microsoft.com/en-us/library/windows/desktop/ms680760%28v=vs.85%29.aspx

Proc InitializeDiaInterface:
    Arguments @lpLibFileName, @pClsid, @pIID, @pOut, @phDiaLib
    Local @hpfDllGetClassObject, @ppv
    Uses esi, edi, ecx, ebx, edx

    call 'KERNEL32.LoadLibraryExW' D@lpLibFileName, 0, &LOAD_WITH_ALTERED_SEARCH_PATH
    .If eax = 0
        call ReportWinError {'LoadLibraryExW' 0} ; show cause of failure
        mov eax &E_FAIL
        ExitP
    .End_If

    mov edi D@phDiaLib | mov D$edi eax

    call 'KERNEL32.GetProcAddress' eax, {B$ "DllGetClassObject", 0}
    .If eax = 0
        call ReportWinError {'msdia120.dll' 0} ; show cause of failure
        mov eax &E_FAIL
        ExitP
    .End_If
    mov D@hpfDllGetClassObject eax

    lea ecx D@ppv | mov D$ecx 0
    call D@hpfDllGetClassObject D@pClsid, IID_IClassFactory, ecx
    mov esi eax
    .If eax = &S_OK ; points to CreateInstance
        icall ICLASS_FACTORY_CREATE_INSTANCE D@ppv, 0, D@pIID, D@pOut
        mov esi eax
        call ReleaseInterface D@ppv
    .End_If

    mov eax esi

EndP


The function is called like this:

Code Select Expand

[HeaderSaveFilter: U$ 0 #&MAX_PATH] ; The path and name of the pdb file (In unicode string)
[hDiaLib: D$ 0] ; A variable that is used to store the msdia120.dll handle

    call GetMSDiaLibrary HeaderSaveFilter, hDiaLib


MSdia contains as an export this function DllGetClassObject, which is used to load unregistered classes
http://msdn.microsoft.com/en-us/library/windows/desktop/ms680760%28v=vs.85%29.aspx

So, if a dll contains this function, you can simply use this routines i did, to load it without registering the interface. Not sure what are other dlls that have this external Api, but for what i saw, whenever a dll have it (and uses classes) you can use it, and therefore, you can put the dll in whatever folder to avoid relying on the system folder. This is particular good if you have 2 dlls containing tteh same (or similar) interfaces, but you want to use the most recent one on your own directory and not the one that is provided in system32 directory

For loading the symbols from a exe/dll, You example may not work if you don´t set the proper servers like this:

Code Select Expand


Proc TryLoadingSymbolForExe:
    Arguments @SzOutput
    Local @callback

    icall DIA_LOADDATA_FOREXE D$g_pDiaDataSource, D@SzOutput, &NULL, &NULL
    If eax = &S_OK
        mov eax &TRUE
        ExitP
    End_If

    icall DIA_LOADDATA_FOREXE D$g_pDiaDataSource, D@SzOutput, {U$ "srv*C:\temp*http://srv.symbolsource.org/pdb/Public;srv*C:\temp*http://symbols.mozilla.org/firefox;srv*C:\temp*http://referencesource.microsoft.com/symbols;srv*C:\temp*http://msdl.microsoft.com/download/symbols", 0}, &NULL

    ..If eax <> &S_OK
        call ReportWinError {'DiaDataSource: LoadDataFromPdb' 0} ; show cause of failure
        xor eax eax
        ExitP
    ..End_If
    mov eax &TRUE

EndP

The 3rd parameter of the loadDataForExe function (the 2nd one in your function) is used to check where to load the pdb. If it is NULL, the function tries to load the pdb on the same directory as in your main file. If it can´t find it, it returns something else then S_OK. Then you can simply see if on the proper servers there is a pdb to be loaded. (I presume you did that on LireEr_Com ?)

Since i´m testing the loaddataforexe, the downloaded pdbs are saved on "C:\temp". Later i´ll make some routines to allow the user to choose the directory he wants for the downloaded files. ANd also some routines to check if the loaded pdb matches to the exe/dll (Through it´s age and signature). If it don´t match there is a way to force a match, simply resetting it´s age, as long the signature is the same.

Also, it is important that symsrv.dll be at the same directory as your msdia120.dll. Otherwise it may not work as expected.

For the syntax used to load the pdb/dbg from the servers (ms or other), the correct is:
srv*C:\temp*http://msdl.microsoft.com/download/symbols

that means:
srv = the token to activate the download
* = a separator. Must be used
c:\temp = the directory for the downloaded files
* = a separator. Must be used
http://msdl.microsoft.com/download/symbols = the server from where you want to dl the pdb

Also. To use more then 1 servers to be searched and dwonloaded, you must separate the syntaxes with a ";" token, like this:
C:\temp*http://referencesource.microsoft.com/symbols;srv*C:\temp*http://msdl.microsoft.com/download/symbols

And the whole string MUST be in Unicode format. This is why i put ' "U$" datatype in my code and not a "B$" in icall DIA_LOADDATA_FOREXE. In RosAsm the "U$" token refers to a Unicode String

[TouEnMasm]

Actually there is no need to use DllRegisterServer or CLSCTX_INPROC_SERVER

A beautifull affirmation without proof.
Microsoft use CLSCTX_INPROC_SERVER in all of his sample.
To affirm that, you need to know what do the DllRegisterServer function.
And if is not enough,you can search for what your program had so much trouble to work.

[guga]

I´m not following M$ example.
See here
http://stackoverflow.com/questions/2466138/windows-c-how-to-use-a-com-dll-which-is-not-registered
http://masm32.com/board/index.php?topic=3441.0

[TouEnMasm]

Those two links not grant that it work perfectly.
The answer is inside the dll itself with his function.

msdia100.dll
   DllCanUnloadNow,DllGetClassObject,DllRegisterServer,DllUnregisterServer,VSDllRegisterServer
   VSDllUnregisterServer

Find how to use VSDllRegisterServer and  VSDllUnregisterServer and you surely have an anwer.