Resource String Table

[guga]

Hi Guys

How a String Message Table is formed inside the rsrc section of a PE ?

I´m not talking about a Resource Script, but on how the script is created in raw inside a PE .  Ex:

This..

Code: [Select]

#define IDS_HELLO    1
#define IDS_GOODBYE  2

STRINGTABLE
{
    IDS_HELLO,   "Hello"
    IDS_GOODBYE, "Goodbye"
}


How is it organized ? How the Ids are created and where they are being placed ? Do it have any sort of structure related to it ?

I know they are formed by groups of 16 Unicode (unicode16) strings, but i don´t know what is the relation between their indexes and ids on the position they are created inside the resource section.

According to this http://www.csn.ul.ie/~caolan/publink/winresdump/winresdump/doc/resfmt.txt, it says that:

"4.8 String Table Resources
 
These tables are constructed in blocks of 16 strings.  The organization of these blocks of 16 is determined by the IDs given to
the various strings.  The lowest four bits of the ID determine a string's position in the block.  The upper twelve bits determine
which block the string is in.  Each block of 16 strings is stored as one resource entry.  Each string or error table resource block is
stored as follows:
 
    [Normal resource header (type = 6 for strings)]
     
    [Block of 16 strings.  The strings are Pascal style with a WORD
    length preceding the string.  16 strings are always written, even
    if not all slots are full.  Any slots in the block with no string
    have a zero WORD for the length.]
     
It is important to note that the various blocks need not be written  out in numerical order in the resource file.  Each block is assigned
an ordinal ID.  This ID is the high 12 bits of the string IDs in the  block plus one (ordinal IDs can't be zero).  The blocks are written
to the .RES file in the order the blocks are encountered in the .RC  file, while the CVTRES utility will cause them to become ordered in
the COFF object, and hence the image file. "

Also, here is an article explaining t, but i didn´t understood.
https://titanwolf.org/Network/Articles/Article?AID=557d9a19-e07c-4eff-95cd-73306336fbf2#gsc.tab=0

And below it shows some internal structures: https://sourceware.org/legacy-ml/binutils/2007-05/msg00314/windint.h

Code: [Select]


/* A stringtable resource is a pointer to a rc_stringtable.  */

typedef struct rc_stringtable
{
  /* Each stringtable resource is a list of 16 unicode strings.  */
  struct
  {
    /* Length of string.  */
    rc_uint_type length;
    /* String data if length > 0.  */
    unichar *string;
  } strings[16];
} rc_stringtable;

More references:
https://source.surroindustries.com/Cycle0/Foster/-/blob/51ad9f48c7d7e90792293e2312263b07b68bad77/Fedora/x86_64/surro-iso/build/binutils-2.29/binutils/resbin.c

Also..can it have a ID of 0 or it must always starts at 1 ?

When we have a multiline string, when encoding, they are generally appended a Line Feed (hex  = 0A) as a paragraph mark. So, does the next line (That is part of the same string) represents another index ?

How to encode it ?

For example, say i have this as strings to be encoded:

Id Value     String to be encoded
120             Hello World
147             Testing String Table
2580           This i a multiline string /n and i´m continuing the text after the line feed


I succeeded to decode it like this:

Code: [Select]


; References:
; http://www.csn.ul.ie/~caolan/publink/winresdump/winresdump/doc/resfmt.txt
; https://devblogs.microsoft.com/oldnewthing/20040130-00/?p=40813
; https://stackoverflow.com/questions/60564533/how-to-read-string-table-resources-for-given-language-into-map
; https://doxygen.reactos.org/df/d90/resources_8c_source.html#l00077

; Function parameters

; InputData - is the pointer to the IMAGE_RESOURCE_DATA_ENTRY corresponding to the leaf node of the resource. So it points to the structure that will point to the true data (The strings that are in Unicode)
; Output -An output buffer to store the decoded strings to be used later
; SrcId - The resource ID as it was found in the IMAGE_RESOURCE_DIRECTORY_ENTRY structure that lead to the true data.

Proc GetRsrcString:
    Arguments @InputData, @Output, @SrcId
    Local @NameInt, @iCounter, @Src, @FileLen, @Id, @NewSize
    Uses esi, edi, ecx, edx

    mov eax D@SrcId | and eax 0FFFF | mov W@NameInt ax
    mov D@iCounter 0
    mov esi D@InputData
    mov edi D@Output
    xor eax eax

    .While eax < 16 ; Groups of 16
        mov eax esi | movzx ecx W$eax | mov D@FileLen ecx
        add esi 2
        mov eax D@FileLen
        .If eax <> 0
            movzx eax W@NameInt | sub eax 1 | shl eax 4
            movzx ecx W@iCounter | add eax ecx | movzx eax ax | mov D@Id eax
            C_call FormatStr edi, {'#%d ', 0}, eax ; Print the correct Id of the string
            add edi eax
            mov eax D@FileLen
            shl eax 1 | mov D@NewSize eax
            call 'ntdll.RtlUnicodeToMultiByteN' edi, D@FileLen, &NULL, esi, eax ; Convert the Unicode to Ansi
            add esi D@NewSize
            add edi D@FileLen | mov W$edi CRLF | add edi 2
        .End_If
        inc D@iCounter
        mov eax D@iCounter
    .End_While

    mov eax edi

EndP

The complete way the strings are mounted inside a PE is given by the image below. From images 1 to 3 i understood how it works, but on image 4 (the actual string table). How is it formed ? It don´[t seems to be they are created with a 16 Word group. And how they Ids are created on a way that don´pt overlap with each other ?

On this example, the rsrc section contains only 40 Strings Groups, each one of them with  his own ID. The total amount of Strings are something around 380. So, how the IDs are related to each group on a way that it don´t overlap ? And what about the position
of each string inside a group how does it works and what is it relation to the ID ?


It seems to be a Structure formed by:

[StringRsrcGroup:
 Group1Rsrc.Size: W$ 11
 Group1Rsrc.Data: W$ "Hello World"
(...)
 Group16Rsrc.Size: W$ 2
 Group16Rsrc.Data: W$ "Hi"]

But...What if the 1st Chunk in the group is empty ?  How is it related to the id or it´position on the 16 words chunk ? Ex:


[StringRsrcGroup:
 Group1Rsrc.Size: W$ 0
 Group1Rsrc.Data: W$ 0

 Group2Rsrc.Size: W$ 0
 Group2Rsrc.Data: W$ 0

 Group3Rsrc.Size: W$ 11
 Group3Rsrc.Data: W$ "Hello World"
(...)
 Group16Rsrc.Size: W$ 2
 Group16Rsrc.Data: W$ "Hi"]


Also, it seems that the maximum amount of Strings Ids is 4095, is that correct ?

[guga]

Ok, i guess i found a way to calculate this, but i´m not sure if it is the correct way to do it.

The strings inside the resources (in text format, not the raw ones) must be limited to a value of 1 to 65535

The equation to retrieve the String Id (in text format) and their actual Id inside the Raw data, seems to be like this:

    NameInt = ID value on the raw Rsrc. (So, the Name1/Id member of the IMAGE_RESOURCE_DIRECTORY_ENTRY structure)
    Id = The number in the string text format (The valuye found in the rc file or any other text format)
    Limits. Id text must be in between 1 and 65535
    Counter - Seems to be the relative position (offset) of each of the 16 words chunk, that can be used as a counter to get the pos ?

    Id = (NameInt-1)*16+Counter
    Id-Counter = (NameInt-1)*16
    (Id-Counter)/16 = (NameInt-1)
     NameInt = 1 + (Id-Counter)/16


is that correct ? And also, how can i calculate the size of the resource data. Does it takes onto account the position of the 16 words chunks regardless they are filled or not, or only is calculated when it do actually contains a string even if it is 1 byte (1 word, actually) long ?



If that´s correct, it means that we can have a maximum of 4096 groups of 16 Unicode Strings in the resource section. I found this number using the minimum and maximum limits for the "Counter" and "Id", like this:

x = 1 + (65535-y)/16 ; x = NameInt  y = Counter

For counter(pos) = 0 we have:
x = 65551/16 = 4096.9375

For counter(pos) = 16 we have:
x = 65555/16 = 4095.9375

Therefore, we can have a maximum of 4096 groups of Unicode Strings in the resource at IMAGE_RESOURCE_DIRECTORY_ENTRY. Or a maximum of 4096 IMAGE_RESOURCE_DIRECTORY_ENTRY structures

Does this limit also applies to other type of resources or it is only a limit on the string resource ? I mean, can we have 10000 Bitmaps on the resource section (Bitmap i mean, on the same Rsrc Type RT_BITMAP), or have 10000 icons, cursors, dialogs, menus itens etc ? Does this limit applies to all type of resources ?

[jj2007]

The strings inside the resources (in text format, not the raw ones) must be limited to a value of 1 to 65535

You can use index zero, too:

Code: [Select]

  lea edi, buffer
  invoke LoadString, rv(GetModuleHandle, 0), 0, edi, 100 ; 0 is the ID
  print edi
MasmBasic Print Res$(0) will through an error ## Res$ index must be non-zero ## but for other reasons.

Note the index should either start from zero (0, 1, 2, 3) or from a number divisible by 400 (400, 401, 402... 800, 801), otherwise you bloat the resource section.

[guga]

The strings inside the resources (in text format, not the raw ones) must be limited to a value of 1 to 65535

You can use index zero, too:

Code: [Select]

  lea edi, buffer
  invoke LoadString, rv(GetModuleHandle, 0), 0, edi, 100 ; 0 is the ID
  print edi
MasmBasic Print Res$(0) will through an error ## Res$ index must be non-zero ## but for other reasons.

Note the index should either start from zero (0, 1, 2, 3) or from a number divisible by 400 (400, 401, 402... 800, 801), otherwise you bloat the resource section.

Hi JJ. Many thanks

Indeed, using 0 as a text index, is possible. It fits to the equation. NameInt = 1 + (Id-Counter)/16 ; Counter = Pos of the Word chunk
NameInt = 1 + (0-Counter)/16
NameInt = 1 - Counter/16

Therefore, if Counter(pos) is at 0, the Id inside the Raw data will be 1
(16 - 0)/16 = 1

If pos = 16, then the raw index will be 0
(16-16)/16 = 0

I´ll make the fix in RosAsm.

About...

Note the index should either start from zero (0, 1, 2, 3) or from a number divisible by 400 (400, 401, 402... 800, 801), otherwise you bloat the resource section.

This index limits are the ones from the raw data or the ones we insert as text (like the ones found in rc files) ? I´m struggling to understand how this works exactly.

I´m asking because the old w32dasm starts with a text index of 120

Code: [Select]

120 Finds the Next Match
121 Set Disassembly Listing at the Current Instruction Pointer
2000 Access Online Help
2001 Help Table of Contents
2002 Help on Using Online Help
(...)

Btw..The maximum limit for the Id (in text format) is 65535 or we can insert a higher number, like 200000 for example ?

[wjr]

Resource Section NameID 0001h covers StringIDs 0000h - 000Fh
:
Resource section NameID 1000h covers StringIDs FFF0h - FFFFh

So the maximum StringID is 65535 (with sequential ranges preferably starting from a number divisible by 16 to avoid bloat).

Edit: corrected maximum StringID

[guga]

Resource Section NameID 0001h covers StringIDs 0000h - 000Fh
:
Resource section NameID 1000h covers StringIDs FFF0h - FFFFh

So the maximum StringID is 65536 (with sequential ranges preferably starting from a number divisible by 16 to avoid bloat).

Hi Wayne. Tks   

Let me see if i understood it correctly. So, if i create a string (in text format, like rc script) with  a Id of 2004, in fact, the true id stored in the resource section will be:

value = 2004 = 00__0000_0111__1101_0100 (binary format)
Id in Raw (what will be used as an id in IMAGE_RESOURCE_DIRECTORY_ENTRY ) = 125+1 = 126 = 00__0000_0111__1101 (plus 1)
Position inside the 16 groups chunk = 4 = 00_0100

And the maximum limit (inside the raw data) of the array of IMAGE_RESOURCE_DIRECTORY_ENTRY will be 4097 ?

Max String Id (in text format) = 65536 = 00__0000_0001__0000_0000__0000_0000
Therefore:
Raw Id = 00__0000_0001__0000_0000__0000 + 1 = 4097
Position (index from 0 to 15) = 00_0000= 0 (The 1st position)

Right ?

But..wouldn´t it extrapolate the limit of the raw id (the one inside the resource section) ? I mean, wouldn´t the correct be ?

Max String Id (in text format) = 65535 = 00__1111_1111__1111_1111
Therefore:
Raw Id = 00__1111_1111__1111 + 1 = 4096
Position (index from 0 to 15) = 00_1111= 15 (The last 16th position)

[wjr]

Correct on the 2004 example. Within the 16 group chunk, 4 (0 based) would be in the 5th position.

For the maximum case, 4096 has the correct figures (maximum NameID just for STRINGTABLE).

The 4097 comes in as the maximum length of the string itself. With multiline strings the next lines are part of the same index. Overall resource size would need to take into account the 16 WORD string size fields per group.

GoRC can handle the embedded carriage returns:
2580           "This is a multiline string \n and i´m continuing \012 the text after
   the various line feeds"
but may do things differently by not requiring a line continuation character (instead, inserts a space and line feed character, then ignores leading spaces on next line).

[guga]

Correct on the 2004 example. Within the 16 group chunk, 4 (0 based) would be in the 5th position.

For the maximum case, 4096 has the correct figures (maximum NameID just for STRINGTABLE).

The 4097 comes in as the maximum length of the string itself. With multiline strings the next lines are part of the same index. Overall resource size would need to take into account the 16 WORD string size fields per group.

GoRC can handle the embedded carriage returns:
2580           "This is a multiline string \n and i´m continuing \012 the text after
   the various line feeds"
but may do things differently by not requiring a line continuation character (instead, inserts a space and line feed character, then ignores leading spaces on next line).

Great.

One question. When using multiline strings, does it encodes only using a Line Feed (0xA), or it also encodes  using a carriage return and a line feed (0x0A0D) ? or even a single carriage return (0xD) ?

I´m asking because the tests i´m doing all the resources strings i found that contains multiline they are using mainly Line Feed (0xA) for it.

I think i got it correct now how to handle these string table. I made a small test function to see if the output was correct.

Code: [Select]

Proc RsrcCountStringGroup:
    Arguments @InputTmpList
    Local @iCount, @RawId, @StringPos
    Uses esi, ecx

    ; 1st get the 1st string id in the temporary list
    mov esi D@InputTmpList
    mov eax D$esi+TmpStringListArray.CountDis | mov D@iCount eax ; Pointer to my test table to be encoded later
    add esi TmpStringListArray.DataDis ; Poins to the actuall data to be encoded.

    .Do
        ; get 1st id of the 16 group array. This is the Id in string format, not the one that will be located in raw data
        mov ecx D$esi+TmpString_Rsrc_Data.IdDis ; 1st member of my test structure is always the Text id format. Let´s decode it to find the Pos and True Id inside the rsrc
        ; or x = NameInt, y = Id, z = Counter(pos)
        ; x = 1 + (y-z)/16
        ; x = y/16 - z/16 + 1
        ; x = 1 + (y-z)/16. So, (y-z) must be a integer multiple of 16. K = (y-z)/16. 16*k = y-z, and y >= z
        ; divide Index by 16 <--- No longer necessary, all i have to do is get the 1st 4 bits for get the string Pos in the chunk and the remainder 12 bits for the Raw Id.

        movzx eax cl | and eax 0F ; <----- 00__0000_0000__0000_1111
        mov D@StringPos eax
        shr ecx 4 | inc ecx | mov D@RawId ecx

        add esi Size_Of_TmpString_Rsrc_Data
        dec D@iCount
    .Loop_Until D@iCount = 0


EndP


I´m considering build a converter for this Numbered Ids in order to avoid any overlappings. Currently, in RosAsm strings Table are writen with a "#" char followed by the text ID plus a space followd by the text to be included (I allowed multiline too).

It works like this (In text formats as existent in rc files . Range = From 0 to 65535 as i understood from you and JJ´s comments):

#51549 SC_LOCK
#51550 PSID_NAME_USE
#51551 LPQUERY_SERVICE_CONFIG
#51552 LPQUERY_SERVICE_LOCK_STATUS

or with multiline

#58019 The thread used up its stack.
 Another exception code is likely
to occur when debugging console processes. It does not arise because of a programming error. The DBG_CONTROL_C
meant to be handled by applications. It is raised only for the benefit of the debugger, and is raised only when a debugger is attached to the console process.
#58020 TOKEN_INFORMATION_CLASS

The strings are created on a Dialog box on this format, like the image below, but perhaps i´ll do a converter option on that same dialog using a checkbox to allow us to see the true Raw Id and the position of the string as well. So, when one checkbox (or radio button) is activated, it displays the text format, and if other checkbox or radiobutton is activated, it shows the actual raw id and position. Maybe something like this would be better to understand how the string table works and also being easier to avoid overlappings:

1st # with digits = Raw ID (From 1 to 4096)
2nd #with digits = Position in the group (from 0 to 15)
#126#0 This is a string
#3627#1 This is another string from group 3627 at pos 1
#3627#4 This is another string from group 3627 at pos 4
#3627#8 This is another string from group 3627 at pos 8

(Doing like above, could also be handy because the position could also be written unordered, and when the file would be assembled, the parser will simply points to the correct values and positions of the table to be created and then place them on the correct positions)




I really need to create a better resources compiler for RosAsm on a more user friendly style, but i have no time right now, except for the updates i´m making. Those part of the code weren´t touched in decades.

[wjr]

Apparently encoded using just a line feed (0Ah).

An API function like LoadString expects the StringID, so perhaps best to keep it simple using just that (let the 'resource compiler' under the hood deal with its more complicated layout - but documentation could go into further detail suggesting to keep identifiers sequential within groups of 16 to save some space).

[guga]

Apparently encoded using just a line feed (0Ah).

An API function like LoadString expects the StringID, so perhaps best to keep it simple using just that (let the 'resource compiler' under the hood deal with its more complicated layout - but documentation could go into further detail suggesting to keep identifiers sequential within groups of 16 to save some space).

Tks wayne. Indeed, it seems to be only encoded as line feed

About keeping simple, i agree. You made a good point here. I already have settled the limits of Id numbers (the text ones) from 0 to 65535. So, considering that when it is encoded it creates only 4096 groups of 16 words, it will never overlap if we stay within the text limit.

For example in the raw Id we can have things like this:

4000 = 00__0000_1111__1010_0000 + pos 1 = 00__0000_1111__1010_0000_0001 = 64001 (resultant String Id)
4001 = 00__0000_1111__1010_0001 + pos 1 = 00__0000_1111__1010_0001_0001 = 64017 (resultant String Id)

4000 = 00__0000_1111__1010_0000 + pos 8 = 00__0000_1111__1010_0000_1000 = 64008 (resultant String Id)
4001 = 00__0000_1111__1010_0001 + pos 8 = 00__0000_1111__1010_0001_1000 = 64024 (resultant String Id)

So we have on this example only 2 slots (pos 1 and 8) being occupied in Id groups 4000 and 4001

Since we always use the lower 4 bits for the position and the next 12 for Raw Id, then it will never overlap (as long we keep the Text Id limited to 0 to 65535). We will always have 16 chunks for each Raw Id group, no matter what, right ?

So, i guess you are right, adding an extra routine seems unnecessary. All encoding can be done under the hood, and also warning the user in simple situations, such as when he insert duplicated strings Ids or if the limit of 65535 was passed.

[jj2007]

Just for fun, here is another approach to creating string tables.

Code: [Select]

include \masm32\MasmBasic\MasmBasic.inc
$Data Enter text here
$Data Click on this button
$Data Welcome
$Data Введите текст здесь ; Enter text here in Russian
$Data Нажмите на эту кнопку ; Click on this button in Russian
$Data Добро пожаловать ; Welcome in Russian
  Init
  Cls
  Read x$()
  For_ ecx=0 To eax-1
PrintLine x$(ecx)
  Next
EndOfCode
Output:

Code: [Select]

Enter text here
Click on this button
Welcome
Введите текст здесь
Нажмите на эту кнопку
Добро пожаловать
Requires MasmBasic 6 October; max #strings is about 24,000 or 800kB, so Windows.inc does not qualify for a test... and I'm afraid it requires a modern assembler like UAsm or AsmC

Attached \Masm32\include\kernl32p.inc as a "stringtable" ;-)

[hutch--]

Guga,

> Those part of the code weren´t touched in decades.

Aha, Betov ware ! You are truly a brave man Guga to make Betov's plaything a working assembler, I hope it all work well for you.

[guga]

Guga,

> Those part of the code weren´t touched in decades.

Aha, Betov ware ! You are truly a brave man Guga to make Betov's plaything a working assembler, I hope it all work well for you.

Hi Steve..I´m fine, tks :) 

About Rene work.... I´m trying to update the best i can. In these years i made several updates on the versions i was working with, but never released what i was finishing. The main problem of RosAsm is that hundreds of functions and variables were extremelly attached to the interface So it was not easy to find where a bug happens.

Rene did his best to try to make the assembler as fast as possible, and even if it was one of the fastest in the past decade to work on a P4 or PIII or even PI, now it is useless to keep those routines, specially due to maintenance issues. Recently i succeeded to implement SSE4 and SSE3 opcodes that another user tried to create on a non official release. JE! did a great job, btw, but i needed to review what he did to upgrade it on a way that i could also try to fix the other bugs.

Now, the disassembler and assembler are working fine with the newer opcodes and i also updated the debugger and i´m currently fixing some annoying bugs that existed in the old Resource routines (Some of them dated from 2006 or something )

My goal is to detach as many functions i can, and make the proper dlls to it work with not only with RosAsm but with other apps as well. I created a dedicated memory management library (finally fixed that monster ) and also created a dll for CRT, and another for the icon editor.

It´s a true hell, but i´m slowly fixing what i can for the next release.

My goal is to succeed to create a dll for the assembler, another for the disassembler, other for the resource editor, debugger and allow RosAsm to work with user made plugins as well (Don't know how to do yet), and then finally, only after the major parts are fixed, i can touch the interface to a more modern style and rebuild the help file completelly. Ouuuuch !

[guga]

Ok, Guys...Finishing the fixes for the StringTable in RosAsm.

I made a small test allowing RosAsm to export the StringTables in rc script for masm syntax.

Is this the proper masm syntax for the string table inside a resource script file ?

[hutch--]

Guga,

It looks OK but the simplest way to test it is to build it with RC.EXE. The version you posted is single character ANSI text version but you can also create RC scripts in UNICODE if you need to, all you need is a UNICODE text editor.