Author Topic: Resource compiler for console applications  (Read 17655 times)

japheth

  • Guest
Re: Resource compiler for console applications
« Reply #30 on: October 26, 2012, 07:04:04 PM »
It's not child's play, of course, but IMHO a pretty basic exercise (sorry, can't help you with C code....):

Even a macro that consists of just EXITM may not work, since the argument behind EXITM may be another macro. So you'll probably have to add recursion in your source sample ( which I cannot read, because I'm suffering from a severe BaSiCphobia , sorry! ).

jj2007

  • Member
  • *****
  • Posts: 7542
  • Assembler is fun ;-)
    • MasmBasic
Re: Resource compiler for console applications
« Reply #31 on: October 26, 2012, 08:53:54 PM »
... your source sample ( which I cannot read,

Strange, very strange: JWasm has no problems to read my sources :icon_mrgreen:

nidud

  • Member
  • *****
  • Posts: 1370
    • https://github.com/nidud/asmc
Re: Resource compiler for console applications
« Reply #32 on: October 26, 2012, 11:43:56 PM »
That's indeed very good news! Since it is obviously a child's play, could you please tell me how to do it? With detailed source code, if you don't mind!
I could help you implement this providing you agree to do as follows: You will copy the content of this code and complement it to cover all the cases needed.

Code: [Select]
l0: upcode
l1: upcode,macro
l2: .if upcode
l3: .if macro
l4: .elseif upcode
l5: .elseif macro
l6: .while upcode
l7: .while macro
l8: .until upcode
l9: .until macro
lA: .continue .if upcode
lB: .continue .if macro
lC: .break .if upcode
lD: .break .if macro
lE: .endw
lF: .repeat
The first thing to do is to create a test case for this code, and insert 4 rows of X’s in the new post (made by you). In the first row you will provide the result using the oldest version of MASM you could find, the second row the newest. The next two rows will be the oldest version (J)WASM and the newest.

The next thing to do is to find the highest code level of the line-feed. This will be the proc using the fgets(..) I assume, and then the location of the first tokenize-call. Make shore that the tokenizer does not have a side job, and only do what it supposes to do: convert the line into tokens. If this is not the case, the code must be moved to a lower level. You should be in the if (GetLine()) if (Tokenize()) Parseline(tokens) situation.

Rename Parseline to Parseline2, and make a new Parseline proc. For now this proc only passes args to Passline2.

The third thing to do: Remove all exceptions below Parsline2 that contain code that test for the if (tokencount > 1 && token[0] == label).

The fourth thing to do: mail me the updated code. I will then insert a ugly hack into the Parseline proc and return the binary for testing. If the binary pass the test code (not the Regression test), then you will remove the labels from the first test and produce a new one, only including the HLL functions.

The next step will be to implement full support for the HLL code, and test to see that this work as expected.

Then comes the ugly Regression test that will reveal all the bugs created in this process. I will the retreat back into a neutral corner and stay very quiet for a month or two till somebody else fixes the mess created. :P

jj2007

  • Member
  • *****
  • Posts: 7542
  • Assembler is fun ;-)
    • MasmBasic
Re: Resource compiler for console applications
« Reply #33 on: October 27, 2012, 01:03:49 AM »
Hi nidud,

AFAICS the problem is mainly relevant for two cases: elseif macro() and or'ed macros like this:
   ; Let esi="This is a stupid test"
   .Break .if Instr_(esi, "stuupid")   ; works correctly
   ; both macros get expanded at the label, but only the second one gets tested
   .Break .if Instr_(esi, "stuupid") || Instr_(esi, "stupid")

There is, of course, a simple workaround for the latter case: two .Break .if lines.
Again, a warning for the noobs would be luxury, but Japheth seems not so eager to dig into it, so I'll pull out and leave him in peace  :icon14:

nidud

  • Member
  • *****
  • Posts: 1370
    • https://github.com/nidud/asmc
Re: Resource compiler for console applications
« Reply #34 on: October 27, 2012, 04:12:12 AM »
I think I have solved the label issue.
I moved the expansion from GetPreprocessedLine to the ParseLine proc,
below the create a global or local code label function.

I compiled 1264 files, 16 and 32-bit, no problem.

However, using the –Zm switch created a General Failure on this code:
Code: [Select]
assert al,0,jne,"tiflushl: al=0"
And the problem line in the macro: (%)
Code: [Select]
% db "cmp &A&,&B& (&J&)  -: ",message,0

The next trick is to halt expansion of these two:
.elseif macro
.while macro
And expand them in the HLL section (with or without macros that is).


jj2007

  • Member
  • *****
  • Posts: 7542
  • Assembler is fun ;-)
    • MasmBasic
Re: Resource compiler for console applications
« Reply #35 on: October 27, 2012, 05:00:44 AM »
The next trick is to halt expansion of these two:
.elseif macro
.while macro

Right, I forgot the .While macro() case in my post above.

nidud

  • Member
  • *****
  • Posts: 1370
    • https://github.com/nidud/asmc
Re: Resource compiler for console applications
« Reply #36 on: October 27, 2012, 07:53:41 AM »
Ok, I’m in the HLL section :P

This code is called by GetPreprocessedLine:
Code: [Select]
ret_code ExpandLine_xlabel( char *string, struct asm_tok tokenarray[] )
{
    if (Token_Count > 2 && (tokenarray[1].token == T_COLON || tokenarray[1].token == T_DBL_COLON))
return 0;
    return ExpandLine( string, tokenarray );
}
At the end of ExpandLine:
Code: [Select]
std_expansion:
    i = 0;
    for (j = 0; j < Token_Count; j++) {
    if (!stricmp(tokenarray[j].string_ptr,".elseif") ||
        !stricmp(tokenarray[j].string_ptr,".while")) {
    i = 1; /* block expansion */
    break;
}
    }
    for(  ; count < Token_Count; count++ ) {
if ( ExpandToken( count, tokenarray, string, addbrackets, i ) == STRING_EXPANDED )
    rc = STRING_EXPANDED;
    }
The proc GetExpression(“.elseif func(foo)”)
Code: [Select]
    for ( ;; ) {
        ptr = ptr + strlen( ptr );
        cur_pos = *i;
        if ( ERROR == GetAndExpression( hll, i, tokenarray, ilabel, is_true, ptr, lastjmp, opndx ) )
            return( ERROR );
Return
Code: [Select]
elseif.asm(13) : Error A2243: Invalid symbol type in expression: func
Not shore how to do the actual expansion here, RunMacro() ?


nidud

  • Member
  • *****
  • Posts: 1370
    • https://github.com/nidud/asmc
Re: Resource compiler for console applications
« Reply #37 on: October 27, 2012, 08:12:52 AM »
I forgot..
The new declaration for ParseLine:
Code: [Select]
ret_code ParseLine ( char *line, struct asm_tok tokenarray[] )
Inserted below label function:
Code: [Select]
    /* expand the line */
    if ( CurrIfState == BLOCK_ACTIVE ) {
/* expand (text) macros. If expansion occured, rescan the line */
while ( Token_Count > 0 && ExpandLine( line, tokenarray ) == STRING_EXPANDED ) {
    DebugMsg1(("GetPreprocessedLine: expanded line is >%s<\n", line));
    Token_Count = Tokenize( line, 0, TRUE );
}
    }

    /* handle directives and (anonymous) data items */
I just copyed this in there, so this needs some help.

japheth

  • Guest
Re: Resource compiler for console applications
« Reply #38 on: October 27, 2012, 06:46:02 PM »
I forgot..
The new declaration for ParseLine:
Code: [Select]
ret_code ParseLine ( char *line, struct asm_tok tokenarray[] )
Inserted below label function:
Code: [Select]
    /* expand the line */
    if ( CurrIfState == BLOCK_ACTIVE ) {
/* expand (text) macros. If expansion occured, rescan the line */
while ( Token_Count > 0 && ExpandLine( line, tokenarray ) == STRING_EXPANDED ) {
    DebugMsg1(("GetPreprocessedLine: expanded line is >%s<\n", line));
    Token_Count = Tokenize( line, 0, TRUE );
}
    }

    /* handle directives and (anonymous) data items */
I just copyed this in there, so this needs some help.

Brrr, what an ugly mess - just for such a VEEEEERRRRY minor issue!? If you mix preprocessor and parser functionality like in your proposal, you'll end in a totally unreadable and unmaintainable pile of sh...

btw, you don't want to query CurrIfState once the conditional assembly directives (IF, ELSE,...) have been handled in the preprocessor.

nidud

  • Member
  • *****
  • Posts: 1370
    • https://github.com/nidud/asmc
Re: Resource compiler for console applications
« Reply #39 on: October 27, 2012, 08:27:29 PM »
Brrr, what an ugly mess - just for such a VEEEEERRRRY minor issue!? If you mix preprocessor and parser functionality like in your proposal, you'll end in a totally unreadable and unmaintainable pile of sh...

btw, you don't want to query CurrIfState once the conditional assembly directives (IF, ELSE,...) have been handled in the preprocessor.
My mane augment is the same:
I expect the same capability of the IF statement as the ELSEIF statement: there is logic in the argument.

So, humour me a bit here without wasting too much energy on it :bgrin:

Lets take a .break from the .while issue and focus on the label situation for a moment.
The code below will illustrait the problem:
Code: [Select]
l1: mov edx,func(foo)

00000000      1  invoke foo
00000000  E800000000        *1   call foo
00000005  8BD0  l1: mov edx,eax
The line argument may be replaced with CurrSource here and just use the old declaration of Parseline.
Code: [Select]
    if (i == 2) {
while (Token_Count > 0 && ExpandLine(line, tokenarray) == STRING_EXPANDED)
    Token_Count = Tokenize(line,0,TRUE);
    }
Good (or at least better)?

result:
Code: [Select]
00000000 l1:
00000000      1  invoke foo
00000000  E800000000        *1   call foo
00000005  8BD0  mov edx,eax

japheth

  • Guest
Re: Resource compiler for console applications
« Reply #40 on: October 27, 2012, 09:55:58 PM »
Good (or at least better)?

I don't dare to judge.

The only approach that has a faint chance to be implemented is to delay macro expansion by some kind of syntax extension. You can see this approach in effect in struct initialization, when a macro call is located within the initialization literal. Quite the same approach may be added to the hll directives, that is, the macro call has to be enclosed in <>:

    .if <whatever()>

with this approach, you just have to add a few lines in hll.c ( determine if there's a literal following the directive and if so, expand the rest of the line ), the rest of the code remains as it is.


nidud

  • Member
  • *****
  • Posts: 1370
    • https://github.com/nidud/asmc
Re: Resource compiler for console applications
« Reply #41 on: October 28, 2012, 06:26:02 AM »
I looked at the source code for v2.08, and there have been some major changes there. And, as you mention above, a delayed expansion mechanism are already included. This means that I have to wait for a stable source base to play with (v2.08 have some issues, so the last stable source is still v2.06).

I did however apply some changes to the current version. The code above was removed (didn’t work very well), and a more brutal approach was inserted instead (oh yes, its very very ugly): :lol:
Code: [Select]
    /* Does line start with a code label? */
    if ( tokenarray[0].token == T_ID && ( tokenarray[1].token == T_COLON || tokenarray[1].token == T_DBL_COLON ) ) {
DebugMsg1(("ParseLine T_COLON, code label=%s\n", tokenarray[0].string_ptr ));
if( DefineProc == TRUE ) write_prologue();
/* create a global or local code label */
if( CreateLabel( tokenarray[0].string_ptr, MT_NEAR, NULL,
    ( CurrProc != NULL && tokenarray[1].token != T_DBL_COLON  ) ) == ERROR ) {
    DebugMsg(("ParseLine, CreateLabel(%s) failed, exit\n", tokenarray[0].string_ptr ));
    return( ERROR );
}
if (tokenarray[2].token != T_FINAL)
    AddLineQueue(tokenarray[2].tokpos);
strcpy(CurrSource, tokenarray[0].string_ptr);
strcat(CurrSource, tokenarray[1].string_ptr);
Token_Count = Tokenize(CurrSource, 0, TRUE);
FStoreLine();
if (CurrFile[LST])
    LstWrite(LSTTYPE_LABEL, 0, NULL);
return( NOT_ERROR );
    }
from
label: <whatever>
to
label:
<whatever>

.while and .elseif are now expanded in the hll.c file: (still incorrectly)
Code: [Select]
    case T_DOT_WHILE:
for(q = i; q < Token_Count; q++) {
        if (ExpandToken(q, tokenarray, CurrSource, 0, 0) == STRING_EXPANDED)
Token_Count = Tokenize( CurrSource, 0, TRUE );
    else
break;
}
...
    case T_DOT_ELSEIF:
for(q = i+1; q < Token_Count; q++) {
        if (ExpandToken(q, tokenarray, CurrSource, 0, 0) == STRING_EXPANDED)
Token_Count = Tokenize( CurrSource, 0, TRUE );
    else
break;
}
    case T_DOT_ELSE:
Strangely enough this seems to work very well, but a regression test will probably tell a different story.

jj2007

  • Member
  • *****
  • Posts: 7542
  • Assembler is fun ;-)
    • MasmBasic
Re: Resource compiler for console applications
« Reply #42 on: October 28, 2012, 09:26:45 AM »
...but a regression test will probably tell a different story.

Unfortunately yes. My little testbed (attached) with the .While Instr_(esi, ... stuff reports plenty of errors, while the latest JWasm build (25-Oct-2012 04:03) works just fine.
Strangely enough, your modified version assembles my fat testbed without any problems. So it must really be the specific "warning" code.

In the meantime, I have instructed my editor (RichMasm, part of the MasmBasic package) to issue a warning for
.While mac()
and
.elseif mac()
- imho the most relevant cases. That was really hard work for a noob who understands only BASIC, but it seems to fit the purpose, and the performance loss is acceptable: overall build time with JWasm is 0.2% slower for the RichMasm source with its 14,000 lines :biggrin:

nidud

  • Member
  • *****
  • Posts: 1370
    • https://github.com/nidud/asmc
Re: Resource compiler for console applications
« Reply #43 on: October 28, 2012, 08:36:01 PM »
You could read the error out of the expansion code above. The break statement must be removed. They will only work if the first token is a macro, if not, the expansion of the rest of the line is dropped.

This works: .while mac() > edx
This will fail: .while edx <= mac()

However, you may have to look at the ones that fail since they could expand code in the wrong place.

I compiled two new versions, one that only fixes the label issue, and one that delay the .elseif and .while.

nidud

  • Member
  • *****
  • Posts: 1370
    • https://github.com/nidud/asmc
Re: Resource compiler for console applications
« Reply #44 on: October 28, 2012, 09:51:47 PM »
Problem executing .EXE in local directory

This configuration will not execute jwasm in the current directory:
Quote
[ShiftF1]
asm=err jwasm -I%doszip%\include -Fl -Sg -Zd

If I delete the version in the %PATH% it will, or if I use .\jwasm.exe.
In DOS the current directory will be scanned first, then the %PATH%.
So, is this normal behavior?

Code: [Select]
; PIPE.ASM--
; Error Message Pipe for Edit
;
; PIPE <command> <args>
;
; Creates a buffer of stdout and stderr from Shell(command)
; If Shell() returns a value (errorlevel) ParseOutput() is called
;
include clib.inc
include io.inc
include dir.inc
include dos.inc
include alloc.inc
include stdio.inc
include string.inc
include process.inc

; extern parser for output
ParseOutput proto stdcall Buffer:dword, bSize:dword

.code

BUFFERSIZE equ 40000h

main proc c uses ebx edi esi
local cmd[WMAXPATH]:byte
local sa:SECURITY_ATTRIBUTES
local pi:PROCESS_INFORMATION
local sinfo:STARTUPINFO
local hRd:dword
local hWr:dword
local hDup:dword
local buffer:dword
local dwCount:dword
local ExitCode:dword
mov edi,_argc
mov esi,_argv
.if _argc > 1
    lea ebx,cmd
    lodsd
    lodsd
    invoke strcpy,ebx,eax
    sub edi,2
    .if edi
    .repeat
    invoke strcat,ebx,cstr(" ")
    lodsd
    invoke strcat,ebx,eax
    dec edi
.until !edi
    .endif
    sub eax,eax
    mov ExitCode,eax
    mov dwCount,eax
    lea edi,sinfo
    mov ecx,sizeof(STARTUPINFO)/4
    rep stosd
    mov sa.bInheritHandle,1
    mov sa.lpSecurityDescriptor,0
    mov sa.nLength,sizeof(SECURITY_ATTRIBUTES)
    .if func(CreatePipe,addr hRd,addr hWr,addr sa,0)
call GetCurrentProcess
mov edi,eax
invoke DuplicateHandle,edi,hRd,edi,addr hDup,0,0,DUPLICATE_SAME_ACCESS
mov esi,eax
invoke CloseHandle,hRd
.if esi
    mov sinfo.cb,sizeof(STARTUPINFO)
    invoke GetStartupInfoA,addr sinfo
    mov eax,hWr
    mov sinfo.hStdOutput,eax
    mov sinfo.hStdError,eax
    mov sinfo.dwFlags,STARTF_USESHOWWINDOW+STARTF_USESTDHANDLES
    mov sinfo.wShowWindow,SW_HIDE
    .if func(CreateProcessA,0,addr cmd,0,0,10h,0,0,0,addr sinfo,addr pi)
invoke CloseHandle,hWr
mov esi,BUFFERSIZE
.if func(GlobalAlloc,GMEM_FIXED,esi)
    mov buffer,eax
    mov edi,eax
    sub eax,eax
    mov ecx,BUFFERSIZE/4
    rep stosd
    mov edi,buffer
    .repeat
invoke ReadFile,hDup,edi,esi,addr dwCount,0
.break .if !eax
mov eax,dwCount
add edi,eax
sub esi,eax
.break .if !esi
    .until !eax
.endif
      .repeat
    invoke GetExitCodeProcess,pi.hProcess,ADDR ExitCode
.until ExitCode != STILL_ACTIVE
invoke CloseHandle,pi.hProcess
invoke CloseHandle,pi.hThread
mov eax,BUFFERSIZE
sub eax,esi
.if eax
    invoke ParseOutput,buffer,eax
.endif
.if buffer
    invoke GlobalFree,buffer
.endif
sub eax,eax
    .else
    invoke _print,cstr(<"Unable to create process",10>)
mov eax,1
    .endif
.else
        invoke _print,cstr(<"Unable to duplicate handle",10>)
        mov eax,1
.endif
    .else
    invoke _print,cstr(<"Unable to create pipe",10>)
    mov eax,1
    .endif
.else
    invoke _print,cstr(<"Usage: PIPE <command> <args>",10>)
    sub eax,eax
.endif
    ret
main endp

end