The MASM Forum

64 bit assembler => UASM Assembler Development => Topic started by: HSE on February 08, 2019, 12:18:48 PM

Title: Long lines UASM32
Post by: HSE on February 08, 2019, 12:18:48 PM
Hi UASM team!

   I'm triyng to build a UASM32. Apparently inttypes.h is missing (included in macho64spec.h).

        Because I have others errors wich maybe I can't solve, perhaps some of you can build a release with long lines, macrolevels, etc. In globals.h:
Code: [Select]
#define MAX_LINE_LEN            2048  /* no restriction for this number */
#define MAX_TOKEN  MAX_LINE_LEN - 32  /* max tokens in one line */
#define MAX_STRING_LEN          MAX_LINE_LEN - 32 /* must be < MAX_LINE_LEN */
#define MAX_ID_LEN              247  /* must be < MAX_LINE_LEN */
#define MAX_STRUCT_ALIGN        32
#define MAX_SEGMENT_ALIGN       4096 /* maximum alignment/packing setting for segments */
#define MAX_IF_NESTING          32 /* IFxx block nesting. Must be <=32, see condasm.c */
#define MAX_SEG_NESTING         32 /* limit for segment nesting  */
#define MAX_MACRO_NESTING       200 /* macro call nesting  */
#define MAX_STRUCT_NESTING      32 /* limit for "anonymous structs" only */
#define MAX_LNAME              255 /* OMF lnames - length must fit in 1 byte */
#define LNAME_NULL             0   /* OMF first entry in lnames array */

Thanks.
Title: Re: Long lines UASM32
Post by: jimg on February 08, 2019, 01:19:05 PM
deleted
Title: Re: Long lines UASM32
Post by: KradMoonRa on February 08, 2019, 02:55:16 PM
2.47.2 version compiled with large lines and extra macros lib

Check source x86macros.c for see wath happens behind the woods for extra macros lib.

Code: [Select]
https://github.com/eXOAMP/UASM/releases/download/eX_v2.47.2/Uasm-v2.47.2.7z
https://github.com/eXOAMP/UASM/archive/eX_v2.47.2.zip
Title: Re: Long lines UASM32
Post by: HSE on February 09, 2019, 12:58:30 AM
 :biggrin: :biggrin: :biggrin:

@Krad:
Just replace in globals.h the lines posted above.

The maximum line size I measured ever was around 1680 bytes after expansion.
Title: Re: Long lines UASM32
Post by: KradMoonRa on February 09, 2019, 05:20:46 AM
Copy that,

I just use large macros and naming in multi line manner with spaces and line breaks has a lot of hasted characters.

Recompiled with,

Code: [Select]
#define MAX_LINE_LEN            25600  /* no restriction for this number */
#define MAX_TOKEN  MAX_LINE_LEN - 32  /* max tokens in one line */
#define MAX_STRING_LEN          MAX_LINE_LEN - 32 /* must be < MAX_LINE_LEN */
#define MAX_ID_LEN              MAX_LINE_LEN - 32 /*247*/  /* must be < MAX_LINE_LEN */
#define MAX_STRUCT_ALIGN        64
#define MAX_SEGMENT_ALIGN       4096 /* maximum alignment/packing setting for segments */
#define MAX_IF_NESTING          32 /* IFxx block nesting. Must be <=32, see condasm.c */
#define MAX_SEG_NESTING         32 /* limit for segment nesting  */
#define MAX_MACRO_NESTING       200 /* macro call nesting  */
#define MAX_STRUCT_NESTING      32 /* limit for "anonymous structs" only */
#define MAX_LNAME              255 /* OMF lnames - length must fit in 1 byte */
#define LNAME_NULL             0   /* OMF first entry in lnames array */

https://github.com/eXOAMP/UASM/releases/download/eX_v2.47.2/Uasm-v2.47.2.7z
https://github.com/eXOAMP/UASM/archive/eX_v2.47.2.zip
Title: Re: Long lines UASM32
Post by: HSE on February 09, 2019, 05:57:27 AM
No problem Krad. Thanks anyway.  :t

My simplest projects have a thousand lines "before" expansion. There is no memory for 25kb lines, just including windows.inc (one line) you have 27000 lines.
Title: Re: Long lines UASM32
Post by: HSE on February 11, 2019, 09:57:50 AM
Solved!!

I found that UASM code only can be easily compiled by V$  :biggrin:

There is a bunch of modules  that depends of functions that are not in the code but apparently  they are injected by V$ at compilation time  :eusa_boohoo:

Regards.
Title: Re: Long lines UASM32
Post by: johnsa on February 11, 2019, 08:24:40 PM

For building it on Windows I only use VS. For Linux and OSX builds it works with gcc and clang + respective make files.

The default setting in the source at present is 1024 i believe for max line len. We could up this slightly.
Title: Re: Long lines UASM32
Post by: jj2007 on February 11, 2019, 08:59:48 PM
The default setting in the source at present is 1024 i believe for max line len.

I made a quick test: Let esi="12 ...78", it starts choking at 990+ characters with "line too long after expansion". More than enough for my taste, but others may have different needs. Does it slow down assembly of a "normal" source if you allow longer lines?

Masm 10 chokes at 255 chars, same for 6.15.
Title: Re: Long lines UASM32
Post by: johnsa on February 11, 2019, 11:10:12 PM
I'm not sure if it would affect the performance without testing it. In theory it shouldn't, as line operations should only act on what is actually stored in the line.
Title: Re: Long lines UASM32
Post by: HSE on February 12, 2019, 12:20:45 AM
For building it on Windows I only use VS. For Linux and OSX builds it works with gcc and clang + respective make files.
V$ was the last to test, it's not in this machine. I don't know if some switch can change the behaviour or just Win DDK it's designed in that way. C++ configuration is very complex, not so easy like assembly.

The default setting in the source at present is 1024 i believe for max line len. We could up this slightly.
Increasing line len to 2048 result in out of memory here. For that I increased stack and heap reserved and committed, and work for ObjAsm32 programs, but apparently size I randomlly choose it's not  enough for ObjAsm Fusion because have more includes by default .  Asmc (rebuilded with more macrolevels) work with 2048 without any trouble, just don't understand well some macro used in ObjAsmF.
Title: Re: Long lines UASM32
Post by: johnsa on February 12, 2019, 03:16:13 AM
I would think the same compiler settings from asmc should be portable to uasm with ddk or vs and the result should be the same with extra macro levels + line length ?
Title: Re: Long lines UASM32
Post by: HSE on February 12, 2019, 06:29:01 AM
the same compiler settings from asmc should be portable to uasm
I don't think so. AsmC is maked in part in assembly, and mainly use a megalibrary totally in assembly (builded with AsmC). The other part use Watcom libraries and tools.

Just a few thing are missing now to build UASM32 without V$: _alloca_probe_16 (V$'s original depens on vcruntime but perhaps is possible to replace with alloca16.obj from ObjAsm), AddSimdTypes (all the time here) and _report_rangecheckfailure (swich /GS- ). 

LATER: Perfect  :t
Title: Re: Long lines UASM32
Post by: KradMoonRa on February 12, 2019, 07:36:11 AM
I would think the same compiler settings from asmc should be portable to uasm with ddk or vs and the result should be the same with extra macro levels + line length ?

For portability with diverse OS WINDOWS UNIX/LINUX and mach, My assembler reference it is UASM, I achieved wath I wanted to do in one simple manner, that is make the procedures names available for the easy portable call-conventions and OS kernel, in a C runtime library format, with less or none cascade jump-call hastes or/and security cookies catch blocks.

I most say UASM in 64bits assemble time, it easily afford large lines, like 43x6x24=6,192 characters for the jump-table offsets names. It assemble in less than 3 seconds. Awesome...

For array lines untested, still no practical real situation to see wath will break the assemble time.

ASMC has cool things, like UASM also has cool things, both produce fast assemble time and portable between each other.  :t
Title: Re: Long lines UASM32
Post by: TimoVJL on February 12, 2019, 09:04:42 PM
UASM sources are for C++ compilers :(
A pure C and C99 C compiler could have some problems with it.

- types.h line needs a fix: #ifndef _TYPES_H_INCLUDED_
- macho64.c needs  #include <stdint.h>
- winconsole.h use useless annotations ?

Clang can't use WDDK 7.10 headers, where in math.h is struct  _complex _Complex. Also it don't have stdint.h inttypes.h
A fix:  _CRTIMP double  __cdecl _cabs(__in struct _complex __Complex);

Some speed tests with \masm32\m32lib\*.asm , one file at a time ;)
Code: [Select]
asmc.exe 8.845s
uasm64.exe 10.701s
hjwasm64poc.exe 10.686s
uasm32.exe 14.398s
uasm64msvcrt-cl.exe 19.16s clang 8 + msvcrt.dll
uasm64msvcrt.exe 20.732s msvc 19.16 + msvcrt.dll
ml.exe 27.690s 14.15.26726
uasm32msvcrt.exe 31.371s msvc 19.16 + msvcrt.dll
uasm32msvcrt-cl.exe 31.90s clang 8 + msvcrt.dll
with -c -coff \masm32\m32lib\*.asm
Code: [Select]
5.506s        asmc.exe
8.424s        uasm64.exe
8.486s        uasm64_msvc2010.exe
8.751s        uasm64poc.exe
9.640s        uasm32_msvc2010.exe
10.264s       uasm32.exe
10.342s       uasm32poc.exe PellesC 9
10.452s       uasm32vc3.exe VC Toolkit 2003
17.924s       uasm64gcc.exe gcc 7
18.111s       uasm64msvcrt.exe
26.910s       uasm32gcc.exe gcc 7
27.549s       uasm32msvcrt.exe
EDIT: NotePad++ nice for patch-files :icon_cool:
Title: Re: Long lines UASM32
Post by: johnsa on February 13, 2019, 08:26:48 PM
There are definitely a bunch of non-pure C things through out the source which would need cleaning up if we wanted it to compile with any true C/C99 compiler. I guess the important question is..
is it worth it ? Who doesn't have a C/C++ compiler ? I'm open to suggestions on this..
Title: Re: Long lines UASM32
Post by: TimoVJL on February 13, 2019, 09:50:01 PM
Quote
UASM is written in C. The source is portable and has successfully been tested with Open Watcom, MS VC, GCC and more.
;)
Why limit the msvc version requirement to the minimum of 2013?
Title: Re: Long lines UASM32
Post by: johnsa on February 13, 2019, 10:22:37 PM
Fair call ;)

I'm happy to look at making the necessary changes for 2.49 then. 2.48 is just about out the door anyway, it's been taking quite some time!
Title: Re: Long lines UASM32
Post by: TimoVJL on February 14, 2019, 09:47:14 AM
Good to hear that.
In my tests in windows gcc and clang are quite weak in this kind of sources.
Clang don't have own crt, it use M$ crt?
If someone have an icc, please give us compiled exe to test.

BTW: msvc is a right choice for Windows version, as version doesn't matter ;)
A big thanks for Andrea, Branislav, John :icon_cool:
Title: Re: Long lines UASM32
Post by: KradMoonRa on February 17, 2019, 10:08:50 AM
2.48b version compiled with large lines and extra macros lib, and some clang fixe's.

Folder structure compiler used.

v141 = Visual Studio 2017 c++ compiler v15
v141_ICC19 = Visual Studio 2017 Intel c++ Compiler 19
v141_LLVM = Visual Studio 2017 LLVM-Clang c++ Compiler 9.0

Code: [Select]
https://github.com/eXOAMP/UASM/releases/download/eX.v2.48b1/Uasm-v2.48b.7z
https://github.com/eXOAMP/UASM/archive/eX.v2.48b1.zip
Title: Re: Long lines UASM32
Post by: TimoVJL on February 17, 2019, 08:23:02 PM
So the  increased line length and nesting slowed down the speed :(
Code: [Select]
9.438s        uasm64.exe
23.478s       x64\uasm64icc.exe
24.273s       x64\uasm64llvm.exe
24.788s       x64\uasm64v141.exe
Title: Re: Long lines UASM32
Post by: jj2007 on February 17, 2019, 09:20:21 PM
So the  increased line length and nesting slowed down the speed :(

That looks pretty significant - a factor 2.5! Since it's caused by two MAX values, it seems a problem of logic. Have you tested decreasing those values, with a source that doesn't require long lines and/or deep nesting?

Sometimes I feel tempted to test MasmBasic's fast string array engine on UAsm, but unfortunately I am not the right person to dig inside C/C++ code :(
Title: Re: Long lines UASM32
Post by: TimoVJL on February 17, 2019, 09:41:05 PM
Only a one test: MAX_LINE_LEN
Code: [Select]
9.594s        uasm64poc.exe 1024
10.826s       uasm64poc4k.exe 4096
11.388s       uasm64poc25K.exe 25600
EDIT: patch files for version 2.48 in git.
Should help to compile uasm with older msvc too.
Title: Re: Long lines UASM32
Post by: KradMoonRa on February 17, 2019, 10:36:34 PM

Uasm-v2.48b1nolinenox86lib.zip has no modifications for large lines lenght and x86macros addition.
Uasm-v2.48b1withlinex86lib.zip has both modifications described above.

Code: [Select]
https://github.com/eXOAMP/UASM/releases/download/eX_v2.48b1/Uasm-v2.48b1nolinenox86lib.zip
https://github.com/eXOAMP/UASM/releases/download/eX_v2.48b1/Uasm-v2.48b1withlinex86lib.zip
https://github.com/eXOAMP/UASM/archive/eX_v2.48b1.zip
Title: Re: Long lines UASM32
Post by: TimoVJL on February 17, 2019, 11:15:06 PM
Using that KradMoonRa global.h settings the speed difference was only 5% slower.
I still wait a good icc version to test, but if it just use a msvc runtime, nothing changes and in Windows msvc is a best choice.
Title: Re: Long lines UASM32
Post by: johnsa on February 18, 2019, 12:20:26 AM
Hey,

I've been going through some of the patches and fixes and pushed a bunch into 2.48 (updated today).

it's not complete, and given my limited time I'm finding it quite hard to keep track of various messages, emails and forum posts all simultaneously.

If you guys take 2.48 master branch as it is today and generate a fork, could I ask if we could do other changes via Pull Requests ? That way I can keep track of it, and we can ensure we're keeping all the right non-experimental changes in  ?

I'll start a 2.49 branch shortly which we can use to then pull in all the remaining C/C99 and long-line changes. I would definitely consider looking at optimising the tokenizer for 2.49 so that long-lines shouldn't make a difference unless the line IS actually long.

PS Today's update included a fix that Biterider pointed out which is long-standing allowing 64bit imm's to go to memory without error.

John