HJWasm

[rrr314159]

The topic has come up "how to improve HJWasm". (Everyone thought the telepathy idea was impractical.)

First, I'm well aware that I need to learn the program (a few month's work) before making suggestions about improving it. Unfortunately I'm not going to do that in any big hurry.

This brings up the key point: lack of people who know, and can work on, HJWasm. It's a pretty big project for one part-timer (two or three full-timers would be better). For this situation, "job #1" (as Ford used to say) should NOT be "improvements" but maintenance, basic bug-free operation - the nitty-gritty stuff that coding is mainly about. You know, many improvements are easy enough to implement, but every change, even a simple one, requires regression testing which can be a low-level (or major) pain for weeks. Only one type of "improvement" is necessary: upgrades, like RIP, AVX2 or 512.

Seems to me this is quite enough work for one part-time transplanted Aussie; plus habran throws in a few of his favorite improvements now and then, like new flugs. Obviously he can do whatever he wants (as far as I'm concerned; the police might object) but I'm figuring "next big thing" (to quote johnsa) improvement projects would not be a good idea. So until someone else masters HJWasm code (which is certainly an option) we (= everyone but habran) need to give up on the idea of major "improvements" to it.

Fortunately this doesn't mean we can't help improve HJ's effectiveness. One way is to write MACROs - if you don't know already, you'll be amazed what can be done with them. The other way: auxiliary progs that support and work with HJ. That way anyone can do it, in any language they want; habran (or anyone else) doesn't even have to know about it.

There are various types of supporting programs. For instance jj's bug catcher (a "lint" program) analyzes source code, independent of the assembler, in the hope of catching mistakes. But the type I'm most interested in is a pre-processor aux prog, which first massages the ("improved") source code then passes it along to HJ as legal code. With a batch file to call the preprocessor, then HJ (you can even use DOS pipes), it's transparent to the user and takes no time at all.

Of course I didn't invent the idea of pre-processors, HJ and almost every other language has one built-in (some, of course, have separate pp's as I recommend here). The built-in aspect is the problem: to modify HJWasm's preprocessor (macros, etc) you have to modify HJ itself. I'd prefer that HJ had more-or-less none of that stuff, it was all separate, but that's not necessary. HJ can be left exactly as is, with its (primitive but effective) preprocessing capability, but new stuff can be separate.

There are two good reasons to do it this way. The key problem it solves is manpower allocation. What you want is part-time people around the world able to work together. Of course they could work together on HJWasm but that's extremely not easy to coordinate. Instead let them work on separate progs entirely. The usually-big problem of coordinating the interface spec just doesn't exist here. The input to HJWasm is perfectly well known, and anyone writing a pre-processor can try it out on HJ anytime. When they have something worthwhile just post it. I can imagine habran getting involved somewhat; people might request some mod to the interface; but essentially it's a non-problem.

What would one do in one's preprocessing program? Well, I think the best thing about the H/J/Wasm line of assemblers (ML too of course) is actually the macros. You can (with limitations) make it do any HLL function, from simple math to classes with inheritance etcetera. But there are limitations. First it's a very primitive language, with IF statements, loops, simple assignment, some arithmetic, that's all. It should be (a bit) more sophisticated, along the lines of C I would say (the HLL ".if" statements are good - I'd like to see more of that). So I can write a prog to take in a text file (source code with a few new keywords I've invented: CLASS, or SCOPE, curly brackets, MAX, COS, whatever); change the code appropriately; and output a text file with legal HJ code. It's not as easy as it sounds but it's entirely do-able.

Other ideas: it's annoying that a macro can't get certain characters such as a semicolon. I'd like to grab the entire line for parsing and preprocessing - semicolons and all. It could be syntaxed like "comment": grab everything between "pre-comment #" and "#", process as desired, paste the processed code back into the file, ship the legal code to HJ. That would allow, for instance, statements like "myvariable + myothervariable" with the + sign under my control - like overloading an infix operator - very readable. One could include debugging capability for one's preprocessor; and/or gradually implement HJ's macro capability also (write your own "REPEAT" for instance, and stop emitting to HJ), for functional modularity. BTW that's another good point, such a prog can be implemented incrementally; just one well-chosen simple feature, to start, could be quite useful.

The possibilities are endless, but my time/energy/enthusiasm isn't, and I doubt very much I'll do any of this. The point is, someone else can, independently (supported, of course, by the community, who provide ideas, testing, and applause). Six months should be enough to get somewhere. It could possibly in future be integrated with HJ, who knows. Would make a more convenient build, speed it up 25 or 33%, and integration would make it possible to emit better error messages; but it's not necessary.

To quote some of our most eminent members,

Johnsa, if I decide to use some Hll, it would be in the first place preprocessor for precompiled headers

However, why not use a pre-processor, instead of burdening the assembler with this "external" task?

- I agree!

the HLL code could use variables exactly like a normal HLL

- I like your idea of "inline HLL". MasmBasic is something along these lines, but different. Don't you agree it could be done with a separate program as outlined above? A fair amount of start-up is necessary, dealing with the basics like includes and who knows what else, but in a few months you could make a good start on MasmCPlusPlus.

In addition it would be nice to add CLASS and METHOD keywords, which do something similar to our OOP macro layers, but built-in with implicit THIS ptr if it's declared as a method and not a proc.

- You know this can be done pretty well by building on structs. The only real stumbling block, it's impossible to get nice clean OOP syntax ("pre-comment" could cure that). Methods are worthwhile: you just define a code-label pointer as a struct element and write some supporting macros, like "method MACRO myObject, myMethod" to invoke it. E.g, "method CustomerData, PRINT_VERBOSE" meaning CustomerData.PRINT_VERBOSE ("THIS" can be done also of course). The syntax is, necessarily and unfortunately, very clumsy - but that's curable with your new pre-processor prog (JOHNSAWasm).

Either there is no point in us using asm anymore, or we should be dreaming up "the next big thing".

- hmmmm... There was a time I felt there was no point in getting out of bed in the morning if I couldn't achieve something remarkable that day. Now I realize: getting out of bed in the morning is the day's remarkable achievement! Similarly asm is its own justification, doesn't need be next or big.

We had started "FindBugs for Assembler" two years ago, but it was limited to fairly simple things, such as comparing the number of pushs and pops in a procedure.

- One can't go too far with this "telepathy" or "lint" approach. For instance (as discussed in that thread) unmatched pushes and pops are perfectly legal in asm. And, a variable declared as dword might be used as real4 or even char, or of course pointer - there's essentially no type-checking, which is good. You could police proc args via the proto, I suppose; but as u said in that thread, makes sense for a noob tool, that's all.

- But perhaps a concordance might be useful: show all uses of eax or whatever, with options: only assignments, only pushes and pops, include the lines above and below, etc. Also proc/subroutine list, (alpha order, frequency order, proto's, .inc file and lib, etc), etc. And, a flow chart (both for asm code and preprocessor). In general, a prog which shows various aspects of the code, just for info not correction. It should be very aware of macros, equ's: for instance, show the actual term, or else the expanded term, or both side by side, etc. If it understood how &works it could be a real boon. Might as well throw in pretty-print while you're at it. When you're done take a break for lunch then get back to work on MasmBasic :)

[habran]

Hi rrr,
Surprisingly enough I had enough patience to read all through what you wrote, well done :t
I like your way of thinking and I seriously suggest you to write some book soon 8)
I agree with most of things you said and especially with this:

Now I realize: getting out of bed in the morning is the day's remarkable achievement! Similarly asm is its own justification, doesn't need be next or big.

One has to grow up to understand the meaning of that

I am busy at the moment with my w**k, and the most of my little free time I spend on finishing AVX512
As you can Imagine, it is very complex job with all of these new instructions and registers,
However, I am looking forward to the power we will get with them

My idea is to use precompiled headers instead of WinInc, that way we don't have to spend time and effort to keep up with all those new stuff

So, someone of you EMINENT programmers should roll the sleeves and get dirty
The preprocessor should be able to precompile both, 32 and 64 bit

[johnsa]

Hi,

I agree that HJWASM should simply continue to develop through small bug-fixes and enhancements like rip, avx2, avx512.
As a "pure assembler" (assuming you include the basic HLL support and macros) it's pretty close on perfect with all those features in.
There isn't much more I could ask of it or want it to do.

I definitely think that any of these larger projects/ideas we have floating around are far too big for any one person to take on even full-time. We need a community effort to create them. They should either be new products or auxiliary products to complement the assembler.

I started some time back on an idea for a compiler which i called "fusion". The idea was that it supports multiple compilers (extensible even) through plugins. Each compiler shared a common core interface to sections
symbol tables, emit data, global state and so on. The objective was to be able to in a single source file mix and match whichever languages you want even with different execution targets.
In the Amiga days we used to code up copper-lists and stick them in the chip_mem section, which is kind of what inspired the idea.
Xeon Phi and shader code is an example, we could directly compile these in as part of a single source.

I think in the future than even high level languages will land up going this way as we realize that no one language is ideally suited to all problem domains. C# is a classic example of how we're already trying to jam different language-styles into one compiler, however I'm not totally convinced about the approach, or at least if it should be taken much further than it is without seriously harming the core of the language.
So this multi-compiler idea would allow you to write classes in different languages, even different methods of the same class or even blocks of code within the same method or procedure.

HLL inlined in asm would be a good example of this too.

Code Select Expand



.asm

   lea rdi,myArray
   xor rax,rax
   mov rcx,sizeof(myArray)
   rep stosq
   lea rdi,myArray
   lea rsi,mySourceArray

.hll(-rdi,-rsi)

   for(dword i=0;i<myArray.length;i++)
   {
   [rdi++] = (dword)(sin([rsi])*cos([rsi++])*sqrt(i)*1000.0f);
   }

.shader(glsl, fragment) as MyShader

varying vec3 normal;
varying vec3 vertex_to_light_vector;

void main()
{
    // Defining The Material Colors
    const vec4 AmbientColor = vec4(0.1, 0.0, 0.0, 1.0);
    const vec4 DiffuseColor = vec4(1.0, 0.0, 0.0, 1.0);

    // Scaling The Input Vector To Length 1
    vec3 normalized_normal = normalize(normal);
    vec3 normalized_vertex_to_light_vector = normalize(vertex_to_light_vector);

    // Calculating The Diffuse Term And Clamping It To [0;1]
    float DiffuseTerm = clamp(dot(normal, vertex_to_light_vector), 0.0, 1.0);

    // Calculating The Final Color
    gl_FragColor = AmbientColor + DiffuseColor * DiffuseTerm;
}

.xeonphi As MyXeonPhiCode

....

.asm

LoadCodeToXeonPhi(ADDR MyXeonPhiCode)
invoke My3DEngine::SetGPUShader, ADDR MyShader

.cpp

#include <xeonphi.h>

void LoadCodeToXeonPhi()
{
    xeonPhiDriver->LoadCode(&MyXeonPhiCode);
}

.js

var quickIdea = [];
for(var i=0;i<100;i++)
   quickIdea.push( (Math.cos(myArray[i])*100)|0 );




Or something to that affect :)  (asuming you had asm,cpp,phi,js,hll and glsl shader compiler plugins)

I'm not suggesting we go ahead and try to accomplish this (it's a monumental undertaking) but maybe it gets the creative juices flowing about potential new ideas or things we could look to build into a pre-processor.
Just being able to use the standard Windows SDK (and it's headers) in asm, as they constantly update would be a fantastic accomplishment.

[habran]

I am really impressed with your way of thinking, Johnsa
I don't think that it would be very hard to create that preprocessor, but that would give great adwantage to the assembler

[HSE]

Hi marlon!

Fortunatly there is a new entertainment for you:
                       
                                      Error A2102: Symbol not defined : @CatStr

A surprise in JWasm13 was that 80 macro levels is not enough for Biterider projects like ObjectBrowser. But Ml have no problem (Is something in ObjAsm interpreted as macro level by Jwasm and not by Ml?). At the moment the changes in global.h are:

Code Select Expand


#define MAX_LINE_LEN            1600  /* no restriction for this number */
#define MAX_TOKEN  MAX_LINE_LEN / 4  /* max tokens in one line */
#define MAX_STRING_LEN          MAX_LINE_LEN - 32 /* must be < MAX_LINE_LEN */
#define MAX_ID_LEN              547  /* must be < MAX_LINE_LEN */
#define MAX_STRUCT_ALIGN        32
/* v2.06: obsolete */
//#define MAX_RESW_LEN            31 /* max length of a reserved word */

#define MAX_IF_NESTING          20 /* IFxx block nesting. Must be <=32, see condasm.c */
//#define MAX_TEXTMACRO_NESTING   20
#define MAX_SEG_NESTING         20 /* limit for segment nesting  */
#ifdef __I86__
#define MAX_MACRO_NESTING       120
#else
#define MAX_MACRO_NESTING       120 /* macro call nesting  */


There is no hurry because, I see, the project is going to take all that last of the century 

Amazing ideas for improvement. :t  I can add the need for a findBugs capable of locate errors inside the macros. At present compiler very often say main.asm(xx): Exxx: error bla bla with no location of the error (of course there is no error in main.asm).   

[habran]

Thanks HSE
I will look at that soon

[LiaoMi]

Hi,

for the preprocessor could we use the developments of HLA assembly language developed by Randall Hyde?

[habran]

Hi LiaoMi,
I doubt that HLA can deal with 64 bit
We have to look how VC does precompiled headers  and get some idea

[Gunther]

Hi Habran,

I doubt that HLA can deal with 64 bit
We have to look how VC does precompiled headers  and get some idea

but on the other hand, is it real necessary? I think the main things are:

• The assembler is up to date.
• Assembles well all the new instructions (that's a big challenge).
• The assembler is available under different Operating Systems (that would be a huge advantage).

Gunther

[habran]

It is not necessary but it would make great leap forward and solve incompatibility of include files and keep them up to date
I don't think it will be hard to do. I will do it with pleasure, ones when I finish this job

[rrr314159]

My idea is to use precompiled headers instead of WinInc

- that would be great but I hope u don't mean "instead of" literally; rather, "in addition to". I'd want to retain the option of using a custom header file also.

I think the main things are:
•The assembler is up to date.
•Assembles well all the new instructions (that's a big challenge).
•The assembler is available under different Operating Systems (that would be a huge advantage).

- assembling the new instructions shouldn't be a separate bullet from "up to date", but that's nit-picking :) Other OS's would definitely be a huge advantage but also, undoubtedly, a huge job (how much work is involved to port it to just one other OS, for instance, your favorite flavor of linux?)

- Anyway - I hate to be a party pooper - but I must disagree, the "mainest" thing is bug-killing. There is no such thing as a (moderately complex) piece of software that's bug-free, HJWasm no exception. The only way it can be "bug-free" is, nobody uses it, so you never know about the bugs! I personally know of at least two in HJWasm that I'll post someday when I find the time to isolate them in a test prog, and I feel habran isn't working hard enough :)

Fortunatly there is a new entertainment for you: Error A2102: Symbol not defined : @CatStr
...
A surprise in JWasm13 was that 80 macro levels is not enough for Biterider projects like ObjectBrowser. But Ml have no problem (Is something in ObjAsm interpreted as macro level by Jwasm and not by Ml?). ... There is no hurry because, I see, the project is going to take all that last of the century

- Thanks HSE ... I think right now this is actually the most important post for HJWasm. I imagine the @CatStr prob is not a bug, rather HSE is doing something wrong. "80 macro levels" is also no bug, but it is something that the "competition" - ML - does better. Here is a user (the lifeblood of a project like HJ) who instead must go to the competition. Keeping him happy is job #1, not so much for his sake, but to show that the HJ team (of one!) responds - something the ML team (MicroSoft) doesn't do. HSE is nice enough to say "no hurry", but ...

for the preprocessor could we use the developments of HLA assembly language developed by Randall Hyde?

- HLA was implemented as a preprocessor, is that right? If Randall Hyde's source code is available then it's a good place to start. At least I'd look at it for ideas. If RH is still around no doubt he'd be happy to give some tips. Translating to 64-bits is no big deal. Having said that, it's easier to do it yourself than try to adapt someone else's code; but I always review anything similar to my project for ideas, and maybe even some snippets. In the same spirit if I were doing "MasmC++" I'd take a good look at ObjAsm (which I've never done, intend to someday) since it's a related idea; and pick Biterider's brain as appropriate.

I definitely think that any of these larger projects/ideas we have floating around are far too big for any one person to take on even full-time.

- If they are they simply will never happen.

- The only reason for more than one person on such a job is schedule. If it must be done in a timely fashion u need as many people as it takes. But one person is at least 33% more efficient than any team. In a loose situation like this - unpaid volunteers on the internet - one person is infinitely more efficient, because you'll never get two people to cooperate on anything. It's true, I hear success stories about GitHub etc, teams of people working together effectively; that's great; but I bet it's a rare miracle, and temporary. And every time it works, someone (emphasis on one) has already created a basic prog to elaborate on. Most such attempted collaborations end in mutual contempt, with no product in sight.

- For "MasmC++" the first step (a.k.a. milestone) is: make a preprocessor that works seamlessly, and simply replaces all instances of the word "CLASS" with "STRUCT" (for instance). That's a 1-person job which will take anywhere from a few days to a few weeks. Then, take the next step ... and the next ... if it takes 5 or 10 years, so what? No competition is going to eat your lunch by beating you to market. And if they do, great: saves you the trouble of writing it.

- I might be accused of inconsistency, since I said HJWasm could use 2 or 3 people: but that's because it is time-constrained. If things go well there should be a bunch of users all complaining about bugs, or at least requiring a response to show them what they're doing wrong. If they're not dealt with quickly they're gone and your project is pointless. Of course habran could do it just for his own use; nothing wrong with that; but I don't think that's his object here.

- Most likely I have more experience than anyone here technical-directing software projects. I've scoped out 1 man-month projects, on up to 10,000 man-year projects (only assistant t.d. on those). Most commonly, projects of 10 man-years or so.

- Now, even when programmers are in the same building, and are paid to do what you tell them, it's tough: they always know better than you, and spend half their time telling the secretary all about it. on the internet - with unpaid volunteers, all of whom have much better ideas than me (that's a given) - it's impossible. Would you do a job I scoped for you? Of course not. Neither will anyone else. Would I take the assignment of telling a bunch of you people what to do? With no pay? Of course not. It would require a lot of my time to do it right; no one will listen to me anyway; and I'd make plenty of mistakes (I, and every other t.d., always do); you'd expect perfection and wouldn't get it, go away in disgust at my first idiot blunder. Herding cats would be heaven by comparison. There's just no such thing as a multi-person project in these circumstances.

- I still say "MasmC++" can get well underway in 6 months (part-time) and in 5 years, you could have a good product. Plus any work done on it won't be waste: you can post it, and some one else can either build on it, or learn from your mistakes. But (and this is the great thing about amateur software): it's your call.

[Gunther]

Hi rrr,

- assembling the new instructions shouldn't be a separate bullet from "up to date", but that's nit-picking :) Other OS's would definitely be a huge advantage but also, undoubtedly, a huge job (how much work is involved to port it to just one other OS, for instance, your favorite flavor of linux?)

The availability on other operating systems was one of the advantages of jWasm in the past. We had versions for Windows(32- and 64-bit), DOS, OS/2, BSD and Linux. That should be the case in the future, too.

Gunther

[habran]

- that would be great but I hope u don't mean "instead of" literally; rather, "in addition to". I'd want to retain the option of using a custom header file also.

I meant OPTIONALLY

Headers with extension .h will be preprocessed and .inc will be left as they are

[HSE]

I imagine the @CatStr prob is not a bug, rather HSE is doing something wrong.

I agree. Just to note that habran's HJWasm(32) find a problem where habran's JWasm13(32) work fine. Perhaps the observation is usefull for habran, perhaps not.

Some issues I see aren't true compiler problem. Using ObjAsm32 and SmplMath push some limits:

Code Select Expand


fSlv8 [esi].AaDegEIN =  [esi].FAaAla *  [esi].HcAla +  [esi].FAaArg *  [esi].HcArg +  [esi].FAaAsp\
*  [esi].HcAsp +  [esi].FAaGlu *  [esi].HcGlu +  [esi].FAaGly\
*  [esi].HcGly +  [esi].FAaIle *  [esi].HcIle +  [esi].FAaLeu\
*  [esi].HcLeu +  [esi].FAaPro *  [esi].HcPro +  [esi].FAaSer\
*  [esi].HcSer +  [esi].FAaThr *  [esi].HcThr +  [esi].FAaVal\
*  [esi].HcVal +  [esi].FAaPhe *  [esi].HcPhe +  [esi].FAaTyr\
*  [esi].HcTyr;
Lines like this could trouble the compiler, but it's for easy coding, because it's nothing sofisticated and using FPU instruccions there is no problem, except that corrections became a nightmare.

Regards, HSE 

[jj2007]

"80 macro levels" is also no bug, but it is something that the "competition" - ML - does better.

I doubt there is a legit need for 80 levels, and suspect that some macro erroneously goes recursive and, well, stops precisely at the maximum allowed level. It would be nice if somebody could isolate this "bug" (which I've never seen in MasmBasic, with 400+ macros).