Minor Update just to mark the release of HASM (new name of HJWASM) as 2.32.
Changes:
1) Added initial file BOM check for UTF-8
2) Removed string literal support from invoke (samples and documentation updated accordingly) to use CSTR and WSTR.
2 Ensures we keep backwards compatibility in favour of adding this new type of functionality under a totally new procedure calling/invocation scheme.
This modified mineiro example works, but not a same way as asmc..386
.model flat,stdcall
option dllimport:<kernel32.dll>
GetStdHandle proto :dword
WriteFile proto :dword,:dword,:dword,:dword,:dword
ExitProcess proto :dword
SetConsoleOutputCP proto :dword
STD_OUTPUT_HANDLE equ -11
CP_UTF8 equ 65001
.data?
houtput dd ?
nada dd ?
.data
utf8 db "accents: áéíóúçã"
.code
start:
invoke SetConsoleOutputCP,CP_UTF8
invoke GetStdHandle,STD_OUTPUT_HANDLE
mov houtput,eax
invoke WriteFile,houtput,addr utf8,sizeof utf8,addr nada,0
invoke ExitProcess,0
end startI hope that people in this forum decide what is the correct way in Win32 and linux to do that.
Quote from: TWell on May 17, 2017, 02:50:51 AM
This modified mineiro example works, but not a same way as asmc.
For me, it works with both assemblers, source and exes attached. What exactly did not work?
My bad :(
I recheck object files.
Both save UTF-8 in similar ways.
will be nice if you add a switch on command line to deal with ascii/unicode/utf8/utf16 ...files default:ascii
This way nobody needs change their program style.
If a person like to write their unicode strings using ascii editors and skills, ok.
If we have:
.if al == "a"
On ascii scope thats a byte and works ok. If person saved as unicode so that will report a error because "a" will be 0041h or a word size, ok too, valid error. If person saved as utf8, so "a" will be like ascii, 1 byte and will work. If person saved as utf16 that will be a word and report a error.
So if we have this:
.if eax == "㑖"
If person don't like to write if eax == "㑖" using unicode/utf8/... text editor we know that we can do that by ".if eax == hexnumberh" using ascii editor. This way, the talk about wstr and cstr will continue from an ascii point of view only.
--edited--
This way, you don't need improvements on hasm syntax, thats fine sir johnsa. And because all text file is being interpreted into one scope, so you give to us much freedom and we are able to not only deal with unicode chars while constructing strings, but also we are able to create variables and function names using unicode chars.
☕ dd 10
I'm reading on primary sources below about this subject:
https://tools.ietf.org/html/rfc3629
www.unicode.org/versions/Unicode9.0.0/ch02.pdf
I think the big problem with full utf-8 / unicode support is that internally the string comparisons, symbol table, tokenizer and parser have all been designed around ascii only.. changing it to make it fully portable would probably require a rather major re-write. I'm not sure how far Nidud has gotten with utf support in asmc (unicode/utf16/utf8 etc) I know he has added code page support ?
deleted
Quote from: johnsa on May 17, 2017, 06:32:08 AM
I think the big problem with full utf-8 / unicode support is that internally the string comparisons, symbol table, tokenizer and parser have all been designed around ascii only.. changing it to make it fully portable would probably require a rather major re-write. I'm not sure how far Nidud has gotten with utf support in asmc (unicode/utf16/utf8 etc) I know he has added code page support ?
http://masm32.com/board/index.php?topic=6221.msg66580#msg66580
Link above have a utf8 text walker. Function is_valid_utf8_encode return sizeof char in byte(s). Can return 1,2,3,4 bytes to one char that user have seen while coding.
Ops, now I understand, if string comparisions only deal with <=7fh so can't be done, to me strings comparisions was done on hasm source code by hexa comparisions. Now I take the point. Thank you for answering and also by think about this subject.
Quote from: nidud on May 17, 2017, 06:58:24 AM
áéíóúçã
If you use a windows editor like Notepad the string is saved as:
E1 E9 ED F3 FA E7 E3
And if you write it in a text mode editor the string is saved as:
A0 82 A1 A2 A3 87 84 - (860)
...
If you use the right tool for the job you don't need to bother with all this Unicode stuff. However, if you use a Windows editor to write a console application you need to translate the text somehow and use the W functions. This adds code to the application.
hello sir nidud
example that you posted is ambiguous, and to avoid ambiguity they have create unicode (aka Universal). And to preserve latin chars as being one byte only they encoded that by using utf8.
Letter á can be E1 or A0, how we can avoid this? By lucky they have different hexa representation and they do not overlap on this example.
Let's convert these symbols to unicode using that easy way (not correct I suppose) that's insert a zero at left size of that char.
00e1h != 00a0h
á != á
So, as you can see, that's not valid, same supposed char to different opcodes. On unicode we can print letter 'a', and after insert accent on same symbol representation as chapter 2 says.
Quote from: johnsa on May 17, 2017, 06:32:08 AM
I think the big problem with full utf-8 / unicode support is that internally the string comparisons, symbol table, tokenizer and parser have all been designed around ascii only.. changing it to make it fully portable would probably require a rather major re-write.
It is
not a big problem, simply because we need to distinguish two entirely different things:
1. Do we need non-Latin chars in symbols, labels and commands?
2. Do we need non-Latin chars in strings?
Version 1 is this:
Печать "Hello World"Version 2 is this:
print "Привет Мир"1. With current assemblers,
Печать "Hello World" is impossible. No problem,
nobody needs that. Russian programmers do not use Russian commands in their code (if you don't agree, gimme a link to
one example at least); they use English, because that is the language of their compiler. Same for Chinese, Arabic, Japanese and North Korean coders.
2. With current assemblers,
print "Привет Мир" is possible. Even the old Masm 6.14 that comes along with the Masm32 SDK can flawlessly produce code that
displays Привет Мир a) in the editor, and b) in the console.
It is not a problem of the assembler - that is just a dumb tool, and it never had any interest in the stupid things that coders put "inside quotes". They are Chinese anyway for the assembler (no insult intended, I like Chinese characters).
Conclusion: THERE IS NO PROBLEM, except perhaps that very few editors (http://masm32.com/board/index.php?topic=5314.msg63702#msg63702) can handle UTF-8 correctly 8)
deleted
Yes, nidud, I think everybody has now understood that you are not a fan of UTF-8, and that every serious coder in the world should use only the native codepage of his system. Norwegian, in your case. You are free to do that, I am free to ignore you.
deleted
deleted
hello nidud;
This talk about utf8 on windows O.S. don't sound too logic because default is unicode. So we need transform utf8 to unicode to have more malleable on windows, but on linux we have this malleable way from the ground.
I'm talking about utf8 just because xml files are now utf8 encoded. The intention is to create a universal gui with universal chars.
I have insisted with you because this. As you I can survive without utf8. But future is becoming present.
deleted
Quote from: nidud on May 18, 2017, 02:28:51 AM
Well, hopefully you now understand a bit more on how this works, and here's one of the argument for not using UTF-8 as a default for text mode or other applications where it's not needed:
hello nidud;
And here's one of the argument to use UTF-8 :badgrin: as default for text mode or gui mode on linux instead of windows O.S.
;--- assemble: hasm -elf64 -Fo=test.o test.asm
;--- link: gcc test.o -o test
.x64
exit proto systemv status:dword
puts proto systemv pchar:ptr
.data
string db "utf8 rulez: áéíóúçãሴ⍅㑖噸枉",0
.code
main PROC SYSTEMV
invoke puts, addr string
invoke exit,0
main ENDP
end
We know that whats good to one person cannot be good to another.
Is it possible for UASM to support the Intel MPX instructions?
For example it does not assemble the open source _chkstk.asm that comes with Visual Studio because does not understand
"bnd jb" and "bnd ret". MASM does understand.
Bounds registers are exotic stuff, José. Do you have an idea how to use them? Looks interesting, but I'd like to see a concrete example.
Found a manual on MPX here. (https://pdfs.semanticscholar.org/bd11/4878c6471cb5ae28546a594bf25ba5c25c6f.pdf)
BNDMK b, m
Creates LowerBound (LB) and UpperBound (
UB) in
bounds register b.
BNDCL b, r/m
Checks the address of a memory referen
ce or
address in r against the lower bound.
BNDCU b, r/m
Checks the address of a memory referen
ce or
address in r against the upper bound.
BNDCN b, r/m
Checks the address of a memory referen
ce or
address in r against the upper bound in one's
compliment.
We added support for the basic BNDn registers and MPX specific instructions in 2.33
We've not really played with them test wise yet, if we've got a concrete example to test it with we can make sure that we've not missed anything, but given MPX only came out with Skylake it's probably not that useful yet in general practice.
BND in that case is a prefix which we've not added yet, but will in the next update.
Basically a range of instructions can use a BND prefix, for MPX enabled code all opcodes that can use BND should, unless you need to specifically override it with a NOBND.
I'm still thinking about how best to implement this particular change as we'll probably want a new OPTION BND:YES/NO from which point the BND is applied automatically.
For NOBND we'll probably want that to act as a new prefix as far as the parser is concerned.
Quote from: jj2007 on May 21, 2017, 05:44:16 PM
Bounds registers are exotic stuff, José. Do you have an idea how to use them? Looks interesting, but I'd like to see a concrete example.
I have a tiny idea, I would like to experiment more. But basically I just wanted the source of _chkstk to assemble without changes.
Quote from: johnsa on May 21, 2017, 07:05:17 PM
We added support for the basic BNDn registers and MPX specific instructions in 2.33
We've not really played with them test wise yet, if we've got a concrete example to test it with we can make sure that we've not missed anything, but given MPX only came out with Skylake it's probably not that useful yet in general practice.
BND in that case is a prefix which we've not added yet, but will in the next update.
Basically a range of instructions can use a BND prefix, for MPX enabled code all opcodes that can use BND should, unless you need to specifically override it with a NOBND.
I'm still thinking about how best to implement this particular change as we'll probably want a new OPTION BND:YES/NO from which point the BND is applied automatically.
For NOBND we'll probably want that to act as a new prefix as far as the parser is concerned.
The only concrete example I have is the _chkstk. I will experiment more when I can.
We're busy looking into some details as well of exactly how it "should" work and which opcodes are applicable, we should have something for you to test in a day or so.
Quote from: aw27 on May 21, 2017, 07:13:48 PMI just wanted the source of _chkstk to assemble without changes.
Where did you find that source? Nothing in my VC folders :(
Here is one (http://www.jbox.dk/sanos/source/lib/chkstk.asm.html) but no MPX instructions.
Quote from: johnsa on May 22, 2017, 02:20:22 AM
We're busy looking into some details as well of exactly how it "should" work and which opcodes are applicable, we should have something for you to test in a day or so.
Great :t
Quote from: jj2007 on May 22, 2017, 03:20:26 AM
Quote from: aw27 on May 21, 2017, 07:13:48 PMI just wanted the source of _chkstk to assemble without changes.
Where did you find that source? Nothing in my VC folders :(
Here is one (http://www.jbox.dk/sanos/source/lib/chkstk.asm.html) but no MPX instructions.
It is in the crt folder of the VC part. However, I pasted it in one of the last messages about the _alloca function.
Quote from: aw27 on May 22, 2017, 03:53:06 AMIt is in the crt folder of the VC part. However, I pasted it in one of the last messages about the _alloca function.
There is a link to nidud's version here (http://masm32.com/board/index.php?topic=6263.msg66920#msg66920), but that's the only chkstk.asm I can see in that thread.
Quote from: jj2007 on May 22, 2017, 06:26:10 AM
but that's the only chkstk.asm I can see in that thread.
It is in message 28, inside the code section is the function _chkstk. Just paste that function in a new file and save it as chkstk.asm and you have it. :t
OK, got it - pasted "as is" below. This is the mysterious "bound jump":
cmp ecx, eax ; Is new TOS
bnd jb cs20 ; in probed page?If I understand the manual (https://pdfs.semanticscholar.org/bd11/4878c6471cb5ae28546a594bf25ba5c25c6f.pdf) correctly, this would do the job, too:
cmp ecx, eax ; Is new TOS
db 0F2h ; bnd prefix
jb cs20 ; in probed page?QuoteLegacy code does not experience any change in its functionality. Instrumented applications can link with, call into, or be called from legacy software.
::)
_chkstk proc C
_alloca_probe = _chkstk
push ecx
; Calculate new TOS.
lea ecx, [esp] + 8 - 4 ; TOS before entering function + size for ret value
sub ecx, eax ; new TOS (Top of Stack)
; Handle allocation size that results in wraparound.
; Wraparound will result in StackOverflow exception.
sbb eax, eax ; 0 if CF==0, ~0 if CF==1
not eax ; ~0 if TOS did not wrapped around, 0 otherwise
and ecx, eax ; set to 0 if wraparound
mov eax, esp ; current TOS
and eax, not ( _PAGESIZE_ - 1) ; Round down to current page boundary
cs10:
cmp ecx, eax ; Is new TOS
bnd jb short cs20 ; in probed page?
mov eax, ecx ; yes.
pop ecx
xchg esp, eax ; update esp
mov eax, dword ptr [eax] ; get return address
mov dword ptr [esp], eax ; and put it at new TOS
bnd ret
; Find next lower page and probe
cs20:
sub eax, _PAGESIZE_ ; decrease by PAGESIZE
test dword ptr [eax],eax ; probe page.
jmp short cs10
_chkstk endp
QuoteLegacy code does not experience any change in its functionality. Instrumented applications can link with, call into, or be called from legacy software.::)
That's the point, even if does not do anything I would like it to assemble :badgrin:
I've uploaded a pre-release of 2.34 which now has full BND/MPX support.
OPTION BND:ON/OFF has been added to generate bnd call for invoke.
BND prefix is support for call/jmp/ret/retn/Jcc
link: www.terraspace.co.uk/uasm64.zip (http://www.terraspace.co.uk/uasm64.zip)
test piece:
; Intel MPX Test-Piece
option casemap:none
option stackbase:rbp
option win64:7
option bnd:on ; Enable automatic generation of BND on invokes.
testProc PROTO aPointer:PTR
.data
bndStore0 dq 0,0
.code
testProc PROC aPointer:PTR
bnd ret
testProc ENDP
main PROC
LOCAL myBuffer[100]:BYTE
; Create two bounds.
; lower bound is stored as is in the BND0 register, the upper value RCX is size-1 and stored in 1s complement.
lea rax,myBuffer
mov rcx,99
bndmk bnd0,[rax+rcx]
lea rax,myBuffer
bndmk bnd1,[rax+99]
; Verify the bound move operations between registers and oword sized memory.
bndmov bnd2,bnd0
bndmov bndStore0,bnd1
bndmov bnd1,bndStore0
; Verify the bounds of the pointer in RAX pre-write for a dword sized write.
; CL (check lower bound uses the pointer value itself).
; CU (check upper uses the pointer + size - 1).
lea rax,myBuffer
bndcl bnd0,[rax]
bndcu bnd0,[rax+3]
mov dword ptr [rax],ecx
; Test OPTION BND via invoke
invoke testProc, rax
; Test BND call.
mov rcx, rax
bnd call testProc
; Test BND jmp/jcc
mov rcx,10
;bnd jmp firstLabel
firstLabel:
dec rcx
bnd jnz short firstLabel
; Test invalid prefix on opcodes that don't allow BND
;bnd mov rax,[rax]
;bnd iretq
;bnd cmp rax,rbx
ret
main ENDP
end main
Quote from: johnsa on May 22, 2017, 11:42:43 PM
I've uploaded a pre-release of 2.34 which now has full BND/MPX support.
I have tested in the 32-bit chkstk and it assembles and runs fine. I will test later this test piece.
There is a "somewhat unexpected" problem:
This code:
bnd jb short cs20
mov eax, ecx
xchg esp, eax
mov eax, dword ptr [eax]
mov dword ptr [esp], eax
mov eax, esp
add eax, 4
bnd ret
cs20:
Assembles with UASM to:
004FC857 F27210 repnz jb $004fc86a
004FC85A 8BC1 mov eax,ecx
004FC85C 94 xchg eax,esp
004FC85D 8B00 mov eax,[eax]
004FC85F 890424 mov [esp],eax
004FC862 8BC4 mov eax,esp
004FC864 83C004 add eax,$04
004FC867 F2C3 repnz ret
and with MASM to:
004FC857 F2720E repnz jb $004fc868
004FC85A 8BC1 mov eax,ecx
004FC85C 94 xchg eax,esp
004FC85D 8B00 mov eax,[eax]
004FC85F 890424 mov [esp],eax
004FC862 8BC4 mov eax,esp
004FC864 83C004 add eax,$04
004FC867 C3 ret
MASM does not convert the bnd ret.
But when I run the UASM assembled code, it hangs occasionally, the MASM assembled does not hang.
The obvious reason is the "ret", because everything else is equal. Well, not exactly, there is a difference in the bnd jb short cs20 encoding as well.
May be you can find an explanation.
That is really odd, I can think of NO reason why the bnd ret should not be encoded as F2 C3, According to MPX spec all near returns should a) support bnd and b) if used as bnd ret should be encoded as F2 C3, or F2 C2 (imm16)!
And given that MPX promises that any use of BND would result in effectively NOP on non-supporting processors whether present or not it shouldn't ever make it hang... I assume you're testing on a non-mpx processor?
Quote from: johnsa on May 23, 2017, 01:35:36 AM
That is really odd, I can think of NO reason why the bnd ret should not be encoded as F2 C3, According to MPX spec all near returns should a) support bnd and b) if used as bnd ret should be encoded as F2 C3, or F2 C2 (imm16)!
And given that MPX promises that any use of BND would result in effectively NOP on non-supporting processors whether present or not it shouldn't ever make it hang... I assume you're testing on a non-mpx processor?
Even more weird is that Visual Studio (2015 is the latest version I have here) does not compile programs with the same chkstk whose source code they supply, it compiles with a version of chkstk that does not contain the MPX instructions.
No, I don't have any computer with a mpx processor.
I've been trying to test the opcodes under SDE (which should emulate them perfectly, but their behaviour is very odd, sometimes the bnd jmp crashes, sometimes not..)
Maybe this is too cutting edge! :eusa_boohoo:
At least in 32-bit (not tested yet in 64-bit)
.data
align 32
Does not produce an error as it does in MASM, and 50% of the time will work due to pure coincidence.
I was believing it worked but it was only my good star.
Currently, the only guaranteed way to align data to 32-bit is, as mentioned in another thread, is through the old way:
_DATA1 SEGMENT ALIGN(32) FLAT 'DATA'
_DATA1 ends
I suggest that you either fix it or produce an error.
Since ".data" aligns to paragraph by default may be you can use ".data 32", ".data 64" to align to 32, 64 etc bytes.
On the subject of memory alignment - are there any good articles out there
describing the effects & mitigation (besides what can be pragmatically detected
through trial and error / experience) ?
Quote from: Raistlin on May 23, 2017, 03:32:49 PM
On the subject of memory alignment - are there any good articles out there
describing the effects & mitigation (besides what can be pragmatically detected
through trial and error / experience) ?
Nothing that I am aware of.
Quote from: aw27 on May 23, 2017, 03:10:16 PM
At least in 32-bit (not tested yet in 64-bit)
.data
align 32
Does not produce an error as it does in MASM, and 50% of the time will work due to pure coincidence.
I was believing it worked but it was only my good star.
Currently, the only guaranteed way to align data to 32-bit is, as mentioned in another thread, is through the old way:
_DATA1 SEGMENT ALIGN(32) FLAT 'DATA'
_DATA1 ends
Thanks, José - you found it :t (where, btw?)
Short version, tested with ML 8.0 onwards (6.14 and 6.15 don't like it):
include \masm32\include\masm32rt.inc ; plain Masm32
MbData SEGMENT ALIGN(32)
myvarA dd ?
align 32
myvarB dd ?
MbData ENDS
MbData SEGMENT
align 32
myvarC dd ?
MbData ENDS
.code
start:
mov eax, offset myvarB
sub eax, offset myvarA
print hex$(eax), "h bytes difference B-A", 13, 10
mov eax, offset myvarC
sub eax, offset myvarA
inkey hex$(eax), "h bytes difference C-A", 13, 10
exit
end start
OPT_Assembler mlv10
OPT_Linker linkv614Replace 'MbData' with your own name. Note the combination of a recent ML version and a very old linker. This works for almost everything except the very old assembler versions.
I've never test align 32 under 32bit to be honest, it's always worked flawlessly for me on 64bit. I would think, that each section would be loaded into memory by the OS loader on page boundaries (4096).
This will need some investigation.. I suspect emitting an error using align 32 under 32bit mode is probably the easiest solution unless there is a better fix to force 32 alignment on the .data and .data? simplified directives.
Quote from: jj2007 on May 23, 2017, 05:18:02 PM
Thanks, José - you found it :t (where, btw?)
I just remembered that .data is a "simplified directive".
According to the documentation on LINK, the default section alignment is page (4096), but can be set with /ALIGN .. that might be something to check?
I'd be curious to see what your section alignments are by default (depending on the version of linker you use etc), perhaps try PEDUMP and have a look ?
It really should be 4096, in which case align 32 should be fine.
With:MbData SEGMENT ALIGN(32) ALIAS('.data')
myvarA dd ?
align 32
myvarB dd ?
MbData ENDS
we can also test linkers how those combine sections.
EDIT: ml v10 accept this:Data_ SEGMENT ALIGN(32) 'BSS' ALIAS('.bss')
foo1 dd ?
align 32
foo2 dd ?
Data_ ENDS
EDIT: since ml v8 seems to be possible to make a linker friendly static lib from one file for polink.exe:.386
_TEXT1 segment flat align(32) 'CODE' ALIAS('.text')
foo1 proc
ret
foo1 endp
_TEXT1 ends
_TEXT2 segment flat align(32) 'CODE' ALIAS('.text')
foo2 proc
ret
foo2 endp
_TEXT2 ends
end
Quote from: johnsa on May 23, 2017, 06:20:26 PM
According to the documentation on LINK, the default section alignment is page (4096), but can be set with /ALIGN .. that might be something to check?
I'd be curious to see what your section alignments are by default (depending on the version of linker you use etc), perhaps try PEDUMP and have a look ?
It really should be 4096, in which case align 32 should be fine.
We are concerned with the default data alignment within a section not the whole section itself. The default is paragraph (16 bytes). To change that behaviour we can not use the simplified directive, or it appears so. I am talking about 32-bit. I have not checked 64-bit, I can just believe what you say.
Quote from: johnsa on May 23, 2017, 06:11:01 PM
I've never test align 32 under 32bit to be honest, it's always worked flawlessly for me on 64bit. I would think, that each section would be loaded into memory by the OS loader on page boundaries (4096).
Attention, section <> segment. Although, ::) ->see attachment (plain asm).
Quote from: aw27 on May 23, 2017, 07:16:41 PMWe are concerned with the default data alignment within a section not the whole section itself. The default is paragraph (16 bytes). To change that behaviour we can not use the simplified directive
That's the whole point indeed. When using align 32 in a .data? segment, MASM chokes because this is above the default alignment of 16. Question is why the Watcom family doesn't complain.
Well intra-section alignment should just work if the start of the section is aligned >= 32, then you should be able to just use align X (for any X <= 4096 assuming page aligned).
I will run some more tests in 32bit to see how that works as it's definitely fine in 64bit. Are you also running on a 32bit OS ? (The OS loader may also affect this as opposed to running a 32bit app on a 64bit os).
deleted
deleted
I guess we can just change in simseg.c to write out align(32) instead of para, as it seems like a perfectly legitimate option under all conditions?
deleted
I guess we could do that too,
It just seems like yet another setting you have to change to obtain the desired behaviour when using align 32. It doesn't seem to be affecting the outcome in 64bit, so I assume PARA or not the whole section is in memory on a 64kb or page boundary at least.
So.. command line switch or the proposed workaround posted here? - I'll go with whatever we decide here (in the spirit of what we discussed earlier!) :)
Quote from: TWell on May 23, 2017, 06:25:27 PM
With:MbData SEGMENT ALIGN(32) ALIAS('.data')
This appears to be a good point, particularly when linking with HLL, because will prevent multiple section names.
"
ALIAS( string )
This string is used as the section name in the emitted COFF object. Creates multiple sections with the same external name, with distinct MASM segment names.
"
In summary:
_DATA1 SEGMENT ALIGN(32) FLAT 'DATA' ALIAS('.data')
Quote from: jj2007 on May 23, 2017, 07:23:07 PM
Quote from: johnsa on May 23, 2017, 06:11:01 PM
I've never test align 32 under 32bit to be honest, it's always worked flawlessly for me on 64bit. I would think, that each section would be loaded into memory by the OS loader on page boundaries (4096).
Attention, section <> segment. Although, ::) ->see attachment (plain asm).
Quote from: aw27 on May 23, 2017, 07:16:41 PMWe are concerned with the default data alignment within a section not the whole section itself. The default is paragraph (16 bytes). To change that behaviour we can not use the simplified directive
That's the whole point indeed. When using align 32 in a .data? segment, MASM chokes because this is above the default alignment of 16. Question is why the Watcom family doesn't complain.
JWasm did complain, we removed the warning about align 32, as I use it all the time for AVX stuff (but 64bit only), so the segment's para setting seems to have little effect in 64bit as I've not once had an issue with align 32 on 64bit.
So,
command line option -align:X set segment alignment
default is para(16) if not specified, and we'll just apply that to simseg ?
deleted
Quote from: johnsa on May 24, 2017, 01:27:44 AMIt just seems like yet another setting you have to change to obtain the desired behaviour when using align 32.
Indeed. There are people who try to remain compatible with the "old" Masm world, and adding new commandline options can be an obstacle to switching.
Just found another incompatibility. This builds fine in ML 8.0 ... 10.0:
include \masm32\include\masm32rt.inc ; plain Masm32
MbData SEGMENT align(32) 'data' ; MSDN: SEGMENT (https://msdn.microsoft.com/en-us/library/d06y3478.aspx)
myvarA dd ?
align 32
myvarB dd ?
; MbData ENDS
.code
nop
MbData segment
align 32
myvarC dd ?
other SEGMENT ALIGN(32)
align 32
myvarO dd ?
other ENDS
.code
start:
mov eax, offset myvarB
sub eax, offset myvarA
print hex$(eax), "h bytes difference B-A", 13, 10
mov eax, offset myvarC
sub eax, offset myvarA
print hex$(eax), "h bytes difference C-A", 13, 10
mov eax, offset myvarO
sub eax, offset myvarA
inkey hex$(eax), "h bytes difference O-A", 13, 10
exit
end start
This is a bit off topic but its close to content. It has been many years since I have used full segments in a MASM executable so I had to test if they still worked in 64 bit MASM. Seems so. Here is a test piece that does nothing but test if a 4096 byte aligned ymm variable is writable.
; ¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤
include \masm32\include64\masm64rt.inc
aln4096 SEGMENT Align(4096)
align 4096
.ymm0 ymmword ?
.ymm1 ymmword ?
.ymm2 ymmword ?
.ymm3 ymmword ?
.ymm4 ymmword ?
.ymm5 ymmword ?
.ymm6 ymmword ?
.ymm7 ymmword ?
.ymm8 ymmword ?
.ymm9 ymmword ?
.ymmA ymmword ?
.ymmB ymmword ?
.ymmC ymmword ?
.ymmD ymmword ?
.ymmE ymmword ?
.ymmF ymmword ?
aln4096 ENDS
.code
; ¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤
entry_point proc
LOCAL ymmvar :QWORD
mov ymmvar, 1234567890
vmovntdqa ymm0, YMMWORD PTR [rcx+r10]
vmovntdq .ymm0, ymm0
waitkey
invoke ExitProcess,0
ret
entry_point endp
; ¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤¤
end
I found some non-scholarly articles on the alignment issue re: what it is, why we need it, best practice, implications etc.
Here's the links, perhaps it provides elucidation.
https://msdn.microsoft.com/en-us/library/Aa290049(VS.71).aspx
http://www.gamasutra.com/view/feature/3942/data_alignment_part_1.php?print=1
http://www.agner.org/optimize/optimizing_assembly.pdf
http://technion.ac.il/doc/intel/compiler_c/main_cls/intref_cls/common/intref_alignment_support.htm#intref_alignment_support
http://www.obpm.org/download/Intro_to_Intel_AVX.pdf
Thanks for the links. The Agner doc has 331 instances of the word
align, but most concern the 'forced' alignments through 64-bit ABI and SIMD instructions. No real info on performance penalties, except for ... the 'modern' P4 processor:
QuoteThe performance penalty for level-1 cache line contention can be quite considerable on older microprocessors, but on newer processors such as the P4 we are losing only a few clock cycles because the data are likely to be prefetched from the level-2 cache, which is accessed quite fast through a full-speed 256 bit data bus. The improved efficiency of the level-2 cache in the P4 compensates for the smaller level-1 data cache.
On a side note (related to occasional debates whether being stingy about code size is good or bad):
Quote11.3 μop cache
The Intel Sandy Bridge and later processors have a traditional code cache before the decoders and a μop cache after the decoders. The μop cache is a big advantage because instruction decoding is often a bottleneck on Intel processors. The capacity of the μop cache is much smaller than the capacity of the level-1 code cache. The μop cache is such a critical resource that the programmer should economize its use and make sure the critical part of the code fits into the μop cache. One way to economize trace cache use is to avoid loop unrolling.
One way to economize trace cache use is to avoid loop unrolling :icon_eek: :icon_exclaim:
OK did'nt see that !- thanks as always JJ2007 - seems counter intuitive as this was the bread and butter
of optimization until now. Is there a graphical (DIAGRAM) way to represent such ? Just for understanding.....
[EDIT: Never mind it all makes sense - just had to read it twice]
Quote from: Raistlin on May 24, 2017, 04:30:50 PMseems counter intuitive as this was the bread and butter of optimization until now.
Unrolling helps every now and then, in case of doubt: time it :P
My experience is mixed: Sometimes a very short loop can be made a bit faster, but equally often it gets slower.
Apparently shorter code is good for battery life (AnandTech (http://www.anandtech.com/show/6355/intels-haswell-architecture/6)):
QuoteSandy Bridge introduced the 1.5K µop cache that caches decoded micro-ops. When future instruction fetch requests are made, if the instructions are contained within the µop cache everything north of the cache is powered down and the instructions are serviced from the µop cache. The decode stages are very power hungry so being able to skip them is a boon to power efficiency.
@ Raistlin
Here I tested a piece of looped code that fits in 1 cache line on a 64 byte aligned code memory boundary. It executes faster.
http://masm32.com/board/index.php?topic=6141.msg65172#msg65172
Quote from: nidud on May 24, 2017, 06:39:14 AM
A bit long. /Sa is used.
We change the maximum alignment of Segments with the n value required to be a power of two which specifies the new alignment in bytes also used as pack(n) in C for structure alignment.
/Zp[n] Set structure alignment
/Sp[n] Set segment alignment
So, Segment Packing maybe?
That works for me, I'm happy with /Sp.
I think what we should do also , is if align 32 (>16) is used in the section and /Sp is not specified or the align is > that the /Sp value.. emit a warning ?
deleted
Relics from old DOS times? JWasm issues a warning, yes, HJWasm doesn't, but both produce correct code.
include \masm32\include\masm32rt.inc
.data
db "123", 0
align 64
MyArray db 25, 18, 23, 17, 9, 2, 6
align 64
HelloW$ db "Hello World", 0
.code
start:
mov ecx, offset MyArray
print hex$(ecx), 9, "MyArray", 13, 10
mov ecx, offset HelloW$
inkey hex$(ecx), 9, "HelloW$", 13, 10
exit
end start
OPT_Assembler JWasm
OxPT_Assembler AsmC
OPT_linker linkv614
deleted
You may find this stuff a bit long in the tooth. The worlds has changed multiple times since the PIV. Core2 and the later "i" series hardware does different things. Humerously enough unrolling does not work like it used to on old processors but not through such a limitation, its due to the internal logic getting better.
deleted
Quote from: jj2007 on May 24, 2017, 08:27:56 PM
Relics from old DOS times? JWasm issues a warning, yes, HJWasm doesn't, but both produce correct code.
Also with most recent MS linker :eusa_clap:. The problem is that not everybody produce standalone Assembly language programs these days, actually almost nobody does.
This is what happens when I run a program linking a similar module assembled with the .data + align 32 combo with a HLL like Delphi or Free Pascal:
(http://www.atelierweb.com/a/accessViolation.png)
Because the Delphi and FPC linkers understand the full segment definition syntax, probably they consider what JWASM/UASM are doing is just a hack.
You see, Masm issues an error, it is only the
MS Linker that let the bad things pass through the raindrops.
I've applied the same command line switch as Nidud's post to Uasm 2.34 now.. Will update site etc. once i've run some more testing on other changes.
Quote from: aw27 on May 24, 2017, 10:52:10 PM
This is what happens when I run a program linking a similar module assembled with the .data + align 32 combo with a HLL like Delphi or Free Pascal:
(http://www.atelierweb.com/a/accessViolation.png)
Because the Delphi and FPC linkers understand the full segment definition syntax, probably they consider what JWASM/UASM are doing is just a hack.
You see, Masm issues an error, it is only the MS Linker that let the bad things pass through the raindrops.
Interesting.
With this code:
foo.asm
.386
.model flat, c
public foo1
public foo2
public foo3
.data
db "123", 0
align 64
foo1 db 25, 18, 23, 17, 9, 2, 6
align 64
foo2 db 2
foo3 db 2
end
program test;
{$LINK foo.obj}
type pchar = ^char;
function printf(fmt : pchar):longint; cdecl; varargs; external 'msvcrt';
var
foo1 : pchar; cvar; external;
foo2 : pchar; cvar; external;
foo3 : pchar; cvar; external;
begin
PrintF('Hello Free Pascal'#10);
PrintF('ptr: %p'#10, @foo1);
PrintF('ptr: %p'#10, @foo2);
PrintF('ptr: %p'#10, @foo3);
end.
Hello Free Pascal
ptr: 00402040
ptr: 00402080
ptr: 00402081ppc386.exe (https://sourceforge.net/projects/freepascal/files/Bootstrap/)
EDIT: now with uasm64.exe -Sp32
program test;
{$LINK AVXTest1.obj}
{$LINK AVXTest2.obj}
type pchar = ^char;
function printf(fmt : pchar):longint; cdecl; varargs; external 'msvcrt';
var
AbsMask1 : pchar; cvar; external;
NegMask1 : pchar; cvar; external;
AbsMask2 : pchar; cvar; external;
NegMask2 : pchar; cvar; external;
foo1:char =#0;
begin
PrintF('ptr: %p'#10, @foo1);
PrintF('ptr: %p'#10, @AbsMask1);
PrintF('ptr: %p'#10, @NegMask1);
PrintF('ptr: %p'#10, @AbsMask2);
PrintF('ptr: %p'#10, @NegMask2);
end.
.686
.model flat, stdcall
option casemap:none
public AbsMask1
public NegMask1
.data
AbsMask1 dword 7fffffffh,7fffffffh,7fffffffh,7fffffffh, 7fffffffh,7fffffffh,7fffffffh,7fffffffh
NegMask1 dword 80000000h,80000000h,80000000h,80000000h, 80000000h,80000000h,80000000h,80000000h
end.686
.model flat, stdcall
option casemap:none
public AbsMask2
public NegMask2
_DATA1 SEGMENT ALIGN(32) FLAT 'DATA' ALIAS('.data')
AbsMask2 dword 7fffffffh,7fffffffh,7fffffffh,7fffffffh, 7fffffffh,7fffffffh,7fffffffh,7fffffffh
NegMask2 dword 80000000h,80000000h,80000000h,80000000h, 80000000h,80000000h,80000000h,80000000h
_DATA1 ends
endptr: 00402000
ptr: 00402020
ptr: 00402040
ptr: 00402060
ptr: 00402080
Quote from: TWell on May 25, 2017, 02:10:51 AM
Interesting.
Good you have FPC. :t
Try the code in attachment changing in AVX.ASM
.data
align 32
instead of
_DATA1 SEGMENT ALIGN(32) FLAT 'DATA' ALIAS('.data')
AbsMask dword 7fffffffh,7fffffffh,7fffffffh,7fffffffh, 7fffffffh,7fffffffh,7fffffffh,7fffffffh
NegMask dword 80000000h,80000000h,80000000h,80000000h, 80000000h,80000000h,80000000h,80000000h
_DATA1 ends
and tell if it works :bgrin:
update available to test:
http://www.terraspace.co.uk/uasm64.zip (http://www.terraspace.co.uk/uasm64.zip)
1) The new segment alignment switch is in, -Sp so you can see if that helps once linked in to HLL, you can just go back to using standard simplified directives .data .data? etc.
2) AW27, you can also re-check BND/MPX stuff, I found a bug with BND CALL, it was hardcoded to assume a 2 byte instruction when calculating the displacement, obviously with a BND prefix it becomes 3 so that fix is in..
Let me know how it goes.
John
Quote from: johnsa on May 25, 2017, 04:51:47 AM
1) The new segment alignment switch is in, -Sp so you can see if that helps once linked in to HLL, you can just go back to using standard simplified directives .data .data? etc.
I don't notice any benefit (the ASM module is from the attachment of my previous answer):
1- With new switch -Sp32 or -Sp
(http://www.atelierweb.com/a/Image_sp32.png)
2-Using complete segment directives:
(http://www.atelierweb.com/a/Image_nonsp32.png)
Conclusion: 1) does not align to 32-byte
Quote
2) AW27, you can also re-check BND/MPX stuff, I found a bug with BND CALL, it was hardcoded to assume a 2 byte instruction when calculating the displacement, obviously with a BND prefix it becomes 3 so that fix is in..
Does not crash, but I don't know what is the reason :badgrin:
That is very strange.. I can't see any real difference between what .data produces and the full directive :
_DATA SEGMENT ALIGN(32) FLAT 'DATA'
The only thing missing is the ALIAS.. I wonder if the reason is that from Delphi it's combining/merging data sections in a different order..
IE: using it's data section then taking on the asm module's .data, thus loosing the alignment, possibly because internally it probably just uses PARA ..
The base address of .data is definitely aligned, but the offset part isn't which is making me thing it's merging the sections together at link stage and somehow the offsets shift.. I wonder if the ALIAS is affecting that..
Quote from: johnsa on May 25, 2017, 10:26:52 PM
That is very strange.. I can't see any real difference between what .data produces and the full directive :
_DATA SEGMENT ALIGN(32) FLAT 'DATA'
The only thing missing is the ALIAS.. I wonder if the reason is that from Delphi it's combining/merging data sections in a different order..
IE: using it's data section then taking on the asm module's .data, thus loosing the alignment, possibly because internally it probably just uses PARA ..
The base address of .data is definitely aligned, but the offset part isn't which is making me thing it's merging the sections together at link stage and somehow the offsets shift.. I wonder if the ALIAS is affecting that..
This happens both with Delphi and Free Pascal Compiler.
If I don't use the ALIAS with Delphi or FPC, the segment is not linked. In other words, there is no error but the segment simply is not there together with its data. I am not aware of any compiler switch or whatever to force the linkage.
So if you just use .data then you wouldn't get the segment at all surely ?? _ perhaps the alias is required because you've named it _DATA1 and not _DATA.
The only thing I can think is using _DATA1 with the alias forces FPC/Delphi to link that segment in a different order that just "happens" to keep the align(32) where-as with the normal _DATA it gets merged in with it's own one and then doesn't respect align(32)
Quote from: johnsa on May 25, 2017, 10:46:37 PM
So if you just use .data then you wouldn't get the segment at all surely ?? _ perhaps the alias is required because you've named it _DATA1 and not _DATA.
The only thing I can think is using _DATA1 with the alias forces FPC/Delphi to link that segment in a different order that just "happens" to keep the align(32) where-as with the normal _DATA it gets merged in with it's own one and then doesn't respect align(32)
I can't keep the name _DATA, I will receive this error: Error A2078: Segment definition changed: _DATA, alignment
Ahh.. well i guess that is giving us the answer right there
_DATA is the "default", it's used by FPC/Delphi and probably has an alignment of para/16, so trying to merge one in with different attributes isn't possible.
It sounds like there is no fix for this apart from your manual segment work-around! :(
@John
With that new option -Sp32, data sections in object files is quite identical.
PEView is good tool to examine that.
As in my tests, as in message #71, Free Pascal linker works correctly.
Anyone can download that bare compiler ppc386.exe (https://sourceforge.net/projects/freepascal/files/Bootstrap/) and test it how it works without RTL.
Minimal support files are in that message.
We need proper examples for testing, to see when something goes wrong.
deleted
Nidud, if I followed your post correctly it seems like two different link orderings, and the first one the asm module comes first, alignment is correct, the second when it merges in after the HLL data the alignment is no longer respected?
deleted
deleted
So in conclusion, it was a good option to add (-Sp) and we've done it, but it won't solve the Delphi/FPC issue. I guess for that you'll have to resort to the manual work-around to avoid it merging the segments and only
respecting it's own 4 byte alignment.
deleted