Comparing a qword (unsigned) to real8

[daydreamer]

As a follow up, the fact is also that if you load a real4 of 1234567890123456 and compare it to a qword 1234567890123456, the fpu will then definitely NOT make you happy.

The gui editor warns showing red digits when any type has too many digits
Read my sigline 

[hutch--]

This is a comment from a maths illiterate, if you have an integer in a QWORD and the same number in a 64 bit float (REAL 8) then it is only a conversion between comparing the two. 12345678901234 can be directly compared with 12345678901234.0  . The only risk is if the REAL 8 also has a fraction so that if you converted the bare integer to a REAL 8 and a REAL 8 number with a fraction, you are not comparing the same thing.

[raymond]

The only risk is if the REAL 8 also has a fraction .....

Hutch

The main risk is for people, without sufficient knowledge of the fpu, to think that QWORDS being 64-bit integers can be compared directly to 64-bit floats because they have the same memory foot print. The fact is that those REAL 8 floats have a maximum accuracy of only 54 bits.

Then, if the QWORD is larger than 54 bits, it would be almost impossible to determine if it is exactly equal to any 64-bit float, regardless if the latter is a fractional number or not. However, because floats can represent numbers much larger than the integer range, it remains possible to determine if a 64-bit float is greater or smaller than any QWORD (which may be the eventual result if both happen to be equal).

[Gunther]

Raymond,

The fact is that those REAL 8 floats have a maximum accuracy of only 54 bits.

that is absolutely correct. Strictly speaking, there are only 53 significant bits. This makes the comparison with a QWORD complicated.

[jj2007]

It is so simple:

Code Select Expand

MyQ dq 1234567890123456789
  ..
  fld FP4(1234567890123456789.0)
  fild MyQ
  deb 4, "The difference", ST(0), ST(1)

The difference
ST(0)           1234567890123456789.
ST(1)           1234567939550609408.

I really don't know what there is to discuss

[hutch--]

I wonder what the result would be if you use the SSE2 SD instructions ? I would imagine that the only difference would be the fraction.

[jj2007]

The point is you can't "pump up" a REAL4 or REAL8 to the 19 digits precision of a QWORD. It's simply not possible. Only a REAL10 has a 64-bit alias "QWORD" mantissa:

Code Select Expand

  fld FP4(1234567890123456789.0)
  fld FP8(1234567890123456789.0)
  fld FP10(1234567890123456789.0)
  fild MyQ
  deb 4, "The difference", ST(0), ST(1), ST(2), ST(3)

Code Select Expand

The difference
ST(0)           1234567890123456789.
ST(1)           1234567890123456789.
ST(2)           1234567890123456768.
ST(3)           1234567939550609408.

[Gunther]

I really don't know what there is to discuss

I know it. If you read the thread carefully, you quickly realize who is discussing and firing smoke candles here.

The point is you can't "pump up" a REAL4 or REAL8 to the 19 digits precision of a QWORD. It's simply not possible.

Oh, really?

Only a REAL10 has a 64-bit alias "QWORD" mantissa:

The title of the thread is: Comparing a qword (unsigned) to real8. And by the way: The mantissa of a REAL10 value is usually something different than a
QWORD - may it be signed or unsigned.

[JK]

Finally i will stick to this solution:

Code Select Expand

  mov ecx,FFFFFFFD                       
  mov ebx,FFFFFFFF                       

  mov dword ptr ss:[ebp-40],ecx                       ;save qword in ecx/ebx to memory (could be eax/edx as well)
  mov dword ptr ss:[ebp-3C],ebx           
  fld st(0),tword ptr ds:[3C5848]                     ;load real10
  mov edx,dword ptr ss:[ebp-3C]           
  test edx,edx                           

  jns testme.3C1A50                                   ;skip, if sign bit of qword is not set
  shr dword ptr ss:[ebp-3C],1                         ; /2
  rcr dword ptr ss:[ebp-40],1             
  fild st(0),qword ptr ds:[3C58D4]                    ;2 (const)
  fdivp st(1),st(0)                                   ; /2

testme.3C1A50
  fild st(0),qword ptr ss:[ebp-40]                    ;load qword
  fcompp                                 
  fwait                                   
  fnstsw ax                               
  sahf                                   
  jne testme.3C1A9D                                   ;skip, if not equal
 
  ...
 
testme.3C1A9D 

this compares an unsigned qword to a real10 and these are some of the results:

18446744073709551614 = 18446744073709551614.0         -> ok
18446744073709551613 = 18446744073709551614.0         -> not ok
18446744073709551615 = 18446744073709551614.0         -> ok (wrong !!!)
18446744073709551614 = 18446744073709551613.0         -> not ok
18446744073709551614 = 18446744073709551615.0         -> not ok
18446744073709551613 = 18446744073709551613.0         -> not ok (wrong !!!)
18446744073709551613 = 18446744073709551614.0         -> not ok
18446744073709551613 = 18446744073709551615.0         -> not ok
18446744073709551613 = 18446744073709551612.0         -> ok (wrong !!!)

The pattern is obvious, these comparisons don´t always yield a correct result, because a real10 can hold a maximum of 19 valid decimal digits, so you might be off by 1. This gets even worse when comparing an unsigned qword to a real8 or real4. So you better don´t rely on results of such comparisons, unless you know about and understand the limitations.

Thanks for your input

JK 

[jj2007]

Here is an alternative way to deal with unsigned QWORDs (source attached, pure Masm32):

Code Select Expand

MyR10 REAL10 18446744073709551612.0 ; 2^64-4
MyQW QWORD 18446744073709551608 ; 2^64-8
...
  .Repeat
inc byte ptr MyQW
fild MyQW
.if !bl
fld FP10(18446744073709551616.0) ; add 2^64
fadd
.endif
fcomip ST, ST(1)
.if Zero?
print "equal", 13, 10
.elseif !Carry?
print "Qword bigger than Real10", 13, 10
.else
print "Qword smaller than Real10", 13, 10
.endif
inc esi
  .Until esi>=7

Code Select Expand

Qword smaller than Real10
Qword smaller than Real10
Qword smaller than Real10
equal
Qword bigger than Real10
Qword bigger than Real10
Qword bigger than Real10

[JK]

Thanks jj,

i must admit, i currently don´t understand, why your sample works! Playing a bit with it, i made the following changes to get the sign bit for each and every real10

Code Select Expand


...
  mov ebx, 1
  finit            ; important: set 64-bit precision
  fld MyR10
  ftst
  fstsw ax
  sahf
  jb @F

  xor ebx, ebx
@@:
;  movzx ebx, byte ptr MyR10[9]
;  and bl, 1            ; get the sign bit
...


It works for the numbers you gave, but it fails for e.g these numbers:

MyR10    REAL10 9223372036854775808.0    ; 2^63
MyQW    QWORD 9223372036854775806    ; 2^63-2

:

So the basic idea seems to work, but there is something missing (or i did something wrong)

[jj2007]

You are right, something is seriously wrong. Inter alia, the test for the sign bit should be inside the loop. Asap I'll investigate.

[JK]

jj, i think i got it:

Code Select Expand

  finit            ; important: set 64-bit precision
  fld MyR10

  xor esi, esi
  .Repeat
    add dword ptr MyQW, 1
    adc dword ptr MyQW[4], 0
    fild MyQW

  ftst
  fstsw ax
  sahf
  jae @F

  fld my64    ;FP10(18446744073709551616.0)    ; add 2^64
  fadd

@@:
    fcomip ST, ST(1)


you must not act on the sign bit of the float, you act on the sign bit of the qword. The above code passes all my tests so far ...
Which leaves the question, how exactly does it work, what does adding 2^64 do bitwise, so that the comparison yields a correct result - something like a two´s complement insde the FPU ?

JK

[raymond]

Just my "2-bits" if it may help understand some of the peculiarities of the fpu now or in the future:

-Whenever you load an integer onto the fpu, it automatically ALWAYS considers it as a SIGNED integer. Trying to load the qword 2⁶⁴-1 (i.e. FFFFFFFFFFFFFFFF), the fpu would consider it as the value of -1.

However, because the fpu has a 64-bit mantissa in REAL10 mode, and the sign bit is defined separately, it could represent the 2⁶⁴-1 value as a negative number even though that cannot be achieved as a 64-bit negative integer.

[jj2007]

Code Select Expand

    add dword ptr MyQW, 1
    adc dword ptr MyQW[4], 0

Excellent

you must not act on the sign bit of the float, you act on the sign bit of the qword

Indeed, that's what I meant with "the test for the sign bit should be inside the loop".

what does adding 2^64 do bitwise

As Raymond also showed, it simply turns a number like -3 into its unsigned equivalent. And that's what you want, right?

Simplified version attached (sahf & friends are awfully slow).