Comparing a qword (unsigned) to real8

[JK]

The FPU treats integer numbers signed by default. In case of a DWORD i can have it unsigned by zero extending it to a QWORD and then loading it into the FPU. The largest integer number, which can be loaded in the FPU is a QWORD, so this extension trick won´t work for a QWORD and the FPU.

This makes it impossible to compare an unsigned QWORD to a float inside the FPU in general, if i didn´t miss something. So how else could i do it?


Thanks

JK

[jj2007]

Check the sign bits of both operands in memory, then see which one is "bigger".

[HSE]

  I forget to read title.

You can take sign bit and load number without that bit, then add sign_bit*2^63.

[raymond]

Let's make one thing clear related to the title of this thread.

A REAL8 float only has a maximum accuracy of 53 significant bits. Thus, ANY dword having an absolute value exceeding 53 bits cannot be compared with full accuracy to a REAL8, almost regardless of any trick you may want to devise.

Secondly, zero extending a (unsigned) dword to a qword and loading it to the FPU requires the FPU to be in REAL10 mode. Comparing that accurate number to a REAL8 number in REAL10 mode
a- would NOT improve the accuracy of that REAL8 number,
b- would not be making a comparison in REAL8 mode.

[JK]

Maybe i should have written more general "floating point values" instead of "real8"!

It´s not about some trick for gaining accuracy, which i know is impossible. It´s about comparing unsigned integer numbers (qword beeing the largest) to floating point values. But i see your point now - such comparisons can get "problematic" because of accuracy.

The highest decimal value an unsigned qword can represent is 18.446.744.073.709.551.615 (2^64 - 1). This requires 20 decimal digits. A real4 has a mximum of 6 digits of precision, a real8 has 15 to 16 and a real10 has 18 to 19 digits of decimal precision.

So an exact comparison is not always possible, the best you can get sometimes is comparing "ranges" depending on the type of floating point and it´s accuracy. In other words the "range of inaccuracy" comparing an unsigned qword and a real10 could be up to be 2 decimal digits. We might get inaccurate results for qword values beyond 18 digits.

The only solution i see at the moment would be dividing both values by 100 (cut off 2 digits of precision, accept a granularity of 100) before loading and/or comparing an unsigned qword and a real10, if the qword needs more than 18 digits. Maybe dividing by 10 would be enough, i will test this ...


Thanks

JK

[HSE]

JK,

In that way, you only have to divide by 2.( "shr reg, 1" for uqword).

HSE

[jj2007]

Sorry to come back to MasmBasic, but ArraySort does Qword comparisons all the time, and fast. It's not rocket science. Do a lab search on ArraySort. I am on a dumbphone, can't help you more.

[JK]

@jj
comparing qwords to qwords or real8 to real8 is easy and straightforeward, the trouble begins, when you want to compare mixed types.

@all:
Having run some tests all i can say by now is: Comparisons between a signed qword and a real10 always yield a correct result. A signed qword has a maximum of 19 decimal places, which are fully covered by a real10. An unsigned qword needs a maximum of 20 decimal places, which doesn´t fit into the FPU.

And as a rule of thumb it is safe to compare word sized integers (signed and unsigned) to real4, dword sized integers (signed and unsigned) to real8. When trying to compare e.g a dword to a real4, you must consider that a dword with more than 6 decimal places cannot be exactly represented by a real4. Of course you will always get a result, but it might not be, what you expect.

It seems i must accept an inaccuracy in the last decimal place, when an unsigned qword reaches and exceeds 2^63. This is for comparing it to a real10, i must accept even more inaccuracy for real8 and real4.

I´m going to try, what HSE proposes ...


JK

[jj2007]

@jj
comparing qwords to qwords or real8 to real8 is easy and straightforeward, the trouble begins, when you want to compare mixed types

Oops, my fault. So it's apples vs oranges. No problem, once your real4 apple and your qword orange are on the fpu, they are both real10 fruit and can easily be compared using ucomisd or similar

[raymond]

@jj
comparing qwords to qwords or real8 to real8 is easy and straightforeward, the trouble begins, when you want to compare mixed types

Oops, my fault. So it's apples vs oranges. No problem, once your real4 apple and your qword orange are on the fpu, they are both real10 fruit and can easily be compared using ucomisd or similar

They may both be "fruits" BUT one is peeled and the other is not, even when both on the FPU.

[jj2007]

I like how you spin the analogy further, Raymond ;-)

[HSE]

They may both be "fruits" BUT one is peeled and the other is not, even when both on the FPU.

 Yes  There is something wrong in program design.

[jj2007]

Fact is that when you load a real4 of 1234567890123456.001 and compare it a qword 1234567890123456, the fpu will make a heroic effort to make you happy

[raymond]

As a follow up, the fact is also that if you load a real4 of 1234567890123456 and compare it to a qword 1234567890123456, the fpu will then definitely NOT make you happy.

[jj2007]

The fpu will make you unhappy, but the answer will be correct: there is NO real4 1234567890123456.