Author Topic: Small favour needed, test precision with sprintf.  (Read 7986 times)

Siekmanski

  • Member
  • *****
  • Posts: 2326
Re: Small favour needed, test precision with sprintf.
« Reply #30 on: September 11, 2018, 04:05:42 AM »
Hi Raymond,

As far as I understand correctly, the maximum number of digits:

real4     7 digits (32/4-1)
real8   15 digits (64/4-1)
real10 19 digits (80/4-1)

Is this correct?
Creative coders use backward thinking techniques as a strategy.

nidud

  • Member
  • *****
  • Posts: 1980
    • https://github.com/nidud/asmc
Re: Small favour needed, test precision with sprintf.
« Reply #31 on: September 11, 2018, 04:24:07 AM »
Not really, no, but in order to be converted to a 64-bit integer value the number has to be <= to 64-bit, else the return value will be cropped to zero/max|min int.
Code: [Select]
    DoubleToInt(1000000000000000000000000000000000000000000000000000000.0)
    printf("%I64d.%I64d\n", rdx, rax)
    DoubleToInt(1.0E100)
    printf("%I64d.%I64d\n", rdx, rax)

-9223372036854775808.-9223372036854775808
-9223372036854775808.-9223372036854775808

AW

  • Member
  • *****
  • Posts: 2583
  • Let's Make ASM Great Again!
Re: Small favour needed, test precision with sprintf.
« Reply #32 on: September 11, 2018, 05:29:26 AM »
Hi Raymond,

As far as I understand correctly, the maximum number of digits:

real4     7 digits (32/4-1)
real8   15 digits (64/4-1)
real10 19 digits (80/4-1)

Is this correct?

Interesting rule of thumb, but is not based on the number of available decimal digits according to the IEEE format.
These are: 64, 52 and 23 bits for REAL10, REAL8 and REAL4 respectively
My calculation is the following (who knows better please contradict):

X=(ln 10)/(ln 2)=3.3219280948873623478703194294894

64/X = 19.26 => 19/20

52/X 15.65 => 15/16

23/X = 6.92 => 6/7

So, to be on the safe side, the number of guaranteed significant digits is: 19, 15, 6 respectively for REAl10, REAL8 and REAL4.



Siekmanski

  • Member
  • *****
  • Posts: 2326
Re: Small favour needed, test precision with sprintf.
« Reply #33 on: September 11, 2018, 09:42:43 AM »
From MSDN,

The mantissa is stored as a binary fraction greater than or equal to 1 and less than 2. For types float and double, there is an implied leading 1 in the mantissa in the most-significant bit position, so the mantissas are actually 24 and 53 bits long, respectively, even though the most-significant bit is never stored in memory.

https://msdn.microsoft.com/en-us/library/hd7199ke.aspx
Creative coders use backward thinking techniques as a strategy.

jj2007

  • Member
  • *****
  • Posts: 10543
  • Assembler is fun ;-)
    • MasmBasic
Re: Small favour needed, test precision with sprintf.
« Reply #34 on: September 11, 2018, 10:03:31 AM »
so the mantissas are actually 24 and 53 bits long

Test it... note the last digits in the range 8.xx and above (53/3.321928095=15.95 digits):
Code: [Select]
include \masm32\include\masm32rt.inc

.data
TmpR8 REAL8 ?
.code
start:
  xor ebx, ebx ; set two non-volatile
  finit
  fld1
  fld FP10(1.2345678901234567890)
  .Repeat
fld st
fstp TmpR8
printf("%.17f\n", TmpR8)
fadd ST, ST(1)
inc ebx
  .Until ebx>=8
  .Repeat
fld st
fstp TmpR8
printf("%.17f\n", TmpR8)
fsub ST, ST(1)
dec ebx
  .Until Sign?
  inkey
  exit
end start

nidud

  • Member
  • *****
  • Posts: 1980
    • https://github.com/nidud/asmc
Re: Small favour needed, test precision with sprintf.
« Reply #35 on: September 11, 2018, 10:15:57 AM »
Same for real16, 113/112-bit. The bit is always set unless the value is zero.

raymond

  • Member
  • **
  • Posts: 244
    • Raymond's page
Re: Small favour needed, test precision with sprintf.
« Reply #36 on: September 11, 2018, 12:37:12 PM »
Hi Raymond,

As far as I understand correctly, the maximum number of digits:

real4     7 digits (32/4-1)
real8   15 digits (64/4-1)
real10 19 digits (80/4-1)

Is this correct?

I generally agree with those limits, except that I consider that the range for the real8 to be 16 significant digits for all intents and purposes.

Quote
Same for real16, 113/112-bit. The bit is always set unless the value is zero.

I'm not familiar with the real16. However, for real4 and real8, the above statement is not 100% correct. It should read:
"The bit is always set unless the biased exponent is zero." i.e. the value of the real number is either 0 or that of a 'denormalized real number'. For those following this thread but not entirely familiar with this subject, some more info on 'denormalized real numbers' is available at
 http://www.ray.masmcode.com/tutorial/fpuchap2.htm#denormal
Whenever you assume something, you risk being wrong half the time.
http://www.ray.masmcode.com/

hutch--

  • Administrator
  • Member
  • ******
  • Posts: 7537
  • Mnemonic Driven API Grinder
    • The MASM32 SDK
Re: Small favour needed, test precision with sprintf.
« Reply #37 on: September 11, 2018, 01:08:33 PM »
Hi Ray,

I well understand what can be fitted into a REAL8 variable but when defining a .data section entry, the assembler simply truncates values that are larger than REAL8 so I can routinely do things like this.

  .data
  align 16
  pi REAL8 3.14159265358979323846

A large number library is probably the only way to do really big numbers, REAL10 gets some extra precision but probably not enough for some of the scientific folk.
hutch at movsd dot com
http://www.masm32.com    :biggrin:  :skrewy:

jj2007

  • Member
  • *****
  • Posts: 10543
  • Assembler is fun ;-)
    • MasmBasic
Re: Small favour needed, test precision with sprintf.
« Reply #38 on: September 11, 2018, 08:03:41 PM »
that I consider that the range for the real8 to be 16 significant digits for all intents and purposes.

Simple pure Masm32 test, using REAL8 precision with CRT printf (source & exe attached):
Code: [Select]
include \masm32\include\masm32rt.inc

.data
TmpR8 REAL8 ?
.code
start:
  xor ebx, ebx ; set two non-volatile
  finit
  fld1
  fld FP10(1.0000000000000011111111)
  print "1.2345678901234567", 13, 10
  print "1.0000000000000011111 as REAL10", 13, 10
  .Repeat
fld st
fstp TmpR8
printf("%.16f\n", TmpR8)
fadd ST, ST(1)
inc ebx
  .Until ebx>=8
  .Repeat
fld st
fstp TmpR8
printf("%.16f\n", TmpR8)
fsub ST, ST(1)
dec ebx
  .Until Sign?
  inkey "0.0000000000000011111 as REAL10 on exit"
  exit
end start

Output:
Code: [Select]
1.2345678901234567
1.0000000000000011111 as REAL10
1.0000000000000011  +-0
2.0000000000000013  +2 at pos 17
3.0000000000000013  +2
4.0000000000000009  -2
5.0000000000000009
6.0000000000000009
7.0000000000000009
8.0000000000000018  +7
9.0000000000000018  +7    <<<<< max deviation
8.0000000000000018  +7
7.0000000000000009
6.0000000000000009
5.0000000000000009
4.0000000000000009  -2
3.0000000000000013  +2
2.0000000000000013
1.0000000000000011  +-0
0.0000000000000011111 as REAL10 on exit

It is the middle range where the 15.95 digits show a small distortion.

hutch--

  • Administrator
  • Member
  • ******
  • Posts: 7537
  • Mnemonic Driven API Grinder
    • The MASM32 SDK
Re: Small favour needed, test precision with sprintf.
« Reply #39 on: September 12, 2018, 01:33:05 AM »
I had to laugh, working with formulas involves stuff I learnt in primary school over 60 years ago. Back then before the digital era it was taught as fractions and little has changed apart from it being taught in digital format.
hutch at movsd dot com
http://www.masm32.com    :biggrin:  :skrewy:

raymond

  • Member
  • **
  • Posts: 244
    • Raymond's page
Re: Small favour needed, test precision with sprintf.
« Reply #40 on: September 12, 2018, 03:02:07 AM »
If the accuracy of that 16th digit with the real8 becomes sooooooooo important, the use of a real10 should then be investigated!!!!

The accuracy of a 17th digit with real8 is definitely not probable.
Whenever you assume something, you risk being wrong half the time.
http://www.ray.masmcode.com/

Siekmanski

  • Member
  • *****
  • Posts: 2326
Re: Small favour needed, test precision with sprintf.
« Reply #41 on: September 12, 2018, 04:05:19 AM »
Code: [Select]
1.2345678901234567
1.0000000000000011111 as REAL10
1.000000000000001
2.000000000000001
3.000000000000001
4.000000000000001
5.000000000000001
6.000000000000001
7.000000000000001
8.000000000000002
9.000000000000002
8.000000000000002
7.000000000000001
6.000000000000001
5.000000000000001
4.000000000000001
3.000000000000001
2.000000000000001
1.000000000000001
0.0000000000000011111 as REAL10 on exit
Creative coders use backward thinking techniques as a strategy.

AW

  • Member
  • *****
  • Posts: 2583
  • Let's Make ASM Great Again!
Re: Small favour needed, test precision with sprintf.
« Reply #42 on: September 12, 2018, 06:23:27 AM »
I am missing something for sure but when Raymond says  "I generally agree with those limits, except that I consider that the range for the real8 to be 16 significant digits for all intents and purposes." and I look at the JJ output where almost half the values have 15  significant digits, can I conclude that it is really bad luck for a calculated 15.95 value?

nidud

  • Member
  • *****
  • Posts: 1980
    • https://github.com/nidud/asmc
Re: Small favour needed, test precision with sprintf.
« Reply #43 on: September 12, 2018, 06:40:24 AM »
The size of the mantissa is in bits and digits have a different radix. The same logic for a byte. Three digits may fit if the value is less than 256.

jj2007

  • Member
  • *****
  • Posts: 10543
  • Assembler is fun ;-)
    • MasmBasic
Re: Small favour needed, test precision with sprintf.
« Reply #44 on: September 12, 2018, 06:43:20 AM »
almost half the values have 15  significant digits

When using rounding, only 3 out of 17 are off by one at the 16th digit - see Marinus' output.