Math10 library

[jj2007]

also - Raymond's FPU lib is likely to have a pretty good routine

Dave, it uses the BCD method     

Test yourself:
   MovVal MyR10, "-123456789012345678.90123E123"
   PrintLine Str$(MyR10)
   invoke FpuAtoFL, Chr$("-123456789012345678.90123E123"), offset MyR10, DEST_MEM
   PrintLine Str$(MyR10)

-1.234567890123456789e+140
-1.234567890123456780e+140

It seems one digit gets lost in translation.

[RuiLoureiro]

Jochen,
«It seems one digit gets lost in translation»

            yes it lose digits !
            I tested some cases

[jj2007]

Indeed. For example,

  invoke FpuAtoFL, chr$("-123456789012345678.99E123"), 0, DEST_FPU
yields -1.234567890123456781e+140 instead of ...79

It could have to do with limits of the BCD conversion (the MasmBasic Val algo uses qwords, not BCD numbers).

[dedndave]

yuppers - from Ray's FpuAtoFL...

Code Select Expand

integer:
      cmp   al,"."
      jnz   @F
      lea   edi,bcdstri
      call  load_integer
      lodsb
      lea   edi,bcdstrf+8
      mov   cl,18              ;<----
      and   bh,4
      jmp   decimals
personally, i would accept up to 21 user-entered decimal digits - and round the last one to 20 digits
that should always yield the closest binary float

[RuiLoureiro]

Dave,

        If i get only 3 decimals from the string below
        the BCD converter gives the same as MASM

_r10MASM1       real10 -123456789012345678.987654321098765432E4914

                7FFB_D484 A3B0_4487 4A03h     <- masm
                7FFB_D484 A3B0_4487 4A03h     <- my BCD converter

[dedndave]

i was just reading the Sun Numerical Computation Guide
(it was on my desktop, and i clicked on it by accident - lol)

it is telling me that x86 extended reals can yield 18 to 21 signifigant decimal digits
that's news to me   :redface:

[jj2007]

21 sounds unlikely - no miracles under the Sun
In reply #41 above, you can see that 3 LSBs cover the range of 1.234567890123456789. With some imagination, you could get something like
1.2345678901234567892
1.2345678901234567895
1.2345678901234567898
when analysing the LSB, but that is still 20 digits, not 21. And remember that we got a theoretical limit of 18 for numbers that start with 9.22xxx and higher. Statistically speaking, you could neglect these 7% of all numbers, but I wouldn't trust a library that suggests this strategy

[dedndave]

i haven't tried it
but, maybe if the number is something like 1.00000000000000000001, it might work   ::)
you'd have to try a few adjacent values to see
1.00000000000000000001
1.00000000000000000002
1.00000000000000000003

[MichaelW]

i haven't tried it
but, maybe if the number is something like 1.00000000000000000001, it might work   ::)
you'd have to try a few adjacent values to see
1.00000000000000000001
1.00000000000000000002
1.00000000000000000003

It seems to me that the adjacent values of interest are the representable values.

[RuiLoureiro]

    I wrote a new procedure to convert ascii (19 digits) to real10 format

    Here are some results (first word is the exponent than qword)
    _r10MASM_  -> converted by masm
    _r10RCL_     ->         "        by my proc StrToRelCnv
    In some cases the last byte is not equal
   

Code Select Expand


_r10MASM_1   real10 123456789012345678.0     4037_DB4D A5D3_1879 A700h 
_r10RCL_1                                    4037_DB4D A5D3_1879 A700h

_r10MASM_2  real10 1.23456789012345678E17 4037_DB4D A5D3_1879 A700h     
_r10RCL_2                                 4037_DB4D A5D3_1879 A700h

_r10MASM_3   real10 123456789012345678.1     4037_DB4D A5D3_1879 A70Dh
_r10RCL_3                                    4037_DB4D A5D3_1879 A70Eh

_r10MASM_4   real10 123456789012345678.2     4037_DB4D A5D3_1879 A71Ah
_r10RCL_4                                    4037_DB4D A5D3_1879 A71Bh

_r10MASM_5   real10 123456789012345678.3     4037_DB4D A5D3_1879 A726h
_r10RCL_5                                    4037_DB4D A5D3_1879 A728h

_r10MASM_6   real10 123456789012345678.4     4037_DB4D A5D3_1879 A733h
_r10RCL_6                                    4037_DB4D A5D3_1879 A734h

_r10MASM_7   real10 123456789012345678.5     4037_DB4D A5D3_1879 A740h
_r10RCL_7                                    4037_DB4D A5D3_1879 A741h

_r10MASM_8   real10 123456789012345678.6     4037_DB4D A5D3_1879 A74Dh
_r10RCL_8                                    4037_DB4D A5D3_1879 A74Eh

_r10MASM_9   real10 123456789012345678.7     4037_DB4D A5D3_1879 A75Ah
_r10RCL_9                                    4037_DB4D A5D3_1879 A75Bh

_r10MASM_10  real10 123456789012345678.8     4037_DB4D A5D3_1879 A766h
_r10RCL_10                                   4037_DB4D A5D3_1879 A768h

_r10MASM_11 real10 123456789012345678.9     4037_DB4D A5D3_1879 A773h
_r10RCL_11                                  4037_DB4D A5D3_1879 A774h

_r10MASM_12 real10 123456789012345678.98    4037_DB4D A5D3_1879 A77Dh
_r10RCL_12                                  4037_DB4D A5D3_1879 A77Eh

_r10MASM_13   real10 -123456789.0E320         4440_EE43 C218_CA57 BB04h
_r10RCL_13      db "-123456789.0E320", 0      4440_EE43 C218_CA57 BB62h

_r10MASM_14   real10 -123456789.12345E320     4440_EE43 C21C_C9A0 1C66h
_r10RCL_14      db "-123456789.12345E320",0   4440_EE43 C21C_C9A0 1A1Ah  **

_r10MASM_15   real10 1.0E320                  4426_8184 2F29_F2CC E376h
_r10RCL_15      db "1.0E320",0                4426_8184 2F29_F2CC E3A9h

;-----------------------------------------------
_r10MASM_30   real10 -0.1                     3FFB_CCCC CCCC_CCCC CCCDh
_r10RCL_30      db "-0.1",0                   3FFB_CCCC CCCC_CCCC CCCEh

_r10MASM_31   real10 -0.01                    3FF8_A3D7 0A3D_70A3 D70Ah
_r10RCL_31      db "-0.01",0                  3FF8_A3D7 0A3D_70A3 D70Bh

_r10MASM_32   real10 -0.001                   3FF5_8312 6E97_8D4F DF3Bh
_r10RCL_32      db "-0.001",0                 3FF5_8312 6E97_8D4F DF3Fh

_r10MASM_33   real10 -0.0001                  3FF1_D1B7 1758_E219 652Ch
_r10RCL_33      db "-0.0001",0                3FF1_D1B7 1758_E219 652Eh

_r10MASM_34   real10 -0.00001                 3FEE_A7C5 AC47_1B47 8423h
_r10RCL_34      db "-0.00001",0               3FEE_A7C5 AC47_1B47 8422h

_r10MASM_35   real10 -0.000001                3FEB_8637 BD05_AF6C 69B6h
_r10RCL_35      db "-0.000001",0              3FEB_8637 BD05_AF6C 69BEh

_r10MASM_36   real10 -0.0000001               3FE7_D6BF 94D5_E57A 42BCh
_r10RCL_36      db "-0.0000001",0             3FE7_D6BF 94D5_E57A 42C6h

_r10MASM_37   real10 -0.00000001              3FE4_ABCC 7711_8461 CEFDh
_r10RCL_37      db "-0.00000001",0            3FE4_ABCC 7711_8461 CF01h  **

_r10MASM_38   real10 -0.000000001             3FE1_8970 5F41_36B4 A597h
_r10RCL_38      db "-0.000000001",0           3FE1_8970 5F41_36B4 A598h

_r10MASM_39   real10 -0.0000000001            3FDD_DBE6 FECE_BDED D5BFh
_r10RCL_39      db "-0.0000000001",0          3FDD_DBE6 FECE_BDED D5BCh

_r10MASM_40   real10 -0.00000000001           3FDA_AFEB FF0B_CB24 AAFFh
_r10RCL_40      db "-0.00000000001",0         3FDA_AFEB FF0B_CB24 AAFAh

_r10MASM_41   real10 -0.000000000001          3FD7_8CBC CC09_6F50 88CCh
_r10RCL_41      db "-0.000000000001",0        3FD7_8CBC CC09_6F50 88DDh

_r10MASM_42   real10 -0.0000000000001         3FD3_E12E 1342_4BB4 0E13h
_r10RCL_42      db "-0.0000000000001",0       3FD3_E12E 1342_4BB4 0E2Bh
...
_r10MASM_47   real10 -0.000000000000000001    3FC3_9392 EE8E_921D 5D07h
_r10RCL_47      db "-0.000000000000000001",0  3FC3_9392 EE8E_921D 5D09h


_r10MASM_50   real10 -12345678901234567.987654321098765432E4915
_r10RCL_50      db "-12345678901234567.987654321098765432E4915", 0

                             _r10MASM_50 ->     7FFB_D484 A3B0_4487 4A71h
                              _r10rcl_50  ->    7FFB_D484 A3B0_4487 3331h

_r10MASM_51   real10 -0.1234567890123456789     3FFB_FCD6 E9BA_37B2 F8E2h
_r10RCL_51      db "-0.1234567890123456789", 0  3FFB_FCD6 E9BA_37B2 F8E1h

_r10RCL_52      real10 -1.2345678912345678934E-4930   
                                              0006_92E1 63E1_EAE4 BDF5h
_r10RCL_52      db "-1.2345678912345678934E-4930", 0   
                                              0006_92E1 63E1_EAE4 BDEFh



[RuiLoureiro]

More data starting with 922.., 93...

Code Select Expand


_r10MASM_60     real10 9223372036854775807.0    403D_FFFF FFFF_FFFF FFFEh
_r10RCL_60      db "9223372036854775807.0",0    403D_FFFF FFFF_FFFF FFFEh
     
_r10MASM_61     real10 9223372036854775808.0    403E_8000 0000_0000 0000h
_r10RCL_61      db "9223372036854775808.0",0    403E_8000 0000_0000 0000h

_r10MASM_62     real10 9233372036854775808.0    403E_8023 86F2_6F61 0000h
_r10RCL_62      db "9233372036854775808.0", 0   403E_8023 86F2_6F61 0000h


_r10MASM_63     real10 9333372036854775808.0    403E_8186 CC6A_CD4B 0000h
_r10RCL_63      db "9333372036854775808.0", 0   403E_8186 CC6A_CD4B 0000h

       
_r10MASM_64     real10 9333372036854775809.9333372036854775808       
                                                403E_8186 CC6A_CD4B 0001h
_r10RCL_64      db"9333372036854775809.9333372036854775808", 0       
                                                403E_8186 CC6A_CD4A FFF9h

_r10MASM_65     real10 9333372036854775809.9333372036854775808E4913
                                                7FFE_C8D4 9910_070B BD08h
_r10RCL_65      db "9333372036854775809.9333372036854775808E4913", 0
                                                7FFE_C8D4 9910_070B B1E9h


[RuiLoureiro]

Hi
    Here we have a math library of real10 numbers version 0.04

    We can test each function using TestMath10

    We can find the documentation in the file Math10.txt

    We have more 2 procedures to convert real10 to ascci
    and ascii to real10 with 19 digits

                StrToRealCnv and RealToStrCnv

    Test it with TestMath10.exe. Choose Table1 to Table12
    and ConvertDn or ConvertUp.
    All values are also converted by MASM. Compare them.
   
    Good Luck

Jochen,
        Now you can see PI with 19 digits (use TestMath10.exe)                   
        Try it and say something !

[jj2007]

Rui,

Congrats, so you made it to full precision :t
How do you handle the 9.22x case? Check for the 18-digits range?

Best, Jochen

[RuiLoureiro]

Jochen,
        Thanks ! :t
       
       «How do you handle the 9.22x case? Check for the 18-digits range?»

        No. I convert and test for 7FFFFFFFh. If above i move the last digit
        from integer part to the decimal part, so i convert only 18 digits.

[jj2007]

Yes, that is what I meant. Good job, Rui :t