Post Reply 
Casio FX991EX
07-14-2018, 05:08 PM
Post: #12
RE: Casio FX991EX
Quote:I'm not sure the calculator really has an internal 15-digit precision answer, as simple calculations
such as 1e-12+1-1 = 1e-12 versus 1e-13+1-1 = 0 suggests it only has 13 digits.

That was Casio trying to be "smart", by flushing close-to-zero to zero

http://people.ku.edu/~wzhuang/CAMseminar...ecrets.pdf

I was unable to figure out what the algorithm is:

Casio fx-3650p:
1e10 + 0.94 - 1e10 ==> 0
1e10 + 0.95 - 1e10 ==> 1

Casio fx-260 Solar
1e10 + 0.79 - 1e10 ==> 0
1e10 + 0.80 - 1e10 ==> 0.8
1e10 + 0.89 - 1e10 ==> 0.8
1e10 + 0.90 - 1e10 ==> 0.9

I really hate this feature. (above look like a bug to me)

With it on, 1e-13+1-1 = 0, can fx-991EX still claim 15-digits precision ?
Find all posts by this user
Quote this message in a reply
Post Reply 


Messages In This Thread
Casio FX991EX - Tugdual - 07-12-2018, 08:09 PM
RE: Casio FX991EX - Csaba Tizedes - 07-12-2018, 08:28 PM
RE: Casio FX991EX - Tugdual - 07-12-2018, 11:08 PM
RE: Casio FX991EX - brickviking - 07-14-2018, 03:24 AM
RE: Casio FX991EX - Csaba Tizedes - 11-06-2018, 10:57 AM
RE: Casio FX991EX - klesl - 11-07-2018, 05:28 PM
RE: Casio FX991EX - Csaba Tizedes - 11-07-2018, 07:03 PM
RE: Casio FX991EX - brickviking - 11-07-2018, 08:31 PM
RE: Casio FX991EX - lrdheat - 07-12-2018, 10:03 PM
RE: Casio FX991EX - Albert Chan - 07-14-2018, 12:21 AM
RE: Casio FX991EX - ijabbott - 07-14-2018, 02:12 AM
RE: Casio FX991EX - Albert Chan - 07-14-2018, 03:32 AM
RE: Casio FX991EX - ijabbott - 07-14-2018, 08:48 AM
RE: Casio FX991EX - Albert Chan - 07-14-2018, 12:00 PM
RE: Casio FX991EX - ijabbott - 07-14-2018, 12:38 PM
RE: Casio FX991EX - grsbanks - 07-15-2018, 04:58 PM
RE: Casio FX991EX - ijabbott - 07-15-2018, 05:04 PM
RE: Casio FX991EX - Albert Chan - 07-14-2018 05:08 PM
RE: Casio FX991EX - lrdheat - 07-15-2018, 12:43 AM
RE: Casio FX991EX - lrdheat - 07-15-2018, 12:45 AM
RE: Casio FX991EX - lrdheat - 07-15-2018, 01:09 AM
RE: Casio FX991EX - John Keith - 07-15-2018, 12:21 PM
RE: Casio FX991EX - Tugdual - 07-15-2018, 07:58 AM
RE: Casio FX991EX - Tugdual - 07-15-2018, 09:17 AM
RE: Casio FX991EX - Albert Chan - 07-15-2018, 11:51 AM
RE: Casio FX991EX - ijabbott - 07-15-2018, 04:39 PM
RE: Casio FX991EX - ijabbott - 07-15-2018, 04:51 PM
RE: Casio FX991EX - Albert Chan - 07-15-2018, 05:39 PM
RE: Casio FX991EX - ijabbott - 07-15-2018, 07:53 PM
RE: Casio FX991EX - rprosperi - 07-16-2018, 12:41 AM
RE: Casio FX991EX - Steve Simpkin - 07-16-2018, 02:58 PM
RE: Casio FX991EX - Albert Chan - 07-16-2018, 03:54 PM
RE: Casio FX991EX - Tugdual - 07-16-2018, 09:33 AM
RE: Casio FX991EX - Albert Chan - 07-16-2018, 11:53 AM
RE: Casio FX991EX - Albert Chan - 08-09-2018, 06:15 PM
RE: Casio FX991EX - rprosperi - 08-09-2018, 06:35 PM
RE: Casio FX991EX - ijabbott - 08-09-2018, 07:33 PM
RE: Casio FX991EX - Albert Chan - 08-09-2018, 08:23 PM
RE: Casio FX991EX - Albert Chan - 09-06-2018, 11:19 PM
RE: Casio FX991EX - Eddie W. Shore - 11-07-2018, 02:22 AM
RE: Casio FX991EX - Csaba Tizedes - 11-07-2018, 12:30 PM
RE: Casio FX991EX - edryer - 12-28-2018, 09:16 PM
RE: Casio FX991EX - Albert Chan - 12-28-2018, 11:04 PM
RE: Casio FX991EX - Luigi Vampa - 12-29-2018, 01:29 AM
RE: Casio FX991EX - Hlib - 10-31-2019, 08:46 PM
RE: Casio FX991EX - ijabbott - 11-01-2019, 09:46 PM



User(s) browsing this thread: 1 Guest(s)