Round off error calculation example
|
07-23-2017, 10:16 AM
Post: #1
|
|||
|
|||
Round off error calculation example
I came across this example from youtube about the round off error on the older computer start from the earlier Pentium CPU which is about the floating point units. Here is the sample: ((2^54)+1)-(2^54) = 0 or 1 ?
I try this on two calculators Casio fx-300ES result is 0 and HP Prime result is 0 and try on the Google Search Calculator result is 0 and on the HP OmniGo 100 result is 0 Only from the WolfarmAlpha website gave result to 1 |
|||
« Next Oldest | Next Newest »
|
Messages In This Thread |
Round off error calculation example - Gamo - 07-23-2017 10:16 AM
RE: Round off error calculation example - Arno K - 07-23-2017, 10:29 AM
RE: Round off error calculation example - Joe Horn - 07-23-2017, 11:57 AM
RE: Round off error calculation example - Csaba Tizedes - 07-23-2017, 12:24 PM
RE: Round off error calculation example - Dieter - 07-23-2017, 05:07 PM
RE: Round off error calculation example - Mark Hardman - 07-23-2017, 06:32 PM
RE: Round off error calculation example - Dieter - 07-23-2017, 07:03 PM
RE: Round off error calculation example - Gamo - 07-24-2017, 01:17 AM
|
User(s) browsing this thread: 1 Guest(s)