Machine epsilon of the HP Prime

[EdS2]

(09-17-2022 05:28 PM)rprosperi Wrote:  I suspect Home mode and CAS mode have different epsilons...

Indeed, I think they should! See for example Joe Horn's explanation:

(10-23-2015 05:41 AM)Joe Horn Wrote:  

(10-23-2015 02:35 AM)luc4as Wrote:  When I try to calculate 3.6^2-4*3.24 in CAS mode, the calculator doesn't return 0, which is the correct answer. Home mode calculates it just fine.

Short answer: It's wrong in CAS for the same reason that .5-.3-.2 doesn't return 0, namely, garbage in garbage out. You THINK you put 3.6 and 3.24 into CAS, but you didn't.

Long answer: It's because Home uses BCD (which can represent 3.6 and 3.24 exactly) and CAS uses binary floating point (which can't). When you type 3.6 in Home, it's EXACTLY 3.6, but when you type 3.6 in CAS, it actually generates this value:

3.599999999999994315658113919198513031005859375 (exactly)

... because that's the largest number less than or equal to 3.6 which is representable with a 48-bit mantissa. For what it's worth, it's stored internally in hex scientific notation as 1.CCCCCCCCCCCCp+1 where "p" means "times 2 to the power of".

...

"If Home and CAS behaved exactly the same, there would be no need for both to exist." -- Joe Horn