HP Prime Miscalculating

[Joe Horn]

(10-23-2015 02:35 AM)luc4as Wrote:  When I try to calculate 3.6^2-4*3.24 in CAS mode, the calculator doesn't return 0, which is the correct answer. Home mode calculates it just fine.

Short answer: It's wrong in CAS for the same reason that .5-.3-.2 doesn't return 0, namely, garbage in garbage out. You THINK you put 3.6 and 3.24 into CAS, but you didn't.

Long answer: It's because Home uses BCD (which can represent 3.6 and 3.24 exactly) and CAS uses binary floating point (which can't). When you type 3.6 in Home, it's EXACTLY 3.6, but when you type 3.6 in CAS, it actually generates this value:

3.599999999999994315658113919198513031005859375 (exactly)

... because that's the largest number less than or equal to 3.6 which is representable with a 48-bit mantissa. For what it's worth, it's stored internally in hex scientific notation as 1.CCCCCCCCCCCCp+1 where "p" means "times 2 to the power of".

Quote: Does anyone know how to fix it?

Yes. You have two options:

Option #1: Use Home for all your floating-point math needs, and ONLY use CAS for EXACT & SYMBOLIC math (that is, integers with no decimal point, and variables).

Option #2: If you INSIST on doing that problem in CAS (why why why?) then type the decimal fractions as exact ratios of integers, like this:

Instead of: \({ 3.6 }^{ 2 }-4\times 3.24\), type this instead: \({ \left( \frac { 36 }{ 10 } \right) }^{ 2 }-4\times \frac { 324 }{ 100 } \)

CAS is designed for exact math, like that. Floating point reals with decimal points in them are usually NOT "exact" in CAS. If that makes CAS seem more complicated that it should be, see Option #1 above. Knowing when to use Home, and when to use CAS, is part of learning how to use the HP Prime.

"If Home and CAS behaved exactly the same, there would be no need for both to exist." -- Joe Horn