CAS precision - Printable Version +- HP Forums ( https://www.hpmuseum.org/forum)+-- Forum: HP Calculators (and very old HP Computers) ( /forum-3.html)+--- Forum: HP Prime ( /forum-5.html)+--- Thread: CAS precision ( /thread-4938.html) |

CAS precision - Han - 10-12-2015 07:19 PM
Does a CAS program behave like built-in CAS commands in the way it handles floating point numbers and intermediate results? RE: CAS precision - parisse - 10-13-2015 06:44 AM
Yes, it should. RE: CAS precision - Han - 10-16-2015 03:16 PM
Does using return(param) within a subroutine truncate any digits? One of the issues I have found is that matrix multiplication (row vector times column vector) gives me a slightly different result than calling a subroutine that does the multiplication and returns the inner product (the former is more accurate than the latter it seems). That is, t:=b*v; seems more accurate in the long run (which I use later on to for something else) than t:=inprod(b,v); where inprod(); is a cas routine whose code is: Code: `inprod(b,v):=` RE: CAS precision - parisse - 10-17-2015 04:05 PM
This might happen because the intermediate conversions double <-> giac::gen are not exactly the same in both processes. Every time a double is stored in a giac::gen, the 5 less significant bits are replaced by the type field, and in the other direction the 5 less significant bits are set to 0. |