Search
Member List
Calendar
Help
|
A quick precision test
|
|
06-03-2014, 12:27 AM
(This post was last modified: 06-03-2014 12:27 AM by pito.)
Post: #5
|
|||
|
|||
|
RE: A quick precison test
Now I am little bit confused. Markus wrote in the cordic topic the standard precision used is 39 digits.
Does it mean you do rounding on 16th place with "single precision" and 34th in "double" after each calculation? Why do you use the 39digit precision as the standard one then? My naive understanding (sorry I am newbie with wp34s) is the 39digit precision is used across any calculations inside the box so it is always maintained as 39. |
|||
|
|
« Next Oldest | Next Newest »
|
User(s) browsing this thread: 13 Guest(s)