Post Reply 
COS(X) dislpay wrong answer
09-02-2018, 01:39 PM (This post was last modified: 09-02-2018 01:41 PM by Albert Chan.)
Post: #8
RE: COS(X) dislpay wrong answer
I was curious of how Prime round the 53 bits float to its internal 48-bits mantissa.
Can someone with a Prime try this for me ? (CAS mode)

sin(3.0) * 2^50 - 158887e9

With IEEE 53-bits mantissa float, I get 3928234.84375

Will Prime store the chopped mantissa, result of 3928234 ?
Or, will it stored better rounded 48 bits, result of 3928235 ?
Find all posts by this user
Quote this message in a reply
Post Reply 


Messages In This Thread
COS(X) dislpay wrong answer - Magnus512 - 09-01-2018, 11:46 PM
RE: COS(X) dislpay wrong answer - JMB - 09-02-2018, 01:02 PM
RE: COS(X) dislpay wrong answer - Albert Chan - 09-02-2018 01:39 PM



User(s) browsing this thread: 4 Guest(s)