COS(X) dislpay wrong answer
|
09-02-2018, 01:39 PM
(This post was last modified: 09-02-2018 01:41 PM by Albert Chan.)
Post: #8
|
|||
|
|||
RE: COS(X) dislpay wrong answer
I was curious of how Prime round the 53 bits float to its internal 48-bits mantissa.
Can someone with a Prime try this for me ? (CAS mode) sin(3.0) * 2^50 - 158887e9 With IEEE 53-bits mantissa float, I get 3928234.84375 Will Prime store the chopped mantissa, result of 3928234 ? Or, will it stored better rounded 48 bits, result of 3928235 ? |
|||
« Next Oldest | Next Newest »
|
Messages In This Thread |
COS(X) dislpay wrong answer - Magnus512 - 09-01-2018, 11:46 PM
RE: COS(X) dislpay wrong answer - Albert Chan - 09-02-2018, 01:10 AM
RE: COS(X) dislpay wrong answer - Magnus512 - 09-02-2018, 02:28 AM
RE: COS(X) dislpay wrong answer - Albert Chan - 09-02-2018, 02:56 AM
RE: COS(X) dislpay wrong answer - Magnus512 - 09-02-2018, 03:40 AM
RE: COS(X) dislpay wrong answer - Albert Chan - 09-02-2018, 12:10 PM
RE: COS(X) dislpay wrong answer - Magnus512 - 09-02-2018, 04:33 PM
RE: COS(X) dislpay wrong answer - rprosperi - 09-03-2018, 01:55 AM
RE: COS(X) dislpay wrong answer - JMB - 09-02-2018, 01:02 PM
RE: COS(X) dislpay wrong answer - Magnus512 - 09-02-2018, 04:22 PM
RE: COS(X) dislpay wrong answer - Albert Chan - 09-02-2018 01:39 PM
RE: COS(X) dislpay wrong answer - Magnus512 - 09-02-2018, 04:16 PM
|
User(s) browsing this thread: 4 Guest(s)