Post Reply 
Python round() broken
04-17-2021, 07:08 PM (This post was last modified: 04-19-2021 07:47 AM by Guenter Schink.)
Post: #1
Python round() broken
Is it just my G2 to show this error in Python?

round(3/7,6) ==> 0.4285709999999999 (if I counted the 9s correctly)
should be 0.2428571
similar error with
round(5/7,6)
round(3/11,6)
round(5/11,6
round(7/11,6)

previously in my post a first digit "2"crept in inadvertently

The virtual calculator 64bit doesn't show this behavior.

Günter

PS: of course reported to calcbeta too
Find all posts by this user
Quote this message in a reply
04-17-2021, 08:48 PM
Post: #2
RE: Python round() broken
(04-17-2021 07:08 PM)Guenter Schink Wrote:  Is it just my G2 to show this error in Python?

round(3/7,6) ==> 0.24285709999999999 (if I counted the 9s correctly)
should be 0.2428571

Wow! Your calculator is really off! I get

0.4285709999999999

Smile

Tom L
Cui bono?
Find all posts by this user
Quote this message in a reply
04-17-2021, 11:10 PM
Post: #3
RE: Python round() broken
(04-17-2021 08:48 PM)toml_12953 Wrote:  
(04-17-2021 07:08 PM)Guenter Schink Wrote:  Is it just my G2 to show this error in Python?

round(3/7,6) ==> 0.24285709999999999 (if I counted the 9s correctly)
should be 0.2428571

Wow! Your calculator is really off! I get

0.4285709999999999

Smile

grrr.. typo - first digit "2" shouldn't be there. But what would you expect with rounding to 6 digits? The result should be 0.428571

Günter
Find all posts by this user
Quote this message in a reply
04-18-2021, 12:32 AM (This post was last modified: 04-18-2021 12:33 AM by toml_12953.)
Post: #4
RE: Python round() broken
(04-17-2021 11:10 PM)Guenter Schink Wrote:  
(04-17-2021 08:48 PM)toml_12953 Wrote:  Wow! Your calculator is really off! I get

0.4285709999999999

Smile

grrr.. typo - first digit "2" shouldn't be there. But what would you expect with rounding to 6 digits? The result should be 0.428571

Günter
Yes, You added an extra 2. I knew that it wasn't your calculator, hence the smiley.

Tom L
Cui bono?
Find all posts by this user
Quote this message in a reply
04-19-2021, 09:48 AM
Post: #5
RE: Python round() broken
Hello,

I have literally NO clue as to what is going on there!
Works on emulator, works on prime G1... but prime G2 fails!

Something buried deep in the float library or something like that?

Note that Python uses doubles... As such, these are binary numbers... not BCD... so things like rounding will/Can yeild unexpected results...

Cyrille

Although I work for the HP calculator group, the views and opinions I post here are my own. I do not speak for HP.
Find all posts by this user
Quote this message in a reply
Post Reply 




User(s) browsing this thread: 1 Guest(s)