FORTRAN floating point accuracy problems
|
04-11-2016, 12:35 PM
Post: #41
|
|||
|
|||
RE: FORTRAN floating point accuracy problems
(04-11-2016 12:03 PM)Dieter Wrote: In this case Excel indeed returns 20 for INT(100*dms). Indeed it does.. but it's an example of Excel 'fiddling' - it adjusted the results to match its decimal audience ;-) Free42 Binary returns 20 only with 1e16 - because then 5e16 + 1 equals 5e16. Excel really should use decimal arithmetic.. Cheers, Werner 41CV†,42S,48GX,49G,DM42,DM41X,17BII,15CE,DM15L,12C,16CE |
|||
« Next Oldest | Next Newest »
|
User(s) browsing this thread: 3 Guest(s)