FORTRAN floating point accuracy problems
|
03-30-2016, 08:32 PM
Post: #14
|
|||
|
|||
RE: FORTRAN floating point accuracy problems
(03-30-2016 06:38 PM)Dieter Wrote: 32 bits mean 9,3 decimal digits, which matches the results. Since 2 digits are used for the integer portion, rounding the mmss part to 7 decimals shoud be OK. Dieter - what is the meaning of "9,3 decimal digits" ? I get that due to internal representation inaccuracies you can only get somewhere between 9 and 10 reliable decimal digits, but what is the meaning or interpretation of the ",3" ? These threads are interesting to follow, I'm always learning something new, but I've not seen this usage before. Or maybe have simply forgotten it in the last 30+ years since doing this stuff on a daily basis. --Bob Prosperi |
|||
« Next Oldest | Next Newest »
|
User(s) browsing this thread: 7 Guest(s)