Post Reply 
Prime: Fraction dependent on FIX setting
12-22-2013, 10:30 PM
Post: #2
RE: Prime: Fraction dependent on FIX setting
The issue is the documentation does not mention that the current digits setting impacts the way the key works. This allows you to control how you are calculating the fractional item. This is kind of similar to how changing the fix setting would make the evaluated integral precision change in older hp calcs. Change it to FIX 3 and it would returns what you expect (basically, rounds to 1 less then the digits setting).

The display of the item does not do anything to the underlying number and just flags it as "display with a fraction". Hence when you convert it to dms it works as expected. Nothing is lost and it is purely a display thing.

TW

Although I work for HP, the views and opinions I post here are my own.
Find all posts by this user
Quote this message in a reply
Post Reply 


Messages In This Thread
RE: Prime: Fraction dependent on FIX setting - Tim Wessman - 12-22-2013 10:30 PM



User(s) browsing this thread: 1 Guest(s)