Longest computation that your calculator performed for you?

[Garth Wilson]

(03-30-2017 07:37 PM)pier4r Wrote:  

(03-30-2017 07:30 PM)Garth Wilson Wrote:  A few years ago I used my HP-71, in BASIC, to generate large look-up tables (most being 65,536 cells each, and 16-bit, meaning 128KB per table) for hyperfast, accurate, 16-bit scaled-integer math, to use as a virtual co-processor for hardware that didn't have a normal co-processor. It took my HP-71 a few weeks to generate the tables and convert to Intel Hex files. I posted them, along with a discussion of the value of scaled-integer math and how to use it effectively, and information on how each table was calculated, on my website, at http://wilsonminesco.com/16bitMathTables/ .

This looks interesting. Is scaled-integer math the same [as] fixed point math? I will check the link, thanks for sharing!

Fixed-point is a limited subset of the broader, more-flexible scaled-integer math. "Fixed" moves the point right or left one in increments of one or more digits at a time, while "scaled" can, in essence, put it within a digit, basically infinitely variable rather than in digit steps.

For example, a 16-bit integer representing 0-360° in fixed-point can go up to 36,000 (for .01° resolution) using base 10, or perhaps even 46,080 (for 1°/128); but then you have the problem of carry/borrow/rollover. But if you scale it so the whole circle is 65,536, it's like the decimal point is, in essence, somewhere in a binary digit, and you get a resolution of 1°/182.044, and the rollover works as it should, so 359°+2°=1°, or 1°-2°=359°. Starting at zero degrees/radians and adding 3π/2 Radians (ie, 270°) puts you in the same place as subtracting π/2 Radians (ie, 90°). The high bit always tells which half of the circle you're in. The odd scale factor remains throughout the calculations until it's time for human-readable output, then you multiply or divide by the appropriate constant. If the output is in the form of graphics rather than printed digits, the number never may never need to exist in any format where the point is tucked neatly between digits in any base. Depending on the size of image, you might multiply by yet another odd scale factor which will change as you change the size of the window.

Or suppose you have a 0-7V range you want to measure with as much precision as possible with a 16-bit A/D converter and represent in memory with 16 bits. You might set the gain of the amplifier ahead of it so 7V gives an output of 65,535, meaning you get 9,362.3 counts per volt, 9.3623 counts per mV, or 106.81µV per count. The odd scale factor remains throughout the calculations until it's time for human-readable output, then you multiply or divide by the appropriate constant. Actually, my workbench computer has a fast 8-bit A/D converter that's 0-5V, or actually 0-255/256*(5.08V) since the regulator is wee bit off; so one count is 19.84mV. A common reference for the converters is 4.096V so that the user can be lazy and used fixed-point rather than scaled-integer, so you get 1mV per count if it's a 12-bit converter, 4mV per count if it's a 10-bit converter, 16mV per count if it's an 8-bit converter, etc., on even boundaries of binary digits.