Accuracy of Quadratic Regression

[jte]

(08-21-2023 05:52 PM)Albert Chan Wrote:  ⋮
Yes, it is due to floating point precision. (HOME have only 12 decimals)

For conceptual simplicity / implementation ease, I used higher precision arithmetics for the curve fitting in the Function app’s ”Sketch” feature. (129-bit sign-magnitude for quadratic fits and 161-bit sign-magnitude for cubic fits.)

My intention to use higher precision there was not just based on wanting to rule out any round-off during the production of the entries of the design matrix, but to also bound the round-off involved when solving for the coefficients (of the quadratic / cubic fit).