(42, all flavours) Integer Division - how?
|
12-15-2020, 10:37 AM
Post: #29
|
|||
|
|||
RE: (42, all flavours) Integer Division - how?
(12-14-2020 04:12 PM)Albert Chan Wrote: ULP consideration is needed. Unlike truncated fmod, floor-mod might not be exact. For idiv3(a, b), with tiny negative a, indeed we may hit with cancellation bug. lua> a = -0.09 lua> b = 1.1 - 1 -- so c = b+1 is exact lua> idiv3(a , b) -- should be -1 -2 For negative tiny a = -ε a1 = a%b - a%c = (b-ε) - (c-ε) = -1 + error b1 = q%c = q = 0, or -1 It is this error that make idiv3 failed. If the problem scaled to integers, the problem goes away. lua> A, B = a*0x1p56, b*0x1p56 lua> C = B+1 lua> A%B - A%C -1 lua> idiv3(A,B) -1 |
|||
« Next Oldest | Next Newest »
|
User(s) browsing this thread: 1 Guest(s)