Post Reply 
(42, all flavours) Integer Division - how?
12-15-2020, 10:37 AM
Post: #29
RE: (42, all flavours) Integer Division - how?
(12-14-2020 04:12 PM)Albert Chan Wrote:  ULP consideration is needed. Unlike truncated fmod, floor-mod might not be exact.

For idiv3(a, b), with tiny negative a, indeed we may hit with cancellation bug.

lua> a = -0.09
lua> b = 1.1 - 1 -- so c = b+1 is exact
lua> idiv3(a , b) -- should be -1
-2

For negative tiny a = -ε

a1 = a%b - a%c = (b-ε) - (c-ε) = -1 + error
b1 = q%c = q = 0, or -1

It is this error that make idiv3 failed.

If the problem scaled to integers, the problem goes away.

lua> A, B = a*0x1p56, b*0x1p56
lua> C = B+1
lua> A%B - A%C
-1
lua> idiv3(A,B)
-1
Find all posts by this user
Quote this message in a reply
Post Reply 


Messages In This Thread
RE: (42, all flavours) Integer Division - how? - Albert Chan - 12-15-2020 10:37 AM



User(s) browsing this thread: 1 Guest(s)