[Bf-blender-cvs] [3a2899cc317] master: Fix T103942 ASAN crash in math_boolean function.

[Howard Trickey]

Commit: 3a2899cc31777308dd16d1f0e6916499564df711
Author: Howard Trickey
Date:   Sun Jan 22 12:48:45 2023 -0500
Branches: master
https://developer.blender.org/rB3a2899cc31777308dd16d1f0e6916499564df711

Fix T103942 ASAN crash in math_boolean function.

The code in questions comes from Shewchuk's triangle code, which
hasn't been updated to fix the out-of-buffer access problem
that ASAN finds in the delaunay unit test. The problem is benign:
the code would exit the loop before using the value fetched from
beyond the end of the buffer, but to make ASAN happy, I put in
a couple extra tests to not fetch values that aren't going to be used.

===================================================================

M	source/blender/blenlib/intern/math_boolean.cc

===================================================================

diff --git a/source/blender/blenlib/intern/math_boolean.cc b/source/blender/blenlib/intern/math_boolean.cc
index 689c23ce092..7c0cf165174 100644
--- a/source/blender/blenlib/intern/math_boolean.cc
+++ b/source/blender/blenlib/intern/math_boolean.cc
@@ -501,11 +501,15 @@ static int fast_expansion_sum_zeroelim(
     while ((eindex < elen) && (findex < flen)) {
       if ((fnow > enow) == (fnow > -enow)) {
         Two_Sum(Q, enow, Qnew, hh);
-        enow = e[++eindex];
+        if (++eindex < elen) {
+          enow = e[eindex];
+        }
       }
       else {
         Two_Sum(Q, fnow, Qnew, hh);
-        fnow = f[++findex];
+        if (++findex < flen) {
+          fnow = f[findex];
+        }
       }
       Q = Qnew;
       if (hh != 0.0) {
@@ -515,7 +519,9 @@ static int fast_expansion_sum_zeroelim(
   }
   while (eindex < elen) {
     Two_Sum(Q, enow, Qnew, hh);
-    enow = e[++eindex];
+    if (++eindex < elen) {
+      enow = e[eindex];
+    }
     Q = Qnew;
     if (hh != 0.0) {
       h[hindex++] = hh;
@@ -523,7 +529,9 @@ static int fast_expansion_sum_zeroelim(
   }
   while (findex < flen) {
     Two_Sum(Q, fnow, Qnew, hh);
-    fnow = f[++findex];
+    if (++findex < flen) {
+      fnow = f[findex];
+    }
     Q = Qnew;
     if (hh != 0.0) {
       h[hindex++] = hh;