[Bf-blender-cvs] [1f75be8a400] master: Fix T80604: BLI_polyfill_calc exceeds stack size allocating points

[Campbell Barton]

Commit: 1f75be8a40041e94711c5d17507d88d8e64c1977
Author: Campbell Barton
Date:   Wed Sep 9 12:48:29 2020 +1000
Branches: master
https://developer.blender.org/rB1f75be8a40041e94711c5d17507d88d8e64c1977

Fix T80604: BLI_polyfill_calc exceeds stack size allocating points

On systems with 512kb stack this happened at around 13k points.

This happened at times with grease-pencil, although callers that
frequently use complex polygons should be using BLI_polyfill_calc_arena.

===================================================================

M	source/blender/blenlib/intern/polyfill_2d.c

===================================================================

diff --git a/source/blender/blenlib/intern/polyfill_2d.c b/source/blender/blenlib/intern/polyfill_2d.c
index d1e2bd58909..eb7e5ca6658 100644
--- a/source/blender/blenlib/intern/polyfill_2d.c
+++ b/source/blender/blenlib/intern/polyfill_2d.c
@@ -907,6 +907,19 @@ void BLI_polyfill_calc(const float (*coords)[2],
                        const int coords_sign,
                        uint (*r_tris)[3])
 {
+  /* Fallback to heap memory for large allocations.
+   * Avoid running out of stack memory on systems with 512kb stack (macOS).
+   * This happens at around 13,000 points, use a much lower value to be safe. */
+  if (UNLIKELY(coords_tot > 8192)) {
+    /* The buffer size only accounts for the index allocation,
+     * worst case we do two allocations when concave, while we should try to be efficient,
+     * any caller that relies on this frequently should use #BLI_polyfill_calc_arena directly. */
+    MemArena *arena = BLI_memarena_new(sizeof(PolyIndex) * coords_tot, __func__);
+    BLI_polyfill_calc_arena(coords, coords_tot, coords_sign, r_tris, arena);
+    BLI_memarena_free(arena);
+    return;
+  }
+
   PolyFill pf;
   PolyIndex *indices = BLI_array_alloca(indices, coords_tot);