[Bf-blender-cvs] [f03df4f] master: Fix crash when changing and using between layer and other brushes in dyntopo

[Antony Riakiotakis]

Commit: f03df4f0241328c1d65e6910a5374935a6f2f78d
Author: Antony Riakiotakis
Date:   Fri Mar 7 16:58:56 2014 +0200
https://developer.blender.org/rBf03df4f0241328c1d65e6910a5374935a6f2f78d

Fix crash when changing and using between layer and other brushes in
dyntopo

Layer brush would not invalidate the layer_disp arrays in dyntopo mode,
checking only for the existence of the array. This means that if a tool
resized the node due to topology changes, the layer brush code could
index (and write!) out of bounds in the array. Solution is to invalidate
the layer data prior to each stroke in dyntopo.

===================================================================

M	source/blender/blenkernel/BKE_pbvh.h
M	source/blender/blenkernel/intern/pbvh.c
M	source/blender/editors/sculpt_paint/sculpt.c

===================================================================

diff --git a/source/blender/blenkernel/BKE_pbvh.h b/source/blender/blenkernel/BKE_pbvh.h
index 34e9659..7a0c8a1 100644
--- a/source/blender/blenkernel/BKE_pbvh.h
+++ b/source/blender/blenkernel/BKE_pbvh.h
@@ -71,6 +71,7 @@ void BKE_pbvh_build_bmesh(PBVH *bvh, struct BMesh *bm, int smooth_shading,
                           struct BMLog *log);
 
 void BKE_pbvh_free(PBVH *bvh);
+void BKE_pbvh_free_layer_disp(PBVH *bvh);
 
 /* Hierarchical Search in the BVH, two methods:
  * - for each hit calling a callback
diff --git a/source/blender/blenkernel/intern/pbvh.c b/source/blender/blenkernel/intern/pbvh.c
index cb7c7c6..426a1e1 100644
--- a/source/blender/blenkernel/intern/pbvh.c
+++ b/source/blender/blenkernel/intern/pbvh.c
@@ -626,6 +626,13 @@ void BKE_pbvh_free(PBVH *bvh)
 	MEM_freeN(bvh);
 }
 
+void BKE_pbvh_free_layer_disp(PBVH *bvh)
+{
+	int i;
+	for (i = 0; i < bvh->totnode; ++i)
+		BKE_pbvh_node_layer_disp_free(&bvh->nodes[i]);
+}
+
 static void pbvh_iter_begin(PBVHIter *iter, PBVH *bvh, BKE_pbvh_SearchCallback scb, void *search_data)
 {
 	iter->bvh = bvh;
diff --git a/source/blender/editors/sculpt_paint/sculpt.c b/source/blender/editors/sculpt_paint/sculpt.c
index 7b10b82..c4c8004 100644
--- a/source/blender/editors/sculpt_paint/sculpt.c
+++ b/source/blender/editors/sculpt_paint/sculpt.c
@@ -3960,6 +3960,12 @@ static void sculpt_update_cache_invariants(bContext *C, Sculpt *sd, SculptSessio
 				}
 			}
 		}
+
+		if (ss->bm) {
+			/* Free any remaining layer displacements from nodes. If not and topology changes
+			 * from using another tool, then next layer toolstroke can access past disp array bounds */
+			BKE_pbvh_free_layer_disp(ss->pbvh);
+		}
 	}
 
 	/* Make copies of the mesh vertex locations and normals for some tools */