[Bf-blender-cvs] [05c4efc9b19] master: Fix Edit Face Set operator not being cancelled when the cursor is not over the mesh

[Pablo Dobarro]

Commit: 05c4efc9b1993f4d11a7d749fe4a9e9d80cc1f2e
Author: Pablo Dobarro
Date:   Thu Nov 12 23:01:51 2020 +0100
Branches: master
https://developer.blender.org/rB05c4efc9b1993f4d11a7d749fe4a9e9d80cc1f2e

Fix Edit Face Set operator not being cancelled when the cursor is not over the mesh

If the cursor is not over the mesh, the operator was still using the
last Face Set ID updated by the drawing cursor code when the cursor was
over the mesh.

This now cancels the operator instead of modifying a Face Set that will
look random to the user.

Reviewed By: sergey

Differential Revision: https://developer.blender.org/D9545

===================================================================

M	source/blender/editors/sculpt_paint/sculpt_face_set.c

===================================================================

diff --git a/source/blender/editors/sculpt_paint/sculpt_face_set.c b/source/blender/editors/sculpt_paint/sculpt_face_set.c
index f67c8f701f7..7c8a4c5a857 100644
--- a/source/blender/editors/sculpt_paint/sculpt_face_set.c
+++ b/source/blender/editors/sculpt_paint/sculpt_face_set.c
@@ -1306,7 +1306,10 @@ static int sculpt_face_set_edit_invoke(bContext *C, wmOperator *op, const wmEven
    * tool without brush cursor. */
   SculptCursorGeometryInfo sgi;
   const float mouse[2] = {event->mval[0], event->mval[1]};
-  SCULPT_cursor_geometry_info_update(C, &sgi, mouse, false);
+  if (!SCULPT_cursor_geometry_info_update(C, &sgi, mouse, false)) {
+      /* The cursor is not over the mesh. Cancel to avoid editing the last updated Face Set ID. */
+      return OPERATOR_CANCELLED;
+  }
   const int active_face_set = SCULPT_active_face_set_get(ss);
 
   switch (mode) {