[Bf-blender-cvs] [bf0a5541149] blender-v2.83-release: Fix T75968: PBVH raycast returns wrong active vertex

[Pablo Dobarro]

Commit: bf0a55411491329aa2ee893c111e66feeab8793b
Author: Pablo Dobarro
Date:   Wed May 13 02:47:46 2020 +0200
Branches: blender-v2.83-release
https://developer.blender.org/rBbf0a55411491329aa2ee893c111e66feeab8793b

Fix T75968: PBVH raycast returns wrong active vertex

nearest_vertex_co was not reset when a new triangle was intersected by
the ray, so it was always returning the closest vertex to the real
cursor position in any triangle, which was not always the triangle under
the cursor.

Reviewed By: sergey

Maniphest Tasks: T75968

Differential Revision: https://developer.blender.org/D7485

===================================================================

M	source/blender/blenkernel/intern/pbvh.c

===================================================================

diff --git a/source/blender/blenkernel/intern/pbvh.c b/source/blender/blenkernel/intern/pbvh.c
index e6d672ad9d9..5756cb8c706 100644
--- a/source/blender/blenkernel/intern/pbvh.c
+++ b/source/blender/blenkernel/intern/pbvh.c
@@ -2155,7 +2155,11 @@ static bool pbvh_faces_node_raycast(PBVH *bvh,
         float location[3] = {0.0f};
         madd_v3_v3v3fl(location, ray_start, ray_normal, *depth);
         for (int j = 0; j < 3; j++) {
-          if (len_squared_v3v3(location, co[j]) < len_squared_v3v3(location, nearest_vertex_co)) {
+          /* Always assign nearest_vertex_co in the first iteration to avoid comparison against
+           * uninitialized values. This stores the closest vertex in the current intersecting
+           * triangle. */
+          if (j == 0 ||
+              len_squared_v3v3(location, co[j]) < len_squared_v3v3(location, nearest_vertex_co)) {
             copy_v3_v3(nearest_vertex_co, co[j]);
             *r_active_vertex_index = mloop[lt->tri[j]].v;
             *r_active_face_index = lt->poly;
@@ -2235,8 +2239,11 @@ static bool pbvh_grids_node_raycast(PBVH *bvh,
             const int y_it[4] = {0, 0, 1, 1};
 
             for (int j = 0; j < 4; j++) {
-              if (len_squared_v3v3(location, co[j]) <
-                  len_squared_v3v3(location, nearest_vertex_co)) {
+              /* Always assign nearest_vertex_co in the first iteration to avoid comparison against
+               * uninitialized values. This stores the closest vertex in the current intersecting
+               * quad. */
+              if (j == 0 || len_squared_v3v3(location, co[j]) <
+                                len_squared_v3v3(location, nearest_vertex_co)) {
                 copy_v3_v3(nearest_vertex_co, co[j]);
 
                 *r_active_vertex_index = gridkey->grid_area * grid_index +