[Bf-blender-cvs] [29d1db9ed62] master: Fix T55964: Direction not normalized in isect_ray_aabb_v3_simple()?

[Bastien Montagne]

Commit: 29d1db9ed6253f68c5452be1f0125ed364d4a954
Author: Bastien Montagne
Date:   Wed Jul 25 21:05:44 2018 +0200
Branches: master
https://developer.blender.org/rB29d1db9ed6253f68c5452be1f0125ed364d4a954

Fix T55964: Direction not normalized in isect_ray_aabb_v3_simple()?

RNA API Object.ray_cast would not normalize direction vector before
doing first quick bbox intersection test, while using its returned
distance value. This could lead to wrong exclusion of object.
Thanks to @codemanx for finding that issue.

===================================================================

M	source/blender/makesrna/intern/rna_object_api.c

===================================================================

diff --git a/source/blender/makesrna/intern/rna_object_api.c b/source/blender/makesrna/intern/rna_object_api.c
index 2acda5985e1..376a89c65d1 100644
--- a/source/blender/makesrna/intern/rna_object_api.c
+++ b/source/blender/makesrna/intern/rna_object_api.c
@@ -332,8 +332,10 @@ static void rna_Object_ray_cast(
 	/* Test BoundBox first (efficiency) */
 	BoundBox *bb = BKE_object_boundbox_get(ob);
 	float distmin;
-	if (!bb || (isect_ray_aabb_v3_simple(origin, direction, bb->vec[0], bb->vec[6], &distmin, NULL) && distmin <= distance)) {
-
+	normalize_v3(direction);  /* Needed for valid distance check from isect_ray_aabb_v3_simple() call. */
+	if (!bb ||
+	    (isect_ray_aabb_v3_simple(origin, direction, bb->vec[0], bb->vec[6], &distmin, NULL) && distmin <= distance))
+	{
 		BVHTreeFromMesh treeData = {NULL};
 
 		/* no need to managing allocation or freeing of the BVH data. this is generated and freed as needed */
@@ -346,9 +348,6 @@ static void rna_Object_ray_cast(
 			hit.index = -1;
 			hit.dist = distance;
 
-			normalize_v3(direction);
-
-
 			if (BLI_bvhtree_ray_cast(treeData.tree, origin, direction, 0.0f, &hit,
 			                         treeData.raycast_callback, &treeData) != -1)
 			{