[Bf-blender-cvs] [0edda8e4ce4] master: Fix T53311: transform edge/normal orientation

[Campbell Barton]

Commit: 0edda8e4ce4e7dc0d72a105b773896b7b17c480b
Author: Campbell Barton
Date:   Thu Jan 11 19:09:45 2018 +1100
Branches: master
https://developer.blender.org/rB0edda8e4ce4e7dc0d72a105b773896b7b17c480b

Fix T53311: transform edge/normal orientation

When the edge is aligned with it's own normals,
transform orientation wasn't aligned with the edge.

===================================================================

M	source/blender/editors/transform/transform_orientations.c

===================================================================

diff --git a/source/blender/editors/transform/transform_orientations.c b/source/blender/editors/transform/transform_orientations.c
index 54959304d72..f3078eb7824 100644
--- a/source/blender/editors/transform/transform_orientations.c
+++ b/source/blender/editors/transform/transform_orientations.c
@@ -743,10 +743,19 @@ int getTransformOrientation_ex(const bContext *C, float normal[3], float plane[3
 							SWAP(BMVert *, v_pair[0], v_pair[1]);
 						}
 
-						add_v3_v3v3(normal, v_pair[0]->no, v_pair[1]->no);
-						sub_v3_v3v3(plane, v_pair[0]->co, v_pair[1]->co);
-						/* flip the plane normal so we point outwards */
-						negate_v3(plane);
+						add_v3_v3v3(normal, v_pair[1]->no, v_pair[0]->no);
+						sub_v3_v3v3(plane, v_pair[1]->co, v_pair[0]->co);
+
+						if (normalize_v3(plane) != 0.0f) {
+							/* For edges it'd important the resulting matrix can rotate around the edge,
+							 * project onto the plane so we can use a fallback value. */
+							project_plane_normalized_v3_v3v3(normal, normal, plane);
+							if (UNLIKELY(normalize_v3(normal) == 0.0f)) {
+								/* in the case the normal and plane are aligned,
+								 * use a fallback normal which is orthogonal to the plane. */
+								ortho_v3_v3(normal, plane);
+							}
+						}
 					}
 
 					result = ORIENTATION_EDGE;