[Bf-blender-cvs] [d5cefc1844c] master: Fix T50103: Transform not working if scale is zero

[Campbell Barton]

Commit: d5cefc1844cfd679b5d1f134386635f6165ae2e3
Author: Campbell Barton
Date:   Thu Apr 1 21:16:13 2021 +1100
Branches: master
https://developer.blender.org/rBd5cefc1844cfd679b5d1f134386635f6165ae2e3

Fix T50103: Transform not working if scale is zero

If any axis of the scale of an object was zero, transforming failed.
This was because `td->smtx` was set to a zero matrix.

orthogonalize_m3_zero_axes is used to fill in the zeroed axes.

Thanks to @filedescriptor for the initial fix: D10869.

===================================================================

M	source/blender/editors/transform/transform_convert_object.c

===================================================================

diff --git a/source/blender/editors/transform/transform_convert_object.c b/source/blender/editors/transform/transform_convert_object.c
index b546f1d0b4c..c217478bd04 100644
--- a/source/blender/editors/transform/transform_convert_object.c
+++ b/source/blender/editors/transform/transform_convert_object.c
@@ -282,9 +282,17 @@ static void ObjectToTransData(TransInfo *t, TransData *td, Object *ob)
      */
     BKE_object_to_mat3(ob, obmtx);
     copy_m3_m4(totmat, ob->obmat);
-    invert_m3_m3(obinv, totmat);
+
+    /* If the object scale is zero on any axis, this might result in a zero matrix.
+     * In this case, the transformation would not do anything, see: T50103. */
+    orthogonalize_m3_zero_axes(obmtx, 1.0f);
+    orthogonalize_m3_zero_axes(totmat, 1.0f);
+
+    /* Use safe invert even though the input matrices have had zero axes set to unit length,
+     * in the unlikely case of failure (float precision for eg) this uses unit matrix fallback. */
+    invert_m3_m3_safe_ortho(obinv, totmat);
     mul_m3_m3m3(td->smtx, obmtx, obinv);
-    invert_m3_m3(td->mtx, td->smtx);
+    invert_m3_m3_safe_ortho(td->mtx, td->smtx);
   }
   else {
     /* no conversion to/from dataspace */