[Bf-blender-cvs] [8c38a994c6b] blender-v3.3-release: Fix Quaternion.rotate(matrix) with negative matrices

[Campbell Barton]

Commit: 8c38a994c6b7728d4eae21626b566bcc13376c49
Author: Campbell Barton
Date:   Wed Aug 24 18:59:04 2022 +1000
Branches: blender-v3.3-release
https://developer.blender.org/rB8c38a994c6b7728d4eae21626b566bcc13376c49

Fix Quaternion.rotate(matrix) with negative matrices

Rotating a quaternion by a negative matrix gave an invalid result.

Follow up fix for T94231 which negated negative matrices too.

===================================================================

M	source/blender/python/mathutils/mathutils_Quaternion.c

===================================================================

diff --git a/source/blender/python/mathutils/mathutils_Quaternion.c b/source/blender/python/mathutils/mathutils_Quaternion.c
index 6994a313237..4972381d29e 100644
--- a/source/blender/python/mathutils/mathutils_Quaternion.c
+++ b/source/blender/python/mathutils/mathutils_Quaternion.c
@@ -543,8 +543,13 @@ static PyObject *Quaternion_rotate(QuaternionObject *self, PyObject *value)
   length = normalize_qt_qt(tquat, self->quat);
   quat_to_mat3(self_rmat, tquat);
   mul_m3_m3m3(rmat, other_rmat, self_rmat);
-
-  mat3_to_quat(self->quat, rmat);
+  normalize_m3(rmat);
+  /* This check could also be performed on `other_rmat`, use the final result instead to ensure
+   * float imprecision doesn't allow the multiplication to make `rmat` negative. */
+  if (is_negative_m3(rmat)) {
+    negate_m3(rmat);
+  }
+  mat3_normalized_to_quat(self->quat, rmat);
   mul_qt_fl(self->quat, length); /* maintain length after rotating */
 
   (void)BaseMath_WriteCallback(self);