[Bf-blender-cvs] [466cd2535b1] blender-v2.91-release: Fix exact boolean coplanar problem due to inexact transform.

[Howard Trickey]

Commit: 466cd2535b1b11f266ffd929c6134647d6d21450
Author: Howard Trickey
Date:   Wed Nov 18 09:53:05 2020 -0500
Branches: blender-v2.91-release
https://developer.blender.org/rB466cd2535b1b11f266ffd929c6134647d6d21450

Fix exact boolean coplanar problem due to inexact transform.

This is an addendum to previous boolean fix, where the object
transformation was "cleaned". Now the operand one is too.
This fixes the issue shown in the video in T82301 when you move
a column around the XY plane with the top and bottom faces
supposedly coplanar with a cube. The transformation matrix when
you do that has a tiny offset in the z component.

===================================================================

M	source/blender/modifiers/intern/MOD_boolean.c

===================================================================

diff --git a/source/blender/modifiers/intern/MOD_boolean.c b/source/blender/modifiers/intern/MOD_boolean.c
index 8d0c7cd6d49..7ddaf15aaa3 100644
--- a/source/blender/modifiers/intern/MOD_boolean.c
+++ b/source/blender/modifiers/intern/MOD_boolean.c
@@ -340,13 +340,16 @@ static void BMD_mesh_intersection(BMesh *bm,
        * other than 0, -1, or 1 in the scaling part of the matrix.
        */
       float cleaned_object_obmat[4][4];
+      float cleaned_operand_obmat[4][4];
       clean_obmat(cleaned_object_obmat, object->obmat);
       invert_m4_m4(imat, cleaned_object_obmat);
+      clean_obmat(cleaned_operand_obmat, operand_ob->obmat);
+      mul_m4_m4m4(omat, imat, cleaned_operand_obmat);
     }
     else {
       invert_m4_m4(imat, object->obmat);
+      mul_m4_m4m4(omat, imat, operand_ob->obmat);
     }
-    mul_m4_m4m4(omat, imat, operand_ob->obmat);
 
     BMVert *eve;
     i = 0;