[Bf-blender-cvs] SVN commit: /data/svn/bf-blender [47686] trunk/blender: Planar tracking support for motion tracking
Sergey Sharybin
sergey.vfx at gmail.com
Sun Jun 10 17:28:20 CEST 2012
Revision: 47686
http://projects.blender.org/scm/viewvc.php?view=rev&root=bf-blender&revision=47686
Author: nazgul
Date: 2012-06-10 15:28:19 +0000 (Sun, 10 Jun 2012)
Log Message:
-----------
Planar tracking support for motion tracking
===========================================
Major list of changes done in tomato branch:
- Add a planar tracking implementation to libmv
This adds a new planar tracking implementation to libmv. The
tracker is based on Ceres[1], the new nonlinear minimizer that
myself and Sameer released from Google as open source. Since
the motion model is more involved, the interface is
different than the RegionTracker interface used previously
in Blender.
The start of a C API in libmv-capi.{cpp,h} is also included.
- Migrate from pat_{min,max} for markers to 4 corners representation
Convert markers in the movie clip editor / 2D tracker from using
pat_min and pat_max notation to using the a more general, 4-corner
representation.
There is still considerable porting work to do; in particular
sliding from preview widget does not work correct for rotated
markers.
All other areas should be ported to new representation:
* Added support of sliding individual corners. LMB slide + Ctrl
would scale the whole pattern
* S would scale the whole marker, S-S would scale pattern only
* Added support of marker's rotation which is currently rotates
only patterns around their centers or all markers around median,
Rotation or other non-translation/scaling transformation of search
area doesn't make sense.
* Track Preview widget would display transformed pattern which
libmv actually operates with.
- "Efficient Second-order Minimization" for the planar tracker
This implements the "Efficient Second-order Minimization"
scheme, as supported by the existing translation tracker.
This increases the amount of per-iteration work, but
decreases the number of iterations required to converge and
also increases the size of the basin of attraction for the
optimization.
- Remove the use of the legacy RegionTracker API from Blender,
and replaces it with the new TrackRegion API. This also
adds several features to the planar tracker in libmv:
* Do a brute-force initialization of tracking similar to "Hybrid"
mode in the stable release, but using all floats. This is slower
but more accurate. It is still necessary to evaluate if the
performance loss is worth it. In particular, this change is
necessary to support high bit depth imagery.
* Add support for masks over the search window. This is a step
towards supporting user-defined tracker masks. The tracker masks
will make it easy for users to make a mask for e.g. a ball.
Not exposed into interface yet/
* Add Pearson product moment correlation coefficient checking (aka
"Correlation" in the UI. This causes tracking failure if the
tracked patch is not linearly related to the template.
* Add support for warping a few points in addition to the supplied
points. This is useful because the tracking code deliberately
does not expose the underlying warp representation. Instead,
warps are specified in an aparametric way via the correspondences.
- Replace the old style tracker configuration panel with the
new planar tracking panel. From a users perspective, this means:
* The old "tracking algorithm" picker is gone. There is only 1
algorithm now. We may revisit this later, but I would much
prefer to have only 1 algorithm. So far no optimization work
has been done so the speed is not there yet.
* There is now a dropdown to select the motion model. Choices:
* Translation
* Translation, rotation
* Translation, scale
* Translation, rotation, scale
* Affine
* Perspective
* The old "Hybrid" mode is gone; instead there is a toggle to
enable or disable translation-only tracker initialization. This
is the equivalent of the hyrbid mode before, but rewritten to work
with the new planar tracking modes.
* The pyramid levels setting is gone. At a future date, the planar
tracker will decide to use pyramids or not automatically. The
pyramid setting was ultimately a mistake; with the brute force
initialization it is unnecessary.
- Add light-normalized tracking
Added the ability to normalize patterns by their average value while
tracking, to make them invariant to global illumination changes.
Additional details could be found at wiki page [2]
[1] http://code.google.com/p/ceres-solver
[2] http://wiki.blender.org/index.php/Dev:Ref/Release_Notes/2.64/Motion_Tracker
Modified Paths:
--------------
trunk/blender/extern/libmv/CMakeLists.txt
trunk/blender/extern/libmv/libmv/image/sample.h
trunk/blender/extern/libmv/libmv/tracking/esm_region_tracker.cc
trunk/blender/extern/libmv/libmv-capi.cpp
trunk/blender/extern/libmv/libmv-capi.h
trunk/blender/release/scripts/startup/bl_ui/space_clip.py
trunk/blender/source/blender/blenkernel/BKE_tracking.h
trunk/blender/source/blender/blenkernel/intern/movieclip.c
trunk/blender/source/blender/blenkernel/intern/tracking.c
trunk/blender/source/blender/blenloader/intern/readfile.c
trunk/blender/source/blender/editors/interface/interface_draw.c
trunk/blender/source/blender/editors/space_clip/clip_buttons.c
trunk/blender/source/blender/editors/space_clip/clip_draw.c
trunk/blender/source/blender/editors/space_clip/space_clip.c
trunk/blender/source/blender/editors/space_clip/tracking_ops.c
trunk/blender/source/blender/editors/transform/transform.c
trunk/blender/source/blender/editors/transform/transform_conversions.c
trunk/blender/source/blender/editors/transform/transform_generics.c
trunk/blender/source/blender/editors/transform/transform_ops.c
trunk/blender/source/blender/makesdna/DNA_movieclip_types.h
trunk/blender/source/blender/makesdna/DNA_tracking_types.h
trunk/blender/source/blender/makesrna/RNA_access.h
trunk/blender/source/blender/makesrna/intern/rna_tracking.c
Added Paths:
-----------
trunk/blender/extern/libmv/libmv/multiview/homography.cc
trunk/blender/extern/libmv/libmv/multiview/homography.h
trunk/blender/extern/libmv/libmv/multiview/homography_parameterization.h
trunk/blender/extern/libmv/libmv/tracking/track_region.cc
trunk/blender/extern/libmv/libmv/tracking/track_region.h
Modified: trunk/blender/extern/libmv/CMakeLists.txt
===================================================================
--- trunk/blender/extern/libmv/CMakeLists.txt 2012-06-10 15:27:41 UTC (rev 47685)
+++ trunk/blender/extern/libmv/CMakeLists.txt 2012-06-10 15:28:19 UTC (rev 47686)
@@ -62,6 +62,7 @@
libmv/multiview/fundamental.cc
libmv/multiview/projection.cc
libmv/multiview/triangulation.cc
+ libmv/multiview/homography.cc
libmv/numeric/numeric.cc
libmv/numeric/poly.cc
libmv/simple_pipeline/bundle.cc
@@ -84,6 +85,7 @@
libmv/tracking/pyramid_region_tracker.cc
libmv/tracking/retrack_region_tracker.cc
libmv/tracking/trklt_region_tracker.cc
+ libmv/tracking/track_region.cc
third_party/fast/fast_10.c
third_party/fast/fast_11.c
Modified: trunk/blender/extern/libmv/libmv/image/sample.h
===================================================================
--- trunk/blender/extern/libmv/libmv/image/sample.h 2012-06-10 15:27:41 UTC (rev 47685)
+++ trunk/blender/extern/libmv/libmv/image/sample.h 2012-06-10 15:28:19 UTC (rev 47686)
@@ -34,25 +34,22 @@
return image(i, j, v);
}
-static inline void LinearInitAxis(float fx, int width,
- int *x1, int *x2,
- float *dx1, float *dx2) {
- const int ix = int(fx);
+inline void LinearInitAxis(float x, int size,
+ int *x1, int *x2,
+ float *dx) {
+ const int ix = static_cast<int>(x);
if (ix < 0) {
*x1 = 0;
*x2 = 0;
- *dx1 = 1;
- *dx2 = 0;
- } else if (ix > width-2) {
- *x1 = width-1;
- *x2 = width-1;
- *dx1 = 1;
- *dx2 = 0;
+ *dx = 1.0;
+ } else if (ix > size - 2) {
+ *x1 = size - 1;
+ *x2 = size - 1;
+ *dx = 1.0;
} else {
*x1 = ix;
- *x2 = *x1 + 1;
- *dx1 = *x2 - fx;
- *dx2 = 1 - *dx1;
+ *x2 = ix + 1;
+ *dx = *x2 - x;
}
}
@@ -60,20 +57,49 @@
template<typename T>
inline T SampleLinear(const Array3D<T> &image, float y, float x, int v = 0) {
int x1, y1, x2, y2;
- float dx1, dy1, dx2, dy2;
+ float dx, dy;
- LinearInitAxis(y, image.Height(), &y1, &y2, &dy1, &dy2);
- LinearInitAxis(x, image.Width(), &x1, &x2, &dx1, &dx2);
+ // Take the upper left corner as integer pixel positions.
+ x -= 0.5;
+ y -= 0.5;
+ LinearInitAxis(y, image.Height(), &y1, &y2, &dy);
+ LinearInitAxis(x, image.Width(), &x1, &x2, &dx);
+
const T im11 = image(y1, x1, v);
const T im12 = image(y1, x2, v);
const T im21 = image(y2, x1, v);
const T im22 = image(y2, x2, v);
- return T(dy1 * ( dx1 * im11 + dx2 * im12 ) +
- dy2 * ( dx1 * im21 + dx2 * im22 ));
+ return T( dy * ( dx * im11 + (1.0 - dx) * im12 ) +
+ (1 - dy) * ( dx * im21 + (1.0 - dx) * im22 ));
}
+/// Linear interpolation, of all channels. The sample is assumed to have the
+/// same size as the number of channels in image.
+template<typename T>
+inline void SampleLinear(const Array3D<T> &image, float y, float x, T *sample) {
+ int x1, y1, x2, y2;
+ float dx, dy;
+
+ // Take the upper left corner as integer pixel positions.
+ x -= 0.5;
+ y -= 0.5;
+
+ LinearInitAxis(y, image.Height(), &y1, &y2, &dy);
+ LinearInitAxis(x, image.Width(), &x1, &x2, &dx);
+
+ for (int i = 0; i < image.Depth(); ++i) {
+ const T im11 = image(y1, x1, i);
+ const T im12 = image(y1, x2, i);
+ const T im21 = image(y2, x1, i);
+ const T im22 = image(y2, x2, i);
+
+ sample[i] = T( dy * ( dx * im11 + (1.0 - dx) * im12 ) +
+ (1 - dy) * ( dx * im21 + (1.0 - dx) * im22 ));
+ }
+}
+
// Downsample all channels by 2. If the image has odd width or height, the last
// row or column is ignored.
// FIXME(MatthiasF): this implementation shouldn't be in an interface file
Added: trunk/blender/extern/libmv/libmv/multiview/homography.cc
===================================================================
--- trunk/blender/extern/libmv/libmv/multiview/homography.cc (rev 0)
+++ trunk/blender/extern/libmv/libmv/multiview/homography.cc 2012-06-10 15:28:19 UTC (rev 47686)
@@ -0,0 +1,267 @@
+// Copyright (c) 2008, 2009 libmv authors.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to
+// deal in the Software without restriction, including without limitation the
+// rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+// sell copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in
+// all copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+// IN THE SOFTWARE.
+
+#include "libmv/logging/logging.h"
+#include "libmv/multiview/homography.h"
+#include "libmv/multiview/homography_parameterization.h"
+
+namespace libmv {
+/** 2D Homography transformation estimation in the case that points are in
+ * euclidean coordinates.
+ *
+ * x = H y
+ * x and y vector must have the same direction, we could write
+ * crossproduct(|x|, * H * |y| ) = |0|
+ *
+ * | 0 -1 x2| |a b c| |y1| |0|
+ * | 1 0 -x1| * |d e f| * |y2| = |0|
+ * |-x2 x1 0| |g h 1| |1 | |0|
+ *
+ * That gives :
+ *
+ * (-d+x2*g)*y1 + (-e+x2*h)*y2 + -f+x2 |0|
+ * (a-x1*g)*y1 + (b-x1*h)*y2 + c-x1 = |0|
+ * (-x2*a+x1*d)*y1 + (-x2*b+x1*e)*y2 + -x2*c+x1*f |0|
+ */
+bool Homography2DFromCorrespondencesLinearEuc(
+ const Mat &x1,
+ const Mat &x2,
+ Mat3 *H,
+ double expected_precision) {
+ assert(2 == x1.rows());
+ assert(4 <= x1.cols());
+ assert(x1.rows() == x2.rows());
+ assert(x1.cols() == x2.cols());
+
+ int n = x1.cols();
+ MatX8 L = Mat::Zero(n * 3, 8);
+ Mat b = Mat::Zero(n * 3, 1);
+ for (int i = 0; i < n; ++i) {
+ int j = 3 * i;
+ L(j, 0) = x1(0, i); // a
+ L(j, 1) = x1(1, i); // b
+ L(j, 2) = 1.0; // c
+ L(j, 6) = -x2(0, i) * x1(0, i); // g
+ L(j, 7) = -x2(0, i) * x1(1, i); // h
+ b(j, 0) = x2(0, i); // i
+
+ ++j;
+ L(j, 3) = x1(0, i); // d
+ L(j, 4) = x1(1, i); // e
+ L(j, 5) = 1.0; // f
+ L(j, 6) = -x2(1, i) * x1(0, i); // g
+ L(j, 7) = -x2(1, i) * x1(1, i); // h
+ b(j, 0) = x2(1, i); // i
+
+ // This ensures better stability
+ // TODO(julien) make a lite version without this 3rd set
+ ++j;
+ L(j, 0) = x2(1, i) * x1(0, i); // a
+ L(j, 1) = x2(1, i) * x1(1, i); // b
+ L(j, 2) = x2(1, i); // c
+ L(j, 3) = -x2(0, i) * x1(0, i); // d
+ L(j, 4) = -x2(0, i) * x1(1, i); // e
+ L(j, 5) = -x2(0, i) ; // f
+ }
+ // Solve Lx=B
+ Vec h = L.fullPivLu().solve(b);
+ Homography2DNormalizedParameterization<double>::To(h, H);
+ if ((L * h).isApprox(b, expected_precision)) {
+ return true;
+ } else {
+ return false;
+ }
+}
+
+/** 2D Homography transformation estimation in the case that points are in
+ * homogeneous coordinates.
+ *
+ * | 0 -x3 x2| |a b c| |y1| -x3*d+x2*g -x3*e+x2*h -x3*f+x2*1 |y1| (-x3*d+x2*g)*y1 (-x3*e+x2*h)*y2 (-x3*f+x2*1)*y3 |0|
+ * | x3 0 -x1| * |d e f| * |y2| = x3*a-x1*g x3*b-x1*h x3*c-x1*1 * |y2| = (x3*a-x1*g)*y1 (x3*b-x1*h)*y2 (x3*c-x1*1)*y3 = |0|
+ * |-x2 x1 0| |g h 1| |y3| -x2*a+x1*d -x2*b+x1*e -x2*c+x1*f |y3| (-x2*a+x1*d)*y1 (-x2*b+x1*e)*y2 (-x2*c+x1*f)*y3 |0|
+ * X = |a b c d e f g h|^t
+ */
+bool Homography2DFromCorrespondencesLinear(const Mat &x1,
+ const Mat &x2,
+ Mat3 *H,
+ double expected_precision) {
+ if (x1.rows() == 2) {
+ return Homography2DFromCorrespondencesLinearEuc(x1, x2, H,
+ expected_precision);
+ }
+ assert(3 == x1.rows());
+ assert(4 <= x1.cols());
+ assert(x1.rows() == x2.rows());
+ assert(x1.cols() == x2.cols());
+
+ const int x = 0;
+ const int y = 1;
+ const int w = 2;
+ int n = x1.cols();
+ MatX8 L = Mat::Zero(n * 3, 8);
+ Mat b = Mat::Zero(n * 3, 1);
+ for (int i = 0; i < n; ++i) {
+ int j = 3 * i;
+ L(j, 0) = x2(w, i) * x1(x, i);//a
+ L(j, 1) = x2(w, i) * x1(y, i);//b
+ L(j, 2) = x2(w, i) * x1(w, i);//c
+ L(j, 6) = -x2(x, i) * x1(x, i);//g
+ L(j, 7) = -x2(x, i) * x1(y, i);//h
+ b(j, 0) = x2(x, i) * x1(w, i);
+
+ ++j;
+ L(j, 3) = x2(w, i) * x1(x, i);//d
+ L(j, 4) = x2(w, i) * x1(y, i);//e
+ L(j, 5) = x2(w, i) * x1(w, i);//f
+ L(j, 6) = -x2(y, i) * x1(x, i);//g
+ L(j, 7) = -x2(y, i) * x1(y, i);//h
+ b(j, 0) = x2(y, i) * x1(w, i);
+
+ // This ensures better stability
+ ++j;
+ L(j, 0) = x2(y, i) * x1(x, i);//a
+ L(j, 1) = x2(y, i) * x1(y, i);//b
+ L(j, 2) = x2(y, i) * x1(w, i);//c
+ L(j, 3) = -x2(x, i) * x1(x, i);//d
+ L(j, 4) = -x2(x, i) * x1(y, i);//e
+ L(j, 5) = -x2(x, i) * x1(w, i);//f
+ }
+ // Solve Lx=B
+ Vec h = L.fullPivLu().solve(b);
+ if ((L * h).isApprox(b, expected_precision)) {
+ Homography2DNormalizedParameterization<double>::To(h, H);
+ return true;
+ } else {
+ return false;
+ }
+}
+/**
+ * x2 ~ A * x1
+ * x2^t * Hi * A *x1 = 0
+ * H1 = H2 = H3 =
+ * | 0 0 0 1| |-x2w| |0 0 0 0| | 0 | | 0 0 1 0| |-x2z|
+ * | 0 0 0 0| -> | 0 | |0 0 1 0| -> |-x2z| | 0 0 0 0| -> | 0 |
+ * | 0 0 0 0| | 0 | |0-1 0 0| | x2y| |-1 0 0 0| | x2x|
+ * |-1 0 0 0| | x2x| |0 0 0 0| | 0 | | 0 0 0 0| | 0 |
+ * H4 = H5 = H6 =
+ * |0 0 0 0| | 0 | | 0 1 0 0| |-x2y| |0 0 0 0| | 0 |
+ * |0 0 0 1| -> |-x2w| |-1 0 0 0| -> | x2x| |0 0 0 0| -> | 0 |
+ * |0 0 0 0| | 0 | | 0 0 0 0| | 0 | |0 0 0 1| |-x2w|
+ * |0-1 0 0| | x2y| | 0 0 0 0| | 0 | |0 0-1 0| | x2z|
+ * |a b c d|
+ * A = |e f g h|
+ * |i j k l|
@@ Diff output truncated at 10240 characters. @@
More information about the Bf-blender-cvs
mailing list