[Bf-blender-cvs] SVN commit: /data/svn/bf-blender [41817] trunk/blender: Assorted camera tracker improvements
Sergey Sharybin
sergey.vfx at gmail.com
Mon Nov 14 07:41:26 CET 2011
Revision: 41817
http://projects.blender.org/scm/viewvc.php?view=rev&root=bf-blender&revision=41817
Author: nazgul
Date: 2011-11-14 06:41:23 +0000 (Mon, 14 Nov 2011)
Log Message:
-----------
Assorted camera tracker improvements
- Add support for refining the camera's intrinsic parameters
during a solve. Currently, refining supports only the following
combinations of intrinsic parameters:
f
f, cx, cy
f, cx, cy, k1, k2
f, k1
f, k1, k2
This is not the same as autocalibration, since the user must
still make a reasonable initial guess about the focal length and
other parameters, whereas true autocalibration would eliminate
the need for the user specify intrinsic parameters at all.
However, the solver works well with only rough guesses for the
focal length, so perhaps full autocalibation is not that
important.
Adding support for the last two combinations, (f, k1) and (f,
k1, k2) required changes to the library libmv depends on for
bundle adjustment, SSBA. These changes should get ported
upstream not just to libmv but to SSBA as well.
- Improved the region of convergence for bundle adjustment by
increasing the number of Levenberg-Marquardt iterations from 50
to 500. This way, the solver is able to crawl out of the bad
local minima it gets stuck in when changing from, for example,
bundling k1 and k2 to just k1 and resetting k2 to 0.
- Add several new region tracker implementations. A region tracker
is a libmv concept, which refers to tracking a template image
pattern through frames. The impact to end users is that tracking
should "just work better". I am reserving a more detailed
writeup, and maybe a paper, for later.
- Other libmv tweaks, such as detecting that a tracker is headed
outside of the image bounds.
This includes several changes made directly to the libmv extern
code rather expecting to get those changes through normal libmv
channels, because I, the libmv BDFL, decided it was faster to work
on libmv directly in Blender, then later reverse-port the libmv
changes from Blender back into libmv trunk. The interesting part
is that I added a full Levenberg-Marquardt loop to the region
tracking code, which should lead to a more stable solutions. I
also added a hacky implementation of "Efficient Second-Order
Minimization" for tracking, which works nicely. A more detailed
quantitative evaluation will follow.
Original patch by Keir, cleaned a bit by myself.
Modified Paths:
--------------
trunk/blender/extern/libmv/CMakeLists.txt
trunk/blender/extern/libmv/libmv/simple_pipeline/bundle.cc
trunk/blender/extern/libmv/libmv/simple_pipeline/bundle.h
trunk/blender/extern/libmv/libmv/simple_pipeline/camera_intrinsics.cc
trunk/blender/extern/libmv/libmv/simple_pipeline/camera_intrinsics.h
trunk/blender/extern/libmv/libmv/simple_pipeline/initialize_reconstruction.cc
trunk/blender/extern/libmv/libmv/simple_pipeline/pipeline.cc
trunk/blender/extern/libmv/libmv/tracking/klt_region_tracker.cc
trunk/blender/extern/libmv/libmv/tracking/pyramid_region_tracker.cc
trunk/blender/extern/libmv/libmv/tracking/trklt_region_tracker.cc
trunk/blender/extern/libmv/libmv-capi.cpp
trunk/blender/extern/libmv/libmv-capi.h
trunk/blender/extern/libmv/third_party/ssba/Geometry/v3d_metricbundle.cpp
trunk/blender/extern/libmv/third_party/ssba/Geometry/v3d_metricbundle.h
trunk/blender/release/scripts/startup/bl_ui/space_clip.py
trunk/blender/source/blender/blenkernel/BKE_tracking.h
trunk/blender/source/blender/blenkernel/intern/tracking.c
trunk/blender/source/blender/editors/space_clip/tracking_ops.c
trunk/blender/source/blender/makesdna/DNA_tracking_types.h
trunk/blender/source/blender/makesrna/intern/rna_tracking.c
Added Paths:
-----------
trunk/blender/extern/libmv/libmv/tracking/esm_region_tracker.cc
trunk/blender/extern/libmv/libmv/tracking/esm_region_tracker.h
trunk/blender/extern/libmv/libmv/tracking/lmicklt_region_tracker.cc
trunk/blender/extern/libmv/libmv/tracking/lmicklt_region_tracker.h
Modified: trunk/blender/extern/libmv/CMakeLists.txt
===================================================================
--- trunk/blender/extern/libmv/CMakeLists.txt 2011-11-14 06:11:40 UTC (rev 41816)
+++ trunk/blender/extern/libmv/CMakeLists.txt 2011-11-14 06:41:23 UTC (rev 41817)
@@ -52,8 +52,10 @@
libmv/image/array_nd.cc
libmv/tracking/pyramid_region_tracker.cc
libmv/tracking/sad.cc
+ libmv/tracking/esm_region_tracker.cc
libmv/tracking/trklt_region_tracker.cc
libmv/tracking/klt_region_tracker.cc
+ libmv/tracking/lmicklt_region_tracker.cc
libmv/tracking/retrack_region_tracker.cc
libmv/multiview/projection.cc
libmv/multiview/conditioning.cc
@@ -99,8 +101,10 @@
libmv/tracking/retrack_region_tracker.h
libmv/tracking/sad.h
libmv/tracking/pyramid_region_tracker.h
+ libmv/tracking/esm_region_tracker.h
libmv/tracking/trklt_region_tracker.h
libmv/tracking/klt_region_tracker.h
+ libmv/tracking/lmicklt_region_tracker.h
libmv/base/id_generator.h
libmv/base/vector.h
libmv/base/scoped_ptr.h
Modified: trunk/blender/extern/libmv/libmv/simple_pipeline/bundle.cc
===================================================================
--- trunk/blender/extern/libmv/libmv/simple_pipeline/bundle.cc 2011-11-14 06:11:40 UTC (rev 41816)
+++ trunk/blender/extern/libmv/libmv/simple_pipeline/bundle.cc 2011-11-14 06:41:23 UTC (rev 41817)
@@ -27,6 +27,8 @@
#include "libmv/multiview/fundamental.h"
#include "libmv/multiview/projection.h"
#include "libmv/numeric/numeric.h"
+#include "libmv/simple_pipeline/camera_intrinsics.h"
+#include "libmv/simple_pipeline/bundle.h"
#include "libmv/simple_pipeline/reconstruction.h"
#include "libmv/simple_pipeline/tracks.h"
#include "third_party/ssba/Geometry/v3d_cameramatrix.h"
@@ -38,6 +40,18 @@
void EuclideanBundle(const Tracks &tracks,
EuclideanReconstruction *reconstruction) {
+ CameraIntrinsics intrinsics;
+ EuclideanBundleCommonIntrinsics(tracks,
+ BUNDLE_NO_INTRINSICS,
+ reconstruction,
+ &intrinsics);
+}
+
+void EuclideanBundleCommonIntrinsics(const Tracks &tracks,
+ int bundle_intrinsics,
+ EuclideanReconstruction *reconstruction,
+ CameraIntrinsics *intrinsics) {
+ LG << "Original intrinsics: " << *intrinsics;
vector<Marker> markers = tracks.AllMarkers();
// "index" in this context is the index that V3D's optimizer will see. The
@@ -69,19 +83,21 @@
}
}
- // Make a V3D identity matrix, needed in a few places for K, since this
- // assumes a calibrated setup.
- V3D::Matrix3x3d identity3x3;
- identity3x3[0][0] = 1.0;
- identity3x3[0][1] = 0.0;
- identity3x3[0][2] = 0.0;
- identity3x3[1][0] = 0.0;
- identity3x3[1][1] = 1.0;
- identity3x3[1][2] = 0.0;
- identity3x3[2][0] = 0.0;
- identity3x3[2][1] = 0.0;
- identity3x3[2][2] = 1.0;
+ // Convert libmv's K matrix to V3d's K matrix.
+ V3D::Matrix3x3d v3d_K;
+ for (int i = 0; i < 3; ++i) {
+ for (int j = 0; j < 3; ++j) {
+ v3d_K[i][j] = intrinsics->K()(i, j);
+ }
+ }
+ // Convert libmv's distortion to v3d distortion.
+ V3D::StdDistortionFunction v3d_distortion;
+ v3d_distortion.k1 = intrinsics->k1();
+ v3d_distortion.k2 = intrinsics->k2();
+ v3d_distortion.p1 = intrinsics->p1();
+ v3d_distortion.p2 = intrinsics->p2();
+
// Convert libmv's cameras to V3D's cameras.
std::vector<V3D::CameraMatrix> v3d_cameras(index_to_camera.size());
for (int k = 0; k < index_to_camera.size(); ++k) {
@@ -98,7 +114,7 @@
}
t[i] = t_libmv(i);
}
- v3d_cameras[k].setIntrinsic(identity3x3);
+ v3d_cameras[k].setIntrinsic(v3d_K);
v3d_cameras[k].setRotation(R);
v3d_cameras[k].setTranslation(t);
}
@@ -134,28 +150,82 @@
}
LG << "Number of residuals: " << num_residuals;
- // This is calibrated reconstruction, so use zero distortion.
- V3D::StdDistortionFunction v3d_distortion;
- v3d_distortion.k1 = 0;
- v3d_distortion.k2 = 0;
- v3d_distortion.p1 = 0;
- v3d_distortion.p2 = 0;
+ // Convert from libmv's specification for which intrinsics to bundle to V3D's.
+ int v3d_bundle_intrinsics;
+ if (bundle_intrinsics == BUNDLE_NO_INTRINSICS) {
+ LG << "Bundling only camera positions.";
+ v3d_bundle_intrinsics = V3D::FULL_BUNDLE_METRIC;
+ } else if (bundle_intrinsics == BUNDLE_FOCAL_LENGTH) {
+ LG << "Bundling f.";
+ v3d_bundle_intrinsics = V3D::FULL_BUNDLE_FOCAL_LENGTH;
+ } else if (bundle_intrinsics == (BUNDLE_FOCAL_LENGTH |
+ BUNDLE_PRINCIPAL_POINT)) {
+ LG << "Bundling f, px, py.";
+ v3d_bundle_intrinsics = V3D::FULL_BUNDLE_FOCAL_LENGTH_PP;
+ } else if (bundle_intrinsics == (BUNDLE_FOCAL_LENGTH |
+ BUNDLE_PRINCIPAL_POINT |
+ BUNDLE_RADIAL)) {
+ LG << "Bundling f, px, py, k1, k2.";
+ v3d_bundle_intrinsics = V3D::FULL_BUNDLE_RADIAL;
+ } else if (bundle_intrinsics == (BUNDLE_FOCAL_LENGTH |
+ BUNDLE_PRINCIPAL_POINT |
+ BUNDLE_RADIAL |
+ BUNDLE_TANGENTIAL)) {
+ LG << "Bundling f, px, py, k1, k2, p1, p2.";
+ v3d_bundle_intrinsics = V3D::FULL_BUNDLE_RADIAL_TANGENTIAL;
+ } else if (bundle_intrinsics == (BUNDLE_FOCAL_LENGTH |
+ BUNDLE_RADIAL |
+ BUNDLE_TANGENTIAL)) {
+ LG << "Bundling f, px, py, k1, k2, p1, p2.";
+ v3d_bundle_intrinsics = V3D::FULL_BUNDLE_RADIAL_TANGENTIAL;
+ } else if (bundle_intrinsics == (BUNDLE_FOCAL_LENGTH |
+ BUNDLE_RADIAL)) {
+ LG << "Bundling f, k1, k2.";
+ v3d_bundle_intrinsics = V3D::FULL_BUNDLE_FOCAL_AND_RADIAL;
+ } else if (bundle_intrinsics == (BUNDLE_FOCAL_LENGTH |
+ BUNDLE_RADIAL_K1)) {
+ LG << "Bundling f, k1.";
+ v3d_bundle_intrinsics = V3D::FULL_BUNDLE_FOCAL_AND_RADIAL_K1;
+ } else {
+ LOG(FATAL) << "Unsupported bundle combination.";
+ }
+ // Ignore any outliers; assume supervised tracking.
+ double v3d_inlier_threshold = 500000.0;
+
// Finally, run the bundle adjustment.
- double const inlierThreshold = 500000.0;
- V3D::CommonInternalsMetricBundleOptimizer opt(V3D::FULL_BUNDLE_METRIC,
- inlierThreshold,
- identity3x3,
+ V3D::CommonInternalsMetricBundleOptimizer opt(v3d_bundle_intrinsics,
+ v3d_inlier_threshold,
+ v3d_K,
v3d_distortion,
v3d_cameras,
v3d_points,
v3d_measurements,
v3d_camera_for_measurement,
v3d_point_for_measurement);
- opt.maxIterations = 50;
+ opt.maxIterations = 500;
opt.minimize();
- LG << "Bundle status: " << opt.status;
+ if (opt.status == V3D::LEVENBERG_OPTIMIZER_TIMEOUT) {
+ LG << "Bundle status: Timed out.";
+ } else if (opt.status == V3D::LEVENBERG_OPTIMIZER_SMALL_UPDATE) {
+ LG << "Bundle status: Small update.";
+ } else if (opt.status == V3D::LEVENBERG_OPTIMIZER_CONVERGED) {
+ LG << "Bundle status: Converged.";
+ }
+ // Convert V3D's K matrix back to libmv's K matrix.
+ Mat3 K;
+ for (int i = 0; i < 3; ++i) {
+ for (int j = 0; j < 3; ++j) {
+ K(i, j) = v3d_K[i][j];
+ }
+ }
+ intrinsics->SetK(K);
+
+ // Convert V3D's distortion back to libmv's distortion.
+ intrinsics->SetRadialDistortion(v3d_distortion.k1, v3d_distortion.k2, 0.0);
+ intrinsics->SetTangentialDistortion(v3d_distortion.p1, v3d_distortion.p2);
+
// Convert V3D's cameras back to libmv's cameras.
for (int k = 0; k < num_cameras; k++) {
V3D::Matrix3x4d const Rt = v3d_cameras[k].getOrientation();
@@ -173,6 +243,7 @@
index_to_point[k]->X(i) = v3d_points[k][i];
}
}
+ LG << "Final intrinsics: " << *intrinsics;
}
void ProjectiveBundle(const Tracks & /*tracks*/,
Modified: trunk/blender/extern/libmv/libmv/simple_pipeline/bundle.h
===================================================================
--- trunk/blender/extern/libmv/libmv/simple_pipeline/bundle.h 2011-11-14 06:11:40 UTC (rev 41816)
+++ trunk/blender/extern/libmv/libmv/simple_pipeline/bundle.h 2011-11-14 06:41:23 UTC (rev 41817)
@@ -23,6 +23,7 @@
namespace libmv {
+class CameraIntrinsics;
class EuclideanReconstruction;
class ProjectiveReconstruction;
class Tracks;
@@ -50,6 +51,46 @@
/*!
Refine camera poses and 3D coordinates using bundle adjustment.
+ This routine adjusts all cameras positions, points, and the camera
+ intrinsics (assumed common across all images) in \a *reconstruction. This
+ assumes a full observation for reconstructed tracks; this implies that if
+ there is a reconstructed 3D point (a bundle) for a track, then all markers
+ for that track will be included in the minimization. \a tracks should
+ contain markers used in the initial reconstruction.
+
+ The cameras, bundles, and intrinsics are refined in-place.
+
+ The only supported combinations of bundle parameters are:
+
+ BUNDLE_NO_INTRINSICS
+ BUNDLE_FOCAL_LENGTH
+ BUNDLE_FOCAL_LENGTH | BUNDLE_PRINCIPAL_POINT
+ BUNDLE_FOCAL_LENGTH | BUNDLE_PRINCIPAL_POINT | BUNDLE_RADIAL
+ BUNDLE_FOCAL_LENGTH | BUNDLE_PRINCIPAL_POINT | BUNDLE_RADIAL | BUNDLE_TANGENTIAL
+
+ \note This assumes an outlier-free set of markers.
+
+ \sa EuclideanResect, EuclideanIntersect, EuclideanReconstructTwoFrames
+*/
+enum BundleIntrinsics {
+ BUNDLE_NO_INTRINSICS = 0,
+ BUNDLE_FOCAL_LENGTH = 1,
+ BUNDLE_PRINCIPAL_POINT = 2,
+ BUNDLE_RADIAL_K1 = 4,
+ BUNDLE_RADIAL_K2 = 8,
+ BUNDLE_RADIAL = 12,
+ BUNDLE_TANGENTIAL_P1 = 16,
+ BUNDLE_TANGENTIAL_P2 = 32,
+ BUNDLE_TANGENTIAL = 48,
+};
+void EuclideanBundleCommonIntrinsics(const Tracks &tracks,
+ int bundle_intrinsics,
+ EuclideanReconstruction *reconstruction,
+ CameraIntrinsics *intrinsics);
+
+/*!
@@ Diff output truncated at 10240 characters. @@
More information about the Bf-blender-cvs
mailing list