[Soc-2016-dev] Weekly Report 2, Multiview reconstruction

[Tianwei Shen]

Hi Sergey,

Thanks for the reply. Actually I am separating the operators (add CLIP_OT_solve_multivew, rather than revising the code related to CLIP_OT_solve_camera), not the functions in libmv. This multi-view solve button can be similar to ‘solve’. So I basically start from your CLIP_OT_solve_camera implementation and made changes on it. 

The problem with the current implementation is that it is assumed that the tracks are all from one camera throughout the whole reconstruction process. It is straightforward to implement like this for single camera, since matched markers are easily managed because they are all from the same track. 

For multi-camera case, which I assume to be more than 2 cameras, much of the stuff here need to become a list. For example, the following data structure can be modified:

typedef struct {
	Scene *scene;
	ListBase *clips;
	MovieClipUser user;
	ReportList *reports;
	char stats_message[256];
	struct MovieMultiviewReconstructContext *context;
} SolveMultiviewJob;

clip is changed to a list of clips, which contains the primary camera and witness camera(s). It holds a struct call MovieMultiviewReconstructContext as follows:

typedef struct MovieMultiviewReconstructContext {
	ListBase *tracks_base;		// from libmv_Tracks *tracks
	ListBase* reconstruction_clips;			// from libmv_Reconstruction *reconstruction
	ListBase* all_camera_intrisics;
	ListBase* tracks_map_base;
	ListBase* correspondence_base;
	bool select_keyframes;
	int keyframe1, keyframe2;
	int refine_flags;
	char object_name[MAX_NAME];
	bool is_camera;
	short motion_flag;
	float reprojection_error;
	int *sfra, *efra;		// start frame and end frame
} MovieMultiviewReconstructContext;

This basically extends from MovieReconstructContext. The key is that it records a set of correspondences (correspondence_base) specified by users, so I can arrange and connect these tracks in the back end (libmv). 

//TODO(tianwei): expand the fields, now only two tracks
typedef struct MovieTrackingCorrespondence {
	struct MovieTrackingCorrespondence *next, *prev;
	char name[64];  /* MAX_NAME */
	MovieTrackingTrack *primary_track;
	MovieTrackingTrack *witness_track;
	int primary_clip_id;
	int witness_clip_id;
} MovieTrackingCorrespondence;

For now this just keeps tracks of two track pointers and their clip ids, since MovieTrackingTrack does not have this information. The indexing of clip id can go with ListBase *clips in SolveMultiviewJob. For example, the tracks in the main camera has primary_clip_id equal 0, the first witness camera is indexed 1, so on and so forth. We may also go with the id field in MovieClip.

So the CLIP_OT_solve_multiview will work like this. First we prepare everything that libmv needs and pass them to MovieMultiviewReconstructContext, including the primary tracks, witness tracks, correspondences, tracks_map, key frames, etc. Then I process the options and pass all the tracks and correspondences to libmv_solve_multiview_reconstruction. 

Then we come to the back end, libmv. It seems to me that libmv::Tracks is also agnostic about the video clip which the track comes from, so I need to complete that information using the correspondences (between video clips). Maybe I need a wrapper around libmv::Tracks, called it libmv::IdenticalTracks, which serve as a container for tracks regarded as the same stuff. The reconstruction can be similar to the previous single-view one, which we first compute a two-view geometry from the two keyframes, then inserted other images both in the primary camera and witness cameras. After bundle adjustment over all the images, we can pass back the info to the blender side and do the cleanup. 

This is basically my thoughts on the multi-view pipeline. I haven’t figure out how object tracking plays the role. Let me first finish the camera tracking part. Comment and suggestions are very much welcomed and appreciated. :) I will also add this to the blender wiki so that you can review and comment. Thanks.


Thanks,
Tianwei

> On 2016 M06 4, at 21:00, Sergey Sharybin <sergey.vfx at gmail.com> wrote:
> 
> Hi,
> 
> Do you have some design overview of what your new data structures are and how they all fit together?
> 
> Would also be cool to hear how exactly you're approaching to the camera solver. While it might be separate Blender operator, you can't really (and should not in fact) separate things in Libmv.
> 
> On Sat, Jun 4, 2016 at 9:50 AM, Tianwei Shen <shentianweipku at gmail.com <mailto:shentianweipku at gmail.com>> wrote:
> Hi everyone,
> 
> This is report of week 2 for my multi-view reconstruction project. You can also view this report at blender wiki:
> https://wiki.blender.org/index.php/User:Tianwei37/Report_%26_Blog/Week_2 <https://wiki.blender.org/index.php/User:Tianwei37/Report_&_Blog/Week_2>
> 
> What I did:
> - organize correspondence: add several utility functions and data structures. A complete correspondence data structure is still in development, until I figure out what’s needed in the back end.
> - add 'solve multi-view' operator and implement it. I separate the function of ‘solve single camera’ and ‘solve multi-camera’ for now. This allows me to do the multi-view reconstruction code without interfering the original solve-camera code. After I finish this function. I will integrate the multi-view reconstruction with the single-view reconstruction.
> - watch TrackMatchBlend and get a full picture of the tracking system. (I thought TrackMatchBlend is an episode and introduction, while it is actually a series of videos, great tutorials!)
> 
> What I will do next week:
> - I thought I could finish a basic multi-view reconstruction operator this week. But it turned out that it involves a great deal of changes to the original codebase. So I will continue the work of multi-view reconstruction.
> - look into the UI side and consider making a separate mode for specifying correspondences.
> 
> 
> Thanks,
> Tianwei
> _______________________________________________
> Soc-2016-dev mailing list
> Soc-2016-dev at blender.org <mailto:Soc-2016-dev at blender.org>
> https://lists.blender.org/mailman/listinfo/soc-2016-dev <https://lists.blender.org/mailman/listinfo/soc-2016-dev>
> 
> 
> 
> -- 
> With best regards, Sergey Sharybin
> _______________________________________________
> Soc-2016-dev mailing list
> Soc-2016-dev at blender.org
> https://lists.blender.org/mailman/listinfo/soc-2016-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.blender.org/pipermail/soc-2016-dev/attachments/20160605/2ca1914f/attachment-0001.htm