<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi Sergey,<div class=""><br class=""></div><div class="">Thanks for the reply. Actually I am separating the operators (add CLIP_OT_solve_multivew, rather than revising the code related to CLIP_OT_solve_camera), not the functions in libmv. This multi-view solve button can be similar to ‘solve’. So I basically start from your CLIP_OT_solve_camera implementation and made changes on it. </div><div class=""><br class=""></div><div class="">The problem with the current implementation is that it is assumed that the tracks are all from one camera throughout the whole reconstruction process. It is straightforward to implement like this for single camera, since matched markers are easily managed because they are all from the same track. </div><div class=""><br class=""></div><div class="">For multi-camera case, which I assume to be more than 2 cameras, much of the stuff here need to become a list. For example, the following data structure can be modified:</div><div class=""><pre style="margin-top: 0px; margin-bottom: 0px;" class="">
<pre style="margin-top: 0px; margin-bottom: 0px;" class=""><!--StartFragment--><span style=" color:#f82670;" class="">typedef</span><span style=" color:#808080;" class=""> </span><span style=" color:#f82670;" class="">struct</span><span style=" color:#808080;" class=""> </span><span style=" color:#e6e5e2;" class="">{</span></pre><pre style="margin-top: 0px; margin-bottom: 0px;" class=""><span style=" color:#808080;" class=""> </span><span style=" color:#f92672;" class="">Scene</span><span style=" color:#808080;" class=""> </span><span style=" color:#e6e5e2;" class="">*</span><span style=" color:#abdc49;" class="">scene</span><span style=" color:#e6e5e2;" class="">;</span></pre><pre style="margin-top: 0px; margin-bottom: 0px;" class=""><span style=" color:#808080;" class=""> </span><span style=" color:#f92672;" class="">ListBase</span><span style=" color:#808080;" class=""> </span><span style=" color:#e6e5e2;" class="">*</span><span style=" color:#abdc49;" class="">clips</span><span style=" color:#e6e5e2;" class="">;</span></pre><pre style="margin-top: 0px; margin-bottom: 0px;" class=""><span style=" color:#808080;" class=""> </span><span style=" color:#f92672;" class="">MovieClipUser</span><span style=" color:#808080;" class=""> </span><span style=" color:#abdc49;" class="">user</span><span style=" color:#e6e5e2;" class="">;</span></pre><pre style="margin-top: 0px; margin-bottom: 0px;" class=""><span style=" color:#808080;" class=""> </span><span style=" color:#f92672;" class="">ReportList</span><span style=" color:#808080;" class=""> </span><span style=" color:#e6e5e2;" class="">*</span><span style=" color:#abdc49;" class="">reports</span><span style=" color:#e6e5e2;" class="">;</span></pre><pre style="margin-top: 0px; margin-bottom: 0px;" class=""><span style=" color:#808080;" class=""> </span><span style=" color:#f92672;" class="">char</span><span style=" color:#808080;" class=""> </span><span style=" color:#abdc49;" class="">stats_message</span><span style=" color:#e6e5e2;" class="">[</span><span style=" color:#ae81ff;" class="">256</span><span style=" color:#e6e5e2;" class="">];</span></pre><pre style="margin-top: 0px; margin-bottom: 0px;" class=""><span style=" color:#808080;" class=""> </span><span style=" color:#f82670;" class="">struct</span><span style=" color:#808080;" class=""> </span><span style=" color:#f92672;" class="">MovieMultiviewReconstructContext</span><span style=" color:#808080;" class=""> </span><span style=" color:#e6e5e2;" class="">*</span><span style=" color:#abdc49;" class="">context</span><span style=" color:#e6e5e2;" class="">;</span></pre><pre style="margin-top: 0px; margin-bottom: 0px;" class=""><span style=" color:#e6e5e2;" class="">}</span><span style=" color:#808080;" class=""> </span><span style=" color:#f92672;" class="">SolveMultiviewJob</span><span style=" color:#e6e5e2;" class="">;</span></pre>
<pre style="margin-top: 0px; margin-bottom: 0px;" class=""><div style="font-family: Helvetica; white-space: normal;" class="">clip is changed to a list of clips, which contains the primary camera and witness camera(s). It holds a struct call MovieMultiviewReconstructContext as follows:</div><div style="font-family: Helvetica; white-space: normal;" class=""><br class=""></div><div style="font-family: Helvetica; white-space: normal;" class="">
<pre style="margin-top: 0px; margin-bottom: 0px;" class=""><!--StartFragment--><span style=" color:#f82670;" class="">typedef</span><span style=" color:#808080;" class=""> </span><span style=" color:#f82670;" class="">struct</span><span style=" color:#808080;" class=""> </span><span style=" color:#f92672;" class="">MovieMultiviewReconstructContext</span><span style=" color:#808080;" class=""> </span><span style=" color:#e6e5e2;" class="">{</span></pre>
<pre style="margin-top: 0px; margin-bottom: 0px;" class=""><span style=" color:#808080;" class=""> </span><font color="#f82670" class="">ListBase</font><span style=" color:#808080;" class=""> </span><span style=" color:#e6e5e2;" class="">*</span><span style=" color:#abdc49;" class="">tracks_base</span><span style=" color:#e6e5e2;" class="">;<span class="Apple-tab-span" style="white-space:pre"> </span>// from libmv_Tracks *tracks</span></pre><pre style="margin-top: 0px; margin-bottom: 0px;" class=""><pre style="margin-top: 0px; margin-bottom: 0px;" class=""><span style="color: rgb(128, 128, 128);" class=""> </span><span style="color: rgb(248, 38, 112);" class="">ListBase*</span><span style="font-family: Helvetica; white-space: normal;" class=""> </span><span style="color: rgb(171, 220, 73); font-family: Helvetica; white-space: normal;" class="">reconstruction_clips</span><span style="font-family: Helvetica; white-space: normal; color: rgb(230, 229, 226);" class="">;<span class="Apple-tab-span" style="white-space: pre;"> </span></span><span style="color: rgb(230, 229, 226); font-family: Helvetica; white-space: normal;" class="">// from libmv_Reconstruction *reconstruction</span></pre><pre style="margin-top: 0px; margin-bottom: 0px;" class=""><pre style="margin-top: 0px; margin-bottom: 0px;" class=""><span style="color: rgb(249, 38, 114);" class=""><span class="Apple-tab-span"> </span>ListBase*</span><span style="color: rgb(128, 128, 128);" class=""> </span><span style="color: rgb(171, 220, 73);" class="">all_camera_intrisics</span><span style="color: rgb(230, 229, 226);" class="">;</span></pre><pre style="margin-top: 0px; margin-bottom: 0px;" class=""><font color="#f92672" class=""><span class="Apple-tab-span"> </span>ListBase*</font><span style="color: rgb(128, 128, 128);" class=""> </span><span style="color: rgb(171, 220, 73);" class="">tracks_map_base</span><span style="color: rgb(230, 229, 226);" class="">;</span></pre><pre style="margin-top: 0px; margin-bottom: 0px;" class=""><font color="#f92672" class=""><span class="Apple-tab-span"> </span>ListBase*</font><span style="color: rgb(128, 128, 128);" class=""> </span><span style="color: rgb(171, 220, 73);" class="">correspondence_base</span><span style="color: rgb(230, 229, 226);" class="">;</span></pre></pre></pre>
<pre style="margin-top: 0px; margin-bottom: 0px;" class=""><span style=" color:#808080;" class=""> </span><span style=" font-weight:600; color:#cc8c5b;" class="">bool</span><span style=" color:#808080;" class=""> </span><span style=" color:#abdc49;" class="">select_keyframes</span><span style=" color:#e6e5e2;" class="">;</span></pre>
<pre style="margin-top: 0px; margin-bottom: 0px;" class=""><span style=" color:#808080;" class=""> </span><span style=" color:#f92672;" class="">int</span><span style=" color:#808080;" class=""> </span><span style=" color:#abdc49;" class="">keyframe1</span><span style=" color:#e6e5e2;" class="">,</span><span style=" color:#808080;" class=""> </span><span style=" color:#abdc49;" class="">keyframe2</span><span style=" color:#e6e5e2;" class="">;</span></pre>
<pre style="margin-top: 0px; margin-bottom: 0px;" class=""><span style=" color:#808080;" class=""> </span><span style=" color:#f92672;" class="">int</span><span style=" color:#808080;" class=""> </span><span style=" color:#abdc49;" class="">refine_flags</span><span style=" color:#e6e5e2;" class="">;</span></pre>
<pre style="margin-top: 0px; margin-bottom: 0px;" class=""><span style=" color:#808080;" class=""> </span><span style=" color:#f92672;" class="">char</span><span style=" color:#808080;" class=""> </span><span style=" color:#abdc49;" class="">object_name</span><span style=" color:#e6e5e2;" class="">[</span><span style=" font-weight:600; color:#cc8c5b;" class="">MAX_NAME</span><span style=" color:#e6e5e2;" class="">];</span></pre>
<pre style="margin-top: 0px; margin-bottom: 0px;" class=""><span style=" color:#808080;" class=""> </span><span style=" font-weight:600; color:#cc8c5b;" class="">bool</span><span style=" color:#808080;" class=""> </span><span style=" color:#abdc49;" class="">is_camera</span><span style=" color:#e6e5e2;" class="">;</span></pre>
<pre style="margin-top: 0px; margin-bottom: 0px;" class=""><span style=" color:#808080;" class=""> </span><span style=" color:#f92672;" class="">short</span><span style=" color:#808080;" class=""> </span><span style=" color:#abdc49;" class="">motion_flag</span><span style=" color:#e6e5e2;" class="">;</span></pre>
<pre style="margin-top: 0px; margin-bottom: 0px;" class=""><span style=" color:#808080;" class=""> </span><span style=" color:#f92672;" class="">float</span><span style=" color:#808080;" class=""> </span><span style=" color:#abdc49;" class="">reprojection_error</span><span style=" color:#e6e5e2;" class="">;</span></pre>
<pre style="margin-top: 0px; margin-bottom: 0px;" class=""><span style=" color:#808080;" class=""> </span><span style=" color:#f92672;" class="">int</span><span style=" color:#808080;" class=""> </span><span style=" color:#abdc49;" class="">*sfra</span><span style=" color:#e6e5e2;" class="">,</span><span style=" color:#808080;" class=""> </span><span style=" color:#abdc49;" class="">*efra</span><span style=" color:#e6e5e2;" class="">;<span class="Apple-tab-span" style="white-space:pre"> </span>// start frame and end frame</span></pre>
<pre style="margin-top: 0px; margin-bottom: 0px;" class=""><span style=" color:#e6e5e2;" class="">}</span><span style=" color:#808080;" class=""> </span><span style=" color:#f92672;" class="">MovieMultiviewReconstructContext</span><span style=" color:#e6e5e2;" class="">;</span></pre></div><div style="font-family: Helvetica; white-space: normal;" class=""></div><!--EndFragment--></pre><div class=""><pre style="margin-top: 0px; margin-bottom: 0px;" class=""><div style="font-family: Helvetica; white-space: normal;" class=""><br class=""></div><div style="font-family: Helvetica; white-space: normal;" class="">This basically extends from MovieReconstructContext. The key is that it records a set of correspondences (correspondence_base) specified by users, so I can arrange and connect these tracks in the back end (libmv). </div><div class="">
<pre style="margin-top: 0px; margin-bottom: 0px;" class=""><!--StartFragment--><span style=" color:#808080;" class="">//TODO(tianwei):</span><span style=" color:#808080;" class=""> </span><span style=" color:#808080;" class="">expand</span><span style=" color:#808080;" class=""> </span><span style=" color:#808080;" class="">the</span><span style=" color:#808080;" class=""> </span><span style=" color:#808080;" class="">fields,</span><span style=" color:#808080;" class=""> </span><span style=" color:#808080;" class="">now</span><span style=" color:#808080;" class=""> </span><span style=" color:#808080;" class="">only</span><span style=" color:#808080;" class=""> </span><span style=" color:#808080;" class="">two</span><span style=" color:#808080;" class=""> </span><span style=" color:#808080;" class="">tracks</span></pre><pre style="margin-top: 0px; margin-bottom: 0px;" class=""><span style=" color:#f82670;" class="">typedef</span><span style=" color:#808080;" class=""> </span><span style=" color:#f82670;" class="">struct</span><span style=" color:#808080;" class=""> </span><span style=" color:#f92672;" class="">MovieTrackingCorrespondence</span><span style=" color:#808080;" class=""> </span><span style=" color:#e6e5e2;" class="">{</span></pre><pre style="margin-top: 0px; margin-bottom: 0px;" class=""><span style=" color:#808080;" class=""> </span><span style=" color:#f82670;" class="">struct</span><span style=" color:#808080;" class=""> </span><span style=" color:#f92672;" class="">MovieTrackingCorrespondence</span><span style=" color:#808080;" class=""> </span><span style=" color:#e6e5e2;" class="">*</span><span style=" color:#abdc49;" class="">next</span><span style=" color:#e6e5e2;" class="">,</span><span style=" color:#808080;" class=""> </span><span style=" color:#e6e5e2;" class="">*</span><span style=" color:#abdc49;" class="">prev</span><span style=" color:#e6e5e2;" class="">;</span></pre><pre style="margin-top: 0px; margin-bottom: 0px;" class=""><span style=" color:#808080;" class=""> </span><span style=" color:#f92672;" class="">char</span><span style=" color:#808080;" class=""> </span><span style=" color:#abdc49;" class="">name</span><span style=" color:#e6e5e2;" class="">[</span><span style=" color:#ae81ff;" class="">64</span><span style=" color:#e6e5e2;" class="">];</span><span style=" color:#808080;" class=""> /*</span><span style=" color:#808080;" class=""> </span><span style=" color:#808080;" class="">MAX_NAME</span><span style=" color:#808080;" class=""> </span><span style=" color:#808080;" class="">*/</span></pre><pre style="margin-top: 0px; margin-bottom: 0px;" class=""><span style=" color:#808080;" class=""> </span><span style=" color:#f92672;" class="">MovieTrackingTrack</span><span style=" color:#808080;" class=""> </span><span style=" color:#e6e5e2;" class="">*</span><span style=" color:#abdc49;" class="">primary_track</span><span style=" color:#e6e5e2;" class="">;</span></pre><pre style="margin-top: 0px; margin-bottom: 0px;" class=""><span style=" color:#808080;" class=""> </span><span style=" color:#f92672;" class="">MovieTrackingTrack</span><span style=" color:#808080;" class=""> </span><span style=" color:#e6e5e2;" class="">*</span><span style=" color:#abdc49;" class="">witness_track</span><span style=" color:#e6e5e2;" class="">;</span></pre><pre style="margin-top: 0px; margin-bottom: 0px;" class=""><pre style="margin-top: 0px; margin-bottom: 0px;" class=""><!--StartFragment--><span style=" color:#808080;" class=""> </span><span style=" color:#f92672;" class="">int</span><span style=" color:#808080;" class=""> </span><span style=" color:#abdc49;" class="">primary_clip_id</span><span style=" color:#e6e5e2;" class="">;</span></pre><pre style="margin-top: 0px; margin-bottom: 0px;" class=""><span style=" color:#808080;" class=""> </span><span style=" color:#f92672;" class="">int</span><span style=" color:#808080;" class=""> </span><span style=" color:#abdc49;" class="">witness_clip_id</span><span style=" color:#e6e5e2;" class="">;</span></pre></pre><pre style="margin-top: 0px; margin-bottom: 0px;" class=""><span style=" color:#e6e5e2;" class="">}</span><span style=" color:#808080;" class=""> </span><span style=" color:#f92672;" class="">MovieTrackingCorrespondence</span><span style=" color:#e6e5e2;" class="">;</span><!--EndFragment--></pre><pre style="margin-top: 0px; margin-bottom: 0px;" class=""><div style="font-family: Helvetica; white-space: normal;" class=""><br class=""></div><div style="font-family: Helvetica; white-space: normal;" class="">For now this just keeps tracks of two track pointers and their clip ids, since <span style="color: rgb(249, 38, 114);" class="">MovieTrackingTrack</span> does not have this information. The indexing of clip id can go with <span style="color: rgb(249, 38, 114);" class="">ListBase</span><span style="color: rgb(128, 128, 128);" class=""> </span><span style="color: rgb(230, 229, 226);" class="">*</span><span style="color: rgb(171, 220, 73);" class="">clips</span><font color="#e6e5e2" class=""> </font>in <span style="color: rgb(249, 38, 114);" class="">SolveMultiviewJob</span>. For example, the tracks in the main camera has primary_clip_id equal 0, the first witness camera is indexed 1, so on and so forth. We may also go with the id field in MovieClip.</div><div class=""><br class=""></div><div class=""><font face="Helvetica" class=""><span style="white-space: normal;" class="">So the CLIP_OT_solve_multiview will work like this. First we prepare everything that libmv needs and pass them to MovieMultiviewReconstructContext, including the primary tracks, witness tracks, correspondences, tracks_map, key frames, etc. Then I process the options and pass all the tracks and correspondences to libmv_solve_multiview_reconstruction. </span></font></div><div class=""><font face="Helvetica" class=""><span style="white-space: normal;" class=""><br class=""></span></font></div><div class=""><font face="Helvetica" class=""><span style="white-space: normal;" class="">Then we come to the back end, libmv. It seems to me that libmv::Tracks is also agnostic about the video clip which the track comes from, so I need to complete that information using the correspondences (between video clips). Maybe I need a wrapper around libmv::Tracks, called it libmv::IdenticalTracks, which serve as a container for tracks regarded as the same stuff. The reconstruction can be similar to the previous single-view one, which we first compute a two-view geometry from the two keyframes, then inserted other images both in the primary camera and witness cameras. After bundle adjustment over all the images, we can pass back the info to the blender side and do the cleanup. </span></font></div><div class=""><font face="Helvetica" class=""><span style="white-space: normal;" class=""><br class=""></span></font></div><div class=""><font face="Helvetica" class=""><span style="white-space: normal;" class="">This is basically my thoughts on the multi-view pipeline. I haven’t figure out how object tracking plays the role. Let me first finish the camera tracking part. Comment and suggestions are very much welcomed and appreciated. :) I will also add this to the blender wiki so that you can review and comment. Thanks.</span></font></div><div class=""><font face="Helvetica" class=""><span style="white-space: normal;" class=""><br class=""></span></font></div><div class=""><font face="Helvetica" class=""><span style="white-space: normal;" class=""><br class=""></span></font></div><div class=""><font face="Helvetica" class=""><span style="white-space: normal;" class="">Thanks,</span></font></div><div class=""><font face="Helvetica" class=""><span style="white-space: normal;" class="">Tianwei</span></font></div><div class=""><br class=""></div></pre></div></pre></div></pre></div><div class=""><div><blockquote type="cite" class=""><div class="">On 2016 M06 4, at 21:00, Sergey Sharybin <<a href="mailto:sergey.vfx@gmail.com" class="">sergey.vfx@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Hi,<div class=""><br class=""></div><div class="">Do you have some design overview of what your new data structures are and how they all fit together?</div><div class=""><br class=""></div><div class="">Would also be cool to hear how exactly you're approaching to the camera solver. While it might be separate Blender operator, you can't really (and should not in fact) separate things in Libmv.</div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Sat, Jun 4, 2016 at 9:50 AM, Tianwei Shen <span dir="ltr" class=""><<a href="mailto:shentianweipku@gmail.com" target="_blank" class="">shentianweipku@gmail.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi everyone,<br class="">
<br class="">
This is report of week 2 for my multi-view reconstruction project. You can also view this report at blender wiki:<br class="">
<a href="https://wiki.blender.org/index.php/User:Tianwei37/Report_&_Blog/Week_2" rel="noreferrer" target="_blank" class="">https://wiki.blender.org/index.php/User:Tianwei37/Report_%26_Blog/Week_2</a><br class="">
<br class="">
What I did:<br class="">
- organize correspondence: add several utility functions and data structures. A complete correspondence data structure is still in development, until I figure out what’s needed in the back end.<br class="">
- add 'solve multi-view' operator and implement it. I separate the function of ‘solve single camera’ and ‘solve multi-camera’ for now. This allows me to do the multi-view reconstruction code without interfering the original solve-camera code. After I finish this function. I will integrate the multi-view reconstruction with the single-view reconstruction.<br class="">
- watch TrackMatchBlend and get a full picture of the tracking system. (I thought TrackMatchBlend is an episode and introduction, while it is actually a series of videos, great tutorials!)<br class="">
<br class="">
What I will do next week:<br class="">
- I thought I could finish a basic multi-view reconstruction operator this week. But it turned out that it involves a great deal of changes to the original codebase. So I will continue the work of multi-view reconstruction.<br class="">
- look into the UI side and consider making a separate mode for specifying correspondences.<br class="">
<br class="">
<br class="">
Thanks,<br class="">
Tianwei<br class="">
_______________________________________________<br class="">
Soc-2016-dev mailing list<br class="">
<a href="mailto:Soc-2016-dev@blender.org" class="">Soc-2016-dev@blender.org</a><br class="">
<a href="https://lists.blender.org/mailman/listinfo/soc-2016-dev" rel="noreferrer" target="_blank" class="">https://lists.blender.org/mailman/listinfo/soc-2016-dev</a><br class="">
</blockquote></div><br class=""><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div class="gmail_signature" data-smartmail="gmail_signature"><div class=""><span style="color:rgb(102,102,102)" class="">With best regards, Sergey Sharybin</span></div></div>
</div>
_______________________________________________<br class="">Soc-2016-dev mailing list<br class=""><a href="mailto:Soc-2016-dev@blender.org" class="">Soc-2016-dev@blender.org</a><br class="">https://lists.blender.org/mailman/listinfo/soc-2016-dev<br class=""></div></blockquote></div><br class=""></div></body></html>