<div class="gmail_quote">On Tue, May 10, 2011 at 8:49 AM, Sergey I. Sharybin <span dir="ltr"><<a href="mailto:g.ulairi@gmail.com">g.ulairi@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div bgcolor="#ffffff" text="#000000">
<small>Hi everybody,<br>
<br>
If somebody still not sure, i'm exactly that student who'll be
working under integrating libmv to Blender.<br>
<br>
After all discussions, i could imagine workflow we want to reach
(all this pre-/post-processing and so on). But let's be
sequentially. We still got nothing implemented but arguing about
place of workflow where undistortion should happen.<br></small></div></blockquote><div><br></div><div>I couldn't agree more.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div bgcolor="#ffffff" text="#000000"><small>I'd prefer to stick to initial proposal with new editor type and
new data ID named MovieClip. I'll describe why i prefer this a bit
later.<br></small></div></blockquote><div><br></div><div>Sounds good to me.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div bgcolor="#ffffff" text="#000000">
<small>So, proposal with working under tracker is the following:<br>
<br>
- Switch to "Tracking" scene layout where there'll be SpaceClip
editor opened. Also, there'll be timeline for
searching/bplayback-ing image sequence.<br>
Initially, implement tools-ish approach of working with this
sequence/tracker data. No nodes at all for first time and working
with such editor would be "iterative" -- you'll be working under
clearing/refining/resolving sequence until you've got good result.<br>
- In this SpaceClip you'll open (or choose already opened)
video/image sequence and start working with it.<br>
- There'll be Toolshelf with panels like "Lens tools", "Marker
tools", "Solver tools" (i hope meaning of this panels is
clear)roperties region to set such parameters, as marker's search
area and so on.<br>
So to change marker's properties, i.e. you'll simply select it
in SpaceClip, open Peoperties panel (if it's stil lclosed) and
input some parameters there. The same approach could be user for
other objects in clip (camera or whatever else).<br>
- It was useful for me to see tracker points (i could be a bit
wrong here with terminology, but i mean representation of markers
in 3d world). I think displaying them (as set of empties or bones,
i.e.) in 3d view would be what we need. Ofcourse we'll need
selection synchronization between markers and trackers. I don't
think it's an issue.<br></small></div></blockquote><div><br></div><div>We need to be clear on definitions. Take a look at the doc I shared, which I've renamed as a simple blender design doc; there are more definitions there. Let's call the 3D point associated with a 2D track a "bundle".</div>
<div><br></div><div>I also think we want to see the 2D tracks somehow in the 3D view. It's also required to be able to see the 3D bundles (3D points reconstructed from each tracked marker) in the normal scene view. I argue those empties should be combined into one "object" that can get moved around the 3D scene, and any linked cameras get moved as well.</div>
<div><br></div><div>This is a requirement even for the simple tracker.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div bgcolor="#ffffff" text="#000000">
<small>
<br>
This should be enough for first-generation tracker in Blender.
It'll be already big step forward if it'll be working. Ofcource
more cool things would be implemenetd after this.<br>
<br>
Now a bit about internal side.<br>
<br>
Data structure overview. There were some discussions about where
all this markers/lens/etc data would be stored and how it'll be
applied. One of variants was to use modifiers' like approach --
modifier for tracking, modifier for solving.. Well, it's a bit
complicated due to there'll be almost always auto tracking+manual
tracks+solving, But i like idea of separating data from VideoClip
and keeping data outside of VideoClip could help with mosing to
node-based design. I can't give exact image of this structures,
but i'd prefer keep it separatelly (not "hardcoded" to clip).</small></div></blockquote><div><br></div><div>Yes, let's split the VideoClip from tracking. </div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div bgcolor="#ffffff" text="#000000"><small> Something like<br>
<br>
struct VideoClip*{<br>
...<br>
struct TrackerData *tracker_data;<br>
...<br>
}<br>
struct TrackerData {<br>
...<br>
struct MarkersData *markers_data;<br>
struct LensData *lens_data;<br>
struct WhateverelseData *...;<br>
...<br>
}<br>
<br>
This MarkersData/LensData/etc could be easily stored in nodes in
the future. And ofcourse all this things related to RNA.. <br></small></div></blockquote><div><br></div><div>Looks great to me. Note that this will all have to go into SDNA.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div bgcolor="#ffffff" text="#000000"><small>But there's some issues i can't solve here myself. They are
related to data types and API which are required/acceptable by
libmv. I hope Keir would help me with this issues. So, questions
are:<br>
<br>
- I suppose we should sotre some kind of libmv context, right?<br></small></div></blockquote><div><br></div><div>Yes, but don't worry about that for now. The very first version will use frame-to-frame tracking for 2D, so there will only be per-frame data and a bit of per-bundle data.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div bgcolor="#ffffff" text="#000000"><small>
- How/where markers should be stored? How their "animation" should
be stored?<br></small></div></blockquote><div><br></div><div>This is a critical question. Ton or other experienced blender dev should hop in here. There will be very little libmv data for this part; it will all have to be in blender.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div bgcolor="#ffffff" text="#000000"><small>
- The same question about trackers.<br></small></div></blockquote><div><br>Yes, see above.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div bgcolor="#ffffff" text="#000000">
<small>
- How would we pass image sequence to libmv? Also it'll be cool to
know which API Blender should use (functions and so on).<br></small></div></blockquote><div><br></div><div>+1; has this changed from 2.4 to 2.5? It used to be that there was an image sequence API, but it was not trivial to use.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div bgcolor="#ffffff" text="#000000"><small>
- Not sure if it's time for such questions, but it'll be nice if
tracking/solving woudl happen in separated thread and displaying
kind progress bar. Some kind of "special" libmv API could be
needed for this, not sure.<br></small></div></blockquote><div><br></div><div>While that would be great, let's punt on threading for the first version. The key is to not write the code such that you can't multithread later.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div bgcolor="#ffffff" text="#000000"><small>
- And ofcourse issues related to cache. Would be nice to know if
there's some cache in libmv already and whar it supports/would
support.<br></small></div></blockquote><div><br>libmv has a cache for the image transforms necessary for tracking. What's less clear is that it's a good fit for blender. Currently there aren't any users of that code that I know of, so I'm happy to tweak it so that it's a better fit.</div>
<div><br></div><div>The caching image sequence:</div><div><meta http-equiv="content-type" content="text/html; charset=utf-8"><a href="http://code.google.com/p/libmv/source/browse/trunk/src/libmv/image/cached_image_sequence.h">http://code.google.com/p/libmv/source/browse/trunk/src/libmv/image/cached_image_sequence.h</a></div>
<div><br></div><div>On top of the caching image sequence, I built filtering cached image sequences:</div><div><meta http-equiv="content-type" content="text/html; charset=utf-8"><a href="http://code.google.com/p/libmv/source/browse/trunk/src/libmv/image/filtered_sequence.h">http://code.google.com/p/libmv/source/browse/trunk/src/libmv/image/filtered_sequence.h</a></div>
<div><br></div><div>Then composing that, I built caching image pyramid sequences:</div><div><meta http-equiv="content-type" content="text/html; charset=utf-8"><a href="http://code.google.com/p/libmv/source/browse/trunk/src/libmv/image/pyramid_sequence.h">http://code.google.com/p/libmv/source/browse/trunk/src/libmv/image/pyramid_sequence.h</a></div>
<div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div bgcolor="#ffffff" text="#000000"><small>If we'll decide that there would be nicer to have tracker ID as
initial entity (this could happen because of different issues --
tracking from multiplie sources, i.e.), there wouldn't be much
changes in DNA -- we'll just move VIdeoClip inside TrackerData and
make TrackerData "root" of tracking stuff. Also, after such switch
we coudl use Image instead of Clip (we've been discussing it today
in IRC).<br></small></div></blockquote><div><br></div><div>I suspect this (have the VideoClip inside the TrackerData) is what we want.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div bgcolor="#ffffff" text="#000000"><small>Note: do not treat this data structures as something finished, i
just wanted to show it'll be better to keep data "separated" and
not keeping all markers/lens/trackers data in the single
structure. I've got feeling it'll help to make design more easily
adaptable for nodes.<br></small></div></blockquote><div><br></div><div>Totally.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div bgcolor="#ffffff" text="#000000">
<small>A bit about tools. Tools (like auto-track, solve, so on) would be
implemented in SpaceClip to deal with very basic tracking. In the
future, new tools would be added or even moved to other editors --
it shouldn't be difficult. I think we could start from auto
track/markers/solving tools and extend them when very basic
workflow would work.<br></small></div></blockquote><div><br></div><div>Definitely. Let's keep things simple to start and not get distracted with non-core features until we have a working tracker.</div><div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div bgcolor="#ffffff" text="#000000"><small>Now a bit about that new ID and editor we've been arguing about
this afternoon. Yep, i'l lagree it could be some duplicated work,
but we'll never know which of changes are really necessery and
which not (before we'll try ofcourse) and such kind of
"experiment" could have negative result on existing tools. And i'm
also warrying about further support of code -- fixing bug in
tracker could break texture stuff (and main reason of it would be
that there're different developers who works for textures and
tarcker and they can't predict if their change would breac
something in other place). It will also help to keep UI clear and
logical. There'll be no paiting/UV tools while match-moving. This
looks like an advantage. And.. never of us was really working
under tracking software, so why not accept this plan as initial?
It could be easult changed in the future (if we'll decide it's
necessery).<br></small></div></blockquote><div><br></div><div>+1</div><div><br></div><div>A straightforward tracker-specific UI is the way to go to start.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div bgcolor="#ffffff" text="#000000"><small>And one more question mostly to Keir. Last time we've been talking
in IRC he proposed to contentrate on 2d stuff without touching
3d/solving. But i've been able to use SfM from libmv
(reconstruct_video) which gave me quite good result on some
sequences from lessons for other tracker's software. So maybe it's
not too much to be finished there to be useful from Bledner?</small></div></blockquote><div><br>I think 3D is totally doable. What I want is to make a "deliverable" 2D tracking system before we do 3D. This is in the spirit of release-early-release-often. In all honesty, a solid 2D tracking system with usability tweaks is larger than a GSoC project already, so even this is non-trivial.</div>
<div><br></div><div>Since there are two of you, this makes the work division easier:</div><div><br></div><div>Sergey:</div><div><ol><li>Race to simplest 2D tracker possible</li><li>Polish 2D tracker and release</li><li>If there's time, work on 3D UI with Matthias</li>
</ol><div>Matthais:</div></div><div><ol><li>libmv build system integration with blender (or decide how this will be done)</li><li>Flesh out components of libmv needed for solving (it's not production ready)</li><li>Add simple 3D solving on top of the simplest 2D tracker (Sergey's #1). This will involve both libmv work and Blender UI work.</li>
</ol></div><div> Keir<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div bgcolor="#ffffff" text="#000000"><div class="im"><br>
<pre cols="72">--
With best regards, Sergey I. Sharybin</pre>
</div></div>
<br>_______________________________________________<br>
Bf-vfx mailing list<br>
<a href="mailto:Bf-vfx@blender.org">Bf-vfx@blender.org</a><br>
<a href="http://lists.blender.org/mailman/listinfo/bf-vfx" target="_blank">http://lists.blender.org/mailman/listinfo/bf-vfx</a><br>
<br></blockquote></div><br>