[Bf-vfx] Blender data, big picture

[Keir Mierle]

On Tue, May 10, 2011 at 8:49 AM, Sergey I. Sharybin <g.ulairi at gmail.com>wrote:

>  Hi everybody,
>
> If somebody still not sure, i'm exactly that student who'll be working
> under integrating libmv to Blender.
>
> After all discussions, i could imagine workflow we want to reach (all this
> pre-/post-processing and so on). But let's be sequentially. We still got
> nothing implemented but arguing about place of workflow where undistortion
> should happen.
>

I couldn't agree more.


> I'd prefer to stick to initial proposal with new editor type and new data
> ID named MovieClip. I'll describe why i prefer this a bit later.
>

Sounds good to me.


> So, proposal with working under tracker is the following:
>
> - Switch to "Tracking" scene layout where there'll be SpaceClip editor
> opened. Also, there'll be timeline for searching/bplayback-ing image
> sequence.
> Initially, implement tools-ish approach of working with this
> sequence/tracker data. No nodes at all for first time and working with such
> editor would be "iterative" -- you'll be working under
> clearing/refining/resolving sequence until you've got good result.
> - In this SpaceClip you'll open (or choose already opened) video/image
> sequence and start working with it.
> - There'll be Toolshelf with panels like "Lens tools", "Marker tools",
> "Solver tools" (i hope meaning of this panels is clear)roperties region to
> set such parameters, as marker's search area and so on.
>   So to change marker's properties, i.e. you'll simply select it in
> SpaceClip, open Peoperties panel (if it's stil lclosed) and input some
> parameters there. The same approach could be user for other objects in clip
> (camera or whatever else).
> - It was useful for me to see tracker points (i could be a bit wrong here
> with terminology, but i mean representation of markers in 3d world). I think
> displaying them (as set of empties or bones, i.e.) in 3d view would be what
> we need. Ofcourse we'll need selection synchronization between markers and
> trackers. I don't think it's an issue.
>

We need to be clear on definitions. Take a look at the doc I shared, which
I've renamed as a simple blender design doc; there are more definitions
there. Let's call the 3D point associated with a 2D track a "bundle".

I also think we want to see the 2D tracks somehow in the 3D view. It's also
required to be able to see the 3D bundles (3D points reconstructed from each
tracked marker) in the normal scene view. I argue those empties should be
combined into one "object" that can get moved around the 3D scene, and any
linked cameras get moved as well.

This is a requirement even for the simple tracker.


> This should be enough for first-generation tracker in Blender. It'll be
> already big step forward if it'll be working. Ofcource more cool things
> would be implemenetd after this.
>
> Now a bit about internal side.
>
> Data structure overview. There were some discussions about where all this
> markers/lens/etc data would be stored and how it'll be applied. One of
> variants was to use modifiers' like approach -- modifier for tracking,
> modifier for solving.. Well, it's a bit complicated due to there'll be
> almost always auto tracking+manual tracks+solving, But i like idea of
> separating data from VideoClip and keeping data outside of VideoClip could
> help with mosing to node-based design. I can't give exact image of this
> structures, but i'd prefer keep it separatelly (not "hardcoded" to clip).
>

Yes, let's split the VideoClip from tracking.


> Something like
>
>     struct VideoClip*{
>         ...
>         struct TrackerData *tracker_data;
>         ...
>     }
>     struct TrackerData {
>         ...
>         struct MarkersData *markers_data;
>         struct LensData *lens_data;
>         struct WhateverelseData *...;
>         ...
>     }
>
> This MarkersData/LensData/etc could be easily stored in nodes in the
> future. And ofcourse all this things related to RNA..
>

Looks great to me. Note that this will all have to go into SDNA.


> But there's some issues i can't solve here myself. They are related to data
> types and API which are required/acceptable by libmv. I hope Keir would help
> me with this issues. So, questions are:
>
> - I suppose we should sotre some kind of libmv context, right?
>

Yes, but don't worry about that for now. The very first version will use
frame-to-frame tracking for 2D, so there will only be per-frame data and a
bit of per-bundle data.


> - How/where markers should be stored? How their "animation" should be
> stored?
>

This is a critical question. Ton or other experienced blender dev should hop
in here. There will be very little libmv data for this part; it will all
have to be in blender.


> - The same question about trackers.
>

Yes, see above.


> - How would we pass image sequence to libmv? Also it'll be cool to know
> which API Blender should use (functions and so on).
>

+1; has this changed from 2.4 to 2.5? It used to be that there was an image
sequence API, but it was not trivial to use.


> - Not sure if it's time for such questions, but it'll be nice if
> tracking/solving woudl happen in separated thread and displaying kind
> progress bar. Some kind of "special" libmv API could be needed for this, not
> sure.
>

While that would be great, let's punt on threading for the first version.
The key is to not write the code such that you can't multithread later.


> - And ofcourse issues related to cache. Would be nice to know if there's
> some cache in libmv already and whar it supports/would support.
>

libmv has a cache for the image transforms necessary for tracking. What's
less clear is that it's a good fit for blender. Currently there aren't any
users of that code that I know of, so I'm happy to tweak it so that it's a
better fit.

The caching image sequence:
http://code.google.com/p/libmv/source/browse/trunk/src/libmv/image/cached_image_sequence.h

On top of the caching image sequence, I built filtering cached image
sequences:
http://code.google.com/p/libmv/source/browse/trunk/src/libmv/image/filtered_sequence.h

Then composing that, I built caching image pyramid sequences:
http://code.google.com/p/libmv/source/browse/trunk/src/libmv/image/pyramid_sequence.h

If we'll decide that there would be nicer to have tracker ID as initial
> entity (this could happen because of different issues -- tracking from
> multiplie sources, i.e.), there wouldn't be much changes in DNA -- we'll
> just move VIdeoClip inside TrackerData and make TrackerData "root" of
> tracking stuff. Also, after such switch we coudl use Image instead of Clip
> (we've been discussing it today in IRC).
>

I suspect this (have the VideoClip inside the TrackerData) is what we want.


> Note: do not treat this data structures as something finished, i just
> wanted to show it'll be better to keep data "separated" and not keeping all
> markers/lens/trackers data in the single structure. I've got feeling it'll
> help to make design more easily adaptable for nodes.
>

Totally.


> A bit about tools. Tools (like auto-track, solve, so on) would be
> implemented in SpaceClip to deal with very basic tracking. In the future,
> new tools would be added or even moved to other editors -- it shouldn't be
> difficult. I think we could start from auto track/markers/solving tools and
> extend them when very basic workflow would work.
>

Definitely. Let's keep things simple to start and not get distracted with
non-core features until we have a working tracker.


> Now a bit about that new ID and editor we've been arguing about this
> afternoon. Yep, i'l lagree it could be some duplicated work, but we'll never
> know which of changes are really necessery and which not (before we'll try
> ofcourse) and such kind of "experiment" could have negative result on
> existing tools. And i'm also warrying about further support of code --
> fixing bug in tracker could break texture stuff (and main reason of it would
> be that there're different developers who works for textures and tarcker and
> they can't predict if their change would breac something in other place). It
> will also help to keep UI clear and logical. There'll be no paiting/UV tools
> while match-moving. This looks like an advantage. And.. never of us was
> really working under tracking software, so why not accept this plan as
> initial? It could be easult changed in the future (if we'll decide it's
> necessery).
>

+1

A straightforward tracker-specific UI is the way to go to start.


> And one more question mostly to Keir. Last time we've been talking in IRC
> he proposed to contentrate on 2d stuff without touching 3d/solving. But i've
> been able to use SfM from libmv (reconstruct_video) which gave me quite good
> result on some sequences from lessons for other tracker's software. So maybe
> it's not too much to be finished there to be useful from Bledner?
>

I think 3D is totally doable. What I want is to make a "deliverable" 2D
tracking system before we do 3D. This is in the spirit of
release-early-release-often. In all honesty, a solid 2D tracking system with
usability tweaks is larger than a GSoC project already, so even this is
non-trivial.

Since there are two of you, this makes the work division easier:

Sergey:

   1. Race to simplest 2D tracker possible
   2. Polish 2D tracker and release
   3. If there's time, work on 3D UI with Matthias

Matthais:

   1. libmv build system integration with blender (or decide how this will
   be done)
   2. Flesh out components of libmv needed for solving (it's not production
   ready)
   3. Add simple 3D solving on top of the simplest 2D tracker (Sergey's #1).
   This will involve both libmv work and Blender UI work.

 Keir


> --
> With best regards, Sergey I. Sharybin
>
>
> _______________________________________________
> Bf-vfx mailing list
> Bf-vfx at blender.org
> http://lists.blender.org/mailman/listinfo/bf-vfx
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.blender.org/pipermail/bf-vfx/attachments/20110511/b8b30d91/attachment-0001.htm