[Bf-vfx] Blender camera tracking approach
Ton Roosendaal
ton at blender.org
Mon May 9 14:48:41 CEST 2011
Hi Sebastian (and everyone!),
Thanks, great feedback!
Apart from getting good insight in general tracking workflows, we'll
have to be practical as well. Target is to first give the libmv team a
Blender testbed so they can more efficiently work too.
A lot of design issues depend on how libmv will handle tracking
itself: which features they'll require to have in Blender, what
options it will support. This info has to come from the libmv team
first before we can elaborate design and workflow further.
The simplest approach forward (right now) is to keep the Blender
system as a separate 'black box', a video clip & tracking editor in
Blender that allows to edit markers and which communicates with libmv.
This not only keeps things simple & feasible, but also will keep such
work manageable. Blender already suffers too much from many
overlapping modules and functionality, having this work - both in UI
as in code design - organized as a new 'module' with distinct coder
ownership is going to help us well.
Same will go for (possible) image preprocess filter options and
masking. If libmv requires or supports it, we can look at how to
integrate it well. Note that the concept of 'mask' is already having
thought out, to be a new general type in Blender which could be edited
in many places; sequencer/compositor/image/tracker, and to be used as
compo node, or as texture and sequence strip even.
I'm also not really interested to provide an open source replacement
for top products like pfMatchit or Syntheyes or Nuke, but to provide a
useful addition for Blender artists to combine film footage with 3D
first. We need to adjust our requirements with our competences too.
First get some totally simple 3D camera track case work. That would be
already a major step forward! :)
-Ton-
------------------------------------------------------------------------
Ton Roosendaal Blender Foundation ton at blender.org www.blender.org
Blender Institute Entrepotdok 57A 1018AD Amsterdam The Netherlands
On 9 May, 2011, at 12:55, Sebastian König wrote:
> That video shows some impressive results.
> 3D stabilisation gives much better results than just 2d stabilizing.
> Syntheyes does that as well, but i don't think it has the warping,
> which really seems to give good results.
> However, as seen in the samples, it also crops the image quite a bit
> to get stable camera. That can work if you shoot 4k for a 2k end-
> result, but otherwise you sacrifice a lot of image information.
>
> I have tried to visualize 3 different tracking workflows, with
> preprocessing, without pp, and with a real 3d camera lens distortion.
> The tracking workflow itself is not described in detail, just how it
> could integrate into blender.
> The flowchart shows the usage of proxy-nodes, which could be viewer-
> nodes or something like that, just to have something to render into
> some sort of image cache, without having to save the temporarily
> preprocessed images to disk. Or better yet: The proxy-node saves to
> disk, just like cache in physx baking, but makes the images
> immediately available with out having to use an extra image input
> node.
>
> Sebastian
>
> <3_Workflows.pdf>
>
>
> Am 09.05.2011 um 12:36 schrieb ibkanat:
>
>> I was looking at a paper the other day that saving the tracking and
>> distortion data might help. I sent this to Julian he had just
>> implemented a fix for rolling shutter in libmv. Cmos chips have this
>> trouble panning.
>>
>> http://pages.cs.wisc.edu/%7Efliu/project/3dstab.htm It uses warping
>> to
>> stablize video. I am not sure if is a goal for Blender's VE to become
>> more robust at video editing. But a few tweaks could make blender
>> quite
>> a bit more capable in that area. One being the ability to stablize
>> video. Others being(three point edit, ripple editing, improve the ui
>> for color balance, etc).
>>
>> As far a work flow I havent done as much as others. But I read quite
>> often that they get a great track stablize then add the vfx then add
>> back a controlled amount of what ever they want(shake, distortion,
>> etc)
>> If there could be a way to store this data there could be lots of non
>> standard uses(particle effects following optical flow patterns, etc)
>>
>>
>> On Mon, 2011-05-09 at 11:16 +0200, Sebastian König wrote:
>>> Hey all!
>>> That is a pretty good and detailed workflow, Tom. I also like
>>> Tobias' graph.
>>> I want to a few thoughts as well.
>>>> From my experience with Syntheyes, 2d) from Tom's list
>>>> (undistorting the footage) can also happen after or during the
>>>> solving process.
>>> Of course you can create the distortion parameters with a grid
>>> before the tracking happens, but since the whole tracking thing is
>>> just an approximation anyway it can also happen based on the
>>> initial track itself. At least that's how it is done in Syntheyes.
>>> Though i don't have the slightest idea how that is achieved code-
>>> wise. I guess it happens in a similar way as the 3d camera
>>> solution itself is calculated, based on the movements of the
>>> tracked points in relation to each other.
>>> In most cases it improves tracking quality dramatically, because
>>> the tracker doesn't get confused buy the apparently "wrong"
>>> movements of the feature points. Once you have the lens distortion
>>> data available you have 2 options, either undistort the footage
>>> and export that, or just export the camera-movement and re-distort
>>> the final CG afterwards, before bringing original footage and
>>> distorted CG together as a final step.
>>> If we could bypass that step by rendering distorted images
>>> directly it would be cool, but, as Tom said, distorting CG is a
>>> little bit of a loss in image quality of CG, but barely noticeable
>>> in most cases. Usually you'd have to add grain and a bit of blur
>>> to the CG anyways to match the look of the footage, which is
>>> rarely as perfect and clean as 3d renders.
>>>
>>> I wanted to add another thing, following the discussion in IRC
>>> yesterday whether or not to have the tracker integrated into the
>>> compositor or vice versa, because of image preprocessing.
>>> Preprocessing can indeed enhance the track quite a bit. A slightly
>>> blurred black and white image with enhanced contrast can make it
>>> easier for the tracker to recognize feature points in the footage
>>> and track them over time, as that preprocessing can eliminate
>>> noise in the footage.
>>> If I understood correctly Ton wants to add a new editor for
>>> tracking (clip editor?). This will then be used to track the
>>> features, create masks, start the solving, and also maybe manage
>>> the 3d (and also 2d) tracking points. I like that idea, and I
>>> think it will play nicely with the compositor and the whole
>>> concept of editors in Blender.
>>> But what you add as footage in the clip editor could just as well
>>> be the output from a viewer-node, that has a preprocessed version
>>> of the footage. But for that to work efficiently the viewer-node
>>> (or the compositor in general) needs a new feature: image cache.
>>> We do need a good caching system anyway.
>>> So a workflow that deals with tracking and the compositor could
>>> look like this:
>>>
>>> 1) Import footage in clip editor.
>>> 2) If no preprocessing is needed because tracking is just fine
>>> then continue with 3) from Tom's workflow
>>> 2a) If you realize that the feature points do not track
>>> sufficiently because there is too much noise or too little
>>> contrast, then
>>> 3) Exchange the clip input in clip editor with a viewer-node-
>>> output (or a new cache-node or whatever) and
>>> 4) go to the compositor
>>> 5) load image sequence into an image-input-node
>>> 6) preprocess with blur, scale, rgb-curves etc.
>>> 7) If we go with a workflow that lets you correct lens-distortion
>>> before tracking, we could use a lens-node for that.
>>> 8) pipe the preprocessed result into a viewer node and go back to
>>> clip editor
>>> 8a) also create a scaled down lowres proxy version of the footage
>>> to have that in the viewport background for fast playback. I guess
>>> this has to be undistorted to fit to the geometry in the viewport.
>>> --
>>> All this would ideally require really fast preprocessing
>>> capabilities of the compositor and a good caching system. But if
>>> that is not possible we can always preprocess into a new proxy
>>> image sequence on harddrive.
>>> --
>>> 9) If we had a blender camera that can use distortion values from
>>> the tracker we could just render our CG and put that on top of our
>>> original footage.
>>> 9a) If that is not possible we would have to use the lens
>>> distortion values and distort the rendered images with that.
>>>
>>> So far.
>>> Masks could be generated in clip editor too, but accessed from the
>>> compositor as well, i don't know.
>>>
>>> Cheers!
>>>
>>>
>>> Seb
>>>
>>>
>>>
>>> Am 08.05.2011 um 23:05 schrieb Tom M:
>>>
>>>> Tobias,
>>>>
>>>> here is a 'complete' workflow for tracking, it might be good to
>>>> provide details for some of your flow areas off to the side.
>>>>
>>>> 1) load image data - video or sequence of images
>>>> 2a) (optional) correct rolling shutter
>>>> 2b) (optional) input known camera parameters
>>>> 2c) (optional) provide hints to camera lens undistortion solver
>>>> (identify straight lines in the image) to help with undistorting
>>>> the
>>>> image
>>>> 2d) (optional) undistort the camera lens using a distortion solver
>>>> 3) (optional) create mask for data you don't want tracked (or for
>>>> 'grouping'/seperating of tracking ie tracking the motion of a
>>>> camera
>>>> and tracking the motion of an object)
>>>> 4) (optional) adjust color/contrast/etc so that feature points to
>>>> track have increased contrast and are thus easier for the tracker
>>>> to
>>>> find them (and easier for the tracker to ignore stuff you don't
>>>> want
>>>> tracked - ie uneven lighting on a green screen creating spurious
>>>> motion)
>>>> 5) (optional) select specific features you want tracked (either by
>>>> mask or placement of tracking markers)
>>>> 5a) if you specified specific features to the tracker you may also
>>>> want to specify how the points should be tracked (bounding box
>>>> for the
>>>> tracker to look for the point in; whether the object being
>>>> tracked is
>>>> 'rigid' or can deform, and knowledge of the types of camera/object
>>>> motion - object translation; rotation; camera translation;
>>>> rotation)
>>>> 6) send image data, specified track data; camera data; and mask
>>>> data to tracker
>>>> 7) tracker library can automatically identify feature points to
>>>> track
>>>> and or use user specified features, ignoring areas that are masked
>>>> 8) do a 'solve' of the camera motion based on the track points,
>>>> including statistics on how 'good' the tracks are in contributing
>>>> to
>>>> the solve
>>>> 9) return track point data and camera solve to software,
>>>> including the
>>>> statistical analysis of the track points and their 3d projection
>>>> 10) based on the statistical analysis pick error thresholds for
>>>> what
>>>> track points to automatically delete (automatic threshold picking
>>>> or
>>>> manually adjusted)
>>>> 11) (optional) manually delete any track points
>>>> 12) (optional) create a mask to 'hide' unwanted track points
>>>> 12a) (optional) mask can be assigned to follow a set of track
>>>> points
>>>> to automatically mask a moving object from the tracker/solver
>>>> 12b) (optional) mask can be manually keyframed so that it moves and
>>>> deforms over time to mask a moving object from the tracker/solver
>>>> 12) (optional) provide a manually created camera curve to 'hint'
>>>> the
>>>> tracker/solver what you expect the actual curve to look like
>>>> 13) retrack if additional tracker points are now needed
>>>> 14) pass the tracker points, camera hints, etc. to the camera
>>>> solver
>>>> 15) return the solved camera track and the 3d location of the track
>>>> points to the software
>>>> 16) visualize the camera motion and track points in the 3d view
>>>> 17) define a ground plane reference to the 3d track points and
>>>> camera
>>>> 18) define the scene origin relative to the 3d track points and
>>>> camera
>>>> 19) define the world scale and orientation relative to the 3d track
>>>> points and data
>>>> 20) add a test object into the 3d view and see if it stays in the
>>>> proper location
>>>> 21) (optional) stabalize the camera view based on the solved
>>>> camera track
>>>> 22) (optional) smooth the camera track curve
>>>> 23) repeat until the error in the solve is good enough
>>>>
>>>> On Sun, May 8, 2011 at 12:33 PM, Tobias Kummer
>>>> <supertoilet at gmx.net> wrote:
>>>>> After some more tinkering, I came up with the following revised
>>>>> proposal:
>>>>> http://www.pasteall.org/pic/12084
>>>>>
>>>>> This takes some of your (Remos) concerns into account.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Tobi
>>>>>
>>>>> On 05/08/2011 10:09 PM, Remo Pini wrote:
>>>>>> Hm... I have never noticed the slight distortions to cause
>>>>>> issues with
>>>>>> the tracker... might be a question of the implementation, but I
>>>>>> don't
>>>>>> see, why the material would have to be converted twice, even IF
>>>>>> "undistorting" is required...
>>>>>>
>>>>>> (1) take original footage
>>>>>> (2) undistort to track (temporary copy)
>>>>>> (3) track
>>>>>> (4) distort tracking data according to undistort
>>>>>> (5) use original footage + track data + lensinfo from source (or
>>>>>> undistort function)
>>>>>>
>>>>>> Syntheyes seems to do that under the hood, I guess, never seen
>>>>>> tracking
>>>>>> issues due to distortions...
>>>>>>
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: bf-vfx-bounces at blender.org [mailto:bf-vfx-bounces at blender.org
>>>>>>> ]
>>>>>>> On Behalf Of Tom M
>>>>>>> Sent: Sonntag, 8. Mai 2011 8:20
>>>>>>> To: Blender motion tracking& VFX
>>>>>>> Subject: Re: [Bf-vfx] Blender camera tracking approach
>>>>>>>
>>>>>>> On Sun, May 8, 2011 at 10:06 AM, Remo Pini<remo.pini at avexys.com>
>>>>>>> wrote:
>>>>>>>> I kind of disagree...
>>>>>>>>
>>>>>>>> If you have lens distortion in you footage, this would most
>>>>>>>> likely
>>>>>> be
>>>>>>>> bad and you would not want the final result to have that in it
>>>>>> anymore,
>>>>>>>> sooo....
>>>>>>> You are thinking extreme distortion - this is about the very
>>>>>>> slight
>>>>>>> distortion that comes from all camera lenses due to their
>>>>>>> lense shape
>>>>>>> (as well as lense imperfections - but the imperfections are
>>>>>>> mostly
>>>>>>> ignored by most lense solvers).
>>>>>>>
>>>>>>> Our brains ignore the distortion but it confuses trackers
>>>>>>> since travel
>>>>>>> in a straight line isn't straight with the distortion.
>>>>>>>
>>>>>>> Undistorting the lense reduces the image quality because it
>>>>>>> shifts
>>>>>>> pixel locations and thus 'blurs' pixels that the undistorted
>>>>>>> location
>>>>>>> of a pixel will not be exactly on the pixel boundary of a
>>>>>>> previous
>>>>>>> pixel.
>>>>>>>
>>>>>>> Thus undistortion is only used to provide straighter lines to
>>>>>>> the
>>>>>> tracker.
>>>>>>> Applying the lens distortion to your rendered image is so that
>>>>>>> it's
>>>>>>> straight lines are curved similarly to that of the original
>>>>>>> image.
>>>>>>> Again this introduces blurring of your render if it is applied
>>>>>>> as a
>>>>>>> post process - so either you need to render larger or do the
>>>>>>> distortion at render time. In practice the amount of bluring
>>>>>>> from
>>>>>>> doing it as a post process is probably not significant enough
>>>>>>> to be
>>>>>>> noticable in most situations.
>>>>>>>
>>>>>>> LetterRip
>>>>>>> _______________________________________________
>>>>>>> Bf-vfx mailing list
>>>>>>> Bf-vfx at blender.org
>>>>>>> http://lists.blender.org/mailman/listinfo/bf-vfx
>>>>>> _______________________________________________
>>>>>> Bf-vfx mailing list
>>>>>> Bf-vfx at blender.org
>>>>>> http://lists.blender.org/mailman/listinfo/bf-vfx
>>>>>>
>>>>> _______________________________________________
>>>>> Bf-vfx mailing list
>>>>> Bf-vfx at blender.org
>>>>> http://lists.blender.org/mailman/listinfo/bf-vfx
>>>>>
>>>> _______________________________________________
>>>> Bf-vfx mailing list
>>>> Bf-vfx at blender.org
>>>> http://lists.blender.org/mailman/listinfo/bf-vfx
>>>
>>> _______________________________________________
>>> Bf-vfx mailing list
>>> Bf-vfx at blender.org
>>> http://lists.blender.org/mailman/listinfo/bf-vfx
>>
>>
>> _______________________________________________
>> Bf-vfx mailing list
>> Bf-vfx at blender.org
>> http://lists.blender.org/mailman/listinfo/bf-vfx
>
> _______________________________________________
> Bf-vfx mailing list
> Bf-vfx at blender.org
> http://lists.blender.org/mailman/listinfo/bf-vfx
More information about the Bf-vfx
mailing list