[Bf-vfx] Blender camera tracking approach

Sun May 8 23:26:45 CEST 2011

Hey Tom,

What you described happens in the "filtering/masking" and "tracker" 
stages of my chart. I wanted to outline an approach avoiding distortion 
the rendered CG plates on a post-pro image level, which might lead to 
inferior quality (depending on the amount of distortion), since that was 
a topic on IRC today. I think this post-pro approach isn't avoidable for 
a tracker-only package, but since blender has 3D capabilities we should 
use them and go for a more complete integration instead of a merely 
"tracking-only" approach (therefore the "set camera properties" stage). 
I didn't want to describe the inner workings of the masking/filtering 
and tracking stages in detail, but I can include them into a more 
complete flowchart if you want.

Greets,
Tobi

On 05/08/2011 11:05 PM, Tom M wrote:
> Tobias,
>
> here is a 'complete' workflow for tracking, it might be good to
> provide details for some of your flow areas off to the side.
>
> 1) load image data - video or sequence of images
> 2a) (optional) correct rolling shutter
> 2b) (optional) input known camera parameters
> 2c) (optional) provide hints to camera lens undistortion solver
> (identify straight lines in the image) to help with undistorting the
> image
> 2d) (optional) undistort the camera lens using a distortion solver
> 3) (optional) create mask for data you don't want tracked (or for
> 'grouping'/seperating of tracking ie tracking the motion of a camera
> and tracking the motion of an object)
> 4) (optional) adjust color/contrast/etc so that feature points to
> track have increased contrast and are thus easier for the tracker to
> find them (and easier for the tracker to ignore stuff you don't want
> tracked - ie uneven lighting on a green screen creating spurious
> motion)
> 5) (optional) select specific features you want tracked (either by
> mask or placement of tracking markers)
> 5a) if you specified specific features to the tracker you may also
> want to specify how the points should be tracked (bounding box for the
> tracker to look for the point in; whether the object being tracked is
> 'rigid' or can deform, and knowledge of the types of camera/object
> motion - object translation; rotation; camera translation; rotation)
> 6) send image data, specified track data; camera data; and mask data to tracker
> 7) tracker library can automatically identify feature points to track
> and or use user specified features, ignoring areas that are masked
> 8) do a 'solve' of the camera motion based on the track points,
> including statistics on how 'good' the tracks are in contributing to
> the solve
> 9) return track point data and camera solve to software, including the
> statistical analysis of the track points and their 3d projection
> 10) based on the statistical analysis pick error thresholds for what
> track points to automatically delete (automatic threshold picking or
> manually adjusted)
> 11) (optional) manually delete any track points
> 12) (optional) create a mask to 'hide' unwanted track points
> 12a) (optional) mask can be assigned to follow a set of track points
> to automatically mask a moving object from the tracker/solver
> 12b) (optional) mask can be manually keyframed so that it moves and
> deforms over time to mask a moving object from the tracker/solver
> 12) (optional) provide a manually created camera curve to 'hint' the
> tracker/solver what you expect the actual curve to look like
> 13) retrack if additional tracker points are now needed
> 14) pass the tracker points, camera hints, etc. to the camera solver
> 15) return the solved camera track and the 3d location of the track
> points to the software
> 16) visualize the camera motion and track points in the 3d view
> 17) define a ground plane reference to the 3d track points and camera
> 18) define the scene origin relative to the 3d track points and camera
> 19) define the world scale and orientation relative to the 3d track
> points and data
> 20) add a test object into the 3d view and see if it stays in the
> proper location
> 21) (optional) stabalize the camera view based on the solved camera track
> 22) (optional) smooth the camera track curve
> 23) repeat until the error in the solve is good enough
>
> On Sun, May 8, 2011 at 12:33 PM, Tobias Kummer<supertoilet at gmx.net>  wrote:
>> After some more tinkering, I came up with the following revised proposal:
>> http://www.pasteall.org/pic/12084
>>
>> This takes some of your (Remos) concerns into account.
>>
>> Regards,
>>
>> Tobi
>>
>> On 05/08/2011 10:09 PM, Remo Pini wrote:
>>> Hm... I have never noticed the slight distortions to cause issues with
>>> the tracker... might be a question of the implementation, but I don't
>>> see, why the material would have to be converted twice, even IF
>>> "undistorting" is required...
>>>
>>> (1) take original footage
>>> (2) undistort to track (temporary copy)
>>> (3) track
>>> (4) distort tracking data according to undistort
>>> (5) use original footage + track data + lensinfo from source (or
>>> undistort function)
>>>
>>> Syntheyes seems to do that under the hood, I guess, never seen tracking
>>> issues due to distortions...
>>>
>>>
>>>> -----Original Message-----
>>>> From: bf-vfx-bounces at blender.org [mailto:bf-vfx-bounces at blender.org]
>>>> On Behalf Of Tom M
>>>> Sent: Sonntag, 8. Mai 2011 8:20
>>>> To: Blender motion tracking&    VFX
>>>> Subject: Re: [Bf-vfx] Blender camera tracking approach
>>>>
>>>> On Sun, May 8, 2011 at 10:06 AM, Remo Pini<remo.pini at avexys.com>
>>>> wrote:
>>>>> I kind of disagree...
>>>>>
>>>>> If you have lens distortion in you footage, this would most likely
>>> be
>>>>> bad and you would not want the final result to have that in it
>>> anymore,
>>>>> sooo....
>>>> You are thinking extreme distortion - this is about the very slight
>>>> distortion that comes from all camera lenses due to their lense shape
>>>> (as well as lense imperfections - but the imperfections are mostly
>>>> ignored by most lense solvers).
>>>>
>>>> Our brains ignore the distortion but it confuses trackers since travel
>>>> in a straight line isn't straight with the distortion.
>>>>
>>>> Undistorting the lense reduces the image quality because it shifts
>>>> pixel locations and thus 'blurs' pixels that the undistorted location
>>>> of a pixel will not be exactly on the pixel boundary of a previous
>>>> pixel.
>>>>
>>>> Thus undistortion is only used to provide straighter lines to the
>>> tracker.
>>>> Applying the lens distortion to your rendered image is so that it's
>>>> straight lines are curved similarly to that of the original image.
>>>> Again this introduces blurring of your render if it is applied as a
>>>> post process - so either you need to render larger or do the
>>>> distortion at render time.  In practice the amount of bluring from
>>>> doing it as a post process is probably not significant enough to be
>>>> noticable in most situations.
>>>>
>>>> LetterRip
>>>> _______________________________________________
>>>> Bf-vfx mailing list
>>>> Bf-vfx at blender.org
>>>> http://lists.blender.org/mailman/listinfo/bf-vfx
>>> _______________________________________________
>>> Bf-vfx mailing list
>>> Bf-vfx at blender.org
>>> http://lists.blender.org/mailman/listinfo/bf-vfx
>>>
>> _______________________________________________
>> Bf-vfx mailing list
>> Bf-vfx at blender.org
>> http://lists.blender.org/mailman/listinfo/bf-vfx
>>
> _______________________________________________
> Bf-vfx mailing list
> Bf-vfx at blender.org
> http://lists.blender.org/mailman/listinfo/bf-vfx
>