[Bf-vfx] Improving the 2D stabilization tool :: Problem with choice of pivot point

[Ichthyostega]

Hello Sean,

thanks for your review! I am happy you liked the overall approach
and the UI integration. (I've noted the minor point to rename
"zoom" to "scale" in the UI label)

On 11.06.2016 07:49, Sean Kennedy wrote:
> I finally got a chance to play around with the stabilizer rework today, and 
> while it looks really promising, I was having an issue where it wasn't 
> working correctly. I made a video to demonstrate.

> https://www.dropbox.com/s/wdzs97xdpjs37rq/stabilizer_problems.mp4?dl=0

> Let me know if maybe I'm doing something wrong. Hopefully it's user error. 
> 😊

No, not user error. It is a known shortcoming, I am well aware of that problem.

It is closely linked to the choice of "pivot point" for rotation/scale.
This is the rotation centre to base detection of rotation angle(s) on
and to use for compensative rotation.

Unfortunately this choice is tricky, and I am not aware of any simple yet
universally correct solution to work with any kind of footage. Of course
there are several conceivable options to deal with that problem.

For this initial version presented here for review, I just used the
geometric centre of blender's image window/cadre (not the centre
of the footage's image). The reason for this choice is a technical
one: blender does not include this pivot point into the data feed
passed on for stabilisation nodes. Thus, my choice is the only
way to make nodes and the clip editor behave the same, without
also having to change the API for this data feed and probably more.
I didn't want to get into any further extensions prior to review
and any follow-ups from that review process.


So what happens in your example?

(1) you use one point for translation stabilisation, which causes
    that point to be absolutely fixed within the frame.

(2) you use the other point for rotation, but due to the camera's
    movement in 3D space, i.e. the perspective, this other point
    still moves relative to the resulting image centre. When you
    switch on rot/scale stabilisation, my code attempts to keep
    the angle of the line from image centre to this second point
    constant and rotates accordingly to compensate. If you
    additionally switch on scale("zoom") stabilisation, it
    also attempts to keep the distance of this point from
    window centre constant.


What can be done here?

(a) pragmatic solution: use both points for translation stabilisation
    and both points for rotation as well. Chances are that the spurious
    rotation will be less (but I wouldn't expect it to go away entirely).
    You can then use the target rotation parameter to compensate, and
    set keyframes to animate this compensation. Urghs. Oh well....
    sometimes it also helps to use more tracking points on different
    parts of the image (probably best when all points are at roughly
    the same distance from the camera). But all of this is tweaking
    and heuristics. Not the clean solution we aspire.

(b) we could use the weight centre of all translation tracking points
    as pivot. In your case then the result should be perfect.
    Shortcomings:
    - nodes, would have to pass this calculated pivot position to
      the nodes, which means to change more public APIs
    - what to do when some tracking points have gaps or are not
      present over the full duration of the clip? This would cause
      the pivot and thus the result to jump suddenly. This again
      could be treated with animating the track weights, i.e
      by a fade-in of the problematic tracks
    - in many cases of camera movement, the weight centre of the
      translation points is just not correct and would induce
      additional spurious rotation.

(c) we could use the weight centre of all rotation tracking points.
    Basically the same argument as (b)
    Shortcomings
    - nodes, need to pass pivot position data
    - you'd need at least two points for rotation stabilisation,
      otherwise we wouldn't be able to detect any angle data
    - again, would not be correct for a lot of conceivable
      camera movements.

(d) let the user define the pivot point explicitly. Including
    the ability to animate that point.
    Shortcomings
    - nodes, need to pass pivot position data
    - is is manual, might be confusing, needs additional UI

(e) solve for the pivot point which causes the least rotation.
    Shortcomings
    - nodes, need to pass pivot position data
    - this is advanced stuff and probably needs some more
      considerations to get it correct and performant
    - it is not clear what range of frames (i.e. temporal scope)
      to use for this solving process. Whatever you choose here,
      it will for sure worsen the result in some cases
    - so, if we give the user this parameter for tweaking, we
      add yet more complexity. Also we'd need a second parameter:
      the time interval after which this whole solving has to
      be re-done.

(f) do a full 3D reconstruction, then apply a physical model
    on the reconstructed camera trajectory
    Shortcomings
    - requires a lot more tracking points to work. This is
      prohibitive: in most takes I did for my own work meanwhile,
      I wouldn't be able to get sufficient coverage for this.
    - requires strong lateral (parallax) movements in the footage
      to work. Thus it breaks down on "almost perfect" takes
      with just minor residual movement, and it breaks down
      on "2d texture takes"

My personal vote goes towards (d), i.e. that is what I intend
to implement after successful review and after resolving that
problem with the data feed for the nodes. Of course, also (b)
and (c) are rather low hanging fruit, so I'd offer these
as choice.

Cheers,
Hermann V.
(aka. Ichthyo)