[Bf-vfx] Keying nodes

[Keir Mierle]

A couple months ago I spent some time surveying the literature on keying to
see what I could find. I found almost nothing. It seems there is little to
no published research in the area?  In most areas that I survey for
reconstruction, I can eventually point at 2 or 3 papers which are the state
of the art.

I have a bunch of ideas on how you could use modern computer vision
techniques to do nice keying, but it would be research rather than
implementing something we know works.

Has anyone seen papers describing good chroma keying techniques?

Keir

P.S. some random ideas:

- Use a mixture mode + EM to estimate P(key|pixel, pixels in the
neighhbourhood)
- Use MRF's or something similar for the above model
- Do k-means clustering on the pixels where the user can specify some
keying regions and non-keying regions, then the pixels are clustered by
some sort of distance metric. This can be smarter than a single central
point for the clusters; each cluster can be modeled as e.g. a gaussian
(determined by PCA of linear RGB pixels, etc)
- So on and so forth.

On Sun, Apr 15, 2012 at 9:50 AM, Bob Holcomb <bob_holcomb at hotmail.com>wrote:

> I think we definitely should look at improving the keying nodes, but I
> also think there is a bit of misunderstanding about what is already
> present.  This is largely my fault for not documenting them better-I'll
> update the wiki after this short novel.  Here's what we got:
>
> Difference Key:
> This takes the difference between a clean plate and another plate and
> determines at a per-pixel level if the two pixels are different within
> an RGB color space (with an adjustable tolerance and falloff).  This
> node works really well if your clean plate and keying plate are lined up
> perfectly, such as when using a tripod to shoot the image or using a
> motion control camera.
>
> This could be made better by allowing for positional tolerances in
> addition to account for small movements between the two shots.   This
> key requires the patch I submitted a couple of weeks ago to fix the
> sliders from being dependent upon one another, they should be independent.
>
> Luminance Key:
> This uses the luminance value (in YUV color space) and a high/low slider
> to determine if a pixels should be keyed or not.  This is useful for
> anytime the foreground is significantly brighter than the background.
> Such as  keying special effects like fire/explosions/fireworks/etc for
> comping over other scenes later.
>
> Distance Key:
> This keys pixels based on the 3 dimensional distance between colors in a
> 3D RGB color space.  This key works well when trying to single out a
> specific color in a background (not necessarily green).  This key
> requires the patch I submitted a couple of weeks ago to fix the sliders
> from being dependent upon one another, they should be independent.
>
> Color Key:
> This is pretty weak.  It keys based off the color in HSV color space.
> It allows for tolerances for each of the components (H,S, and V)
> independently.   It was more of an experiment than anything else and
> probably isn't useful.
>
> Channel Key:
> This is the (most well known) simple green minus red (or the max of the
> red/blue) channel keying.  The node can work in other color spaces and
> allows for any channel to be used to limit the other channels, but I
> think it has very limited uses outside of the RGB color space.  It does
> allow for arbitrary channels to be used (e.g.  red minus max of
> green/blue).  It has some high/low tolerance slider to vary the
> impact.   This is probably the most useful keying node.
>
> Chroma Key:
> This node was based on the method described in the book "Video
> Demistified."  It converts the image to YCbCr color space, then
> "rotates" the pixels color (in the CbCr plane) to align it with the
> keying color space.  If the angle between the color and the key (the key
> laying on the X axis in the rotated color space) is less than the angle
> tolerance, then the pixel is keyed.  This key node is useful when the
> background is unevenly lit, but is consistent in chroma.  This is
> probably the second most useful keying node.  This node usually looks
> crappy because it works in YCbCr color space, and most digital cameras
> compress the hell out of the CbCr components.  Separating these channels
> out and blurring them a little, then recombining them helps pull a
> better matte.
>
>
> None of these keying nodes do spill suppression, so there's a separate
> node for that.
>
> I would expect that none of these nodes will produce a good matte on
> their own.  You'll probably need to combine a garbage matte, keying
> nodes (more than one) and edge filtering nodes to get a good matte.
> Each shot is different and will probably require its own node setup.
>
> I think an advanced keyer such as the bayesian approach would be an
> awesome addition.
>
> cheers,
>
> Bob
>
>
>
> _______________________________________________
> Bf-vfx mailing list
> Bf-vfx at blender.org
> http://lists.blender.org/mailman/listinfo/bf-vfx
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.blender.org/pipermail/bf-vfx/attachments/20120416/8c1c0be9/attachment.htm