[Bf-vfx] Keying nodes

[Ton Roosendaal]

Hi Keir,

I agree, computer vision and 3d reconstruction potentially gives superior and high quality mattes. I'm afraid we need to invent it ourselves :)

Best keyer in market I saw is www.ultimatte.com, they do realtime greenscreen for tv etc. It even does glass :) I suspect they work with references (images without the keyed objects). Motion detection (not camera) + temporal filter is also interesting.

-Ton-

------------------------------------------------------------------------
Ton Roosendaal  Blender Foundation   ton at blender.org    www.blender.org
Blender Institute   Entrepotdok 57A  1018AD Amsterdam   The Netherlands

On 16 Apr, 2012, at 19:32, Keir Mierle wrote:

> A couple months ago I spent some time surveying the literature on keying to see what I could find. I found almost nothing. It seems there is little to no published research in the area?  In most areas that I survey for reconstruction, I can eventually point at 2 or 3 papers which are the state of the art.
> 
> I have a bunch of ideas on how you could use modern computer vision techniques to do nice keying, but it would be research rather than implementing something we know works.
> 
> Has anyone seen papers describing good chroma keying techniques?
> 
> Keir
> 
> P.S. some random ideas:
> 
> - Use a mixture mode + EM to estimate P(key|pixel, pixels in the neighhbourhood)
> - Use MRF's or something similar for the above model
> - Do k-means clustering on the pixels where the user can specify some keying regions and non-keying regions, then the pixels are clustered by some sort of distance metric. This can be smarter than a single central point for the clusters; each cluster can be modeled as e.g. a gaussian (determined by PCA of linear RGB pixels, etc)
> - So on and so forth.
> 
> On Sun, Apr 15, 2012 at 9:50 AM, Bob Holcomb <bob_holcomb at hotmail.com> wrote:
> I think we definitely should look at improving the keying nodes, but I
> also think there is a bit of misunderstanding about what is already
> present.  This is largely my fault for not documenting them better-I'll
> update the wiki after this short novel.  Here's what we got:
> 
> Difference Key:
> This takes the difference between a clean plate and another plate and
> determines at a per-pixel level if the two pixels are different within
> an RGB color space (with an adjustable tolerance and falloff).  This
> node works really well if your clean plate and keying plate are lined up
> perfectly, such as when using a tripod to shoot the image or using a
> motion control camera.
> 
> This could be made better by allowing for positional tolerances in
> addition to account for small movements between the two shots.   This
> key requires the patch I submitted a couple of weeks ago to fix the
> sliders from being dependent upon one another, they should be independent.
> 
> Luminance Key:
> This uses the luminance value (in YUV color space) and a high/low slider
> to determine if a pixels should be keyed or not.  This is useful for
> anytime the foreground is significantly brighter than the background.
> Such as  keying special effects like fire/explosions/fireworks/etc for
> comping over other scenes later.
> 
> Distance Key:
> This keys pixels based on the 3 dimensional distance between colors in a
> 3D RGB color space.  This key works well when trying to single out a
> specific color in a background (not necessarily green).  This key
> requires the patch I submitted a couple of weeks ago to fix the sliders
> from being dependent upon one another, they should be independent.
> 
> Color Key:
> This is pretty weak.  It keys based off the color in HSV color space.
> It allows for tolerances for each of the components (H,S, and V)
> independently.   It was more of an experiment than anything else and
> probably isn't useful.
> 
> Channel Key:
> This is the (most well known) simple green minus red (or the max of the
> red/blue) channel keying.  The node can work in other color spaces and
> allows for any channel to be used to limit the other channels, but I
> think it has very limited uses outside of the RGB color space.  It does
> allow for arbitrary channels to be used (e.g.  red minus max of
> green/blue).  It has some high/low tolerance slider to vary the
> impact.   This is probably the most useful keying node.
> 
> Chroma Key:
> This node was based on the method described in the book "Video
> Demistified."  It converts the image to YCbCr color space, then
> "rotates" the pixels color (in the CbCr plane) to align it with the
> keying color space.  If the angle between the color and the key (the key
> laying on the X axis in the rotated color space) is less than the angle
> tolerance, then the pixel is keyed.  This key node is useful when the
> background is unevenly lit, but is consistent in chroma.  This is
> probably the second most useful keying node.  This node usually looks
> crappy because it works in YCbCr color space, and most digital cameras
> compress the hell out of the CbCr components.  Separating these channels
> out and blurring them a little, then recombining them helps pull a
> better matte.
> 
> 
> None of these keying nodes do spill suppression, so there's a separate
> node for that.
> 
> I would expect that none of these nodes will produce a good matte on
> their own.  You'll probably need to combine a garbage matte, keying
> nodes (more than one) and edge filtering nodes to get a good matte.
> Each shot is different and will probably require its own node setup.
> 
> I think an advanced keyer such as the bayesian approach would be an
> awesome addition.
> 
> cheers,
> 
> Bob
> 
> 
> 
> _______________________________________________
> Bf-vfx mailing list
> Bf-vfx at blender.org
> http://lists.blender.org/mailman/listinfo/bf-vfx
> 
> _______________________________________________
> Bf-vfx mailing list
> Bf-vfx at blender.org
> http://lists.blender.org/mailman/listinfo/bf-vfx