[Bf-vfx] Keying nodes

Mon Apr 16 21:35:19 CEST 2012

I might be wrong, but my understanding of the ultimatte technique is 
that they work in 3d RGB color space and instead of using a point for 
color distance keying, they use a 3d shape that envelopes sampled points 
(user selects inside/outside colors) to fine tune the keying.

If our single point system (distance keying node), in essence creates a 
sphere, then maybe we could do some kind of multipoint sample and 
generate a metaball kinda shape that is used for sampling.

Cheers,

Bob

On 4/16/2012 1:52 PM, Ton Roosendaal wrote:
> Hi Keir,
>
> I agree, computer vision and 3d reconstruction potentially gives superior and high quality mattes. I'm afraid we need to invent it ourselves :)
>
> Best keyer in market I saw is www.ultimatte.com, they do realtime greenscreen for tv etc. It even does glass :) I suspect they work with references (images without the keyed objects). Motion detection (not camera) + temporal filter is also interesting.
>
> -Ton-
>
> ------------------------------------------------------------------------
> Ton Roosendaal  Blender Foundation   ton at blender.org    www.blender.org
> Blender Institute   Entrepotdok 57A  1018AD Amsterdam   The Netherlands
>
> On 16 Apr, 2012, at 19:32, Keir Mierle wrote:
>
>> A couple months ago I spent some time surveying the literature on keying to see what I could find. I found almost nothing. It seems there is little to no published research in the area?  In most areas that I survey for reconstruction, I can eventually point at 2 or 3 papers which are the state of the art.
>>
>> I have a bunch of ideas on how you could use modern computer vision techniques to do nice keying, but it would be research rather than implementing something we know works.
>>
>> Has anyone seen papers describing good chroma keying techniques?
>>
>> Keir
>>
>> P.S. some random ideas:
>>
>> - Use a mixture mode + EM to estimate P(key|pixel, pixels in the neighhbourhood)
>> - Use MRF's or something similar for the above model
>> - Do k-means clustering on the pixels where the user can specify some keying regions and non-keying regions, then the pixels are clustered by some sort of distance metric. This can be smarter than a single central point for the clusters; each cluster can be modeled as e.g. a gaussian (determined by PCA of linear RGB pixels, etc)
>> - So on and so forth.
>>
>> On Sun, Apr 15, 2012 at 9:50 AM, Bob Holcomb<bob_holcomb at hotmail.com>  wrote:
>> I think we definitely should look at improving the keying nodes, but I
>> also think there is a bit of misunderstanding about what is already
>> present.  This is largely my fault for not documenting them better-I'll
>> update the wiki after this short novel.  Here's what we got:
>>
>> Difference Key:
>> This takes the difference between a clean plate and another plate and
>> determines at a per-pixel level if the two pixels are different within
>> an RGB color space (with an adjustable tolerance and falloff).  This
>> node works really well if your clean plate and keying plate are lined up
>> perfectly, such as when using a tripod to shoot the image or using a
>> motion control camera.
>>
>> This could be made better by allowing for positional tolerances in
>> addition to account for small movements between the two shots.   This
>> key requires the patch I submitted a couple of weeks ago to fix the
>> sliders from being dependent upon one another, they should be independent.
>>
>> Luminance Key:
>> This uses the luminance value (in YUV color space) and a high/low slider
>> to determine if a pixels should be keyed or not.  This is useful for
>> anytime the foreground is significantly brighter than the background.
>> Such as  keying special effects like fire/explosions/fireworks/etc for
>> comping over other scenes later.
>>
>> Distance Key:
>> This keys pixels based on the 3 dimensional distance between colors in a
>> 3D RGB color space.  This key works well when trying to single out a
>> specific color in a background (not necessarily green).  This key
>> requires the patch I submitted a couple of weeks ago to fix the sliders
>> from being dependent upon one another, they should be independent.
>>
>> Color Key:
>> This is pretty weak.  It keys based off the color in HSV color space.
>> It allows for tolerances for each of the components (H,S, and V)
>> independently.   It was more of an experiment than anything else and
>> probably isn't useful.
>>
>> Channel Key:
>> This is the (most well known) simple green minus red (or the max of the
>> red/blue) channel keying.  The node can work in other color spaces and
>> allows for any channel to be used to limit the other channels, but I
>> think it has very limited uses outside of the RGB color space.  It does
>> allow for arbitrary channels to be used (e.g.  red minus max of
>> green/blue).  It has some high/low tolerance slider to vary the
>> impact.   This is probably the most useful keying node.
>>
>> Chroma Key:
>> This node was based on the method described in the book "Video
>> Demistified."  It converts the image to YCbCr color space, then
>> "rotates" the pixels color (in the CbCr plane) to align it with the
>> keying color space.  If the angle between the color and the key (the key
>> laying on the X axis in the rotated color space) is less than the angle
>> tolerance, then the pixel is keyed.  This key node is useful when the
>> background is unevenly lit, but is consistent in chroma.  This is
>> probably the second most useful keying node.  This node usually looks
>> crappy because it works in YCbCr color space, and most digital cameras
>> compress the hell out of the CbCr components.  Separating these channels
>> out and blurring them a little, then recombining them helps pull a
>> better matte.
>>
>>
>> None of these keying nodes do spill suppression, so there's a separate
>> node for that.
>>
>> I would expect that none of these nodes will produce a good matte on
>> their own.  You'll probably need to combine a garbage matte, keying
>> nodes (more than one) and edge filtering nodes to get a good matte.
>> Each shot is different and will probably require its own node setup.
>>
>> I think an advanced keyer such as the bayesian approach would be an
>> awesome addition.
>>
>> cheers,
>>
>> Bob
>>
>>
>>
>> _______________________________________________
>> Bf-vfx mailing list
>> Bf-vfx at blender.org
>> http://lists.blender.org/mailman/listinfo/bf-vfx
>>
>> _______________________________________________
>> Bf-vfx mailing list
>> Bf-vfx at blender.org
>> http://lists.blender.org/mailman/listinfo/bf-vfx
> _______________________________________________
> Bf-vfx mailing list
> Bf-vfx at blender.org
> http://lists.blender.org/mailman/listinfo/bf-vfx
>
>