A couple months ago I spent some time surveying the literature on keying to see what I could find. I found almost nothing. It seems there is little to no published research in the area? In most areas that I survey for reconstruction, I can eventually point at 2 or 3 papers which are the state of the art.<div>
<br></div><div>I have a bunch of ideas on how you could use modern computer vision techniques to do nice keying, but it would be research rather than implementing something we know works.<div>
<br>Has anyone seen papers describing good chroma keying techniques?</div><div><br>Keir</div><div><br></div><div>P.S. some random ideas:</div><div><br></div><div>- Use a mixture mode + EM to estimate P(key|pixel, pixels in the neighhbourhood)</div>
<div>- Use MRF's or something similar for the above model</div><div>- Do k-means clustering on the pixels where the user can specify some keying regions and non-keying regions, then the pixels are clustered by some sort of distance metric. This can be smarter than a single central point for the clusters; each cluster can be modeled as e.g. a gaussian (determined by PCA of linear RGB pixels, etc)</div>
<div>- So on and so forth.</div>
<div><br><div class="gmail_quote">On Sun, Apr 15, 2012 at 9:50 AM, Bob Holcomb <span dir="ltr"><<a href="mailto:bob_holcomb@hotmail.com" target="_blank">bob_holcomb@hotmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I think we definitely should look at improving the keying nodes, but I<br>
also think there is a bit of misunderstanding about what is already<br>
present. This is largely my fault for not documenting them better-I'll<br>
update the wiki after this short novel. Here's what we got:<br>
<br>
Difference Key:<br>
This takes the difference between a clean plate and another plate and<br>
determines at a per-pixel level if the two pixels are different within<br>
an RGB color space (with an adjustable tolerance and falloff). This<br>
node works really well if your clean plate and keying plate are lined up<br>
perfectly, such as when using a tripod to shoot the image or using a<br>
motion control camera.<br>
<br>
This could be made better by allowing for positional tolerances in<br>
addition to account for small movements between the two shots. This<br>
key requires the patch I submitted a couple of weeks ago to fix the<br>
sliders from being dependent upon one another, they should be independent.<br>
<br>
Luminance Key:<br>
This uses the luminance value (in YUV color space) and a high/low slider<br>
to determine if a pixels should be keyed or not. This is useful for<br>
anytime the foreground is significantly brighter than the background.<br>
Such as keying special effects like fire/explosions/fireworks/etc for<br>
comping over other scenes later.<br>
<br>
Distance Key:<br>
This keys pixels based on the 3 dimensional distance between colors in a<br>
3D RGB color space. This key works well when trying to single out a<br>
specific color in a background (not necessarily green). This key<br>
requires the patch I submitted a couple of weeks ago to fix the sliders<br>
from being dependent upon one another, they should be independent.<br>
<br>
Color Key:<br>
This is pretty weak. It keys based off the color in HSV color space.<br>
It allows for tolerances for each of the components (H,S, and V)<br>
independently. It was more of an experiment than anything else and<br>
probably isn't useful.<br>
<br>
Channel Key:<br>
This is the (most well known) simple green minus red (or the max of the<br>
red/blue) channel keying. The node can work in other color spaces and<br>
allows for any channel to be used to limit the other channels, but I<br>
think it has very limited uses outside of the RGB color space. It does<br>
allow for arbitrary channels to be used (e.g. red minus max of<br>
green/blue). It has some high/low tolerance slider to vary the<br>
impact. This is probably the most useful keying node.<br>
<br>
Chroma Key:<br>
This node was based on the method described in the book "Video<br>
Demistified." It converts the image to YCbCr color space, then<br>
"rotates" the pixels color (in the CbCr plane) to align it with the<br>
keying color space. If the angle between the color and the key (the key<br>
laying on the X axis in the rotated color space) is less than the angle<br>
tolerance, then the pixel is keyed. This key node is useful when the<br>
background is unevenly lit, but is consistent in chroma. This is<br>
probably the second most useful keying node. This node usually looks<br>
crappy because it works in YCbCr color space, and most digital cameras<br>
compress the hell out of the CbCr components. Separating these channels<br>
out and blurring them a little, then recombining them helps pull a<br>
better matte.<br>
<br>
<br>
None of these keying nodes do spill suppression, so there's a separate<br>
node for that.<br>
<br>
I would expect that none of these nodes will produce a good matte on<br>
their own. You'll probably need to combine a garbage matte, keying<br>
nodes (more than one) and edge filtering nodes to get a good matte.<br>
Each shot is different and will probably require its own node setup.<br>
<br>
I think an advanced keyer such as the bayesian approach would be an<br>
awesome addition.<br>
<br>
cheers,<br>
<br>
Bob<br>
<br>
<br>
<br>
_______________________________________________<br>
Bf-vfx mailing list<br>
<a href="mailto:Bf-vfx@blender.org" target="_blank">Bf-vfx@blender.org</a><br>
<a href="http://lists.blender.org/mailman/listinfo/bf-vfx" target="_blank">http://lists.blender.org/mailman/listinfo/bf-vfx</a><br>
</blockquote></div><br></div>
</div>