[Bf-compositor] Image sampling issue

Tue Nov 12 12:55:06 CET 2013

The basic problem i see is the difference between upsampling vs.
downsampling (essentially scaling input image by >1 vs <1). The sampling
methods used by default like NEAREST/BILINEAR/BICUBIC as well as the new
pre-filter method implemented by Troy all work very well for the upsampling
case (interpolating between image pixels for screen space), but they are
not good for downsampling (accumulating multiple input texels for one
screen space pixel), especially when the input image is deformed
anisotropically which is generally the case for UVMap, Displace and
PlaneTrackWarp nodes.

The EWA sampling does a pretty good job for downsampling now. There may be
other algorithms for downsampling, but i'm quite happy with the results so
far.

Biggest challenge is to provide useful partial derivatives for the UV
coordinates. In the case of PlaneTrack this is straightforward, since the
UV coordinates are generated from the corners (could use a 3D transform
later, which also gives direct 2D derivatives). For UVMap and Displace
nodes, however, the jacobian has to be estimated by doing multiple
evaluations and getting a 1st order estimate. This sort of works most of
the time, but breaks down with more extreme scaling and is not accurate at
discontinuities (UV map edges for example). An ideal solution would be to
use "dual" values in the compositor where needed, i.e. provide not only the
UV vector but also its derivatives as an input value (this concept is used
heavily in OSL for all data types). The derivatives could then be
calculated based on the actual image data, which is as accurate as it gets.

Here is a preliminary patch, in 2 parts:

1) Cleanup patch, renaming the different executePixel variants to make it
easier to distinguish them (my IDE can not tell the different signatures
apart ...). Not much to check here if you agree to the basic idea, just
needed for compiling.
http://www.pasteall.org/47200/diff

2) Actual EWA improvement patch. This is implemented only for PlaneTrack
and UVMap nodes atm, Displace node is still disabled.
http://www.pasteall.org/47201/diff

On Mon, Nov 11, 2013 at 9:50 PM, Jeroen Bakker <j.bakker at atmind.nl> wrote:

>  Hi Lukas,
>
> I will be on-line tomorrow for a short discussion. Instead of EWA
> filtering we should also investigate other filter-types. As long as it can
> be switched by a parameter I think it should be oke to implement. Currently
> the read_pixel with dx and dy only selects EWA, but has a pixelsampler
> parameter that can be inspected.
>
> Jeroen.
>
> On 11/08/2013 02:02 PM, Lukas Tönne wrote
>
> I've made some progress with the EWA (elliptical weighted average) filter,
> which is one of the main grievances with tracking+compositor according to
> Sebastian.
>
>  http://www.pasteall.org/pic/show.php?id=62224
>  http://www.pasteall.org/pic/show.php?id=62225
>
>  The Plane Track Warp operation now calculates actual partial derivatives
> from the plane corners (via extended UV resolve function) instead of trying
> to estimate them from insufficient data. This allows the EWA filtering to
> use accurate elliptical regions in the image space for filtering. Such
> derivatives can be calculated in different ways as pointed out in the
> previous mail, the important thing is that they are actually available.
>
>  The code is not quite ready yet, i'll put it up for review soon. There
> are still some details i need to look into, like optimal filter sizes, some
> optimizations and safeguards that can be implemented and just a bunch of
> debug code i need to clean up.
> @Jeroen: please poke me when you are on IRC, i'd like to discuss this
> whole sampling issue a bit to get a broader picture.
>
>
> On Sun, Nov 3, 2013 at 6:31 PM, Lukas Tönne <lukas.toenne at gmail.com>wrote:
>
>>  @Jeroen: I've been looking into the EWA sampling. Here are some
>> thoughts, please tell me if this would make any sense, just to see if i
>> understand you correctly:
>>
>>  1) Give each operation an explicit coordinate transform function which,
>> given x/y coordinates, calculates both the transformed coordinates u/v of
>> the source image(s) as well as the partial derivatives du/dx, du/dy, dv/dx,
>> dv/dy (jacobian matrix)
>>
>>  2) For most nodes this transformation is linear, i.e. the derivatives
>> are straightforward. A "LinearTransformOperation" base class or so could
>> simplify it then by keeping a transform matrix. Only a few nodes such as
>> UVMap (?) node currently do non-linear transforms afaik. PlaneTrack could
>> perhaps use a full 3D transform.
>>
>>  3) Remove integrated pixel transform calculations from executePixel
>> functions and let the SocketReader handle this by passing x/y through the
>> transform functions and giving source-space coordinates to the input node.
>> Nodes such as Translate or PlaneTrack would not do any actual color
>> manipulation but only coordinate transforms.
>>
>>  4) For canvas compositing (later): Add a transform matrix to bNode
>> types that support it, with all the necessary UI bells and whistles for
>> manipulating these in the editor. Then use these matrices to set up the
>> associated operations.
>>
>>
>> On Sun, Nov 3, 2013 at 1:09 PM, Francesco Paglia <f.paglia.80 at gmail.com>wrote:
>>
>>> Hi Jeroen,
>>> as I can see every node that involves pixel transformation gives a much
>>> more blurred image.
>>> the example in the link clearly shows what I'm talking about.
>>>
>>>  http://www.pasteall.org/pic/show.php?id=61835
>>>
>>>   there's an image input rotated twice in both direction to get back to
>>> the original position then I compared the resulting image with the original
>>> one using a difference operand.
>>> The image "should" result in a completely black image but as you can see
>>> there are difference bigger than a pixel using the bicubic filter.
>>>
>>>   This result shows up 2 consideration:
>>> 1. a sub-pixel transformation should be taken into account for all the
>>> distortion nodes (maybe matrices should be the right choice?)
>>> 2. there's no optimization for two nodes whom do the same operation the
>>> so called "concatenation" in other software.
>>>
>>>  To further explain concatenation is an evaluation of all the node that
>>> are of the same kind and that are linked one to each other prior to apply
>>> the final transformation to the image.
>>> here a reference link:
>>>
>>> http://www.nukepedia.com/written-tutorials/concatenation-of-transforms-in-nuke/
>>>
>>>
>>>
>>>
>>>
>>> 2013/11/3 Sebastian König <koenig.sebastian at gmx.net>
>>>
>>>>  Hi all!
>>>>
>>>>  I have created a new wiki page:
>>>> http://wiki.blender.org/index.php/Dev:Ref/Proposals/Compositor
>>>> I will collect the proposals and requests there, and also try to merge
>>>> it with requests and proposals from here:
>>>> http://wiki.blender.org/index.php/Dev:Ref/Requests/Compositing2.6
>>>>
>>>>  I will also start threads in this list mailing list, as Francesco
>>>> suggested, so that we can discuss things more clearly and structured.
>>>> What I would like to see at some point is some kind of priority list.
>>>> Everyone has his main issues and problems with the compositor and also
>>>> probably a pretty clear vision what should be changed and how. Would be
>>>> cool if we can get agreement on the most important issues.
>>>>
>>>>  Cheers!
>>>>
>>>>  Seb
>>>>
>>>>
>>>>
>>>> On 3. November 2013 at 09:21:30, Jeroen Bakker (j.bakker at atmind.nl<//j.bakker at atmind.nl>)
>>>> wrote:
>>>>
>>>>   Hi!
>>>>
>>>> Just to clarify the issue, the quick solution and an draft-idea of the
>>>> possible solution:
>>>>
>>>> Issue: the compositor can not handle sub-sampling with good quality by
>>>> default. Nodes that want to use it should currently implement a style like
>>>> UV mapping (EWA sampling). To use EWA sampling the pixel size needs to be
>>>> evaluated (DX, DY).
>>>>
>>>> Quick solution: let the plane tracker node calculate the pixel-size and
>>>> EWA filtering methods. This solution will work for most cases, but is not
>>>> the ideal solution. It would be better that the pixel size is known by all
>>>> nodes.
>>>>
>>>> Possible solution (draft idea): In stead of evaluating pixels with just
>>>> an X, Y coordinate or even X, Y, DX, DY. We should investigate on using 3x3
>>>> matrices. This solution should also help to implement canvas compositor,
>>>> but it could lead to strange results as matrices are limited by the
>>>> optimization structure (Complex vs Simple nodes) of the compositor.
>>>>
>>>> Jeroen
>>>>
>>>> On 11/02/2013 04:48 PM, Sebastian König wrote:
>>>>
>>>> Hi!
>>>> Here’s a file that illustrates the problem with sampling of plane
>>>> track: http://www.pasteall.org/blend/24971
>>>> It’s a UV grid that is being blurred quite badly, even though the plane
>>>> track has only straight lines.
>>>> I was talking to Jeroen and Sergey about that, and they said there are
>>>> issues with the entire sampling code in compositor, so it seems not to be a
>>>> local issue.
>>>> But I don’t know about the technical issues. It just looks crappy. :)
>>>>
>>>>  Cheers!
>>>>
>>>>  Seb
>>>>
>>>>
>>>>
>>>>
>>>>  --
>>>> Sebastian König
>>>> Schenkendorfstrasse 45
>>>> 04275 Leipzig
>>>> Germany
>>>> +49 176 20319318 <%2B49%20176%2020319318>
>>>> www.3dzentrale.com
>>>> koenig.sebastian at gmx.net
>>>>
>>>>
>>>> On 2. November 2013 at 13:20:56, Lukas Tönne (lukas.toenne at gmail.com<//lukas.toenne at gmail.com>)
>>>> wrote:
>>>>
>>>>  Sebastian: Could you create a few test files to demonstrate the
>>>> sampling issues?
>>>>
>>>>
>>>>  _______________________________________________
>>>> Bf-compositor mailing list
>>>> Bf-compositor at blender.org
>>>> http://lists.blender.org/mailman/listinfo/bf-compositor
>>>>
>>>>
>>>> _______________________________________________
>>>> Bf-compositor mailing list
>>>> Bf-compositor at blender.org
>>>> http://lists.blender.org/mailman/listinfo/bf-compositor
>>>>
>>>>
>>>
>>>
>>>   --
>>> Francesco Paglia
>>> Vfx and Production Supervisor
>>>
>>>  mobile  +39 347.82.12.473 <%2B39%20347.82.12.473>
>>>  e-mail   f.paglia.80 at gmail.com
>>>
>>>
>>> _______________________________________________
>>> Bf-compositor mailing list
>>> Bf-compositor at blender.org
>>> http://lists.blender.org/mailman/listinfo/bf-compositor
>>>
>>>
>>
>
>
> _______________________________________________
> Bf-compositor mailing listBf-compositor at blender.orghttp://lists.blender.org/mailman/listinfo/bf-compositor
>
>
>
> _______________________________________________
> Bf-compositor mailing list
> Bf-compositor at blender.org
> http://lists.blender.org/mailman/listinfo/bf-compositor
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.blender.org/pipermail/bf-compositor/attachments/20131112/a537f9c3/attachment-0001.htm 