[Bf-funboard] New compositing and VSE monitor tools

[Troy Sobotka]

On Sat, Feb 20, 2016 at 1:06 AM Knapp <magick.crow at gmail.com> wrote:

> OK, so then working in the compositor and doing color grading in the
> compositor is a valid concept that will work. This leaves the VSE good for
> high speed scrubbing.
>

You have just stumbled into precisely the reason that an offline production
pipeline exists; there is simply too much stuff moving around to cram all
of that vital image science and complexity into an NLE.

Full stop.

So what is the option? Very close to what the current implementation of the
VSE is currently, which is basically a method to arrive at an editorial
endpoint (based on offlines, aka proxies) and commence the heavy lifting in
a non-realtime environment.


> This also means that having scopes as output nodes is a valid idea.
>

The scopes in the UV Image Editor were relatively robust, but have hard
coded elements which make them somewhat broken in the modern day version of
Blender with OCIO. Having scopes / waveforms / etc. crammed into a sidebar
is far from optimal, of course.


> Also a VSE channel input node in the compositor might be of some value.
> Correct?
>

I'm unsure that the solution is optimal, and I'm not entirely sure how best
to solve this one, although some ideas have rooted over the past decade.

I'd personally prefer _not_ to see folks mixing and matching phases, flip
flopping in the pipeline, back and forth. Rather, I'd like to see a clear
path from editorial, to post production CGI / Vfx, to grading / finishing
be plausible. I believe that something close to what the VSE is today is a
critical part of that; a Temporal Viewer if you will.

Grading comes with a unique set of challenges, not the least of which is
per shot grades. This means that the temporal placement of the shots needs
to be maintained for something like a dissolve from A to B. Shot A requires
grading as does shot B, completely independent of one-another. Somewhat
obvious perhaps, but the temporal view of independent shots is rather
important in contrast to a strictly node based non-temporal view.

Would a strip to node alleviate that complexity? I don't think so. Nodes
_are_ the solution, but having 1000 shots would add 1000 nodes. It would
seem that more granularity is required in each node view rather like our
existing Material nodes, for example.

We also have to remember that grading is an _extremely complex_ operation
that covers all aspects of looks. This could be heavily processor based,
using tracking, mattes or restricted regions, and many other things
including even Z depth etc. Now couple that with the understanding that you
are frequently operating on scene referred data (zero to infinity) of both
footage and CGI elements, and you can begin to see the power needs. Again,
a per-shot Material node-like view makes sense here, with a Temporal Strip
View being the navigation mechanism.


> I would love to know more about all your whys at the end!
>

> Why not use a linearized reference space in the VSE then? Why not use
> linearized reference spaces everywhere?

If we think about what an NLE should do, we can probably peel apart _how_
it might do it.

Here are a few points that we can (hopefully) agree upon at this point:
 * If we are performing manipulations, we _must_ use a linearized reference
space. This includes blending, compositing, blurring, mixing, etc.
 * Grading is complex, requiring massive horsepower.
 * CGI animation / visual effects are complex, requiring massive horsepower.

All three of these points, as trivial as they seem, challenge the notion of
the KitchenSink™ NLE application. If you care about the above, you can't
have everything moving along realtime. Nor do you want it to.

Linearized data, especially scene linear zero to infinity data, requires a
bare minimum of half float data. That means that all of the incoming
imagery needs to be either already at float or half float, or the codec
requires promoting an eight bit buffer to a half float representation. The
View on the reference then converts that not-quite-viewable-directly data
into useful display referred output. Much like what already exists in the
UV / Image Viewer chain for nodes.

Immediately we can see though, that we effectively double our memory
requirement, and likely significantly increase our draw on processing,
taxing our concepts of realtime. Now load 600 of those in a simple short
project[1].

The essence of an editorial system is to deliver editorial, which means
rapid changes and realtime display of the things that matter to editorial:
pacing, beats, duration, timing, audio cues, etc. All of those realtime
facets cannot be negotiated with the extremely granular needs of CGI,
visual effects, grading, etc. Worse, Blinn's Law seems in full effect. Deep
compositing might be a good example here.

The TL;DR is that there _is_ a path forward for the VSE, or rather, its
next generation iteration. There _is_ a need. It _can_ be solved for
imagers within not only a Blender paradigm, but also one that works beyond
Blender for mixing and matching tools.

We just need to sort out the noise from the design goals, and focus on
those very specific needs in relation to Blender as it exists now. Blender
imagers have the advantage of having seen some of the complexity involved
across the may facets of imaging, and as such, can likely more easily
understand statements like "You can't do that in an NLE." This makes a
typical Blender imager much more likely to be able to agree to and
understand a clear design path forward without the ridiculous noise that
typically surrounds NLE discussions.

With respect,
TJS

[1] I'd actually insist we upgrade the next strip viewer's implementation
to half float representation. We could then get a much closer idea as to
what an editorial temporary look might be in terms of a rough grade or
blending. Again, even though it would still be a proxy, having half float
makes the guesswork a little closer, allowing us to reuse our existing
curves and such 1:1 on the full resolution, full bit depth sources. Cycles
too has a need for a library such as this, as well as painting, so it is an
effort that would span across many areas.