[Bf-funboard] New compositing and VSE monitor tools

[Knapp]

On Sat, Feb 20, 2016 at 8:02 PM, Troy Sobotka <troy.sobotka at gmail.com>
wrote:

> On Sat, Feb 20, 2016 at 1:06 AM Knapp <magick.crow at gmail.com> wrote:
>
> > OK, so then working in the compositor and doing color grading in the
> > compositor is a valid concept that will work. This leaves the VSE good
> for
> > high speed scrubbing.
> >
>
> You have just stumbled into precisely the reason that an offline production
> pipeline exists; there is simply too much stuff moving around to cram all
> of that vital image science and complexity into an NLE.
>
> Full stop.
>
> So what is the option? Very close to what the current implementation of the
> VSE is currently, which is basically a method to arrive at an editorial
> endpoint (based on offlines, aka proxies) and commence the heavy lifting in
> a non-realtime environment.
>
>
> > This also means that having scopes as output nodes is a valid idea.
> >
>
> The scopes in the UV Image Editor were relatively robust, but have hard
> coded elements which make them somewhat broken in the modern day version of
> Blender with OCIO. Having scopes / waveforms / etc. crammed into a sidebar
> is far from optimal, of course.
>

I don't know the code but was a  programmer in the distant past. It would
seem that a lot of the code could be reused and refactored into a nodal
system. I can't of course say how much work this is but it does seem in
line with the other nodal projects that are starting right now.


> Also a VSE channel input node in the compositor might be of some value.
> > Correct?
> >
>
> I'm unsure that the solution is optimal, and I'm not entirely sure how best
> to solve this one, although some ideas have rooted over the past decade.
>
> I'd personally prefer _not_ to see folks mixing and matching phases, flip
> flopping in the pipeline, back and forth. Rather, I'd like to see a clear
> path from editorial, to post production CGI / Vfx, to grading / finishing
> be plausible. I believe that something close to what the VSE is today is a
> critical part of that; a Temporal Viewer if you will.
>

Sounds good.


>
> Grading comes with a unique set of challenges, not the least of which is
> per shot grades. This means that the temporal placement of the shots needs
> to be maintained for something like a dissolve from A to B. Shot A requires
> grading as does shot B, completely independent of one-another. Somewhat
> obvious perhaps, but the temporal view of independent shots is rather
> important in contrast to a strictly node based non-temporal view.
>

Yes.


>
> Would a strip to node alleviate that complexity? I don't think so. Nodes
> _are_ the solution, but having 1000 shots would add 1000 nodes. It would
> seem that more granularity is required in each node view rather like our
> existing Material nodes, for example.
>

Sounds like a good plan. One compositor window of each strip. The active
window would change with the active strip. BTW this does bring up a point.
Should the strips not be in the Outliner? Seems that they should to me.


>
> We also have to remember that grading is an _extremely complex_ operation
> that covers all aspects of looks. This could be heavily processor based,
> using tracking, mattes or restricted regions, and many other things
> including even Z depth etc. Now couple that with the understanding that you
> are frequently operating on scene referred data (zero to infinity) of both
> footage and CGI elements, and you can begin to see the power needs. Again,
> a per-shot Material node-like view makes sense here, with a Temporal Strip
> View being the navigation mechanism.
>

Agreed with one side point. Most of the Blender users are not making 2 hour
long films of 4k quality. Most of my projects have about 20 strips max and
I think this is normal and mostly just HD and not too complex.


>
> > Why not use a linearized reference space in the VSE then? Why not use
> > linearized reference spaces everywhere?
>
> If we think about what an NLE should do, we can probably peel apart _how_
> it might do it.
>
> Here are a few points that we can (hopefully) agree upon at this point:
>  * If we are performing manipulations, we _must_ use a linearized reference
> space. This includes blending, compositing, blurring, mixing, etc.
>

Yes.


>  * Grading is complex, requiring massive horsepower.
>

At a pro level yes but at a normal Blender user's level I don't think this
is a big factor. Am I wrong here? Most of us are doing shorts or tutorials
not 2 hour movies. A modern computer with 2 titans and a good i7 CPU has a
lot of power and about 32gb ram. This should be enough for the average user
or even above average user. I have a Nvidia 580 and even it is getting the
short end of the stick (5% speed loss) do to its age. So there is a fair
amount of power and the average user can afford to wait. Pros can through
cash at it and have server farm.


>  * CGI animation / visual effects are complex, requiring massive
> horsepower.
>

As above.



>
> All three of these points, as trivial as they seem, challenge the notion of
> the KitchenSink™ NLE application. If you care about the above, you can't
> have everything moving along realtime. Nor do you want it to.
>
> Agreed.



> Linearized data, especially scene linear zero to infinity data, requires a
> bare minimum of half float data. That means that all of the incoming
> imagery needs to be either already at float or half float, or the codec
> requires promoting an eight bit buffer to a half float representation. The
> View on the reference then converts that not-quite-viewable-directly data
> into useful display referred output. Much like what already exists in the
> UV / Image Viewer chain for nodes.
>
> Immediately we can see though, that we effectively double our memory
> requirement, and likely significantly increase our draw on processing,
> taxing our concepts of realtime. Now load 600 of those in a simple short
> project[1].
>
> The essence of an editorial system is to deliver editorial, which means
> rapid changes and realtime display of the things that matter to editorial:
> pacing, beats, duration, timing, audio cues, etc. All of those realtime
> facets cannot be negotiated with the extremely granular needs of CGI,
> visual effects, grading, etc. Worse, Blinn's Law seems in full effect. Deep
> compositing might be a good example here.
>
> Yes.


> The TL;DR is that there _is_ a path forward for the VSE, or rather, its
> next generation iteration. There _is_ a need. It _can_ be solved for
> imagers within not only a Blender paradigm, but also one that works beyond
> Blender for mixing and matching tools.
>
> OK.



> We just need to sort out the noise from the design goals, and focus on
> those very specific needs in relation to Blender as it exists now. Blender
> imagers have the advantage of having seen some of the complexity involved
> across the may facets of imaging, and as such, can likely more easily
> understand statements like "You can't do that in an NLE." This makes a
> typical Blender imager much more likely to be able to agree to and
> understand a clear design path forward without the ridiculous noise that
> typically surrounds NLE discussions.
>
>
Yes. It makes full sense to me to make the VSE the real time viewer and
focus on speed and the compositor the place for the heavy lifting. To make
its node view per strip in the VSE and to include strips in the full assets
management system, IE Outliner view.

A first step would be to have the scopes as output nodes in the compositor.



> With respect,
> TJS
>
> [1] I'd actually insist we upgrade the next strip viewer's implementation
> to half float representation. We could then get a much closer idea as to
> what an editorial temporary look might be in terms of a rough grade or
> blending. Again, even though it would still be a proxy, having half float
> makes the guesswork a little closer, allowing us to reuse our existing
> curves and such 1:1 on the full resolution, full bit depth sources. Cycles
> too has a need for a library such as this, as well as painting, so it is an
> effort that would span across many areas.
>
> Sounds good.

Thanks!


-- 
Douglas E Knapp, MSAOM, LAc.