[Bf-committers] GSoC 2018 BVH8

Sergey Sharybin sergey.vfx at gmail.com
Fri Feb 2 10:25:28 CET 2018 

Hi,

Answers are inlined.

On Tue, Jan 30, 2018 at 11:56 AM, Brecht Van Lommel <
brechtvanlommel at pandora.be> wrote:

> Hi,
>
> On Tue, Jan 30, 2018 at 9:32 AM, Sergey Sharybin <sergey.vfx at gmail.com>
> wrote:
>
> > Was more like discussed, and agreed that it's still lots of open ends in
> > there?
> >
> > With the current state it is not really convincing that it's THE way to
> > go.It will become THE way to go when we will know answers to those open
> > ends, which will not increase already tedious burden of GPU support.
>
>
> The way I see it is that there is still a lot of work, not so much open
> questions.
>

The amount of work of is only measurable once you have a goal. What is the
goal here: ditch everything and replace with Embree? It has consequences on
a longer term.

I do get your point about not duplicating code from Embree when they
already have done everything. But, this is only true for CPU. When you are
mixing GPU support into account, things becomes less obvious.

Consequences i see for GPU:

- We would need to have some way of "extracting" BVH from Embree.

Surely, this could be solved with some more comprehensive Embree API. So
maybe at the end of a day it's not that much of a code even. But this is to
be done, and ONLY to be done for GPU. Which leads to potential issues of
either Embree changing something in their library which will break our
"extraction", or that we will make some improvement in BVH which will break
GPU.

- We need to mimic primitives intersection from Embree.

It is POSSIBLE to use own intersection code in Embree, but this is NOT
recommended due to performance impacts. So if we want ultimate performance
on CPU, we would have to accept DUPLICATION of code for GPU (remember, the
idea of using Embree was to avoid code or work duplication, which is
violated here).

- We need to mimic OBB BVH intersection on GPU.

Surely, we do have SOME code for this, but i'm not sure orientation is
still denoted the same way as it originally was in Embree, and will it ever
change? If Embree changes something related on this, we would need to
duplicate work for GPU once again.

- We need to support proper motion BVH on GPU.

While on CPU we do have nodes interpolation "for free" with Embree, we
would NEED to re-implement this on GPU. Again, duplicated work and code,
which defeats original idea of reducing burden of maintaining own BVH
builder/traversal.

Or, it brings us back to question: do we care about GPU? Or do we just
stagnate at current state, and do not improve rays intersection at all?

> Well, sure, as Stefan mentioned above, there is no way for a custom node
> > builder function to know orientation for hair BVH, but i don't see why
> > would AABB BVH take long time to do such a test.
> >
> > Further, (as Stefan mentioned already), you would HAVE to implement that
> in
> > one way or another in order to use BVH built by Embree on GPU.
> >
>
> I don't think we have to implement 8-wide BVH traversal for the GPU, or any
> of the SSE optimizations or CPU specific tuning.
>

Well, maybe not 8 wide,  but what about 4? Is it possible to extract GOOD
QUALITY 2-wide BVH from Embree? To my knowledge, for final result they
removed binary BVH support, but i'm not sure they are building binary first
and then merge, or if they build 4-BVH to begin with.

If it's not possible to extract binary BVH, then we would need to have some
node splitting implemented on our side. Doing simple split will degrade
quality of BVH, defeating the whole idea of using Embree for higher quality
BVH on GPU. Or we are doomed to (re)implement some smart heuristics to find
a good split of the node.

I'm not sure what you would expect from such a test. If it's slower than
> Embree's traversal, then we can attribute that to our implementation not
> being tuned as much as theirs, which is what I would expect. If it's
> faster, then that would be interesting but it seems unlikely to happen?
>

I would expect better understanding what is the actual problem, and see
what consequences it causes on a bigger picture.

Currently all the work i see is more like "let's go full-speed hacks and
what not for Embree on CPU, and GPU support we will hack on later, and hope
for the best in there".


> For me the big reason to adopt Embree is so we don't have to do all this
> optimizations for the CPU, we don't have to worry about SSE, AVX, 16-wide,
> pack triangle intersection, ray packet traversal, etc.
>

Yes, this is indeed good thing.


> > And last but not least here, you would HAVE to duplicate Embree code in
> one
> > way or another: primitives intersection on GPU is something you would
> HAVE
> > to re-implement.
> >
>
> Yes, but we only need a simpler scalar version of it, with one BVH layout
> type.
>

Not sure what do you mean by scalar here. Sounds like a fiasco of our
float{3,4} abstraction to me if we need to worry about whether something is
vectorized or not.

Or, do you mean non-packeted-triangles-intersection by scalar here?

Also, see above, there is fair amount of things we have to support in BVH
traversal still.

Now, if we still need to duplicate that work for GPU, what are we winning
here by using Embree? What is the shortest way to give artists faster
renders:

- Wrap up existing BVH8 and packeted intersection patches, and use Embree
as a builder only

OR

- Use Embree for builder AND CPU traversal AND solve all the TODOs
mentioned in the discussion earlier AND do something about GPU support?

-----

Don't get me wrong, i'm not against of all the Embree integration by any
means, but to me cost function for mowing forward is quite complicated and
consists of the following residual blocks:

- How soon we can have master-ready speedup for artists?
- How much code duplication we are doomed to have duplicated for GPU
support?
- How much time we invest into keeping GPU support?
- How much time we REALLY save, keeping in mind that we need to
re-implement things on GPU?
- How much time we will be spending by investigating CPU / GPU related
artifacts?
- How soon we can ship fixes if bug is discovered in Embree?

There are probably more blocks, and some of them should probably be 1/x.

> > > On Mon, Jan 29, 2018 at 8:34 PM, Stefan Werner <stewreo at gmail.com>
> > > wrote:
> > > >
> > > > > Hi Brecht, Anton,
> > > > >
> > > > > to get Embree to be a full replacement of Cycles’ own BVH, there
> are
> > > > still
> > > > > a number of tasks to be done. From the top of my head, here’s
> what’s
> > > > > missing from my current branch:
> > > > >
> > > > > 1. Update to Embree 3.0.0, of which a first beta was just released.
> > > This
> > > > > should allow us to build with an unpatched Embree.
> > > > > 2. If we decide that it’s needed, implement a solid line
> > > segment/cylinder
> > > > > primitive for Embree. Embree has flat (ribbon) line segments, flat
> > > curves
> > > > > and solid curves but not solid cylinders.
> > > > > 3. Reduce the amount of duplicated data. Ideally, Cycles keeps all
> > the
> > > > > geometry data in its own buffers and Embree accesses it directly
> > > through
> > > > > pointers. That may require resolving race conditions in interactive
> > > > > rendering.
> > > > > 4. Extract BVH data from Embree and write GPU traversal code.
> > > > > 5. Find out if it’s possible in the latest Embree to do quaternion
> > > > > interpolation of motion transforms - the 2.x versions are still
> > limited
> > > > to
> > > > > linear straight interpolation. If this isn’t there, motion blur
> > quality
> > > > > will degrade.
> > > > >
> > > > > Once parity is reached, there are several features in Embree that
> > could
> > > > > improve Cycles. Some of these may not work with the GPU:
> > > > > 6. Using Embree’s ray stream API for the CPU split kernel.
> > > > > 7. Embree’s built-in displacement.
> > > > > 8. Embree’s quad primitive.
> > > > > 9. Compare Embree’s subdivision implementation to Cycles’ and use
> it
> > if
> > > > > it’s better. The on-demand cache could help reduce memory usage at
> > the
> > > > > expense of performance.
> > > > >
> > > > > Feel free to take over any of these tasks. I would be happy to work
> > on
> > > > > them myself, but as it stands I can’t make any promises regarding a
> > > time
> > > > > line. The embree integration in its current state is sufficient for
> > our
> > > > > current production needs. I don’t expect to have a lot of spare
> time
> > > for
> > > > > extra polish in the next few months.
> > > > >
> > > > > The current state of my Embree integration is here:
> > > > > https://git.blender.org/gitweb/gitweb.cgi/blender.git/
> > > > > shortlog/refs/heads/cycles_embree <https://git.blender.org/
> > > > > gitweb/gitweb.cgi/blender.git/shortlog/refs/heads/cycles_embree>
> > > > >
> > > > > It does rely on a patched version of Embree which you can find in
> > this
> > > > > branch:
> > > > > https://github.com/skwerner/embree/tree/cycles_compatible
> > > > >
> > > > > Don’t hesitate to ask if you have any questions about this.
> > > > >
> > > > > -Stefan
> > > > >
> > > > > > On 29. Jan 2018, at 18:58, Brecht Van Lommel <
> > > > brechtvanlommel at pandora.be>
> > > > > wrote:
> > > > > >
> > > > > > Hi Anton,
> > > > > >
> > > > > > We're planning to move to Embree, so improving our existing BVH
> > > builder
> > > > > and traversal is not the best use of time.
> > > > > > https://lists.blender.org/pipermail/bf-committers/2017-
> > > > > November/048936.html <https://lists.blender.org/
> > > > > pipermail/bf-committers/2017-November/048936.html>
> > > > > >
> > > > > > However a lot of work is still needed to get the Embree
> integration
> > > > > stable, and I imagine whatever the state will be this summer, there
> > > will
> > > > be
> > > > > a lot room for improvement. Finding optimizations in the
> integration,
> > > > > memory usage reductions, etc. Some of the bigger topics would be:
> > > > > > * Use Embree for building the GPU BVH, so we can remove our own
> > > > builder.
> > > > > > * For subdivision surfaces, implement more memory efficient grid
> > > > > primitive to intersect directly.
> > > > > > * Optimizations for tracing short rays on a single object for
> > > > subsurface
> > > > > scattering, bevel, curvature.
> > > > > >
> > > > > > Stefan, do you have an idea about which tasks will be open still
> > this
> > > > > summer?
> > > > > >
> > > > > > Regards,
> > > > > > Brecht.
> > > > > >
> > > > > > On Sun, Jan 28, 2018 at 11:07 PM, Антон Гавриков <
> > > > > gavrikovantonkapi at gmail.com <mailto:gavrikovantonkapi at gmail.com>>
> > > wrote:
> > > > > > Hello, I worked on this (https://developer.blender.org/D2875 <
> > > > > https://developer.blender.org/D2875>) project as a
> > > > > > summer intern in Intel. I would like to know if there is any
> > interest
> > > > in
> > > > > > working on this project in GSoC 2018. Or maybe there are any
> > similar
> > > > > tasks
> > > > > > to optimize Cycles for new architectures.
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > Anton.
> > > > > > _______________________________________________
> > > > > > Bf-committers mailing list
> > > > > > Bf-committers at blender.org <mailto:Bf-committers at blender.org>
> > > > > > https://lists.blender.org/mailman/listinfo/bf-committers <
> > > > > https://lists.blender.org/mailman/listinfo/bf-committers>
> > > > > >
> > > > >
> > > > > _______________________________________________
> > > > > Bf-committers mailing list
> > > > > Bf-committers at blender.org
> > > > > https://lists.blender.org/mailman/listinfo/bf-committers
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > With best regards, Sergey Sharybin
> > > > _______________________________________________
> > > > Bf-committers mailing list
> > > > Bf-committers at blender.org
> > > > https://lists.blender.org/mailman/listinfo/bf-committers
> > > >
> > > _______________________________________________
> > > Bf-committers mailing list
> > > Bf-committers at blender.org
> > > https://lists.blender.org/mailman/listinfo/bf-committers
> > >
> >
> >
> >
> > --
> > With best regards, Sergey Sharybin
> > _______________________________________________
> > Bf-committers mailing list
> > Bf-committers at blender.org
> > https://lists.blender.org/mailman/listinfo/bf-committers
> >
> _______________________________________________
> Bf-committers mailing list
> Bf-committers at blender.org
> https://lists.blender.org/mailman/listinfo/bf-committers
>



-- 
With best regards, Sergey Sharybin

More information about the Bf-committers mailing list