[Bf-committers] Is there a parallel pipeline in blender?

[Ruan Beihong]

I did some googling about the TBB you mentioned. It provide a similar
function set just as my thought.
The thing that I want to discuss is a much more specific implement in
C without a lot of change in code, not a generic template version in
C++ that try to solve everything but need a lot of work to change
code. And the data dependency is not taken care by this approach, it's
up to the programmer. This approach only promise that each task in
pipeline is running in order and more than one piece of data is
started and ended in order. For example:

data[0] in
data[1] in
==here they go in parallel==
data[0] is running task 1
data[1] is running task 1
data[1] is running task 2
data[1] finishes running
data[0] is running task 2
data[0] finishes running
==here they go in series==
data[0] out
data[1] out

The two tasks of each piece of data is promised to be run in its order
and the data's in and out of the pipeline is promised to be run in the
calling order.

To programmer, the lock-free way is recommended which means the task
functions should behave somewhat functional: only do things with input
and output and should not alter any variable that hasn't been pass to
it as an argument. Like a Monad in Haskell this pipeline warps tasks
input and output and pass it from the previous task function to the
next. If the task function is not lock-free, it's up to the programmer
to put is the way it is wanted.

My original design includes locks and programmer controllable buffers
between each task functions in one pipeline. But I then abandon it
because its take too much thread controlling work: each buffer should
behave FIFO, so a semaphore is need for each buffer; buffers takes
memory space; it may also allow BYPASSING (which in the CPU
microarchitecture design means data pass from a half-done pipeline to
one of following half-done pipeline) which need data dependency be
defined before running and that makes things ugly for both me and
future users for we are actually simulate a CPU internal structure.

That's not the thing we wanted, we hope it be easy to use, be good in
look and be efficient as the original purpose.

2009/8/7 Lars Krueger <lars_e_krueger at gmx.de>:
> If you want a library to do this in C++, you might want to take a look at the INTEL thread building blocks.
>
> http://www.threadingbuildingblocks.org/
>
> Depending on how far reaching you want to try this, it might be a lot of work. Most problematic are data dependencies. You have to make sure that data computed by one function, but used by a second function later, is either computed so far that the second function can start or completely. That's why most people do their parallelisation block-wise.
>
> I don't think manually specifying these dependencies would be a good idea. If you find an at least semi-automatic way to specify these dependencies, more parts of the code can benefit from it.
> Other options are lock-free data structures (interesting, but usually require complete rewrite), parallel functional programming (possible in C, but requires a lot of discipline and good libraries, e.g. garbage collection) or more esoteric concepts, e.g. tuple space (requires rewrite).
>
> If you can find such an approach to the dependencies, you are better off investing your time in parallising different modules.
>
> I don't know which of the following modules are already parallel, but I guess the nodes and the particle system are prime candidates for parallelisation. On the other hand, if parallelise too fine-grained, things like thread-start overhead and cache utilisation become the determining factors.
>
> --
> Dr. Lars Krueger
>
>
> Jetzt kostenlos herunterladen: Internet Explorer 8 und Mozilla Firefox 3 -
> sicherer, schneller und einfacher! http://portal.gmx.net/de/go/chbrowser
> _______________________________________________
> Bf-committers mailing list
> Bf-committers at blender.org
> http://lists.blender.org/mailman/listinfo/bf-committers
>



-- 
James Ruan