[Bf-committers] Compositor speedup proposal [1/2]

[Jeroen Bakker]

This is a proposal to solve speed-issues of the compositor. It should 
not be considered as the final solution, but should help the most common 
issues.

Problem statement.
The compositor works best when having a good mixture of simple and 
complex nodes. If you have a lot of simple nodes the system is not able 
to find a good balance when converting to execution groups (subprogram 
that will be scheduled to a core of the CPU). It results in a few 
execution groups with many simple operations and a small number of 
buffers that store intermediate results. This slows down the system a 
lot 
[http://projects.blender.org/tracker/?func=detail&aid=33785&group_id=9&atid=498]. 
A workaround for this slowdown was to add a complex node (that doesn't 
do anything, like blur 0) in the setup.

First test shows that good result depends on the node tree setup and the 
available memory of the system. We propose to split up execution groups 
into smaller ones if they get too big. The split up will depend on two 
variables:
1. amount of memory in the system (not free memory)
2. number of operations in an execution group

As this mechanism does a lot of guesses, the user should be able to 
manually control the number of cuts.

During tests we saw the next results
Used file: file attached to issue #33785
Used system: Intel(R) Core(TM) i5 CPU M 580 @ 2.67GHz, with 8GB of 
memory, ubuntu 12.04 64 bit:
  - Baseline (no changes to code): 861MB, 47.49 seconds
  - Limit execution group size to 10: 3424MB, 7.267 seconds
  - Limit execution group size to 15: 3289MB, 7.607 seconds
  - Limit execution group size to 20: 2884MB, 9.393 seconds
  - Limit execution group size to 25: 2884MB, 11.987 seconds

Best regards,
Jeroen & Monique
  - At Mind -