[Bf-committers] Challenges for distributed rendering

[Jonas Eschenburg]

I am developing a  distributed rendering add-on for Blender, based on 
BitWrk (http://bitwrk.net). So far, I've come a long way by studying 
what others have been doing and getting inspiration from add-ons such as 
"netrender" and "render_renderfarmfi".

One challenge in distributed rendering is replicating the complete set 
of files needed for rendering onto the worker computers.

There are a couple of different cases:

- Textures. Blender's Python API makes iterating over all images quite 
easy, even those which were linked from some library. There is 
"File->External Data->Pack all into blend" and the associated Python 
function.

- Libraries, i.e. linked blend files. Just like with images, it is easy 
to iterate over them and get to the file name. Also, there is 
"Object->Make Local->All" which copies all linked objects into the main 
blend file.

- Caches, other resources. Haven't really looked into those yet. Lets 
exclude them for now.

So, in order to have a nice, all-encompassing blend file that can be 
sent over the wire, the user would need to perform two distinct actions 
and then save the file. Might get annoying. Worse yet, it requires the 
user to make modifications to the blend file that have to be rolled back 
later on (you generally don't want to include all linked resources 
permanently).

The renderfarm.fi add-on simply packs all textures, makes all objects 
local and saves the scene, without telling the user. Not very nice, if 
you ask me.

The netrender add-on transmits all linked resources separately, but has 
to go through big troubles in order to restore referential integrity 
between files on the worker side. File names have to be changed for safe 
transmission, e.g. you can't use absolute paths for obvious security 
reasons.

My own approach was inspired by netrender's, in that I also transmit all 
connected files separately (but in one big transmission) and assign a 
unique ID to each file. Then, on the worker side, I store all files 
using their unique IDs, not under the original filenames. Blender is 
then started with a python script that walks through all linked 
resources and replaces the original file names (which can't be resolved 
on the worker) by the new unique IDs.

This has turned out to be very troublesome because
- I can't prevent Blender from trying to load all linked resourced under 
their original file name initially
- and to make things even more complicated, Blender removes all 
references to libraries it can't load on startup.

Ideally, there would be a function 
"save_main_file_under_another_name_with_all_external_resources_included()".

Second best would be a way to manipulate all data on save time, perhaps 
by employing a visitor pattern.

Third best would be a method to defer loading of external resources 
until a python startup script had the chance to replace file names properly.

Ok, anyone has a suggestion how to handle linked files nicely?

Thanks,
Jonas