[Bf-committers] Questions about cloud rendering

[Campbell Barton]

On Thu, Apr 16, 2015 at 11:34 AM, Andrey Chursin <andll at danasoft.ws> wrote:
> Hi guys,
>
>
> I am trying to create a cloud render farm for Blender (because there
> is never enough render power!) and I have a few questions on how
> Blender works. I have a solid background in distributed and cloud
> computing, but I am still new to 3D rendering and Blender
> specifically.
>
>
> The few questions that I have are below. If it’s too much for an
> email, perhaps there is someone who is willing to get on
> skype/hangouts with me to help with these? I would be extremely
> grateful! (happy to pay for your time too). Thanks a lot!
>
>
> Questions:
>
> 1) I saw 'do not run scripts' in blender options. Does it mean the
> rendering process would need to run some arbitrary code injected into
> the blender file?

Yes.

> Are scripts commonly used? If we do not support it on the farm would
> it turn off a lot of people?

For rigging, yes, PyDrivers (typically character animation).

> 2) I want to check files uploaded to my renderer before putting them
> into queue. I figured I can do it by running:
>
> blender -b filename.blender -P myScr.py
>
> Can I assume, that if myScr.py was run by blender, that the file is ok
> (more or less)?

No, even if the file is missing the script will run, though the script
can check Blender state and use `sys.exit(error_value)` if there are
any problems.

> Generally, what is the best way to check most of file's parameters
> before launching computation-heavy rendering?

It depends what you want to check.

You could load the blend file and check values using blendfile module.

But I wouldn't recommend this as a first step, its quite involved and
might not even be necessary (compared with just checking before
render).

On the other hand, its very fast since it only reads parts of the file
you ask for.
https://developer.blender.org/diffusion/BAM/browse/master/bam/blend/blendfile.py

----

Running a script that does sanity checks and initiates the render
operator works well. You can exit if theres any problems too.

See this thread, it covers similar topics.
http://markmail.org/thread/2qyeqgf7pyhobdt7


> 3) Does the packed blender file contain everything needed to render?
>
> What’s the best way for the user to upload data to the farm? A packed
> blend file? Or is it better to let them compress everything into one
> archive?

Packed files aren't 100% assured to contain all data - video files for
example wont pack... and linked libraries complicate things too
(though there is support for packed libs).

The BAM project has a pack command which can run standalone.

This is used for downloading single blends from the cloud and it was
being tested for the Blender institutes renderfarm. (though not sure
if it ended up being used for production).

https://developer.blender.org/diffusion/BAM/browse/master/bam/cli.py;9e92aa773691ea5b4ef7e4cd99e97df179e92f25$1629

> 4) Are all render settings already stored in the file? Is there
> something that can’t be included in the file, that I’d need to ask
> about before starting the render?
>
> I see that I can specify output size, pick frames and so on from the
> command line - but aren’t they already stored in the blender file?

Yes, each scene has its own frame-range, so in fact there are multiple
possible outputs within a single Blend file.

But you may want to split up the frame range between multiple
computers. or change the output size for a preview render.

For producing a full animation its often useful to override many
settings from the blend file, since artists may be doing preview
renders and forget to change some option before sending to the farm.

> Network rendering
>
>
> 5) I noticed (using wireshark) that the master starts an http server
> and accepts POSTs with json commands from clients. These commands
> contain local (to client) paths to files. So how does the client send
> these files to master? Seems like no other data is transferred with
> HTTP. Did I miss something?

As far as I know file paths should be relative, though I only ever did
small tests with netrender.

> 6) I really like the idea of the network renderer, but right now it
> uses network discovery and doesn’t have any authentication, so I can’t
> really use it for my cloud farm. It seems like minor improvements can
> be done in order to create a standard protocol and allow to push user
> tasks to any farm (assuming the farm provides the user with some URL
> like https://my.farm.io/ and username and password).
>
> It seems like the only other solution currently if each farm writes
> their own blender plugin.

Yep, its not ideal, we don't really have developers who are interested
in this area so the netrender addon is only being kept working, not
improved.

I think we may not be able to have an entirely generic solution, but
we could have some generic utilities that different farms can re-use.

- pre-flight checks (don't render with missing files...).
- bundling blend files to send to the farm (something like "bam pack"
extracted out of bam).
- checking blend files for scripts/code.

> Is there someone I can talk about network renderer? Perhaps, I can
> even contribute.

Not sure, see if others like to chime in here :)

> --
> Regards,
> Andrey
> _______________________________________________
> Bf-committers mailing list
> Bf-committers at blender.org
> http://lists.blender.org/mailman/listinfo/bf-committers



-- 
- Campbell