[Bf-committers] Blender Startup Time

[Campbell Barton]

On Sat, Feb 26, 2011 at 4:56 AM, Martin Poirier <theeth at yahoo.com> wrote:
>
>
> --- On Fri, 2/25/11, Campbell Barton <ideasman42 at gmail.com> wrote:
>
>> I think most of this problem can be fixed by doing
>> lazy-importing.
>> Where a scripts will only register its classes on startup
>> but not
>> actually load the majority of the code which can be stored
>> in a
>> submodule and imported as needed.
>> I've already done this for OBJ/FBX/PLY/3DS io scripts and
>> found it a
>> simple change to make and doesn't make the scripts hard to
>> read/maintain.
>>
>> Looking at netrender loading, it indirectly calls 3050
>> open(), stat()
>> and fstat() combine.
>>   strace ./blender --debug > out.txt
>>   # isolate netrender startup
>>   cat out.txt "^open(\|^stat(\|^fstat("
>>
>> Full output
>> http://www.pasteall.org/19495/c
>>
>> Granted, a lot of this is out of control of netrender since
>> pythons
>> internals are searching all over the hard drive for files,
>> nevertheless lazy loading helps a to avoid the problem.
>
> Very interesting stats.
>
> Much of that has to do with order of import. After the module cache is filled in, subsequent imports won't do disk access. Due to loading order, importing netrender probably hits a lot of module not in the module cache that would also be imported by others later.
>
> For example, 391 of these operations are for importing the webbrowser module, which does a lot of guessing for installed software by poking around. That module is also imported at runtime when clicking on a url link, but that then doesn't cost anything as its already in the cache.
>
> If we want to benchmark individual modules/packages, it's important to do them without any others, otherwise the stats are heavily skewed by the module cache.
>
> On the other hand, most of those operators are checks for non-existing files, you could try to force prefetches all you want, that won't do anything useful. It's also important to note that you probably get significantly more of these because of the changes to precompiled module caches in Python 3.2 (if I'm reading your logs correctly).
>
> As for lazy loading, sure, that can be used to offset the loading cost by redistributing it at runtime, but aren't we complicating things for a gain that is very minimal here?
>
> Martin
> PS: Someone also asked me months ago if netrender could be moved to the extensions repo and I said yes, so that can probably help startup time a bit already (though, as I believe, this will probably just mean delaying module cache misses).

Your right that this test should be done without other modules which
may have cached some files already and make some of the modules loaded
after netrender seem fast because they using cached modules.

Still not a perfect test but a quick test is to see performance with &
without netrender, this way any unfair cost attributed to netrender
would be passed onto the next module which uses the same extra
modules.
Times with/without netrender, cold starts, ran both tests twice and averaged.
---
  sync ; echo 3 > /proc/sys/vm/drop_caches
  time /data/src/blender/build/linux/bin/blender -b
---
with: 10.289
without: 8.0595

>From a quick search, netrender is the only module loading json,
socket, http, webbrowser on startup.

As for lazy loading complicating things, I don't think it does
especially though it depends on a lot on each script, for
import/export its very simple.

for the rest of blender I would just move some functions into modules
and load as needed.
keymap load/save for eg, some of the addons functions also.

I don't mean to pick on netrender but all our other scripts have very
simple module requirements in comparison and its really the most
obvious one to speed up.
Having as an addon would be good, but it doesn't really change much,
only that it can be disabled on startup, if its enabled by default we
should still try to get good startup times.

You also mention this will re-distribute the slowdown at runtime. this
is true but IMHO not to the point of being a problem.
For users that don't activate these features its always faster - don't
click on the web-links, don't use renderfarms, don't use X3D == no
overhead of xml/json/socket/webbrowser/http modules loading.

- Campbell