[Bf-committers] Blender Startup Time

[Martin Poirier]

--- On Fri, 2/25/11, Campbell Barton <ideasman42 at gmail.com> wrote:

> I think most of this problem can be fixed by doing
> lazy-importing.
> Where a scripts will only register its classes on startup
> but not
> actually load the majority of the code which can be stored
> in a
> submodule and imported as needed.
> I've already done this for OBJ/FBX/PLY/3DS io scripts and
> found it a
> simple change to make and doesn't make the scripts hard to
> read/maintain.
> 
> Looking at netrender loading, it indirectly calls 3050
> open(), stat()
> and fstat() combine.
>   strace ./blender --debug > out.txt
>   # isolate netrender startup
>   cat out.txt "^open(\|^stat(\|^fstat("
> 
> Full output
> http://www.pasteall.org/19495/c
>
> Granted, a lot of this is out of control of netrender since
> pythons
> internals are searching all over the hard drive for files,
> nevertheless lazy loading helps a to avoid the problem.

Very interesting stats.

Much of that has to do with order of import. After the module cache is filled in, subsequent imports won't do disk access. Due to loading order, importing netrender probably hits a lot of module not in the module cache that would also be imported by others later.

For example, 391 of these operations are for importing the webbrowser module, which does a lot of guessing for installed software by poking around. That module is also imported at runtime when clicking on a url link, but that then doesn't cost anything as its already in the cache.

If we want to benchmark individual modules/packages, it's important to do them without any others, otherwise the stats are heavily skewed by the module cache.

On the other hand, most of those operators are checks for non-existing files, you could try to force prefetches all you want, that won't do anything useful. It's also important to note that you probably get significantly more of these because of the changes to precompiled module caches in Python 3.2 (if I'm reading your logs correctly).

As for lazy loading, sure, that can be used to offset the loading cost by redistributing it at runtime, but aren't we complicating things for a gain that is very minimal here?

Martin
PS: Someone also asked me months ago if netrender could be moved to the extensions repo and I said yes, so that can probably help startup time a bit already (though, as I believe, this will probably just mean delaying module cache misses).