[Bf-committers] Script files layout (proposed changes).

[Campbell Barton]

On Wed, Mar 16, 2011 at 1:26 AM, Martin Poirier <theeth at yahoo.com> wrote:
> Hey,
>
> Didn't you say you were going to get more concrete data on loading speed the last time we talked about this? If you did, now would be a good time to show it.
>
> --- On Tue, 3/15/11, Campbell Barton <ideasman42 at gmail.com> wrote:
>
>> There are a few problems with the
>> current way we load scripts on startup
>> * too many modules in sys.modules (96 for blender alone),
>>   in most cases this is only to make use of
>> auto-loading feature and
>> we don't need them directly imported, they only pollute the
>> namespace.
>
> sys.modules is a dictionary, it doesn't matter how many there are, access speed is constant.
>
> What namespace do they pollute?
>
>> * too many search paths in sys.path, causes python to do
>> over 4000
>> open(...) commands (on my system, at least).
>>   rough guess is we could get this down to 1500-2000.
>>
>> * failed stat/open calls slow startup. see below for
>> example below.
>>   rough guess based on previous tests - 8% - 15%
>> overall slowdown, so
>> not huge but would vary depending on the OS, drivers,
>> hardware.
>
> Is that cold start stats or average stats?
> Did any one else do some timings?
> Did you try on more than one machine?
> Do we have stats for other OS?
>
> Aren't we drawing conclusions with a very limited set of data...
>
>> ---
>>
>> Proposal:
>> Have a single auto-load directory and group scripts as
>> packages.
>> *note* this won't effect addons at all and wont break
>> scripts so its
>> mostly an internal blender/python topic.
>>
>> existing layout
>>  - scripts/addons,
>>  - scripts/keyingsets,
>>  - scripts/io,
>>  - scripts/op,
>>  - scripts/ui,
>>  - scripts/modules,
>>
>> proposed layout
>>  - scripts/addons,
>>  - scripts/startup, <-- only auto-load dir
>>  - scripts/modules,
>>
>> This way we can have...
>> "scripts/ui" --> would move to "scripts/startup/ui" and
>> become its own
>> package-module, with __init__.py
>>
>> Any scripts which don't make sense to have as packages
>> could be placed
>> directly in scripts/startup/ as a file though I'd like to
>> avoid this.
>
> This will not reduce the amount of entries in the module cache (each imported subpackage gets an entry) and won't reduce the number of unique import calls: subpackage defining RNA subtypes will have to be imported anyway, back to square one (or very near). Not without serious repackaging of all of these modules at least.
>
> The only thing this does is remove three entries from sys.path, which is fine but is it worth moving everything around?
>
> Now, I could very well be wrong, but it's hard to be sure without any kind of data on what that change accomplishes.
>
> Martin
> PS: I didn't want this to sound so negative. On second reading, it does a bit. C'est la vie.

You're criticisms reasonable, I shouldn't have confused optimizing
with layout changes, so I'll explain my take on this.

First off, I actually wanted to re-arrange the modules but wasn't sure
if there were big startup gains to be had from doing tricks like
delayed loading of function body or initializing a stub class and
loading the real class later.

So I looked into optimizing load times (see thread on blender startup times).
My conclusion is:
* minimal gains to be had from changing layout or removing body code
from scripts even.
* biggest slowdown is disk access & worst case of this is python
loading many modules, both blenders and python modules.
  we get some speedup from zipping all scripts together but loose a
little on warm startup.
* Tweaking module layout for speedups isn't worthwhile.
* Good speedup but just *not loading* modules altogether.

so I went ahead and made a few imports inline, and made a bare-bones
OrderedDict class to avoid importing pythons heavy collections module,
it wasn't all that much work and avoids reading many files on python
side and the expense of making the scripts a little odd by not having
all imports at the top.

For now I'm quite happy with the speedup this gives.
BUT, this is only on my system.... ok, ok... I could test 3-4 systems
and make better decisions based on more info.
Worst case is that on Win or OSX or different hardware - 1000's of
failed open() or stat() calls give big overhead and I'm not taking
this into account.
ack, you got me there! I should really do more tests...  but if it was
THAT bad we would probably have people saying why is blender 3x slower
starting on Win/OSX?...
Sigh... I know, weak argument for not installing windows and running
tests but I suspect its not totally different on other OS's.
Reading lots of files from different places is known to be slow in general.


Taking speed out of the picture I'd still like to but forward this
proposal on its own merits.
You mention namespace lookup speed not being a problem in sys.modules
but this is still a namespace issue.
blender is adding modules like: wm, object, presets, mesh - we can
rename these to more unique module names, bl_wm, bl_mesh ? - but some
really have no need to be accessed as modules from other scripts and
IMHO they may as well go in a package.

This also makes more sense for the users home dir which currently
matches the blender system scripts layout.
~/.blender/2.56/scripts/ui
~/.blender/2.56/scripts/op
~/.blender/2.56/scripts/io
~/.blender/2.56/scripts/keyingsets
.... etc
IMHO doesn't make much sense for users who probably just want a few
startup scripts but not to write their own blender interface/tools
overlaying blender (for which addons are better suited anyway).
Easier just to have.

~/.blender/2.56/scripts/startup <-- auto-load if users want that, it
handy to drop scripts in there without addon header.

startup more clearly loads modules, similar to startup.blend too.

- Campbell