[Bf-committers] Blender Startup Time

[Campbell Barton]

The difference between py3.1 and 3.2 does seem to be because py3.2 is
testing for ~twice as many files with calls to open(), but the impact
on startup time isnt that bit.
You also are interested in warm startup time for lazy loading, I don't
have exact numbed but I tested this too and it was roughly an overall
40% speedup.

Talked to Martin on IRC and he was interested in some aspects of these tests...

Firstly there are a large number of failed open attempts when loading modules,
Martin was interested to know if the failed opens are taking much time
and if this was worse in py3.2.

Some stats (bets of 3), standard r35253.
 sync ; echo 3 > /proc/sys/vm/drop_caches ; time ./bin/blender -b
times:
 Py3.2 cold start 7.744
 Py3.1 cold start 7.065
 Py3.2 warm start 0.287
 Py3.1 warm start 0.301

calls to open:
Py3.1, 2347 calls to open(), 1932 of them fail.
Py3.2, 4756 calls to open(), 4331 of them fail.

After this I did some more tests by prepending "/usr/bin" 'sys.path'
to see if adding more directories makes it slower, it did but only
slightly for warm and cold startups.
each time I added dir like /usr/lib or /usr/bin (each with over 3k
files in it) it was approx 5-10% slower.

I assume this starts to depend on the filesystem and os drivers,
assume once the dirs cached further calls to open a file are not all
that slow.

Incidentally, the order of the sys.path is important, if the blender
paths are added to the end of sys.path there are 6445/6019(failed)
calls to open() rather then 4756/4331.

Example of loading webbrowser with current unmodified blender and py3.2.
----
open("/b/release/scripts/io/webbrowser.cpython-32d.so", O_RDONLY) = -1
ENOENT (No such file or directory)
open("/b/release/scripts/io/webbrowsermodule.cpython-32d.so",
O_RDONLY) = -1 ENOENT (No such file or directory)
open("/b/release/scripts/io/webbrowser.abi3.so", O_RDONLY) = -1 ENOENT
(No such file or directory)
open("/b/release/scripts/io/webbrowsermodule.abi3.so", O_RDONLY) = -1
ENOENT (No such file or directory)
open("/b/release/scripts/io/webbrowser.so", O_RDONLY) = -1 ENOENT (No
such file or directory)
open("/b/release/scripts/io/webbrowsermodule.so", O_RDONLY) = -1
ENOENT (No such file or directory)
open("/b/release/scripts/io/webbrowser.py", O_RDONLY) = -1 ENOENT (No
such file or directory)
open("/b/release/scripts/io/webbrowser.pyc", O_RDONLY) = -1 ENOENT (No
such file or directory)
open("/b/release/scripts/op/webbrowser.cpython-32d.so", O_RDONLY) = -1
ENOENT (No such file or directory)
open("/b/release/scripts/op/webbrowsermodule.cpython-32d.so",
O_RDONLY) = -1 ENOENT (No such file or directory)
open("/b/release/scripts/op/webbrowser.abi3.so", O_RDONLY) = -1 ENOENT
(No such file or directory)
open("/b/release/scripts/op/webbrowsermodule.abi3.so", O_RDONLY) = -1
ENOENT (No such file or directory)
open("/b/release/scripts/op/webbrowser.so", O_RDONLY) = -1 ENOENT (No
such file or directory)
open("/b/release/scripts/op/webbrowsermodule.so", O_RDONLY) = -1
ENOENT (No such file or directory)
open("/b/release/scripts/op/webbrowser.py", O_RDONLY) = -1 ENOENT (No
such file or directory)
open("/b/release/scripts/op/webbrowser.pyc", O_RDONLY) = -1 ENOENT (No
such file or directory)
open("/b/release/scripts/ui/webbrowser.cpython-32d.so", O_RDONLY) = -1
ENOENT (No such file or directory)
open("/b/release/scripts/ui/webbrowsermodule.cpython-32d.so",
O_RDONLY) = -1 ENOENT (No such file or directory)
open("/b/release/scripts/ui/webbrowser.abi3.so", O_RDONLY) = -1 ENOENT
(No such file or directory)
open("/b/release/scripts/ui/webbrowsermodule.abi3.so", O_RDONLY) = -1
ENOENT (No such file or directory)
open("/b/release/scripts/ui/webbrowser.so", O_RDONLY) = -1 ENOENT (No
such file or directory)
open("/b/release/scripts/ui/webbrowsermodule.so", O_RDONLY) = -1
ENOENT (No such file or directory)
open("/b/release/scripts/ui/webbrowser.py", O_RDONLY) = -1 ENOENT (No
such file or directory)
open("/b/release/scripts/ui/webbrowser.pyc", O_RDONLY) = -1 ENOENT (No
such file or directory)
open("/b/release/scripts/webbrowser.cpython-32d.so", O_RDONLY) = -1
ENOENT (No such file or directory)
open("/b/release/scripts/webbrowsermodule.cpython-32d.so", O_RDONLY) =
-1 ENOENT (No such file or directory)
open("/b/release/scripts/webbrowser.abi3.so", O_RDONLY) = -1 ENOENT
(No such file or directory)
open("/b/release/scripts/webbrowsermodule.abi3.so", O_RDONLY) = -1
ENOENT (No such file or directory)
open("/b/release/scripts/webbrowser.so", O_RDONLY) = -1 ENOENT (No
such file or directory)
open("/b/release/scripts/webbrowsermodule.so", O_RDONLY) = -1 ENOENT
(No such file or directory)
open("/b/release/scripts/webbrowser.py", O_RDONLY) = -1 ENOENT (No
such file or directory)
open("/b/release/scripts/webbrowser.pyc", O_RDONLY) = -1 ENOENT (No
such file or directory)
open("/b/release/scripts/modules/webbrowser.cpython-32d.so", O_RDONLY)
= -1 ENOENT (No such file or directory)
open("/b/release/scripts/modules/webbrowsermodule.cpython-32d.so",
O_RDONLY) = -1 ENOENT (No such file or directory)
open("/b/release/scripts/modules/webbrowser.abi3.so", O_RDONLY) = -1
ENOENT (No such file or directory)
open("/b/release/scripts/modules/webbrowsermodule.abi3.so", O_RDONLY)
= -1 ENOENT (No such file or directory)
open("/b/release/scripts/modules/webbrowser.so", O_RDONLY) = -1 ENOENT
(No such file or directory)
open("/b/release/scripts/modules/webbrowsermodule.so", O_RDONLY) = -1
ENOENT (No such file or directory)
open("/b/release/scripts/modules/webbrowser.py", O_RDONLY) = -1 ENOENT
(No such file or directory)
open("/b/release/scripts/modules/webbrowser.pyc", O_RDONLY) = -1
ENOENT (No such file or directory)
open("/opt/py32/lib/python3.2/webbrowser.cpython-32d.so", O_RDONLY) =
-1 ENOENT (No such file or directory)
open("/opt/py32/lib/python3.2/webbrowsermodule.cpython-32d.so",
O_RDONLY) = -1 ENOENT (No such file or directory)
open("/opt/py32/lib/python3.2/webbrowser.abi3.so", O_RDONLY) = -1
ENOENT (No such file or directory)
open("/opt/py32/lib/python3.2/webbrowsermodule.abi3.so", O_RDONLY) =
-1 ENOENT (No such file or directory)
open("/opt/py32/lib/python3.2/webbrowser.so", O_RDONLY) = -1 ENOENT
(No such file or directory)
open("/opt/py32/lib/python3.2/webbrowsermodule.so", O_RDONLY) = -1
ENOENT (No such file or directory)
open("/opt/py32/lib/python3.2/webbrowser.py", O_RDONLY) = 13
open("/opt/py32/lib/python3.2/__pycache__/webbrowser.cpython-32.pyc",
O_RDONLY) = 14



On Sun, Feb 27, 2011 at 3:07 PM, Martin Poirier <theeth at yahoo.com> wrote:
> I think you're missing a lot of data by concentrating on cold starts, a situation that is very much not representative of the general cases.
>
> The speed gain for non-cold starts is much more interesting, IMHO.
>
> Also, could you do tests with Py 3.1 vs 3.2? Baring differences in hardware that should be in your advantage (if I'm reading this right), there's no good reason to explain the difference in timings between you and Ton. If there's an big measurable slowdown in 3.2, it might be worth it to log a bug upstream.
>
> I think there's some really nice numbers we could squeeze out of this with a bit more rigorous method (different OS, stable conditions, same py version, ...).
>
> Good work digging into this, even though the payoff might appear small.
>
> Martin
>
> --- On Sun, 2/27/11, Campbell Barton <ideasman42 at gmail.com> wrote:
>
>> From: Campbell Barton <ideasman42 at gmail.com>
>> Subject: Re: [Bf-committers] Blender Startup Time
>> To: "bf-blender developers" <bf-committers at blender.org>
>> Received: Sunday, February 27, 2011, 8:23 AM
>> Some more tests and found lazy
>> loading of modules in other areas,
>> gives good speedup on cold start.
>>
>> Times don't compare to last tests because I disabled some
>> options
>> (BGE, FFMPEG etc).
>> I did lazy importing of modules:
>>   traceback, shutil, time, math and pydoc
>>   I also removed 'collections' import as well as
>> netrender. *
>>
>> Results for "time ./blender -b" after flushing disk cache,
>> Normal: 8.923 sec
>> Lazy Loading: 4.647s
>>
>> * note, removing collections import means we would need to
>> use a
>> different ordered dict metaclass, I'm not worried about
>> this since we
>> don't use many of the OrderedDict features, we could have a
>> cut down
>> version.
>
>
>
> _______________________________________________
> Bf-committers mailing list
> Bf-committers at blender.org
> http://lists.blender.org/mailman/listinfo/bf-committers
>



-- 
- Campbell