[Bf-committers] Import, research on speed improvements

[Campbell Barton]

On Mon, Jan 6, 2014 at 9:00 AM, Sebastian A. Brachi
<sebastianbrachi at gmail.com> wrote:
> Hi all,
> this is the first time writing to the list, I'd like to start learning a
> lot and hopefully help to improve blender in some areas.
> My main interest right now is Blender's import/export pipeline.
> Currently I'm making an addon to import a lot of different formats from
> different game engines and also rewriting/improving some existing ones like
> PSK/PSA and MD5.
>
> I'd like to research the best possible ways to import to blender; my first
> concern besides code style is speed in large or medium files, and I have a
> couple questions. I've been reading
> http://wiki.blender.org/index.php/User:Ideasman42/ImportExport_TODO , and
> the idea of using C/C++ modules is very interesting. Here are some
> observations and questions for importing,
>
> (expect some misconceptions):
>
>
> 1) Python file reading/parsing.
>
>  *Seems fast enough to me in binary data, even with processing many GB's. I
> haven't tested XML/text data though (seems many users complain of the obj
> importer, but where are the main bottlenecks is unknown to me)
>   Also best practices for reading seems to improve the speed (see 2))
>
>  Q: C + Ctypes doesn't seem very good since we don't want to rewrite the
> reading part in C if it's done in python, and compile dlls or OS with the
> addons, right? But if someone
>     like me would do it, does it seem like the best option because we can
> still keep the mesh building or other data handling in more readable
> python?

This is potentially really useful, since it allows speed of C using
Python as glue... but has some drawbacks..

- You need to compile some DLL on every platform you want to deploy
it, generally it complicates anyone getting and using the script.
- communicating with Ctypes can be cumbersome (even more trouble then
its worth in some cases) since you may end up spending CPU time in
python to encode/decode data... this depends a lot on how the CAPI
works for whatever you interface.

>  Q: Is it worth to investigate Cython/pypy for speed improvements here and
> was it used in past addons pre 2.5?
>     I haven't done any tests so far and I'd like to know opinions first,
> haven't found more than a couple threads in the list that mention it, and a
> few in BA:
>     e.g.,
> http://blenderartists.org/forum/showthread.php?278213-TEST-Cython-performances-test

Cython, is just a way to write CPython-API code, so you still have the
problem of binary compatibility, if your OK with that then its easier
then writing CPython-API code directly in C (albeit less flexible - so
depends on your needs).

PyPy - is only usable as an external process at the moment, so you
could pass some data off to PyPy and get it back... otherwise its not
usable with Blender yet (and not even close to being usable - I keep
an eye on their cpython compatibility layer but its very primitive).


> 2) Python binary data unpack (struct module).
>
>   * This seems to add a lot of overhead, specially in big files. Also best
> practices allow for some speedups (like the use of struct.Struct or
> list.extend).
>   Besides the main document in the API for best practices with a few tips
> in string handling when importing, I couldn't fine much info.

Try use `array` module, bytearray, and memoryview's where possible.

As an example using pyfbx, I managed to parse a ~30mb FBX file (with
zlib compressed chunks) in ~0.5sec (cant recall exact details... but
was faster then I expected).

This uses Python arrays see:
https://github.com/ideasman42/pyfbx_i42/blob/master/pyfbx/parse_bin.py

>   Q: Is it worth to start/modify a document for best practices, and also
> add benchmarks? Who could I ask to review it?

Not sure, best practice depends a lot on each use-case, If you wanted
to Im sure others would appreciate, but it may only end up being
relevant to others doing similar scripts to you. Such docs also get
out of date as python changes so... no strong opinion. General Python
optimizations also apply to Blender for the mostpart so not sure its
really worth to write Blender specific docs.

>   * What if Blender could accept/interpret python bytes objects to build
> geometry, avoiding the unpacking in python? E.g., reading a face index
> buffer all at once, and just passing the count,
>   type (short in most cases) and bytes object to Blender. In case of Vertex
> buffer seems more complicated, since many parameters need to be defined,
> such as stride, type of each element,
>   ignore vertex normals if included, etc.

Yes, we could have some way to encode binary data in a way you can
pass off to blender without having to go through RNA or BMesh api's.
Issue here is its very specific to our internal mesh format...

Some generic API could be exposed where you pass in an array for
faces, vertex-coords, uv's, shapekeys, vertex colors... etc... then
the API internally sets up the customdata layers.

>   Q: Would this be a reasonable option to investigate, or even possible to
> do?

Yes, though I worry we already have 2 ways to access meshes (RNA and
BMesh), adding a 3rd for fast access from a language which is by
definition interpreted therefor fairly slow, not sure about this. We
could also investigate ways to include C plugins (or use some LLVM
based language to compiler code which is needs to be fast...).
Speedups here seem like a bit of a half-solution and you still get
speed issues with Python anyway.

>   * Another bottleneck is when binary data uses half-floats (used
> extensively in game formats for UV data).
>     The struct module doesn't have support for them (there is a patch
> though: http://bugs.python.org/issue11734), so a python
>     function must be used instead. I'm using this one:
> http://davidejones.com/blog/1413-python-precision-floating-point/ which is
> pretty slow.

In that case some C utility function could be written to speedup conversion.

>   Q: Is the python function optimal? I couldn't find better ones.
no

>   Q: If not, is it feasible to do one of the following? :
>      a) Implement the patch mentioned above in our blender python
>      b) Create the function in C and expose it to the python API
either can work...

>   Q: If b) is the best option, do you think is an OK task for me, as a
> first approach to Blender's C code? (no much experience in C besides
> college first year)

This is a bit of an odd case since you want to speedup a spesific
operation which may become redundant in the future anyway (if we add
to blender, do we remove once CPython gets it).

It doesnt really fit anywhere in our API, Im not against adding some
module which fills in various short-comings in CPython for use in
Blender, perhaps it can go in `bpy_utils.struct` and be documented as
a stop-gap until CPython implements.

> 3) Python data to blender data (converting lists to C arrays with
> mesh.vertices.add, mesh.polygons.add, etc):
>
>   *I've been doing a lot of tests but not much digging into C code. I don't
> seem to understand the process very well.
>    Using ctypes arrays, or array.array don't seem to have any performance
> improvement when used  in reading the file (it's actually a little slower)
> nor building the mesh.

using foreach_set with arrays should increase performance. if not, you
would have to step over the assignment in a debugger and see exactly
whats going on, also possible bottleneck isnt setting data at all (or
its related - like running RNA update function after data is set).

>   Q: When using python data to import geometry, the python objects need to
> be converted to C arrays and that's the overhead right?
right

>   Q: When using c-like objects in python like ctypes array or array.array,
> they still need to be converted to C arrays, so that's why performance is
> not improved?
it depends, mathutils.Vector for example read from floats direcly when
assigned. We could add a special check for float arrays too.

>   Q: What could be done to avoid the conversion without using C? Something
> like passing a pointer to a ctypes array instead?

With bpy/rna API foreach_set() can speedup assignment too, if you can
use it, try to do so. (its not usable in every instance - depending on
how data access is exposed).

>
> Thanks!
> Regards
> _______________________________________________
> Bf-committers mailing list
> Bf-committers at blender.org
> http://lists.blender.org/mailman/listinfo/bf-committers



-- 
- Campbell