[Bf-committers] Import, research on speed improvements

[Paul Melis]

Hi Sebastian,

I read your interesting thread on blender import improvements. Did you 
make any progress on this topic? Would this be something that a 
summer-of-code student can work on as well?

Regards,
Paul


On 01/05/14 23:00, Sebastian A. Brachi wrote:
> Hi all,
> this is the first time writing to the list, I'd like to start learning a
> lot and hopefully help to improve blender in some areas.
> My main interest right now is Blender's import/export pipeline.
> Currently I'm making an addon to import a lot of different formats from
> different game engines and also rewriting/improving some existing ones like
> PSK/PSA and MD5.
>
> I'd like to research the best possible ways to import to blender; my first
> concern besides code style is speed in large or medium files, and I have a
> couple questions. I've been reading
> http://wiki.blender.org/index.php/User:Ideasman42/ImportExport_TODO , and
> the idea of using C/C++ modules is very interesting. Here are some
> observations and questions for importing,
>
> (expect some misconceptions):
>
>
> 1) Python file reading/parsing.
>
>   *Seems fast enough to me in binary data, even with processing many GB's. I
> haven't tested XML/text data though (seems many users complain of the obj
> importer, but where are the main bottlenecks is unknown to me)
>    Also best practices for reading seems to improve the speed (see 2))
>
>   Q: C + Ctypes doesn't seem very good since we don't want to rewrite the
> reading part in C if it's done in python, and compile dlls or OS with the
> addons, right? But if someone
>      like me would do it, does it seem like the best option because we can
> still keep the mesh building or other data handling in more readable
> python?
>
>   Q: Is it worth to investigate Cython/pypy for speed improvements here and
> was it used in past addons pre 2.5?
>      I haven't done any tests so far and I'd like to know opinions first,
> haven't found more than a couple threads in the list that mention it, and a
> few in BA:
>      e.g.,
> http://blenderartists.org/forum/showthread.php?278213-TEST-Cython-performances-test
>
>
>
> 2) Python binary data unpack (struct module).
>
>    * This seems to add a lot of overhead, specially in big files. Also best
> practices allow for some speedups (like the use of struct.Struct or
> list.extend).
>    Besides the main document in the API for best practices with a few tips
> in string handling when importing, I couldn't fine much info.
>
>    Q: Is it worth to start/modify a document for best practices, and also
> add benchmarks? Who could I ask to review it?
>
>    * What if Blender could accept/interpret python bytes objects to build
> geometry, avoiding the unpacking in python? E.g., reading a face index
> buffer all at once, and just passing the count,
>    type (short in most cases) and bytes object to Blender. In case of Vertex
> buffer seems more complicated, since many parameters need to be defined,
> such as stride, type of each element,
>    ignore vertex normals if included, etc.
>
>    Q: Would this be a reasonable option to investigate, or even possible to
> do?
>
>    * Another bottleneck is when binary data uses half-floats (used
> extensively in game formats for UV data).
>      The struct module doesn't have support for them (there is a patch
> though: http://bugs.python.org/issue11734), so a python
>      function must be used instead. I'm using this one:
> http://davidejones.com/blog/1413-python-precision-floating-point/ which is
> pretty slow.
>
>    Q: Is the python function optimal? I couldn't find better ones.
>    Q: If not, is it feasible to do one of the following? :
>       a) Implement the patch mentioned above in our blender python
>       b) Create the function in C and expose it to the python API
>    Q: If b) is the best option, do you think is an OK task for me, as a
> first approach to Blender's C code? (no much experience in C besides
> college first year)
>
>
> 3) Python data to blender data (converting lists to C arrays with
> mesh.vertices.add, mesh.polygons.add, etc):
>
>    *I've been doing a lot of tests but not much digging into C code. I don't
> seem to understand the process very well.
>     Using ctypes arrays, or array.array don't seem to have any performance
> improvement when used  in reading the file (it's actually a little slower)
> nor building the mesh.
>
>    Q: When using python data to import geometry, the python objects need to
> be converted to C arrays and that's the overhead right?
>    Q: When using c-like objects in python like ctypes array or array.array,
> they still need to be converted to C arrays, so that's why performance is
> not improved?
>    Q: What could be done to avoid the conversion without using C? Something
> like passing a pointer to a ctypes array instead?
>
>
> Thanks!
> Regards
> _______________________________________________
> Bf-committers mailing list
> Bf-committers at blender.org
> http://lists.blender.org/mailman/listinfo/bf-committers