[Bf-committers] Import, research on speed improvements

Mon Mar 10 00:42:57 CET 2014

I have been praying for a faster obj loader for years!
I use massive objs on a regular basis (up to 900mb) so this would be 
amazing. IMHO all I/O plugins should be done in C/C++ as python just 
doesn't have the speed.
FWIW, Blender takes about 30mins to export a 31.5m poly mesh while 
ZBrush takes around 2 mins for the same mesh.

Cheers,

-Andy
On 09/03/2014 19:21, Fredrik hansson wrote:
> hi,
> i
>   have recently written a obj loader myself just for fun. now i did try
> to
> optimize it quite a bit by making it multithreaded and a few other
> things.
> i have put it up on https://github.com/FredrikHson/fast_obj_loader
> now
>   granted this is a much simpler example than doing everything that
> blender does
> with pushing it into blenders internal structure and all that.
> anyway tried it + blenders current importer.
> some stats
> blender total 17.8 sec.
> 8.6sec. parse time
>
> mine:
> sigle threaded 0.6seonds
> multi threaded 0.26sec
>
>   this is on a 36mb obj file on a ssd drive with 8threads.
>
> so yes it could probably benefit from being done in c/c++ the question is still if its worth the trouble
> i personally never import anything much heavier than that 36mb file myself due to
> slow viewport/manipulation speeds after actually getting the file into blender and 18seconds is
> a bit annoying but i don't do it that often that its really much of an issue.
> what is much worse imo is export speeds ( i often forget export selected only) when dealing with
> dense subsurfed meshes.
>
> // Fredrik
>
>
>
>
>
> On Saturday, March 8, 2014 3:30 AM, Sebastian A. Brachi <sebastianbrachi at gmail.com> wrote:
>   
> Hi Paul,
> I'm pretty satisfied with the performance of my importers right now, and
> specially compared to max-script.
> Turned out the major bottleneck was from UV data, since I wasn't using
> foreach_set which is a much efficient method than adding uv data in a loop.
> check
> http://blenderartists.org/forum/showthread.php?321210-Importing-UV-coordinates
> This is for video game data that usually isn't very heavy though (although
> I'm also importing whole levels with hundreds of assets, all in ~40
> seconds),
> and also not using an efficient method for unpacking half-floats (used a
> lot in video games for uvs).
>
> As I said before, I heard users complaining .obj importer performance vs
> zbrush's for example in >500 mb data (sculpts mainly).
> There might be room for improvement there, but consider the topics
> discussed (would be good to rewrite it? is C an better option? can small
> tweaks improve performance considerably?)
>
> Regards
>
>
>
> On Thu, Mar 6, 2014 at 12:26 PM, Paul Melis <paul.melis at surfsara.nl> wrote:
>
>> Hi Sebastian,
>>
>> I read your interesting thread on blender import improvements. Did you
>> make any progress on this topic? Would this be something that a
>> summer-of-code student can work on as well?
>>
>> Regards,
>> Paul
>>
>>
>>
>> On 01/05/14 23:00, Sebastian A. Brachi wrote:
>>
>>> Hi all,
>>> this is the first time writing to the list, I'd like to start learning a
>>> lot and hopefully help to improve blender in some areas.
>>> My main interest right now is Blender's import/export pipeline.
>>> Currently I'm making an addon to import a lot of different formats from
>>> different game engines and also rewriting/improving some existing ones
>>> like
>>> PSK/PSA and MD5.
>>>
>>> I'd like to research the best possible ways to import to blender; my first
>>> concern besides code style is speed in large or medium files, and I have a
>>> couple questions. I've been reading
>>> http://wiki.blender.org/index.php/User:Ideasman42/ImportExport_TODO , and
>>> the idea of using C/C++ modules is very interesting. Here are some
>>> observations and questions for importing,
>>>
>>> (expect some misconceptions):
>>>
>>>
>>> 1) Python file reading/parsing.
>>>
>>>     *Seems fast enough to me in binary data, even with processing many
>>> GB's. I
>>> haven't tested XML/text data though (seems many users complain of the obj
>>> importer, but where are the main bottlenecks is unknown to me)
>>>      Also best practices for reading seems to improve the speed (see 2))
>>>
>>>     Q: C + Ctypes doesn't seem very good since we don't want to rewrite the
>>> reading part in C if it's done in python, and compile dlls or OS with the
>>> addons, right? But if someone
>>>        like me would do it, does it seem like the best option because we can
>>> still keep the mesh building or other data handling in more readable
>>> python?
>>>
>>>     Q: Is it worth to investigate Cython/pypy for speed improvements here
>>> and
>>> was it used in past addons pre 2.5?
>>>        I haven't done any tests so far and I'd like to know opinions first,
>>> haven't found more than a couple threads in the list that mention it, and
>>> a
>>> few in BA:
>>>        e.g.,
>>> http://blenderartists.org/forum/showthread.php?278213-
>>> TEST-Cython-performances-test
>>>
>>>
>>>
>>> 2) Python binary data unpack (struct module).
>>>
>>>      * This seems to add a lot of overhead, specially in big files. Also
>>> best
>>> practices allow for some speedups (like the use of struct.Struct or
>>> list.extend).
>>>      Besides the main document in the API for best practices with a few tips
>>> in string handling when importing, I couldn't fine much info.
>>>
>>>      Q: Is it worth to start/modify a document for best practices, and also
>>> add benchmarks? Who could I ask to review it?
>>>
>>>      * What if Blender could accept/interpret python bytes objects to build
>>> geometry, avoiding the unpacking in python? E.g., reading a face index
>>> buffer all at once, and just passing the count,
>>>      type (short in most cases) and bytes object to Blender. In case of
>>> Vertex
>>> buffer seems more complicated, since many parameters need to be defined,
>>> such as stride, type of each element,
>>>      ignore vertex normals if included, etc.
>>>
>>>      Q: Would this be a reasonable option to investigate, or even possible
>>> to
>>> do?
>>>
>>>      * Another bottleneck is when binary data uses half-floats (used
>>> extensively in game formats for UV data).
>>>        The struct module doesn't have support for them (there is a patch
>>> though: http://bugs.python.org/issue11734), so a python
>>>        function must be used instead. I'm using this one:
>>> http://davidejones.com/blog/1413-python-precision-floating-point/ which
>>> is
>>> pretty slow.
>>>
>>>      Q: Is the python function optimal? I couldn't find better ones.
>>>      Q: If not, is it feasible to do one of the following? :
>>>         a) Implement the patch mentioned above in our blender python
>>>         b) Create the function in C and expose it to the python API
>>>      Q: If b) is the best option, do you think is an OK task for me, as a
>>> first approach to Blender's C code? (no much experience in C besides
>>> college first year)
>>>
>>>
>>> 3) Python data to blender data (converting lists to C arrays with
>>> mesh.vertices.add, mesh.polygons.add, etc):
>>>
>>>      *I've been doing a lot of tests but not much digging into C code. I
>>> don't
>>> seem to understand the process very well.
>>>       Using ctypes arrays, or array.array don't seem to have any performance
>>> improvement when used  in reading the file (it's actually a little slower)
>>> nor building the mesh.
>>>
>>>      Q: When using python data to import geometry, the python objects need
>>> to
>>> be converted to C arrays and that's the overhead right?
>>>      Q: When using c-like objects in python like ctypes array or
>>> array.array,
>>> they still need to be converted to C arrays, so that's why performance is
>>> not improved?
>>>      Q: What could be done to avoid the conversion without using C?
>>> Something
>>> like passing a pointer to a ctypes array instead?
>>>
>>>
>>> Thanks!
>>> Regards
>>> _______________________________________________
>>> Bf-committers mailing list
>>> Bf-committers at blender.org
>>> http://lists.blender.org/mailman/listinfo/bf-committers
>>>
>>
> _______________________________________________
> Bf-committers mailing list
> Bf-committers at blender.org
> http://lists.blender.org/mailman/listinfo/bf-committers
> _______________________________________________
> Bf-committers mailing list
> Bf-committers at blender.org
> http://lists.blender.org/mailman/listinfo/bf-committers
>