[Bf-committers] import_obj.py - Reimplementation

Isaac Lenton isaac at isuniversal.com
Fri Jan 4 15:06:30 CET 2013 

I just built `soc-2012-bratwurst` and had a chance to take a look at the
new assimp and FBX importers.  I wasn't aware that "FBX Importer" also
included assimp (but I am very pleased it does).

Using the assimp importer I can reduce the import time of my 200MB OBJ file
from just over 2 minutes down to 36 seconds.  Watching the memory usage and
processor usage during import I see that assimp runs a single process and
uses slightly more memory then blenders python script.  I think with the
current memory usage you might still run into problems trying to import
a 3.5GB OBJ file (even with 32GB of memory).  I don't have 32GB of memory
handy, but extrapolating the quick data I just collected I would think
about 2.5GB to 3.0GB would be a conservative upper file size limit.

I strongly agree with the line: "IO scripts should be kept simple
IMO, so that scripters can learn from them for the own addons."
The main reason I learnt python 3 years ago was because of Blender's
add-ons.  One of my major learning resources where Blender's existing
scripts (easy to read, efficient and functional).  The changes that I
had made to the script where not easy to read, and potentially very
difficult to bug fix and extend.


Thanks,
Isaac




________________________________
 From: Kursad Karatas <gonderici at plecxus.com>
To: bf-blender developers <bf-committers at blender.org> 
Sent: Friday, 4 January 2013 3:33 AM
Subject: Re: [Bf-committers] import_obj.py - Reimplementation
 
The mesh is around 3.5gb single mesh obj.  Blender puffed up all the way 
to 26.8gb in ram and stopped loading  after sometime. I have 32gb here 
so that should be ok. In any case 26.8 gb for this mesh sounds a bit 
excessive to me.

Keeping the vert order definitely helped. Blender finally showed some 
signs of life after  sometime. But I was not able to interact with 
Blender after that point (massive slowdown due to mesh size I guess) . 
Maybe that is the best it can do at the moment. Just load it. Once it 
loaded the mesh the memory use went down to 11.08 gb although Blender is 
not in usable interaction state once that mesh was in the viewport.

thanks

k




On 01/03/2013 10:58 AM, CoDEmanX wrote:
> OBJ import can take quite some time and use a lot memory, but only if
> you use splitting ("keep vert order" imports way faster). Is there
> something else causing the bad performance?
>
> Multiprocessing would be great, for splitting it might work. But
> probably not for any code that calls bpy api functions...
>
> C/C++ importer: This year's GSoC project "FBX importer" wasn't actually
> about FBX, but assimp integration (= C/C++ importers for FBX, OBJ etc.).
> There are builds available at graphicall.org. However, I'd keep the
> python addons in trunk.
>
> Python + Lib: Why improve performance of a python script if a C/C++
> importer is available (assimp) and faster than the script will ever be?
> In addition, I/O scripts should be kept simple IMO, so that scripters
> can learn from them for their own addons.
>
> If there are any simple python hacks to improve performance, they should
> be applied of course.
>
>
> Am 03.01.2013 03:16, schrieb Isaac Lenton:
>> Hi
>>
>> A few days ago I was using Blender's import_obj.py script to import a few
>> rather bulky OBJ files into blender (200MB and above).  On my little system
>> it takes about 2 minutes for every file.  I have been looking into improving
>> this script with simple optimisation, splitting the work amongst multiple
>> workers and changing the amount of memory the script uses.  Some of these
>> methods are successful, but the results are only minor.  I did manage to
>> reduce the loading time of the 200MB file to 1 minute 40 seconds.
>>
>> Although a quicker method for importing my data into Blender would be to
>> use another format, I quite like the OBJ format and I am "in the mood" for
>> optimising.
>>
>> In order to significantly improve the speed of Blender's OBJ importer I can
>> see a couple of options:
>>
>>       - Implement the importer straight in C/C++ and build into Blender.
>>           I think this option is probably undesirable considering it takes
>>           away one of the really nice things about Blender's add-ons (the
>>           add-on part!).
>>
>>       - Keep the implementation in pure python.
>>           Also, in my opinion, undesirable.  The Python is great, but it can
>>           slither only so fast.  (Also, what are your opinions on using Python's
>>           multiprocessing module in import scripts?)
>>
>>       - Keep the python script but implement the most expensive methods in
>>           a C dynamic library.  This option will keep the interface to the
>>           exporter exactly the same, the module will still be able to be
>>           imported using pythons `import` and Blender will not even notice
>>           the difference.  The library would be placed in the same directory
>>           as the script (scripts/addons/io_scene_obj) and the script would
>>           simply add the line `from . import TheNewLibrary` at the start.
>>           Downside: using plain C and python can be painful (but fun).
>>
>>       - Keep the python script but implement the most expensive methods in
>>           a boost::python C++ dynamic library.  Same as above.
>>           Downside: Boost::Python libraries have a big overhead.
>>
>> I find for my own code, when size is not important, I prefer the last option.
>>
>> I am interested to hear any opinions on the above ideas, and any ideas anyone
>> may have on generally improving the speed of Blender's importers.  Are any
>> of these options Blender compatible (as in, could they be included in the
>> official release)?
>>
>> Thanks,
>> Isaac Lenton
>> (ilent2)
>> _______________________________________________
>> Bf-committers mailing list
>> Bf-committers at blender.org
>> http://lists.blender.org/mailman/listinfo/bf-committers
>>
> _______________________________________________
> Bf-committers mailing list
> Bf-committers at blender.org
> http://lists.blender.org/mailman/listinfo/bf-committers
>


-- 
Kursad Karatas
http://portfolio.plecxus.com/

_______________________________________________
Bf-committers mailing list
Bf-committers at blender.org
http://lists.blender.org/mailman/listinfo/bf-committers

More information about the Bf-committers mailing list