[Bf-committers] Import, research on speed improvements
metalliandy
metalliandy666 at googlemail.com
Mon Mar 10 05:50:17 CET 2014
I'm sure I can get something to you, mate. Be aware that the files are
often 900mb+ though
I will upload something tomorrow.
Cheers,
-Andy
On 10/03/2014 03:12, Sebastian A. Brachi wrote:
> Andy, could you provide a link to an example to do some benchmarks?
> I'd prefer to work with real user cases than a super-subdivided suzzane.
>
>
> On Sun, Mar 9, 2014 at 8:42 PM, metalliandy
> <metalliandy666 at googlemail.com>wrote:
>
>> I have been praying for a faster obj loader for years!
>> I use massive objs on a regular basis (up to 900mb) so this would be
>> amazing. IMHO all I/O plugins should be done in C/C++ as python just
>> doesn't have the speed.
>> FWIW, Blender takes about 30mins to export a 31.5m poly mesh while
>> ZBrush takes around 2 mins for the same mesh.
>>
>> Cheers,
>>
>> -Andy
>> On 09/03/2014 19:21, Fredrik hansson wrote:
>>> hi,
>>> i
>>> have recently written a obj loader myself just for fun. now i did try
>>> to
>>> optimize it quite a bit by making it multithreaded and a few other
>>> things.
>>> i have put it up on https://github.com/FredrikHson/fast_obj_loader
>>> now
>>> granted this is a much simpler example than doing everything that
>>> blender does
>>> with pushing it into blenders internal structure and all that.
>>> anyway tried it + blenders current importer.
>>> some stats
>>> blender total 17.8 sec.
>>> 8.6sec. parse time
>>>
>>> mine:
>>> sigle threaded 0.6seonds
>>> multi threaded 0.26sec
>>>
>>> this is on a 36mb obj file on a ssd drive with 8threads.
>>>
>>> so yes it could probably benefit from being done in c/c++ the question
>> is still if its worth the trouble
>>> i personally never import anything much heavier than that 36mb file
>> myself due to
>>> slow viewport/manipulation speeds after actually getting the file into
>> blender and 18seconds is
>>> a bit annoying but i don't do it that often that its really much of an
>> issue.
>>> what is much worse imo is export speeds ( i often forget export selected
>> only) when dealing with
>>> dense subsurfed meshes.
>>>
>>> // Fredrik
>>>
>>>
>>>
>>>
>>>
>>> On Saturday, March 8, 2014 3:30 AM, Sebastian A. Brachi <
>> sebastianbrachi at gmail.com> wrote:
>>> Hi Paul,
>>> I'm pretty satisfied with the performance of my importers right now, and
>>> specially compared to max-script.
>>> Turned out the major bottleneck was from UV data, since I wasn't using
>>> foreach_set which is a much efficient method than adding uv data in a
>> loop.
>>> check
>>>
>> http://blenderartists.org/forum/showthread.php?321210-Importing-UV-coordinates
>>> This is for video game data that usually isn't very heavy though
>> (although
>>> I'm also importing whole levels with hundreds of assets, all in ~40
>>> seconds),
>>> and also not using an efficient method for unpacking half-floats (used a
>>> lot in video games for uvs).
>>>
>>> As I said before, I heard users complaining .obj importer performance vs
>>> zbrush's for example in >500 mb data (sculpts mainly).
>>> There might be room for improvement there, but consider the topics
>>> discussed (would be good to rewrite it? is C an better option? can small
>>> tweaks improve performance considerably?)
>>>
>>> Regards
>>>
>>>
>>>
>>> On Thu, Mar 6, 2014 at 12:26 PM, Paul Melis <paul.melis at surfsara.nl>
>> wrote:
>>>> Hi Sebastian,
>>>>
>>>> I read your interesting thread on blender import improvements. Did you
>>>> make any progress on this topic? Would this be something that a
>>>> summer-of-code student can work on as well?
>>>>
>>>> Regards,
>>>> Paul
>>>>
>>>>
>>>>
>>>> On 01/05/14 23:00, Sebastian A. Brachi wrote:
>>>>
>>>>> Hi all,
>>>>> this is the first time writing to the list, I'd like to start learning
>> a
>>>>> lot and hopefully help to improve blender in some areas.
>>>>> My main interest right now is Blender's import/export pipeline.
>>>>> Currently I'm making an addon to import a lot of different formats from
>>>>> different game engines and also rewriting/improving some existing ones
>>>>> like
>>>>> PSK/PSA and MD5.
>>>>>
>>>>> I'd like to research the best possible ways to import to blender; my
>> first
>>>>> concern besides code style is speed in large or medium files, and I
>> have a
>>>>> couple questions. I've been reading
>>>>> http://wiki.blender.org/index.php/User:Ideasman42/ImportExport_TODO ,
>> and
>>>>> the idea of using C/C++ modules is very interesting. Here are some
>>>>> observations and questions for importing,
>>>>>
>>>>> (expect some misconceptions):
>>>>>
>>>>>
>>>>> 1) Python file reading/parsing.
>>>>>
>>>>> *Seems fast enough to me in binary data, even with processing many
>>>>> GB's. I
>>>>> haven't tested XML/text data though (seems many users complain of the
>> obj
>>>>> importer, but where are the main bottlenecks is unknown to me)
>>>>> Also best practices for reading seems to improve the speed (see
>> 2))
>>>>> Q: C + Ctypes doesn't seem very good since we don't want to
>> rewrite the
>>>>> reading part in C if it's done in python, and compile dlls or OS with
>> the
>>>>> addons, right? But if someone
>>>>> like me would do it, does it seem like the best option because
>> we can
>>>>> still keep the mesh building or other data handling in more readable
>>>>> python?
>>>>>
>>>>> Q: Is it worth to investigate Cython/pypy for speed improvements
>> here
>>>>> and
>>>>> was it used in past addons pre 2.5?
>>>>> I haven't done any tests so far and I'd like to know opinions
>> first,
>>>>> haven't found more than a couple threads in the list that mention it,
>> and
>>>>> a
>>>>> few in BA:
>>>>> e.g.,
>>>>> http://blenderartists.org/forum/showthread.php?278213-
>>>>> TEST-Cython-performances-test
>>>>>
>>>>>
>>>>>
>>>>> 2) Python binary data unpack (struct module).
>>>>>
>>>>> * This seems to add a lot of overhead, specially in big files.
>> Also
>>>>> best
>>>>> practices allow for some speedups (like the use of struct.Struct or
>>>>> list.extend).
>>>>> Besides the main document in the API for best practices with a
>> few tips
>>>>> in string handling when importing, I couldn't fine much info.
>>>>>
>>>>> Q: Is it worth to start/modify a document for best practices, and
>> also
>>>>> add benchmarks? Who could I ask to review it?
>>>>>
>>>>> * What if Blender could accept/interpret python bytes objects to
>> build
>>>>> geometry, avoiding the unpacking in python? E.g., reading a face index
>>>>> buffer all at once, and just passing the count,
>>>>> type (short in most cases) and bytes object to Blender. In case of
>>>>> Vertex
>>>>> buffer seems more complicated, since many parameters need to be
>> defined,
>>>>> such as stride, type of each element,
>>>>> ignore vertex normals if included, etc.
>>>>>
>>>>> Q: Would this be a reasonable option to investigate, or even
>> possible
>>>>> to
>>>>> do?
>>>>>
>>>>> * Another bottleneck is when binary data uses half-floats (used
>>>>> extensively in game formats for UV data).
>>>>> The struct module doesn't have support for them (there is a
>> patch
>>>>> though: http://bugs.python.org/issue11734), so a python
>>>>> function must be used instead. I'm using this one:
>>>>> http://davidejones.com/blog/1413-python-precision-floating-point/which
>>>>> is
>>>>> pretty slow.
>>>>>
>>>>> Q: Is the python function optimal? I couldn't find better ones.
>>>>> Q: If not, is it feasible to do one of the following? :
>>>>> a) Implement the patch mentioned above in our blender python
>>>>> b) Create the function in C and expose it to the python API
>>>>> Q: If b) is the best option, do you think is an OK task for me,
>> as a
>>>>> first approach to Blender's C code? (no much experience in C besides
>>>>> college first year)
>>>>>
>>>>>
>>>>> 3) Python data to blender data (converting lists to C arrays with
>>>>> mesh.vertices.add, mesh.polygons.add, etc):
>>>>>
>>>>> *I've been doing a lot of tests but not much digging into C code.
>> I
>>>>> don't
>>>>> seem to understand the process very well.
>>>>> Using ctypes arrays, or array.array don't seem to have any
>> performance
>>>>> improvement when used in reading the file (it's actually a little
>> slower)
>>>>> nor building the mesh.
>>>>>
>>>>> Q: When using python data to import geometry, the python objects
>> need
>>>>> to
>>>>> be converted to C arrays and that's the overhead right?
>>>>> Q: When using c-like objects in python like ctypes array or
>>>>> array.array,
>>>>> they still need to be converted to C arrays, so that's why performance
>> is
>>>>> not improved?
>>>>> Q: What could be done to avoid the conversion without using C?
>>>>> Something
>>>>> like passing a pointer to a ctypes array instead?
>>>>>
>>>>>
>>>>> Thanks!
>>>>> Regards
>>>>> _______________________________________________
>>>>> Bf-committers mailing list
>>>>> Bf-committers at blender.org
>>>>> http://lists.blender.org/mailman/listinfo/bf-committers
>>>>>
>>> _______________________________________________
>>> Bf-committers mailing list
>>> Bf-committers at blender.org
>>> http://lists.blender.org/mailman/listinfo/bf-committers
>>> _______________________________________________
>>> Bf-committers mailing list
>>> Bf-committers at blender.org
>>> http://lists.blender.org/mailman/listinfo/bf-committers
>>>
>> _______________________________________________
>> Bf-committers mailing list
>> Bf-committers at blender.org
>> http://lists.blender.org/mailman/listinfo/bf-committers
>>
> _______________________________________________
> Bf-committers mailing list
> Bf-committers at blender.org
> http://lists.blender.org/mailman/listinfo/bf-committers
>
More information about the Bf-committers
mailing list