[Bf-committers] Import, research on speed improvements
Sebastian A. Brachi
sebastianbrachi at gmail.com
Mon Mar 10 04:12:17 CET 2014
Andy, could you provide a link to an example to do some benchmarks?
I'd prefer to work with real user cases than a super-subdivided suzzane.
On Sun, Mar 9, 2014 at 8:42 PM, metalliandy
<metalliandy666 at googlemail.com>wrote:
> I have been praying for a faster obj loader for years!
> I use massive objs on a regular basis (up to 900mb) so this would be
> amazing. IMHO all I/O plugins should be done in C/C++ as python just
> doesn't have the speed.
> FWIW, Blender takes about 30mins to export a 31.5m poly mesh while
> ZBrush takes around 2 mins for the same mesh.
>
> Cheers,
>
> -Andy
> On 09/03/2014 19:21, Fredrik hansson wrote:
> > hi,
> > i
> > have recently written a obj loader myself just for fun. now i did try
> > to
> > optimize it quite a bit by making it multithreaded and a few other
> > things.
> > i have put it up on https://github.com/FredrikHson/fast_obj_loader
> > now
> > granted this is a much simpler example than doing everything that
> > blender does
> > with pushing it into blenders internal structure and all that.
> > anyway tried it + blenders current importer.
> > some stats
> > blender total 17.8 sec.
> > 8.6sec. parse time
> >
> > mine:
> > sigle threaded 0.6seonds
> > multi threaded 0.26sec
> >
> > this is on a 36mb obj file on a ssd drive with 8threads.
> >
> > so yes it could probably benefit from being done in c/c++ the question
> is still if its worth the trouble
> > i personally never import anything much heavier than that 36mb file
> myself due to
> > slow viewport/manipulation speeds after actually getting the file into
> blender and 18seconds is
> > a bit annoying but i don't do it that often that its really much of an
> issue.
> > what is much worse imo is export speeds ( i often forget export selected
> only) when dealing with
> > dense subsurfed meshes.
> >
> > // Fredrik
> >
> >
> >
> >
> >
> > On Saturday, March 8, 2014 3:30 AM, Sebastian A. Brachi <
> sebastianbrachi at gmail.com> wrote:
> >
> > Hi Paul,
> > I'm pretty satisfied with the performance of my importers right now, and
> > specially compared to max-script.
> > Turned out the major bottleneck was from UV data, since I wasn't using
> > foreach_set which is a much efficient method than adding uv data in a
> loop.
> > check
> >
> http://blenderartists.org/forum/showthread.php?321210-Importing-UV-coordinates
> > This is for video game data that usually isn't very heavy though
> (although
> > I'm also importing whole levels with hundreds of assets, all in ~40
> > seconds),
> > and also not using an efficient method for unpacking half-floats (used a
> > lot in video games for uvs).
> >
> > As I said before, I heard users complaining .obj importer performance vs
> > zbrush's for example in >500 mb data (sculpts mainly).
> > There might be room for improvement there, but consider the topics
> > discussed (would be good to rewrite it? is C an better option? can small
> > tweaks improve performance considerably?)
> >
> > Regards
> >
> >
> >
> > On Thu, Mar 6, 2014 at 12:26 PM, Paul Melis <paul.melis at surfsara.nl>
> wrote:
> >
> >> Hi Sebastian,
> >>
> >> I read your interesting thread on blender import improvements. Did you
> >> make any progress on this topic? Would this be something that a
> >> summer-of-code student can work on as well?
> >>
> >> Regards,
> >> Paul
> >>
> >>
> >>
> >> On 01/05/14 23:00, Sebastian A. Brachi wrote:
> >>
> >>> Hi all,
> >>> this is the first time writing to the list, I'd like to start learning
> a
> >>> lot and hopefully help to improve blender in some areas.
> >>> My main interest right now is Blender's import/export pipeline.
> >>> Currently I'm making an addon to import a lot of different formats from
> >>> different game engines and also rewriting/improving some existing ones
> >>> like
> >>> PSK/PSA and MD5.
> >>>
> >>> I'd like to research the best possible ways to import to blender; my
> first
> >>> concern besides code style is speed in large or medium files, and I
> have a
> >>> couple questions. I've been reading
> >>> http://wiki.blender.org/index.php/User:Ideasman42/ImportExport_TODO ,
> and
> >>> the idea of using C/C++ modules is very interesting. Here are some
> >>> observations and questions for importing,
> >>>
> >>> (expect some misconceptions):
> >>>
> >>>
> >>> 1) Python file reading/parsing.
> >>>
> >>> *Seems fast enough to me in binary data, even with processing many
> >>> GB's. I
> >>> haven't tested XML/text data though (seems many users complain of the
> obj
> >>> importer, but where are the main bottlenecks is unknown to me)
> >>> Also best practices for reading seems to improve the speed (see
> 2))
> >>>
> >>> Q: C + Ctypes doesn't seem very good since we don't want to
> rewrite the
> >>> reading part in C if it's done in python, and compile dlls or OS with
> the
> >>> addons, right? But if someone
> >>> like me would do it, does it seem like the best option because
> we can
> >>> still keep the mesh building or other data handling in more readable
> >>> python?
> >>>
> >>> Q: Is it worth to investigate Cython/pypy for speed improvements
> here
> >>> and
> >>> was it used in past addons pre 2.5?
> >>> I haven't done any tests so far and I'd like to know opinions
> first,
> >>> haven't found more than a couple threads in the list that mention it,
> and
> >>> a
> >>> few in BA:
> >>> e.g.,
> >>> http://blenderartists.org/forum/showthread.php?278213-
> >>> TEST-Cython-performances-test
> >>>
> >>>
> >>>
> >>> 2) Python binary data unpack (struct module).
> >>>
> >>> * This seems to add a lot of overhead, specially in big files.
> Also
> >>> best
> >>> practices allow for some speedups (like the use of struct.Struct or
> >>> list.extend).
> >>> Besides the main document in the API for best practices with a
> few tips
> >>> in string handling when importing, I couldn't fine much info.
> >>>
> >>> Q: Is it worth to start/modify a document for best practices, and
> also
> >>> add benchmarks? Who could I ask to review it?
> >>>
> >>> * What if Blender could accept/interpret python bytes objects to
> build
> >>> geometry, avoiding the unpacking in python? E.g., reading a face index
> >>> buffer all at once, and just passing the count,
> >>> type (short in most cases) and bytes object to Blender. In case of
> >>> Vertex
> >>> buffer seems more complicated, since many parameters need to be
> defined,
> >>> such as stride, type of each element,
> >>> ignore vertex normals if included, etc.
> >>>
> >>> Q: Would this be a reasonable option to investigate, or even
> possible
> >>> to
> >>> do?
> >>>
> >>> * Another bottleneck is when binary data uses half-floats (used
> >>> extensively in game formats for UV data).
> >>> The struct module doesn't have support for them (there is a
> patch
> >>> though: http://bugs.python.org/issue11734), so a python
> >>> function must be used instead. I'm using this one:
> >>> http://davidejones.com/blog/1413-python-precision-floating-point/which
> >>> is
> >>> pretty slow.
> >>>
> >>> Q: Is the python function optimal? I couldn't find better ones.
> >>> Q: If not, is it feasible to do one of the following? :
> >>> a) Implement the patch mentioned above in our blender python
> >>> b) Create the function in C and expose it to the python API
> >>> Q: If b) is the best option, do you think is an OK task for me,
> as a
> >>> first approach to Blender's C code? (no much experience in C besides
> >>> college first year)
> >>>
> >>>
> >>> 3) Python data to blender data (converting lists to C arrays with
> >>> mesh.vertices.add, mesh.polygons.add, etc):
> >>>
> >>> *I've been doing a lot of tests but not much digging into C code.
> I
> >>> don't
> >>> seem to understand the process very well.
> >>> Using ctypes arrays, or array.array don't seem to have any
> performance
> >>> improvement when used in reading the file (it's actually a little
> slower)
> >>> nor building the mesh.
> >>>
> >>> Q: When using python data to import geometry, the python objects
> need
> >>> to
> >>> be converted to C arrays and that's the overhead right?
> >>> Q: When using c-like objects in python like ctypes array or
> >>> array.array,
> >>> they still need to be converted to C arrays, so that's why performance
> is
> >>> not improved?
> >>> Q: What could be done to avoid the conversion without using C?
> >>> Something
> >>> like passing a pointer to a ctypes array instead?
> >>>
> >>>
> >>> Thanks!
> >>> Regards
> >>> _______________________________________________
> >>> Bf-committers mailing list
> >>> Bf-committers at blender.org
> >>> http://lists.blender.org/mailman/listinfo/bf-committers
> >>>
> >>
> > _______________________________________________
> > Bf-committers mailing list
> > Bf-committers at blender.org
> > http://lists.blender.org/mailman/listinfo/bf-committers
> > _______________________________________________
> > Bf-committers mailing list
> > Bf-committers at blender.org
> > http://lists.blender.org/mailman/listinfo/bf-committers
> >
>
> _______________________________________________
> Bf-committers mailing list
> Bf-committers at blender.org
> http://lists.blender.org/mailman/listinfo/bf-committers
>
More information about the Bf-committers
mailing list