[Bf-committers] Import, research on speed improvements
Martijn Berger
martijn.berger at gmail.com
Mon Mar 10 08:41:54 CET 2014
Hi all,
Just a side note:
You should be careful to compare blenders obj importer to something awesome
that you have written yourself. There is one thing that can be done to
speed it up greatly but that breaks entities that are split ( started and
finished elsewhere ).
Blenders obj importer needs to build a list now and make sure it got
everything before it can do the actual object import. If you convert this
to a generator you save a ton of memory and get a dramatic speedup. All
"normal" obj files will load but some exotics won't. It is little things
like this that make the importer slow also besides being written in python.
And it is little things like this that will take a couple of orders of
magnitude of your speedups if you implement an obj importer that has exact
feature parity.
On Mon, Mar 10, 2014 at 8:05 AM, Fredrik hansson <
fredrikhansson_12345 at yahoo.com> wrote:
> you can always do as i did and grab some of the stanford models they don't
> have uvs/normals but works anyway,
> the xyzrgb_dragon one is 574mb or so and takes 27seconds to load for me or
> 7seconds if the file is in the file cache.
>
> can't say how long blender takes to do the same as it runs out of memory
> and crashes after a few minutes(32bit build)
> it did manage to do the parsing step however 82seconds.
> note that these numbere would be smaller if i was at home haven't got a
> ssd here at work.
>
> as to Dan's question what i do here is not to thread every line but
> instead load say 1mb of the file at a time split that up
> into sections of1-numcores of lines, that i then parse and stitch
> together in the original order since i know what order the chunks are.
>
> the main speedup i found however is to just load the entire file or parts
> of the file into memory and then parse.
> loading the entire thing is if im not mistaken the fastest but uses way to
> much memory while if i load 1mb at a time,
> i actually use less memory than the file size in the end (338mb for the
> xyzrgb dragon)
>
>
> // Fredrik
>
>
>
>
> On Monday, March 10, 2014 5:50 AM, metalliandy <
> metalliandy666 at googlemail.com> wrote:
>
> I'm sure I can get something to you, mate. Be aware that the files are
> often 900mb+ though
>
> I will upload something tomorrow.
>
> Cheers,
>
> -Andy
>
> On 10/03/2014 03:12, Sebastian A. Brachi wrote:
> > Andy, could you provide a link to an example to do some benchmarks?
> > I'd prefer to work with real user cases than a super-subdivided suzzane.
> >
> >
> > On Sun, Mar 9, 2014 at 8:42 PM, metalliandy
> > <metalliandy666 at googlemail.com>wrote:
> >
> >> I have been praying for a faster obj loader for years!
> >> I use massive objs on a regular basis (up to 900mb) so this would be
> >> amazing. IMHO all I/O plugins should be done in C/C++ as python just
> >> doesn't have the speed.
> >> FWIW, Blender takes about 30mins to export a 31.5m poly mesh while
> >> ZBrush takes around 2 mins for the same mesh.
> >>
> >> Cheers,
> >>
> >> -Andy
> >> On 09/03/2014 19:21, Fredrik hansson wrote:
> >>> hi,
> >>> i
> >>> have recently written a obj loader myself just for fun. now i did
> try
> >>> to
> >>> optimize it quite a bit by making it multithreaded and a few other
> >>> things.
> >>> i have put it up on https://github.com/FredrikHson/fast_obj_loader
> >>> now
> >>> granted this is a much simpler example than doing everything that
> >>> blender does
> >>> with pushing it into blenders internal structure and all that.
> >>> anyway tried it + blenders current importer.
> >>> some stats
> >>> blender total 17.8 sec.
> >>> 8.6sec. parse time
> >>>
> >>> mine:
> >>> sigle threaded 0.6seonds
> >>> multi threaded 0.26sec
> >>>
> >>> this is on a 36mb obj file on a ssd drive with 8threads.
> >>>
> >>> so yes it could probably benefit from being done in c/c++ the question
> >> is still if its worth the trouble
> >>> i personally never import anything much heavier than that 36mb file
> >> myself due to
> >>> slow viewport/manipulation speeds after actually getting the file into
> >> blender and 18seconds is
> >>> a bit annoying but i don't do it that often that its really much of an
> >> issue.
> >>> what is much worse imo is export speeds ( i often forget export
> selected
> >> only) when dealing with
> >>> dense subsurfed meshes.
> >>>
> >>> // Fredrik
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Saturday, March 8, 2014 3:30 AM, Sebastian A. Brachi <
> >> sebastianbrachi at gmail.com> wrote:
> >>> Hi Paul,
> >>> I'm pretty satisfied with the performance of my importers right now,
> and
> >>> specially compared to max-script.
> >>> Turned out the major bottleneck was from UV data, since I wasn't using
> >>> foreach_set which is a much efficient method than adding uv data in a
> >> loop.
> >>> check
> >>>
> >>
> http://blenderartists.org/forum/showthread.php?321210-Importing-UV-coordinates
> >>> This is for video game data that usually isn't very heavy though
> >> (although
> >>> I'm also importing whole levels with hundreds of assets, all in ~40
> >>> seconds),
> >>> and also not using an efficient method for unpacking half-floats (used
> a
> >>> lot in video games for uvs).
> >>>
> >>> As I said before, I heard users complaining .obj importer performance
> vs
> >>> zbrush's for example in >500 mb data (sculpts mainly).
> >>> There might be room for improvement there, but consider the topics
> >>> discussed (would be good to rewrite it? is C an better option? can
> small
> >>> tweaks improve performance considerably?)
> >>>
> >>> Regards
> >>>
> >>>
> >>>
> >>> On Thu, Mar 6, 2014 at 12:26 PM, Paul Melis <paul.melis at surfsara.nl>
> >> wrote:
> >>>> Hi Sebastian,
> >>>>
> >>>> I read your interesting thread on blender import improvements. Did you
> >>>> make any progress on this topic? Would this be something that a
> >>>> summer-of-code student can work on as well?
> >>>>
> >>>> Regards,
> >>>> Paul
> >>>>
> >>>>
> >>>>
> >>>> On 01/05/14 23:00, Sebastian A. Brachi wrote:
> >>>>
> >>>>> Hi all,
> >>>>> this is the first time writing to the list, I'd like to start
> learning
> >> a
> >>>>> lot and hopefully help to improve blender in some areas.
> >>>>> My main interest right now is Blender's import/export pipeline.
> >>>>> Currently I'm making an addon to import a lot of different formats
> from
> >>>>> different game engines and also rewriting/improving some existing
> ones
> >>>>> like
> >>>>> PSK/PSA and MD5.
> >>>>>
> >>>>> I'd like to research the best possible ways to import to blender; my
> >> first
> >>>>> concern besides code style is speed in large or medium files, and I
> >> have a
> >>>>> couple questions. I've been reading
> >>>>> http://wiki.blender.org/index.php/User:Ideasman42/ImportExport_TODO,
> >> and
> >>>>> the idea of using C/C++ modules is very interesting. Here are some
> >>>>> observations and questions for importing,
> >>>>>
> >>>>> (expect some misconceptions):
> >>>>>
> >>>>>
> >>>>> 1) Python file reading/parsing.
> >>>>>
> >>>>> *Seems fast enough to me in binary data, even with processing
> many
> >>>>> GB's. I
> >>>>> haven't tested XML/text data though (seems many users complain of the
> >> obj
> >>>>> importer, but where are the main bottlenecks is unknown to me)
> >>>>> Also best practices for reading seems to improve the speed (see
> >> 2))
> >>>>> Q: C + Ctypes doesn't seem very good since we don't want to
> >> rewrite the
> >>>>> reading part in C if it's done in python, and compile dlls or OS with
> >> the
> >>>>> addons, right? But if someone
> >>>>> like me would do it, does it seem like the best option
> because
> >> we can
> >>>>> still keep the mesh building or other data handling in more readable
> >>>>> python?
> >>>>>
> >>>>> Q: Is it worth to investigate Cython/pypy for speed improvements
> >> here
> >>>>> and
> >>>>> was it used in past addons pre 2.5?
> >>>>> I haven't done any tests so far and I'd like to know opinions
> >> first,
> >>>>> haven't found more than a couple threads in the list that mention it,
> >> and
> >>>>> a
> >>>>> few in BA:
> >>>>> e.g.,
> >>>>> http://blenderartists.org/forum/showthread.php?278213-
> >>>>> TEST-Cython-performances-test
> >>>>>
> >>>>>
> >>>>>
> >>>>> 2) Python binary data unpack (struct module).
> >>>>>
> >>>>> * This seems to add a lot of overhead, specially in big files.
> >> Also
> >>>>> best
> >>>>> practices allow for some speedups (like the use of struct.Struct or
> >>>>> list.extend).
> >>>>> Besides the main document in the API for best practices with a
> >> few tips
> >>>>> in string handling when importing, I couldn't fine much info.
> >>>>>
> >>>>> Q: Is it worth to start/modify a document for best practices,
> and
> >> also
> >>>>> add benchmarks? Who could I ask to review it?
> >>>>>
> >>>>> * What if Blender could accept/interpret python bytes objects
> to
> >> build
> >>>>> geometry, avoiding the unpacking in python? E.g., reading a face
> index
> >>>>> buffer all at once, and just passing the count,
> >>>>> type (short in most cases) and bytes object to Blender. In
> case of
> >>>>> Vertex
> >>>>> buffer seems more complicated, since many parameters need to be
> >> defined,
> >>>>> such as stride, type of each element,
> >>>>> ignore vertex normals if included, etc.
> >>>>>
> >>>>> Q: Would this be a reasonable option to investigate, or even
> >> possible
> >>>>> to
> >>>>> do?
> >>>>>
> >>>>> * Another bottleneck is when binary data uses half-floats (used
> >>>>> extensively in game formats for UV data).
> >>>>> The struct module doesn't have support for them (there is a
> >> patch
> >>>>> though: http://bugs.python.org/issue11734), so a python
> >>>>> function must be used instead. I'm using this one:
> >>>>>
> http://davidejones.com/blog/1413-python-precision-floating-point/which
> >>>>> is
> >>>>> pretty slow.
> >>>>>
> >>>>> Q: Is the python function optimal? I couldn't find better ones.
> >>>>> Q: If not, is it feasible to do one of the following? :
> >>>>> a) Implement the patch mentioned above in our blender python
> >>>>> b) Create the function in C and expose it to the python API
> >>>>> Q: If b) is the best option, do you think is an OK task for me,
> >> as a
> >>>>> first approach to Blender's C code? (no much experience in C besides
> >>>>> college first year)
> >>>>>
> >>>>>
> >>>>> 3) Python data to blender data (converting lists to C arrays with
> >>>>> mesh.vertices.add, mesh.polygons.add, etc):
> >>>>>
> >>>>> *I've been doing a lot of tests but not much digging into C
> code.
> >> I
> >>>>> don't
> >>>>> seem to understand the process very well.
> >>>>> Using ctypes arrays, or array.array don't seem to have any
> >> performance
> >>>>> improvement when used in reading the file (it's actually a little
> >> slower)
> >>>>> nor building the mesh.
> >>>>>
> >>>>> Q: When using python data to import geometry, the python
> objects
> >> need
> >>>>> to
> >>>>> be converted to C arrays and that's the overhead right?
> >>>>> Q: When using c-like objects in python like ctypes array or
> >>>>> array.array,
> >>>>> they still need to be converted to C arrays, so that's why
> performance
> >> is
> >>>>> not improved?
> >>>>> Q: What could be done to avoid the conversion without using C?
> >>>>> Something
> >>>>> like passing a pointer to a ctypes array instead?
> >>>>>
> >>>>>
> >>>>> Thanks!
> >>>>> Regards
> >>>>> _______________________________________________
> >>>>> Bf-committers mailing list
> >>>>> Bf-committers at blender.org
> >>>>> http://lists.blender.org/mailman/listinfo/bf-committers
> >>>>>
> >>> _______________________________________________
> >>> Bf-committers mailing list
> >>> Bf-committers at blender.org
> >>> http://lists.blender.org/mailman/listinfo/bf-committers
> >>> _______________________________________________
> >>> Bf-committers mailing list
> >>> Bf-committers at blender.org
> >>> http://lists.blender.org/mailman/listinfo/bf-committers
> >>>
> >> _______________________________________________
> >> Bf-committers mailing list
> >> Bf-committers at blender.org
> >> http://lists.blender.org/mailman/listinfo/bf-committers
> >>
> > _______________________________________________
> > Bf-committers mailing list
> > Bf-committers at blender.org
> > http://lists.blender.org/mailman/listinfo/bf-committers
> >
>
> _______________________________________________
> Bf-committers mailing list
> Bf-committers at blender.org
> http://lists.blender.org/mailman/listinfo/bf-committers
> _______________________________________________
> Bf-committers mailing list
> Bf-committers at blender.org
> http://lists.blender.org/mailman/listinfo/bf-committers
>
More information about the Bf-committers
mailing list