[Bf-committers] Recent Change in Blender File Format

Bastien Montagne montagne29 at wanadoo.fr
Fri Nov 18 17:08:24 CET 2016

Well… I’d say your 'nice trick' to read .blend file was actually already 
wrong, outliner tree is not the only guy needing special pointer magic 
during writefile time, other areas (aware at least about some mesh data) 
need it too, so you could still have overlapping 'virtual memory areas' 
across whole file.

Further more, there is absolutely nothing saying .blend file should 
ensure a virtual non-overlapping memory map, we are only using pointers 
because it is (in most case) the easiest and cheapest way to get uuids 
for our data blocks, that’s all.

Never a good idea to rely on an implementation-specific behavior in file 
format area, only rely on written specs (though .blend format is not 
that well documented either).

My two cents anyway,

Le 18/11/2016 à 16:55, homac a écrit :
> Hello blender devs!
> I have noticed a change in blender file format, which has some
> consequences, which I would like to discuss. Because the topic is a bit
> difficult to explain, I wrote this email. I'm also sitting in
> #blendercoders.
> This change has no consequences for blender now but it may have for
> future versions of blender or thirdparty applications using .blend files.
> In terms of LoC, this change is tiny, and therefore I thought there is a
> realistic chance to convince you to change it again or accept a patch, I
> could write.
> The Change
> ==========
> Each block in a .blend file contains a copy of a unique area from heap
> memory. The start address stored in its block header is used to identify
> the block (and thereby the memory area) referenced by a pointer.
> Thus, a .blend file represented parts of the heap from the virtual
> address space of a previous run of Blender. This implicitly meant, that
> memory areas in the file could not overlap each other (they where disjunct).
> Some data structures such as trees or node networks, which have elements
> spread over several different areas in the heap, need special care:
> Traditionally, those structures where for example copied (packed) into
> one continuous memory area (e.g. an array) which is temporarily
> allocated on the heap before written to the block. To prevent malloc
> from assigning the same address to another temporarily allocated buffer,
> the buffer is deallocated after the entire file was written. That way,
> the fundamental assumption, that blocks are disjunct from each other,
> still holds.
> Actually, Blender just needs the start address of a block to be unique,
> because pointers, which have to be resolved while reading a file, always
> reference start addresses only. Thus, recently (see [1]) the code for
> writing a tree structure was changed in a way which guarantees just that:
> 	The start address of a block is unique, but its content may
>           (virtually) overlap other blocks.
> In this case, the start address is taken from the area inside the
> preceding block, and thus guaranteed to be unique. This is a neat trick
> to circumvent the deferred deallocation of the temporary buffer after
> writing the file but it changes the concept.
> Consequence
> ===========
> A pointer, referencing an element inside of a block, such as a member of
> a struct, cannot be resolved without knowledge about the structure which
> contains the member.
> For example, in the old model it was possible to map the file in memory
> and fill an address table with the start addresses and length of the
> blocks and its new address in heap. Based on this address table, any
> pointer could be mapped to its new address by identifying the block
> which contains this address.
> This concept allowed for a very lazy method to read a file, just picking
> the data needed. It also allowed for a generic method which is almost
> independent from changes in blenders read and write functions.
> With the new model, this is not possible, because blocks can overlap and
> addresses may be ambiguous.
> Regards
>    Holger Machens
> [1] https://developer.blender.org/rBb98b331d04fe3a335cc0656809b4b09196507500
> _______________________________________________
> Bf-committers mailing list
> Bf-committers at blender.org
> https://lists.blender.org/mailman/listinfo/bf-committers

More information about the Bf-committers mailing list