[Bf-committers] Recent Change in Blender File Format
homac
homac at strace.org
Fri Nov 18 16:55:53 CET 2016
Hello blender devs!
I have noticed a change in blender file format, which has some
consequences, which I would like to discuss. Because the topic is a bit
difficult to explain, I wrote this email. I'm also sitting in
#blendercoders.
This change has no consequences for blender now but it may have for
future versions of blender or thirdparty applications using .blend files.
In terms of LoC, this change is tiny, and therefore I thought there is a
realistic chance to convince you to change it again or accept a patch, I
could write.
The Change
==========
Each block in a .blend file contains a copy of a unique area from heap
memory. The start address stored in its block header is used to identify
the block (and thereby the memory area) referenced by a pointer.
Thus, a .blend file represented parts of the heap from the virtual
address space of a previous run of Blender. This implicitly meant, that
memory areas in the file could not overlap each other (they where disjunct).
Some data structures such as trees or node networks, which have elements
spread over several different areas in the heap, need special care:
Traditionally, those structures where for example copied (packed) into
one continuous memory area (e.g. an array) which is temporarily
allocated on the heap before written to the block. To prevent malloc
from assigning the same address to another temporarily allocated buffer,
the buffer is deallocated after the entire file was written. That way,
the fundamental assumption, that blocks are disjunct from each other,
still holds.
Actually, Blender just needs the start address of a block to be unique,
because pointers, which have to be resolved while reading a file, always
reference start addresses only. Thus, recently (see [1]) the code for
writing a tree structure was changed in a way which guarantees just that:
The start address of a block is unique, but its content may
(virtually) overlap other blocks.
In this case, the start address is taken from the area inside the
preceding block, and thus guaranteed to be unique. This is a neat trick
to circumvent the deferred deallocation of the temporary buffer after
writing the file but it changes the concept.
Consequence
===========
A pointer, referencing an element inside of a block, such as a member of
a struct, cannot be resolved without knowledge about the structure which
contains the member.
For example, in the old model it was possible to map the file in memory
and fill an address table with the start addresses and length of the
blocks and its new address in heap. Based on this address table, any
pointer could be mapped to its new address by identifying the block
which contains this address.
This concept allowed for a very lazy method to read a file, just picking
the data needed. It also allowed for a generic method which is almost
independent from changes in blenders read and write functions.
With the new model, this is not possible, because blocks can overlap and
addresses may be ambiguous.
Regards
Holger Machens
[1] https://developer.blender.org/rBb98b331d04fe3a335cc0656809b4b09196507500
More information about the Bf-committers
mailing list