[Bf-committers] Recent Change in Blender File Format

homac homac at strace.org
Fri Nov 18 16:55:53 CET 2016


Hello blender devs!


I have noticed a change in blender file format, which has some 
consequences, which I would like to discuss. Because the topic is a bit 
difficult to explain, I wrote this email. I'm also sitting in 
#blendercoders.

This change has no consequences for blender now but it may have for 
future versions of blender or thirdparty applications using .blend files.

In terms of LoC, this change is tiny, and therefore I thought there is a 
realistic chance to convince you to change it again or accept a patch, I 
could write.


The Change
==========
Each block in a .blend file contains a copy of a unique area from heap 
memory. The start address stored in its block header is used to identify 
the block (and thereby the memory area) referenced by a pointer.

Thus, a .blend file represented parts of the heap from the virtual 
address space of a previous run of Blender. This implicitly meant, that 
memory areas in the file could not overlap each other (they where disjunct).

Some data structures such as trees or node networks, which have elements 
spread over several different areas in the heap, need special care: 
Traditionally, those structures where for example copied (packed) into 
one continuous memory area (e.g. an array) which is temporarily 
allocated on the heap before written to the block. To prevent malloc 
from assigning the same address to another temporarily allocated buffer, 
the buffer is deallocated after the entire file was written. That way, 
the fundamental assumption, that blocks are disjunct from each other, 
still holds.

Actually, Blender just needs the start address of a block to be unique, 
because pointers, which have to be resolved while reading a file, always 
reference start addresses only. Thus, recently (see [1]) the code for 
writing a tree structure was changed in a way which guarantees just that:

	The start address of a block is unique, but its content may
         (virtually) overlap other blocks.

In this case, the start address is taken from the area inside the 
preceding block, and thus guaranteed to be unique. This is a neat trick 
to circumvent the deferred deallocation of the temporary buffer after 
writing the file but it changes the concept.


Consequence
===========

A pointer, referencing an element inside of a block, such as a member of 
a struct, cannot be resolved without knowledge about the structure which 
contains the member.

For example, in the old model it was possible to map the file in memory 
and fill an address table with the start addresses and length of the 
blocks and its new address in heap. Based on this address table, any 
pointer could be mapped to its new address by identifying the block 
which contains this address.

This concept allowed for a very lazy method to read a file, just picking 
the data needed. It also allowed for a generic method which is almost 
independent from changes in blenders read and write functions.

With the new model, this is not possible, because blocks can overlap and 
addresses may be ambiguous.


Regards
  Holger Machens


[1] https://developer.blender.org/rBb98b331d04fe3a335cc0656809b4b09196507500


More information about the Bf-committers mailing list