[Bf-committers] Recent Change in Blender File Format

[Ton Roosendaal]

Hi Holger,

Very cool analysis. Good to see people spend quality time on reviewing internals like this.

I agree with Bastien's remarks though. And I want to add this (as coder of the system):

The 'treestore' is for Outliner only. It saves the state as the UI shows outliner data.
It's not saving anything related to the scene description or relationships.

It's always been a weak design issue here. I really want UIs to be able to save state, but the way we do it could probably be done much better.

-Ton-

--------------------------------------------------------
Ton Roosendaal  -  ton at blender.org   -   www.blender.org
Chairman Blender Foundation, Director Blender Institute
Entrepotdok 57A, 1018 AD, Amsterdam, the Netherlands

> On 18 Nov 2016, at 17:08, Bastien Montagne <montagne29 at wanadoo.fr> wrote:
> 
> Well… I’d say your 'nice trick' to read .blend file was actually already 
> wrong, outliner tree is not the only guy needing special pointer magic 
> during writefile time, other areas (aware at least about some mesh data) 
> need it too, so you could still have overlapping 'virtual memory areas' 
> across whole file.
> 
> Further more, there is absolutely nothing saying .blend file should 
> ensure a virtual non-overlapping memory map, we are only using pointers 
> because it is (in most case) the easiest and cheapest way to get uuids 
> for our data blocks, that’s all.
> 
> Never a good idea to rely on an implementation-specific behavior in file 
> format area, only rely on written specs (though .blend format is not 
> that well documented either).
> 
> My two cents anyway,
> Cheers,
> Bastien
> 
> Le 18/11/2016 à 16:55, homac a écrit :
>> Hello blender devs!
>> 
>> 
>> I have noticed a change in blender file format, which has some
>> consequences, which I would like to discuss. Because the topic is a bit
>> difficult to explain, I wrote this email. I'm also sitting in
>> #blendercoders.
>> 
>> This change has no consequences for blender now but it may have for
>> future versions of blender or thirdparty applications using .blend files.
>> 
>> In terms of LoC, this change is tiny, and therefore I thought there is a
>> realistic chance to convince you to change it again or accept a patch, I
>> could write.
>> 
>> 
>> The Change
>> ==========
>> Each block in a .blend file contains a copy of a unique area from heap
>> memory. The start address stored in its block header is used to identify
>> the block (and thereby the memory area) referenced by a pointer.
>> 
>> Thus, a .blend file represented parts of the heap from the virtual
>> address space of a previous run of Blender. This implicitly meant, that
>> memory areas in the file could not overlap each other (they where disjunct).
>> 
>> Some data structures such as trees or node networks, which have elements
>> spread over several different areas in the heap, need special care:
>> Traditionally, those structures where for example copied (packed) into
>> one continuous memory area (e.g. an array) which is temporarily
>> allocated on the heap before written to the block. To prevent malloc
>> from assigning the same address to another temporarily allocated buffer,
>> the buffer is deallocated after the entire file was written. That way,
>> the fundamental assumption, that blocks are disjunct from each other,
>> still holds.
>> 
>> Actually, Blender just needs the start address of a block to be unique,
>> because pointers, which have to be resolved while reading a file, always
>> reference start addresses only. Thus, recently (see [1]) the code for
>> writing a tree structure was changed in a way which guarantees just that:
>> 
>> 	The start address of a block is unique, but its content may
>>          (virtually) overlap other blocks.
>> 
>> In this case, the start address is taken from the area inside the
>> preceding block, and thus guaranteed to be unique. This is a neat trick
>> to circumvent the deferred deallocation of the temporary buffer after
>> writing the file but it changes the concept.
>> 
>> 
>> Consequence
>> ===========
>> 
>> A pointer, referencing an element inside of a block, such as a member of
>> a struct, cannot be resolved without knowledge about the structure which
>> contains the member.
>> 
>> For example, in the old model it was possible to map the file in memory
>> and fill an address table with the start addresses and length of the
>> blocks and its new address in heap. Based on this address table, any
>> pointer could be mapped to its new address by identifying the block
>> which contains this address.
>> 
>> This concept allowed for a very lazy method to read a file, just picking
>> the data needed. It also allowed for a generic method which is almost
>> independent from changes in blenders read and write functions.
>> 
>> With the new model, this is not possible, because blocks can overlap and
>> addresses may be ambiguous.
>> 
>> 
>> Regards
>>   Holger Machens
>> 
>> 
>> [1] https://developer.blender.org/rBb98b331d04fe3a335cc0656809b4b09196507500
>> _______________________________________________
>> Bf-committers mailing list
>> Bf-committers at blender.org
>> https://lists.blender.org/mailman/listinfo/bf-committers
>> 
> 
> _______________________________________________
> Bf-committers mailing list
> Bf-committers at blender.org
> https://lists.blender.org/mailman/listinfo/bf-committers