[Bf-committers] Recent Change in Blender File Format

homac homac at strace.org
Fri Nov 18 19:58:19 CET 2016


Hello Bastien and Ton!


Thank you for spending your quality time replying to emails!

Unfortunate for me, that I made my design on a false assumption, but 
I'll find a reasonable solution. Also, I will add this information to my 
documentation of the Blender DNA to prevent others from making the same 
mistake.

And Ton, on your statement:

 > It's always been a weak design issue here. I really want UIs to be
 > able to save state, but the way we do it could probably be done much
 > better.

I don't know, if this is meant to be an invitation to make a proposal. I 
actually don't see the issue there -- maybe because I've never looked 
into UI. I see, that there is a difference between persistent data 
stored for UI and that of the scene, since thirdparty tools just need 
the latter. But that's probably not the 'issue' you referred to.


Anyway, thanks for clarifying things.


Regards
  Holger


On 18.11.2016 18:16, Ton Roosendaal wrote:
> Hi Holger,
>
> Very cool analysis. Good to see people spend quality time on reviewing internals like this.
>
> I agree with Bastien's remarks though. And I want to add this (as coder of the system):
>
> The 'treestore' is for Outliner only. It saves the state as the UI shows outliner data.
> It's not saving anything related to the scene description or relationships.
>
> It's always been a weak design issue here. I really want UIs to be able to save state, but the way we do it could probably be done much better.
>
> -Ton-
>
> --------------------------------------------------------
> Ton Roosendaal  -  ton at blender.org   -   www.blender.org
> Chairman Blender Foundation, Director Blender Institute
> Entrepotdok 57A, 1018 AD, Amsterdam, the Netherlands
>
>> On 18 Nov 2016, at 17:08, Bastien Montagne <montagne29 at wanadoo.fr> wrote:
>>
>> Well… I’d say your 'nice trick' to read .blend file was actually already
>> wrong, outliner tree is not the only guy needing special pointer magic
>> during writefile time, other areas (aware at least about some mesh data)
>> need it too, so you could still have overlapping 'virtual memory areas'
>> across whole file.
>>
>> Further more, there is absolutely nothing saying .blend file should
>> ensure a virtual non-overlapping memory map, we are only using pointers
>> because it is (in most case) the easiest and cheapest way to get uuids
>> for our data blocks, that’s all.
>>
>> Never a good idea to rely on an implementation-specific behavior in file
>> format area, only rely on written specs (though .blend format is not
>> that well documented either).
>>
>> My two cents anyway,
>> Cheers,
>> Bastien
>>
>> Le 18/11/2016 à 16:55, homac a écrit :
>>> Hello blender devs!
>>>
>>>
>>> I have noticed a change in blender file format, which has some
>>> consequences, which I would like to discuss. Because the topic is a bit
>>> difficult to explain, I wrote this email. I'm also sitting in
>>> #blendercoders.
>>>
>>> This change has no consequences for blender now but it may have for
>>> future versions of blender or thirdparty applications using .blend files.
>>>
>>> In terms of LoC, this change is tiny, and therefore I thought there is a
>>> realistic chance to convince you to change it again or accept a patch, I
>>> could write.
>>>
>>>
>>> The Change
>>> ==========
>>> Each block in a .blend file contains a copy of a unique area from heap
>>> memory. The start address stored in its block header is used to identify
>>> the block (and thereby the memory area) referenced by a pointer.
>>>
>>> Thus, a .blend file represented parts of the heap from the virtual
>>> address space of a previous run of Blender. This implicitly meant, that
>>> memory areas in the file could not overlap each other (they where disjunct).
>>>
>>> Some data structures such as trees or node networks, which have elements
>>> spread over several different areas in the heap, need special care:
>>> Traditionally, those structures where for example copied (packed) into
>>> one continuous memory area (e.g. an array) which is temporarily
>>> allocated on the heap before written to the block. To prevent malloc
>>> from assigning the same address to another temporarily allocated buffer,
>>> the buffer is deallocated after the entire file was written. That way,
>>> the fundamental assumption, that blocks are disjunct from each other,
>>> still holds.
>>>
>>> Actually, Blender just needs the start address of a block to be unique,
>>> because pointers, which have to be resolved while reading a file, always
>>> reference start addresses only. Thus, recently (see [1]) the code for
>>> writing a tree structure was changed in a way which guarantees just that:
>>>
>>> 	The start address of a block is unique, but its content may
>>>          (virtually) overlap other blocks.
>>>
>>> In this case, the start address is taken from the area inside the
>>> preceding block, and thus guaranteed to be unique. This is a neat trick
>>> to circumvent the deferred deallocation of the temporary buffer after
>>> writing the file but it changes the concept.
>>>
>>>
>>> Consequence
>>> ===========
>>>
>>> A pointer, referencing an element inside of a block, such as a member of
>>> a struct, cannot be resolved without knowledge about the structure which
>>> contains the member.
>>>
>>> For example, in the old model it was possible to map the file in memory
>>> and fill an address table with the start addresses and length of the
>>> blocks and its new address in heap. Based on this address table, any
>>> pointer could be mapped to its new address by identifying the block
>>> which contains this address.
>>>
>>> This concept allowed for a very lazy method to read a file, just picking
>>> the data needed. It also allowed for a generic method which is almost
>>> independent from changes in blenders read and write functions.
>>>
>>> With the new model, this is not possible, because blocks can overlap and
>>> addresses may be ambiguous.
>>>
>>>
>>> Regards
>>>   Holger Machens
>>>
>>>
>>> [1] https://developer.blender.org/rBb98b331d04fe3a335cc0656809b4b09196507500
>>> _______________________________________________
>>> Bf-committers mailing list
>>> Bf-committers at blender.org
>>> https://lists.blender.org/mailman/listinfo/bf-committers
>>>
>>
>> _______________________________________________
>> Bf-committers mailing list
>> Bf-committers at blender.org
>> https://lists.blender.org/mailman/listinfo/bf-committers
>
> _______________________________________________
> Bf-committers mailing list
> Bf-committers at blender.org
> https://lists.blender.org/mailman/listinfo/bf-committers
>



More information about the Bf-committers mailing list