[Bf-python] Bf-python Digest, Vol 133, Issue 9

[James Crowther]

Hi Bastien! Thanks for those details, very useful!

Kind Regards

James

On Sun, Jun 26, 2016 at 8:00 PM, <bf-python-request at blender.org> wrote:

> Send Bf-python mailing list submissions to
>         bf-python at blender.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://lists.blender.org/mailman/listinfo/bf-python
> or, via email, send a message with subject or body 'help' to
>         bf-python-request at blender.org
>
> You can reach the person managing the list at
>         bf-python-owner at blender.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Bf-python digest..."
>
>
> Today's Topics:
>
>    1. Understanding blend file binary data (James Crowther)
>    2. Re: Understanding blend file binary data (Bastien Montagne)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Sat, 25 Jun 2016 21:55:09 +1000
> From: James Crowther <jamesharrycrowther at gmail.com>
> Subject: [Bf-python] Understanding blend file binary data
> To: bf-python at blender.org
> Message-ID:
>         <
> CAJOwpNsZo1zmhMkx_knOkbYwtKe6zone4OTQT8+Jbt1vRdfgog at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi there,
>              I have a need to understand why a blend file can be different
> on disk after saving it despite nothing actually changing in the actual
> scene data, by that I mean D.objects.
>
> >From a cursory look at the differences using a file diff viewer like
> beyond
> compare, I can tell that there are some metadata like the file name that
> are included in the file's binary, so this would of course change for some
> instances where you copy the file for example.
> However, the file name is only one part of the vast number of changes I can
> see.
>
> To reproduce simply open a blend file, save it. Then copy it. Open the copy
> and save it. Do a file diff on the original file and the copy and you will
> see many differences throughout the file.
>
> The use case I am working with here is I am distributing a file across many
> machines to render parts on each. I need to make sure the file is the same
> on each machine so I have some assurance that the file transfer works
> (whatever method the transport uses, LAN, USB stick etc) and that the
> machines all have the same data.
> Originally I was going to do this using something like CRC32 or MD5
> depending on speed and security needs, but since the file changes once it
> is saved, even if the file is not changed, this presents a problem since
> the data of the scene is the same, but the results of a CRC32 or MD5 will
> be different due to the effects of saving a file.
>
> Could someone educate me as to how to read the blend file format in such a
> way as to remove these differences? I'm more than happy to write a custom
> CRC routine so long as I can know which parts of the file contain the data
> and which are headers that are altered on a save.
>
> Thanks!
>
> James
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://lists.blender.org/pipermail/bf-python/attachments/20160625/620fb8b6/attachment-0001.htm
>
> ------------------------------
>
> Message: 2
> Date: Sat, 25 Jun 2016 15:35:53 +0200
> From: Bastien Montagne <montagne29 at wanadoo.fr>
> Subject: Re: [Bf-python] Understanding blend file binary data
> To: bf-python at blender.org
> Message-ID: <3be9f46a-9c8c-cd16-e7ab-9b1ca092f155 at wanadoo.fr>
> Content-Type: text/plain; charset="utf-8"
>
> Hi James,
>
> Think the main (if not the only) serious cause of those differences here
> are the fact that Blender uses memory addresses of its struct as
> 'in-file uuid' in the .blend. So first thing to do for your usecase
> would be to ignore all and every pointers from DNA structs, since those
> will change every time.
>
> As for reading a .blend file, besides blender code itself, suggest you
> have a look at code in io_blend_utils folder in addons repository, not
> yet complete but should give you a good start. You may also want to have
> a look to blend2json.py script (in our blender-dev-tools git repository,
> https://developer.blender.org/diffusion/BDT/).
>
> Cheers,
> Bastien
>
> Le 25/06/2016 ? 13:55, James Crowther a ?crit :
> > Hi there,
> >              I have a need to understand why a blend file can be
> > different on disk after saving it despite nothing actually changing in
> > the actual scene data, by that I mean D.objects.
> >
> > From a cursory look at the differences using a file diff viewer like
> > beyond compare, I can tell that there are some metadata like the file
> > name that are included in the file's binary, so this would of course
> > change for some instances where you copy the file for example.
> > However, the file name is only one part of the vast number of changes
> > I can see.
> >
> > To reproduce simply open a blend file, save it. Then copy it. Open the
> > copy and save it. Do a file diff on the original file and the copy and
> > you will see many differences throughout the file.
> >
> > The use case I am working with here is I am distributing a file across
> > many machines to render parts on each. I need to make sure the file is
> > the same on each machine so I have some assurance that the file
> > transfer works (whatever method the transport uses, LAN, USB stick
> > etc) and that the machines all have the same data.
> > Originally I was going to do this using something like CRC32 or MD5
> > depending on speed and security needs, but since the file changes once
> > it is saved, even if the file is not changed, this presents a problem
> > since the data of the scene is the same, but the results of a CRC32 or
> > MD5 will be different due to the effects of saving a file.
> >
> > Could someone educate me as to how to read the blend file format in
> > such a way as to remove these differences? I'm more than happy to
> > write a custom CRC routine so long as I can know which parts of the
> > file contain the data and which are headers that are altered on a save.
> >
> > Thanks!
> >
> > James
> >
> >
> > _______________________________________________
> > Bf-python mailing list
> > Bf-python at blender.org
> > https://lists.blender.org/mailman/listinfo/bf-python
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://lists.blender.org/pipermail/bf-python/attachments/20160625/89a17332/attachment-0001.htm
>
> ------------------------------
>
> _______________________________________________
> Bf-python mailing list
> Bf-python at blender.org
> https://lists.blender.org/mailman/listinfo/bf-python
>
>
> End of Bf-python Digest, Vol 133, Issue 9
> *****************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.blender.org/pipermail/bf-python/attachments/20160626/7b753053/attachment.html>