[Bf-committers] Blendfile compression changed to Zstandard

[Lukas Stockner]

Hi all,

Since this is a considerable change to the .blend file format, here's a
mail to make everyone aware of this change: As of yesterday [1], Blender
supports loading .blend files that are compressed using Zstandard and
will save files in this format if "Compressed" is enabled. Of course,
loading gzipped files (the format used before) is still supported.

This doesn't affect you if you only interact with .blend files through
the latest Blender version, but it means that compressed files from
latest builds (and from Blender 3.0+, once released) will not load in
older versions. In order to open them there, you can re-save them
without compression in a recent version or decompress them using
external tools (e.g. "unzstd" on Linux).

Additionally, this means that external tools that open .blend files
(e.g. third-party software importers or pipeline tools) will need to be
updated to support Zstandard compression as well.

The benefit of the new format is that it produces files that have a
similar size to gzipped files, but loading and especially saving are
much faster (roughly -60% and -90%, respectively, for my test files).
For examples, see the release notes [2]. Additionally, Zstandard is
quite flexible - compression levels range from extremely fast (LZ4-like)
to extremely small (LZMA-like). Currently, Blender uses level 3, but it
can load any standard-compliant file - therefore, this level could be
changed (or exposed as a setting) in the future. Additionally, loading
times are mostly independent of compression level, so you can do things
like re-compressing files as part of your pipeline for e.g. network
syncing without affecting usability.

Internally, the format is a Zstandard-wrapped "raw" .blend file, as
before with gzip. However, there is one additional detail: For more
efficient partial reading of linked .blend files, Blender tries to skip
parts of the file. This was not supported by gzip, and it would usually
not be supported by Zstandard. However, since the file contents are
divided into independent sections, we can skip entire sections if we
know the mapping from uncompressed to compressed position. Therefore,
Blender now encodes this mapping into a "skippable frame" - a section of
the file that does not produce any data when decompressed, but can be
used for storing additional info. The format here follows the upstream
seekable Zstandard format [3]. If a file is missing this section (e.g.
because it was compressed manually), it can still be loaded without
issues, but more memory might be required.

- Lukas

[1] https://developer.blender.org/rB2ea66af7
[2] https://wiki.blender.org/wiki/Reference/Release_Notes/3.0/Core
[3]
https://github.com/facebook/zstd/blob/dev/contrib/seekable_format/zstd_seekable_compression_format.md