[Bf-committers] point cache

[Matt Ebb]

Hi,

>  The state of the physics point cache is pretty bad at the moment. There's various issues. Part of those are bugs that need to be fixed, but a part of the problem is also that in the design some issues seem to have been overlooked or not defined well.

I totally agree, it's a major problem, especially with particles. It
seems at a glance (and from this discussion here) that much of the
design of the system was concentrated on technical issues such as
frame advancing, and not really considering wider workflow and use
cases, which brings us to the situation we have here.

I don't know if I can offer any solutions, but I can mention some
thoughts, and some of the problems that I've found that I haven't
mentioned already [1].

A philosophy I like is "make simple things simple, and complicated
things possible". It seems like the point cache system was designed
for the most complicated scenario, and we're forced to endure the
consequences no matter how simple the sim we're using. I personally
think that there are a few main things that should be kept in mind for
the entire implementation:

* For simple tasks, an artist shouldn't need to even know a point
cache even exists at all. It should just work.

* At *no* point should an artist ever see data that's out of date,
unless the artist has specifically and explicitly asked for this (by
explicitly telling Blender to cache, or protect, or whatever)

Personally I think the whole idea of generating external point cache
files should only be done as an additional option. It should just save
the data internal to memory / the .blend file by default. While a
point cache is probably necessary if you're doing cloth sims on 300k
face meshes, the majority of things that artists do with
particles/soft bodies/etc are a lot simpler. I have a lot of point
cache folders lying around, and most of the time they're only a few
hundred KB in size at the most.

Having external point cache files brings in a whole lot of extra
administrative things to worry about and spend time massaging the
system rather than getting work done. Some things that I've run in to
are:

* It's easy to miss the point cache folder when moving .blend files
around, since it's often in a different place in the file list in
Finder/Explorer/etc. Many artists won't even understand what it's for,
and what the significance of it is, especially with the .blend file
format being so good for keeping data self-contained.

* After working for a while, point cache files get littered all over
the system. If you delete an object with (eg.) a particle system, the
point cache folder doesn't get deleted too. When working, my
co-workers and I save a *lot* of versions, throughout the stages of
the project - it's a good backup, and it's very useful to be able to
get data from older versions when clients change their mind. Since a
new point cache folder is created for each file name, you end up with
lots of redundant point cache folders everywhere. This is even more of
a pain when our auto-synced backup system keeps adding the old point
cache folders to backup too.

* It's annoying to work with for our render farm. This isn't really an
issue specific to Blender, but a symptom of the extra file management
that's forced upon us. We're using 'Farmerjoe', a render farm manager
that copies the .blend file of each job submitted to a temporary
staging area, where the frames are subsequently rendered to (good for
knowing exactly what .blend file rendered exactly what frames). Having
the point cache outside the .blend file makes much more work, making
sure that the correct point cache is copied over to the right
location.

I think most of these problems would go away if the sim data was just
stored in the .blend.

But if you're doing something heavy that demands external point cache
files, Blender's still a pain to work with. For complex work you need
a lot of control that doesn't exist now such as:

* Defining a method/schema for where in the file system the files get
saved to / searched for - different people have different filing
setups (version control systems, network file systems, different
drives for different texture/geometry/etc data, backup regimes) and
just putting the point caches in the same folder as the .blend isn't
always the best thing to do.

* Being able to explicitly load, save, import, export object's point
caches to files, and re-map the paths the Blender is looking in (like
you can in the outliner for linked libraries).

* Defining a local/world co-ordinates, so you've got a better chance
of manipulating already cached animation

* Defining interpolation methods - linear? bspline? etc?

* Defining how the point cache is read/written. i.e. loaded per frame?
pre-loaded for the entire time line? loaded explicitly? keep it in
memory for smooth repeat playback?

It's quite annoying when playing back looped animations involving a
point cache, there's a stutter at frame 1, I presume when it's
re-loading the point cache to memory even though there's plenty of
memory available, and nothing has changed. Also, I'm not an expert on
this, but I think that different ways of storing the files would be
faster in different contexts (i.e. a single binary file per timeline
vs lots of small files, with the overhead of network drives etc)

Another system I have a bit of familiarity with is the Point Cache
modifier in 3ds max [2] - there though, the point cache only gets
created explicitly when you tell it to. This isn't so convenient if
you want to keep changing things, but the good thing is that you
always know what it's doing, since it's explicit. You know where it's
saving the files, there's no weird stuff going on behind the scenes
that might change the cache - it's a level of certainty that Blender
is really missing right now. One other cool thing about the Point
Cache modifier in max is that since it's a modifier, you can use it to
cache whatever's below it in the stack - i.e. you can cache complex
armature deformation (i.e. crowds) to speed it up in playback.

I mentioned previously, the lack of certainty about what you're
actually looking at in the 3D View is a big problem, and leads to a
general feeling of flakiness and not being able to trust Blender. This
is really nasty, I want my tools to feel 100% reliable, and I want to
know that what I see is what I will get. Despite or regardless of the
other problems, better feedback about the state of the cache would
help a lot here. Fusion has a good method for this [3] - it displays a
green bar in the timeline under frames that are cached and up to date,
and blank if the frames are not cached.

Anyway, I know I've offered mainly complaints here, rather than too
many suggestions. I hope it's valuable feedback though, and I hope
that these sorts of workflow issues can be thought of and fixed on a
higher level than just the mechanisms of the current system.

cheers

Matt



[1] http://lists.blender.org/pipermail/bf-committers/2008-January/020149.html
[2] http://www.kxcad.net/autodesk/3ds_max/Autodesk_3ds_Max_9_Reference/point_cache_modifier.html
/ http://www.oman3d.com/tutorials/3ds/flag_aziz/
[3] http://www.vfxpedia.com/index.php?title=Eyeon:Manual/Previews_and_Final_Renders#RAM_Cache_Previews