[Bf-committers] point cache

[Roland Hess]

I have to give a big second to what Matt said. Having tried to use cloth 
on my current animation project, I've found that while the sim itself is 
very easy and simple to set up, it is the larger workflow issues and 
general sense of "flakiness" that made me have to take another route. 
I've also found myself wondering why such small (in kb) folders of cache 
files can't be packed into the .blend, especially when I'm sending stuff 
to my render farm. Sure, I can copy the cache folders to the proper 
directory on my farm, because it's my own, but not everyone is going to 
have that kind of access.

Two things that have hampered my workflow in general with cloth, SBs and 
particles are a lack of pre-roll capability and the need to simulate 
every frame. Clearly, the fact that the sim system is tied to frame 
advance with Alt-A is a major factor in both of these. If you're 
thinking about implementing a "Bake" function, then pre-roll (allowing 
negative frame values) should certainly be considered. In addition, in 
many cases (and especially when you're still searching for your correct 
initial settings) it would be beneficial and more than good enough to 
calculate the sim on, say every other frame (or every 3rd, 5th, etc.) 
then interpolate. You lose accuracy and gain speed, but for many 
applications, that's exactly what you want.

> I totally agree, it's a major problem, especially with particles. It
> seems at a glance (and from this discussion here) that much of the
> design of the system was concentrated on technical issues such as
> frame advancing, and not really considering wider workflow and use
> cases, which brings us to the situation we have here.
> 
> I don't know if I can offer any solutions, but I can mention some
> thoughts, and some of the problems that I've found that I haven't
> mentioned already [1].
> 
> A philosophy I like is "make simple things simple, and complicated
> things possible". It seems like the point cache system was designed
> for the most complicated scenario, and we're forced to endure the
> consequences no matter how simple the sim we're using. I personally
> think that there are a few main things that should be kept in mind for
> the entire implementation:
> 
> * For simple tasks, an artist shouldn't need to even know a point
> cache even exists at all. It should just work.
> 
> * At *no* point should an artist ever see data that's out of date,
> unless the artist has specifically and explicitly asked for this (by
> explicitly telling Blender to cache, or protect, or whatever)
> 
> Personally I think the whole idea of generating external point cache
> files should only be done as an additional option. It should just save
> the data internal to memory / the .blend file by default. While a
> point cache is probably necessary if you're doing cloth sims on 300k
> face meshes, the majority of things that artists do with
> particles/soft bodies/etc are a lot simpler. I have a lot of point
> cache folders lying around, and most of the time they're only a few
> hundred KB in size at the most.
> 
> Having external point cache files brings in a whole lot of extra
> administrative things to worry about and spend time massaging the
> system rather than getting work done. Some things that I've run in to
> are:
> 
> * It's easy to miss the point cache folder when moving .blend files
> around, since it's often in a different place in the file list in
> Finder/Explorer/etc. Many artists won't even understand what it's for,
> and what the significance of it is, especially with the .blend file
> format being so good for keeping data self-contained.
> 
> * After working for a while, point cache files get littered all over
> the system. If you delete an object with (eg.) a particle system, the
> point cache folder doesn't get deleted too. When working, my
> co-workers and I save a *lot* of versions, throughout the stages of
> the project - it's a good backup, and it's very useful to be able to
> get data from older versions when clients change their mind. Since a
> new point cache folder is created for each file name, you end up with
> lots of redundant point cache folders everywhere. This is even more of
> a pain when our auto-synced backup system keeps adding the old point
> cache folders to backup too.
> 
> * It's annoying to work with for our render farm. This isn't really an
> issue specific to Blender, but a symptom of the extra file management
> that's forced upon us. We're using 'Farmerjoe', a render farm manager
> that copies the .blend file of each job submitted to a temporary
> staging area, where the frames are subsequently rendered to (good for
> knowing exactly what .blend file rendered exactly what frames). Having
> the point cache outside the .blend file makes much more work, making
> sure that the correct point cache is copied over to the right
> location.
> 
> I think most of these problems would go away if the sim data was just
> stored in the .blend.
> 
> But if you're doing something heavy that demands external point cache
> files, Blender's still a pain to work with. For complex work you need
> a lot of control that doesn't exist now such as:
> 
> * Defining a method/schema for where in the file system the files get
> saved to / searched for - different people have different filing
> setups (version control systems, network file systems, different
> drives for different texture/geometry/etc data, backup regimes) and
> just putting the point caches in the same folder as the .blend isn't
> always the best thing to do.
> 
> * Being able to explicitly load, save, import, export object's point
> caches to files, and re-map the paths the Blender is looking in (like
> you can in the outliner for linked libraries).
> 
> * Defining a local/world co-ordinates, so you've got a better chance
> of manipulating already cached animation
> 
> * Defining interpolation methods - linear? bspline? etc?
> 
> * Defining how the point cache is read/written. i.e. loaded per frame?
> pre-loaded for the entire time line? loaded explicitly? keep it in
> memory for smooth repeat playback?
> 
> It's quite annoying when playing back looped animations involving a
> point cache, there's a stutter at frame 1, I presume when it's
> re-loading the point cache to memory even though there's plenty of
> memory available, and nothing has changed. Also, I'm not an expert on
> this, but I think that different ways of storing the files would be
> faster in different contexts (i.e. a single binary file per timeline
> vs lots of small files, with the overhead of network drives etc)
> 
> Another system I have a bit of familiarity with is the Point Cache
> modifier in 3ds max [2] - there though, the point cache only gets
> created explicitly when you tell it to. This isn't so convenient if
> you want to keep changing things, but the good thing is that you
> always know what it's doing, since it's explicit. You know where it's
> saving the files, there's no weird stuff going on behind the scenes
> that might change the cache - it's a level of certainty that Blender
> is really missing right now. One other cool thing about the Point
> Cache modifier in max is that since it's a modifier, you can use it to
> cache whatever's below it in the stack - i.e. you can cache complex
> armature deformation (i.e. crowds) to speed it up in playback.
> 
> I mentioned previously, the lack of certainty about what you're
> actually looking at in the 3D View is a big problem, and leads to a
> general feeling of flakiness and not being able to trust Blender. This
> is really nasty, I want my tools to feel 100% reliable, and I want to
> know that what I see is what I will get. Despite or regardless of the
> other problems, better feedback about the state of the cache would
> help a lot here. Fusion has a good method for this [3] - it displays a
> green bar in the timeline under frames that are cached and up to date,
> and blank if the frames are not cached.
> 
> Anyway, I know I've offered mainly complaints here, rather than too
> many suggestions. I hope it's valuable feedback though, and I hope
> that these sorts of workflow issues can be thought of and fixed on a
> higher level than just the mechanisms of the current system.
> 
> cheers
> 
> Matt



-- 
Roland Hess
IT Manager and Digital Prepress
Reed & Witting Company
Pittsburgh, PA
412-682-1000