[Bf-committers] Clay Tool Bug

[Jason Wilkins]

I ran into this while re-factoring sculpt.

The clay tool in sculpt is supposed to obey the 'accumulate' flag.
This flag tells sculpt if it should base the brush location on the
original mesh before a stroke started (Accumulate Off) or use the new
position of the mesh that it gets during the stroke (Accumulate On).
Subjectively, accumulate 'off' looks like a brush has less and less
effect as you use it on the same spot without starting a new stroke,
sort of like how painting over the same area over and over doesn't
change the color as much.  You can build up a sculpt in this mode if
you release and restart your stroke in the same spot, but not if you
just hold down the button and keep going over the same area.  With
accumulate 'on' the sculpt will just build higher and higher without
releasing the button but it is harder to get a smooth result.

An option like 'Accumulate' is needed because unlike adding or
blending a color, where it will eventually saturate to 1 or converge
(respectively), a sculpting displacement can be applied indefinitely
with continued effect.  Setting accumulate to 'off' lets sculpting
simulate this saturation to some degree (could possibly be better),
but it isn't obvious that this is always what should be done, because
sculpting isn't painting.

The bug is that clay is partially ignoring the Accumulate setting.
When Accumulate is 'off' the brush location is correctly placed on the
original mesh position, and the displacement vector is correctly
determined from the original surface, however the amount that the clay
brush raises the surface is computed from the new positions when it
should come from the original positions.  The result is that even with
Accumulate 'off', the brush still looks kind of like Accumulate is
'on', but because it is still using the original brush position and
normals it simply isn't doing the right thing.

I would just fix this, but I am unsure because somebody may really
like how clay works now and by changing it to be 'correct' they will
feel that I broke it.  However, I do think it should be fixed.  The
old behavior can be mostly gotten back by turning 'Accumulate On' for
clay, and it will accumulate correctly instead of in the potentially
broken way it is now.  The subjective strength of clay with
'Accumulate Off' will seem weaker after this fix, but it will also be
possible to get much smoother results because it won't build up on top
of itself, allowing you to fill in gaps without applying too much
clay.

I've attached a patch, if anybody would like to test the difference
just try un-patched and patched blender with accumulate turned on and
off for the clay tool and tell me what you think.