[Bf-codereview] Sculpt masking (issue 6135048)

[NicholasBishop at gmail.com]

Reviewers: bf-codereview_blender.org,

Description:
This patch implements a mask layer for sculpt mode. Masking attenuates
other sculpt brushes.

Documentation:
http://wiki.blender.org/index.php/User:Nicholasbishop/PaintMasks

Branch on github:
https://github.com/nicholasbishop/blender-bishop/tree/masking

I'm appending the commit log here, hopefully gives useful explanations
of the changes:

commit addeb52

     Code cleanup for parameters of subsurf_make_derived_from_derived.

     Replaced four boolean parameters with a single flag and a new enum,
     SubsurfFlags.

commit 440c385

     Code cleanup for multires_dm_create_from_derived().

     Changed name to multires_make_derived_from_derived() and parameter
     order to be more similar to subsurf_make_derived_from_derived().

     Added MultiresFlags enum with flag values to replace the local_mmd
and
     useRenderParams parameters.

commit db383bb

     Add an RNA access function to get an enum item name from its value.

     New function is RNA_enum_name_from_value.

commit 6d74d04

     Add new options to PAINT_OT_brush_select, toggle and create_missing.

     The toggle option, if enabled, will toggle back and forth between
two
     brushes. (The first brush of the desired tool type will be toggled
to,
     running the toggle again switches back to the previously selected
     brush.)

     If no brush of the desired type is found, and the create_missing
     option is enabled, a new brush of that type will be created and set.

commit 0c36a12

     Modify CCGSubsurf to subdivide an arbitrary number of (float)
layers.

     The layout of vert data in CCGSubSurf is almost the same; previously
     it was three floats (for xyz coordinate) optionally followed by
three
     floats for the normal. The only change is that the first three
floats
     can now be any number of floats.

     * _getSubSurf takes a numLayers parameter to set the number of
layers,
       stored in CCGMeshIFC.numLayers.

     * All calls to _getSubSurf currently have numLayers set to 3, except
       for UV subsurf, where it is reduced to 2 (with a corresponding
       change when reading the results out to use float (*)[2] rather
than
       float (*)[3].)

     * The various VertData* macros in CCGSubSurf.c are now functions
that
       take a CCGSubSurf pointer, which provides access to CCGMeshIFC,
       which has numLayers.

     * Add ccgSubSurf_setNumLayers() to the API. Only changes the number
of
       layers that get subdivided, doesn't change the amount of memory
       allocated. So if space for N layers is allocated, it's safe to set
       the number of layers to less than N, but not more.

     * The rest of the changes are just adding the 'ss' parameter.

commit 8569e47

     Add CCGKey/CCGElem for accessing CCGSubSurf elements.

     CCGKey caches information about the CCGSubSurf element layout. This
     data, along with the CCG_* inline functions, allows access to
     CCGSubSurf elements with an arbitrary number of layers (as opposed
to
     the hardcoded DMGridData structure which assumes xyz coordinates
     followed by three normal components.)

     The CCGElem structure is declared but not defined anywhere, just
used
     as a convenient type.

commit e1c9b8c

     Replace hardcoded DMGridData structure with CCGElem/CCGKey.

     * Changes to DerivedMesh interface: DMGridData has been removed,
       getGridData() now returns an array of CCGElem pointers. Also added
       getGridKey() to initialize a CCGKey (implemented only by
       CCGDerivedMesh.)

     * PBVH: added BLI_pbvh_get_grid_key().

     * A lot of code is affected, but mainly is just replacing
       DMGridData.co, DMGridData.no, and sizeof(DMGridData) with the
       CCG_*_elem functions, removing the reliance on grid elements of
       exactly six floats.

commit ef319c0

     Add DNA and customdata entries for paint masks.

     CD_PAINT_MASK is a layer of per-vertex floats for non-multires
     meshes. Multires meshes use CD_GRID_PAINT_MASK, which is a layer of
     per-loop GridPaintMask structures. GridPaintMask is similar to
MDisp,
     but contains an array of scalar floats.

     Note: the GridPaintMask could be folded into MDisp, but this way
     should be easier to add mask layers in the future (if we do fold
     GridPaintMask into MDisp, the mask array should probably be an array
     of arrays with a 'totmask' field so that mask layers can be easily
     supported.)

     Includes blenload read/write support for CD_PAINT_MASK and
     CD_GRID_PAINT_MASK.

commit 1663d60

     Add access to mesh vertex customdata to the PBVH.

commit 84f4de4

     Add paint mask access to the PBVH vertex iterator.

commit e51abec

     Add GridPaintMask accessor to paint.c.

commit 93e0537

     Add mask support to CCGSubSurf and multires.

     * Add new CCG function ccgSubSurf_setAllocMask(). Similar to to
       ccgSubSurf_setCalcVertexNormals(), it sets whether the CCG
elements
       have a mask layer and what that layer's offset is. Unlike normals
       however, it doesn't change any behavior during CCG calculation;
it's
       there only to give CCGKey information on the mask.

     * Add a new flag to _getSubSurf(), CCG_ALLOC_MASK. If set, space for
       an extra layer is allocated, but the number of CCG layers is not
set
       to include it. This is done because GridPaintMasks are absolute,
       rather than being relative to the subdivided output (as MDisp
       displacements are), so we skip subdividing paint masks here.

     * Add a new flag to subsurf_make_derived_from_derived(),
       SUBSURF_ALLOC_PAINT_MASK. This controls whether CCG_ALLOC_MASK is
       set for _getSubSurf(). Related, masks are never loaded in during
       ss_sync_from_derivedmesh(). After subdivision is finished, if the
       alloc mask flag is set, the number of CCG layers is increase to 4
       with ccgSubSurf_setNumLayers().

     * Add a new flag to multires_make_from_derived(),
       MULTIRES_ALLOC_PAINT_MASK. Not all multires functions need paint
       mask data (e.g. multiresModifier_base_apply.) This flag is always
       set in MOD_multires.c so that subdividing a mesh with a mask
updates
       properly even when not in sculpt mode.

     * Update multiresModifier_disp_run() to apply, calculate, and add
mask
       elements. It's almost the same as the existing operations with xyz
       coordinates, but treats masks as absolute rather than
displacements
       relative to subdivided values.

     * Update multires_customdata_delete to free CD_GRID_PAINT_MASK in
       addition to CD_MDISPS.

     * Update multires_del_higher() to call the new function
       multires_grid_paint_mask_downsample(), which allocates a
       lower-resolution paint mask grid and copies values over from the
       high-resolution grid.

commit cc8addc

     Copy GridPaintMask to vertex paint mask when applying multires.

     Adds new subsurf_copy_grid_paint_mask() function similar to
     subsurf_copy_grid_hidden().

commit 4945e0b

     Ensure mask layers are always present in sculpt mode.

commit f29b5b2

     Add undo/redo support for paint masks.

commit e6a7082

     Use paint mask when calculating sculpt strength.

commit 03b47bf

     Add new mask-brush icon from Julio Iglesias.

commit ec2835b

     Add mask brush for sculpt mode.

     The mask brush currently has two modes, 'draw' and 'smooth'.

commit ed796af

     Update the keymap for the mask brush.

     * Add MKEY as a toggle for the mask brush. We could use ALT similar
to
       SHIFT toggling the smooth brush, but it would conflict with MMB
       emulation (not to mention many window managers.)

     * When the mask brush is active, SHIFT toggles it into smooth mode.

commit cb6a3c0

     Add a paint mask operator to clear, fill, or invert the mask.

commit dde5522

     Add support for hiding masked regions.

     Add a new mode, PARTIALVIS_MASKED, to the PAINT_OT_hide_show
operator.

commit c817577

     Add keymap and menu entries for masking.

     * Add CTRL+IKEY to invert the mask.

     * Add ALT+MKEY to clear the mask.

     * Change the 'Hide' menu in sculpt mode to 'Hide/Mask', adds entires
       for clearing, filling, and inverting the mask, as well as hiding
       masked regions.

commit 5190202

     Use VertexBufferFormat for multires VBO.

commit 86e61e6

     Add mask-drawing support to GPU_Buffers.

     * For VBO, add color to the VertexBufferFormat structure as three
       unsigned bytes. Since mask elements are scalar the three color
       components are identical to eachother, but the fixed-function
OpenGL
       pipeline requires colors to be either three or four components.

     * For the same reason, multires VBO drawing now copies into the
       VertexBufferFormat format as well.

     * Regression: material colors will not show up correctly now, masks
       colors are overriding. Not sure how to fix this nicely (would be
       much easier to fix if drawing with vertex shaders.)

     * Also, masks will only draw PBVH drawing, so only 'solid' drawing
       will work correctly with masks.


Please review this at http://codereview.appspot.com/6135048/

Affected files:
   source/blender/blenlib/BLI_ghash.h
   source/blender/bmesh/CMakeLists.txt
   source/blender/bmesh/intern/bmesh_error.h
   source/blender/bmesh/intern/bmesh_opdefines.c
   source/blender/bmesh/intern/bmesh_operators_private.h
   source/blender/bmesh/operators/bmo_hull.c
   source/blender/editors/mesh/editmesh_tools.c
   source/blender/editors/mesh/mesh_intern.h
   source/blender/editors/mesh/mesh_ops.c