[Bf-committers] sculpt mode wireframe display issue and question

[David Jeske]

I'm trying to fix a bug in sculpt mode display and could use some input...

User problem...

With edges enabled but draw-all-edges disabled, edges between coplanar
faces are hidden. In sculpt-mode, as the user-sculpts (no dyntopo), the
co-planar status of edges is not re-evaluated. This means that as edges
which started hidden remain hidden even as the sculpt brush causes them to
no-longer be coplanar. --- This is pretty undesirable, because the user
can't see their work without flipping to edit mode and back.

Obviously the user can turn on draw-all-edges.

If that's the only way sculpt mode visuals work, it probably makes sense to
just force draw-all-edges during static sculpt -- like is already done in
weight-paint, and dyntopo sculpt.

If that's a good enough fix, I'll make a patch, and you can all skip the
rest of this email.

If not, read on, as I could use some advice..

Why it happens and possible fixes...

Sculpt mode (no-dyntopo) draws object wireframes with the same rules as
object-mode. This means if wireframe is enabled, but draw-all-edges is not,
it uses the ME_EDGEDRAW status to show/hide each edge. This (basically)
means that coplanar edges are hidden (see bmesh_quick_edgedraw_flag).
However, as far as I can see ME_EDGEDRAW is only setup during bulk-convert
of a BMesh into a Mesh for display.

In theory, I think the "best" solution would be to recalc the ME_EDGEDRAW
flag status only for edges of vertices which are moved by a static sculpt
brush. However, if I'm reading the code correctly, there isn't (currently)
any quick way to correlate the BMEdge edge structure with the MEdge entry
that contains the flag... -- am I reading that right?

If I got that right, some paths to fix this might be...

1) have static-sculpt build a BMEdge to MEdge hashmap so it can efficiently
update edge-draw flags for moved edges and their neighbors (more memory
usage, but should be very little performance hit during sculpt)

2) force recalc of the MEdge list when co-planar edges are moved. (only
costs something when coplanar edges are moved, but that cost is
proportional to mesh size so it could be large)

3) ... and of course, as I mentioned above -- force draw-all-edges during
sculpting (more draw cost to draw all edges, depending on model)

Thoughts?