[Bf-taskforce25] RNA and API wrap-up

[Elia Sarti]

Hi,
I've been almost absent in the last couple of months (which I apologize 
for) and as I'm trying to make a bit of a summary of current state of 
API related stuff I thought it would be useful to write it down in order 
for everyone to have a more clear idea and make it possible to discuss 
further so that then we can put up something definite and have some 
specs for other things to be settled that depend on this (like 
exporters/importers etc).


Data access and performance

Data access as of now for pyscripts always happens through RNA. The 
advantage of having everything abstracted as always presents the same 
problem, which is performance. While it should work fine to access 
almost any data it would get very slow for things like vertices and in 
general anything which is a huge collection of pieces of data.
Talking a bit on IRC Campbell proposed having a special function 
implemented PYAPI side in the form of obj.foreach_set("name", [array]) 
and same for getting obj.foreach_get("name", [array]). For instance (as 
I remember it) mesh.foreach_get("verts", vertex_array) would get all the 
vertices for the mesh "mesh" and put them into "vertex_array". We could 
also extend this to use RNA paths like mesh.foreach_get("verts.co", 
coords). I'm not sure what the complications of this could be, I think 
trying it out is the best option here. As Campbell said this is 
essential to have importers/exporters working which again is essential 
in order to have Blender working.


Operators design

Operators are more or less defined/complete at least C side. API side 
though we still have things undefined or problematic. For instance an 
issue is unique naming. Although the current naming convention fits 
perfectly and it's cohesive on our core application side it presents 
some limitations for extensibility as we cannot really control how many 
pyoperators will be written and handle collisions between these. A way 
could be simply to always prepend pyops with PY_ (or tell py devs to use 
this convention). We could tell devs to prepend their pyoperators with 
their own names/ids of choice. Or we could use python own module/package 
system to combine the final operator name. As of now I don't seem to see 
any check on operator names? We could allow a 
module.OBJECT_OT_do_something() type of name syntax which could be 
translated automatically from py? Not sure if this conflicts with something.
Another issue for operators in general is context constraints. For 
instance as of now as Ton mentioned in IRC add_object operator works in 
node editor. Is this fine? Can it be expected by the user? From my point 
of view being able to run operators from contexts you wouldn't expect is 
totally acceptable architecture-wise as running constraints are defined 
in terms of "needs" and if requirements are respected I see no reason 
why the operator should not run. They should just be hidden from the 
user in order not to create confusion. I think we could do this with 
some type of "flag" both on operators and space types defining the 
constraints of context. Something like CTX_HANDLE_OBJECT, 
CTX_HANDLE_NODE etc. On the operator this is the same as represented by 
the prefix of its idname so we could actually use this on spaces 
although string comparison would introduce some issues but would be more 
solid e.g. for py/plugin spaces which brings me back to pyops naming as 
using this method would leave module/package approach as the only usable 
naming convention.
You might wonder why not to put the space check directly on the 
operator's poll? Well the reason for this is that as I said poll() 
should just check requirements and not interface issues which is instead 
something spaces take care of. This is not just a design choice but also 
has some practical examples. For instance, the same example I made on 
IRC, suppose at some point Outliner space gets capacity of 
listing/handling nodes. To support node tools in outliner you would have 
to go and change every single node operator to add additional check on 
space type while with the former approach you just update outliner space 
to accept NODE_* operators. This of course is even more clear for 
python/plugin spaces.


Type registration

This is less important and more of a personal question than general issue.
Types can get registered for adding customization like it happens for 
Panels or Menus. If I'm not mistaken this is how current architecture works:

script.py -> class SomePanel(Panel): [...] bpy.types.register(SomePanel) ->

    RNA: Panel.register() -> register the type in some custom way -> 
(ARegionType.paneltypes)
        
        ARegionType.draw() ->
        
            ED_region_panels("context") ->
            
                foreach (panel in paneltypes) if (panel.context == 
"context") draw(panel)
                

How it should work in my opinion:

script.py -> class SomePanel(Panel): [...] bpy.types.register(SomePanel) ->

    [RNA: insert SomePanel in optimized lookup list]
        
        ARegionType.init() ->
            
            retrieve all Panel-inherited RNA types and register -> 
(ARegionType.autopaneltypes)
        
        ARegionType.draw() ->
        
            ED_region_panels("context") ->
            
                foreach (panel in paneltypes) draw(panel)
                foreach (panel in autopaneltypes) if (panel.context == 
"context") draw(panel)
                


Advantages of this kind of architecture are that it keeps RNA more 
consistent and self-contained, no messing around (i.e. no real logics) 
with interface. By consequence it also keeps interface stuff more 
self-contained by letting it handle its own supported custom types.
The apparent disadvantage is low performance, having to do lookups at 
every init. I understand this could be why the first method was adopted 
but to me this doesn't seem a valid reason. To solve this problem 
instead we could create a coexistent fast-to-access list of registered 
types. This could be ordered like some type of tree to optimize access 
based on type hierarchy lookup. I know caches are bad but still I think 
they're better than code fragmentation. Once we have this type of cache 
we could further optimize for instance like (in pseudo-code):

if type_tree.is_hierarchy_changed_for(Panel):
    re-init ARegionType.autopaneltypes

This could simply be a flag on the cache-stored type updated by the 
cache itself. This way we would re-do the lookup only when some new 
specific type gets registered/unregistered.



I think I mentioned the main issues that must be either solved, decided 
upon or simply settled, if you think I forgot something feel free to add it.
I'm also thinking about writing some more detailed docs on RNA and its 
internals in order not to turn it into a black box that only a few 
understand. This of course with the assistance of Brecht.