[Bf-taskforce25] RNA and API wrap-up

[joe]

Can we name the the foreach_get/set functions something else?  It's a
rather confusing name, I think.  Maybe getArray/setArray, or just
".as_array" or something :)

Joe

On Mon, Jun 29, 2009 at 4:54 PM, Campbell Barton<ideasman42 at gmail.com> wrote:
> Hey Elia,
>
> On Mon, Jun 29, 2009 at 5:06 AM, Elia Sarti<vekoon at gmail.com> wrote:
>>
>> Hi,
>> I've been almost absent in the last couple of months (which I apologize
>> for) and as I'm trying to make a bit of a summary of current state of
>> API related stuff I thought it would be useful to write it down in order
>> for everyone to have a more clear idea and make it possible to discuss
>> further so that then we can put up something definite and have some
>> specs for other things to be settled that depend on this (like
>> exporters/importers etc).
>>
>>
>> Data access and performance
>>
>> Data access as of now for pyscripts always happens through RNA. The
>> advantage of having everything abstracted as always presents the same
>> problem, which is performance. While it should work fine to access
>> almost any data it would get very slow for things like vertices and in
>> general anything which is a huge collection of pieces of data.
>> Talking a bit on IRC Campbell proposed having a special function
>> implemented PYAPI side in the form of obj.foreach_set("name", [array])
>> and same for getting obj.foreach_get("name", [array]). For instance (as
>> I remember it) mesh.foreach_get("verts", vertex_array) would get all the
>> vertices for the mesh "mesh" and put them into "vertex_array". We could
>> also extend this to use RNA paths like mesh.foreach_get("verts.co",
>> coords). I'm not sure what the complications of this could be, I think
>> trying it out is the best option here. As Campbell said this is
>> essential to have importers/exporters working which again is essential
>> in order to have Blender working.
>
> Python classes (types) are also inspected at runtime, we're just
> forwarding the lookups from python to RNA, So even if we did a fully
> autogenerated Py/C api, Python would still be doing hash lookups
> internally.
> Basically if our hash lookups are as fast as pythons, there should be
> no speed difference.
>
>
> The array access foreach_get/set will allow faster access, and I'm
> still not ruling out having rna support python3's buffer protocol.
> We could have something like this...
>  coords = mesh.verts.as_buffer("co")
>  normals = mesh.verts.as_buffer("no")
>
> This could make a new buffer object, python3 has support for buffer
> stride and offset (like numpy afaik)
>
> Still, this is more of a hassle to define our own buffer type, and
> foreach_get/set can also be more flexible...   with other args we
> might want to use...
>
>  # eg. to only position selected verts.
>  mesh.verts.foreach_set(attr="co", array=myList, check_attr="selected")
>
>
>> Operators design
>>
>> Operators are more or less defined/complete at least C side. API side
>> though we still have things undefined or problematic. For instance an
>> issue is unique naming. Although the current naming convention fits
>> perfectly and it's cohesive on our core application side it presents
>> some limitations for extensibility as we cannot really control how many
>> pyoperators will be written and handle collisions between these. A way
>> could be simply to always prepend pyops with PY_ (or tell py devs to use
>> this convention). We could tell devs to prepend their pyoperators with
>> their own names/ids of choice. Or we could use python own module/package
>> system to combine the final operator name. As of now I don't seem to see
>> any check on operator names? We could allow a
>> module.OBJECT_OT_do_something() type of name syntax which could be
>> translated automatically from py? Not sure if this conflicts with something.
>> Another issue for operators in general is context constraints. For
>> instance as of now as Ton mentioned in IRC add_object operator works in
>> node editor. Is this fine? Can it be expected by the user? From my point
>> of view being able to run operators from contexts you wouldn't expect is
>> totally acceptable architecture-wise as running constraints are defined
>> in terms of "needs" and if requirements are respected I see no reason
>> why the operator should not run. They should just be hidden from the
>> user in order not to create confusion. I think we could do this with
>> some type of "flag" both on operators and space types defining the
>> constraints of context. Something like CTX_HANDLE_OBJECT,
>> CTX_HANDLE_NODE etc. On the operator this is the same as represented by
>> the prefix of its idname so we could actually use this on spaces
>> although string comparison would introduce some issues but would be more
>> solid e.g. for py/plugin spaces which brings me back to pyops naming as
>> using this method would leave module/package approach as the only usable
>> naming convention.
>> You might wonder why not to put the space check directly on the
>> operator's poll? Well the reason for this is that as I said poll()
>> should just check requirements and not interface issues which is instead
>> something spaces take care of. This is not just a design choice but also
>> has some practical examples. For instance, the same example I made on
>> IRC, suppose at some point Outliner space gets capacity of
>> listing/handling nodes. To support node tools in outliner you would have
>> to go and change every single node operator to add additional check on
>> space type while with the former approach you just update outliner space
>> to accept NODE_* operators. This of course is even more clear for
>> python/plugin spaces.
>
> Personally Id prefer if we could completely avoid the C operator
> naming - OBJECT_OT_do_something
>
> though its possible to do this now with the current python api it
> would be cumbersome to fake the different operators into submodules
> eg.
> - bpy.ops.object.do_something
>
> Id propose drop OBJECT_OT_do_something  for the operator ID's and use
> dot syntax instead.
>  ot->idname= "object.do_something";
>
> OBJECT_OT_do_something  cant be kept for the C operator definitions
> only, users and script writers would never see it.
>
> Python can convert this into bpy.ops.object.do_something most
> efficiently if each operator type, OBJECT, ANIM, SEQUENCER etc are
> each in their own list/hash.
>
>> Type registration
>>
>> This is less important and more of a personal question than general issue.
>> Types can get registered for adding customization like it happens for
>> Panels or Menus. If I'm not mistaken this is how current architecture works:
>>
>> script.py -> class SomePanel(Panel): [...] bpy.types.register(SomePanel) ->
>>
>>    RNA: Panel.register() -> register the type in some custom way ->
>> (ARegionType.paneltypes)
>>
>>        ARegionType.draw() ->
>>
>>            ED_region_panels("context") ->
>>
>>                foreach (panel in paneltypes) if (panel.context ==
>> "context") draw(panel)
>>
>>
>> How it should work in my opinion:
>>
>> script.py -> class SomePanel(Panel): [...] bpy.types.register(SomePanel) ->
>>
>>    [RNA: insert SomePanel in optimized lookup list]
>>
>>        ARegionType.init() ->
>>
>>            retrieve all Panel-inherited RNA types and register ->
>> (ARegionType.autopaneltypes)
>>
>>        ARegionType.draw() ->
>>
>>            ED_region_panels("context") ->
>>
>>                foreach (panel in paneltypes) draw(panel)
>>                foreach (panel in autopaneltypes) if (panel.context ==
>> "context") draw(panel)
>>
>>
>>
>> Advantages of this kind of architecture are that it keeps RNA more
>> consistent and self-contained, no messing around (i.e. no real logics)
>> with interface. By consequence it also keeps interface stuff more
>> self-contained by letting it handle its own supported custom types.
>> The apparent disadvantage is low performance, having to do lookups at
>> every init. I understand this could be why the first method was adopted
>> but to me this doesn't seem a valid reason. To solve this problem
>> instead we could create a coexistent fast-to-access list of registered
>> types. This could be ordered like some type of tree to optimize access
>> based on type hierarchy lookup. I know caches are bad but still I think
>> they're better than code fragmentation. Once we have this type of cache
>> we could further optimize for instance like (in pseudo-code):
>>
>> if type_tree.is_hierarchy_changed_for(Panel):
>>    re-init ARegionType.autopaneltypes
>>
>> This could simply be a flag on the cache-stored type updated by the
>> cache itself. This way we would re-do the lookup only when some new
>> specific type gets registered/unregistered.
>>
>>
>>
>> I think I mentioned the main issues that must be either solved, decided
>> upon or simply settled, if you think I forgot something feel free to add it.
>> I'm also thinking about writing some more detailed docs on RNA and its
>> internals in order not to turn it into a black box that only a few
>> understand. This of course with the assistance of Brecht.
>>
>>
>>
>> _______________________________________________
>> Bf-taskforce25 mailing list
>> Bf-taskforce25 at blender.org
>> http://lists.blender.org/mailman/listinfo/bf-taskforce25
>>
>
>
>
> --
> - Campbell
> _______________________________________________
> Bf-taskforce25 mailing list
> Bf-taskforce25 at blender.org
> http://lists.blender.org/mailman/listinfo/bf-taskforce25
>