[Bf-committers] Blender 2.5 api proposal

[Campbell Barton]

On Mon, Dec 1, 2008 at 11:38 PM, Campbell Barton <ideasman42 at gmail.com> wrote:
> On Mon, Dec 1, 2008 at 9:56 PM, Toni Alatalo <antont at kyperjokki.fi> wrote:
>> I agree with Cambpell and Joe that dropping the old api in 2.50, but
>> helping with a py written Blender module with functions people are
>> familiar with, is probably the way to go. People depending on existing
>> scripts that are not ported can stick to 2.4x in the time being.
>>
>> Without knowing the 2.50 work well but based on the reading have done
>> in the wikis etc. the plans seem good to me otherwise too.
>>
>> One concern I have is PyRNA's ability to support NumPy, i.e.
>> fast/direct access to memory buffers like the positions of all vertices
>> of a mesh, all particles of a particle system, or the values of all
>> pixels in a compositing node, to process them efficiently from py
>> (change all of them in one py call).
> Your right, PyRNA does not give especially fast access.
> Id prefer de
>
>> I am not a great expert there, but recently got more experience by
>> writing a real-time thing with pygame & numpy for an artist friend
>> where am manipulating all pixels individually with numpy ops, and after
>> got the hang of it found it a great way to work. So am still guessing
>> that it'll be a killer feat for at least compositing nodes, but
>> possibly for 3d things too (would be fun to try to manipulate mesh
>> geoms, but afaik there is no tool yet that supports that (unless
>> pyopengl directly somehow?)).
>>
>> As Theo de Ridder told and showed us in the 2007 conf, all that's
>> needed to enable numpy manipulation is to implement the minimal Python
>> buffer protocol on the c side, which basically is just returning a
>> pointer to the array. It was revised for Python 3000, which is also
>> backported to 2.6 and documented in
>> http://docs.python.org/c-api/objbuffer.html .. the current 2.5.2 docs
>> at http://www.python.org/doc/api/bufferObjects.html tell more about the
>> usage. For those who were not there, Theo had added buffer access to
>> shape keys, so he could animate a swarm of fishes swimming by having
>> the fishes as dupliverts and manipulating the keybuffer directly in
>> memory using numpy. I asked Jahka and the particles in a particle
>> systems are there similarily as a continuous memory chunk, so the same
>> could be done there.
>>
>> For something that is not a buffer but e.g. a linked list, I guess it
>> may still make sense to implement an interface to numpy in c, in order
>> to not iterate on the py side (which is slow) but just define the
>> operations there, and iterate in native code instead. But I haven't
>> looked into that yet .. dunno if exposing even something like
>> scene.objects as a numpy array might be interesting for some cases? (at
>> least if Blender is restructured internally so that it'd support
>> hundreds of thousands of objects more efficiently ;)
>>
>> So my 2 cents from outside of the inner loop is that please don't make
>> it difficult to use numpy, but consider where it'd be good to support -
>> thank you :) .. and do note that the support for it is *optional*, it
>> does not add any runtime dependencies (that's how pygame does it too -
>> the surfarray module works if you have numpy installed, but you can
>> ignore it otherwise).
>>
>> ~Toni
>
> Hi Toni, your right, though this isnt a problem Im trying to solve :)
> Fast access to memory buffers with numpy has merits but 2.5 wont make
> easier or harder, data structures are unchanged.
>
> Theo's example only worked because shape key buffers are an array of
> floats, this is not the case for most other areas of blenders mesh
> storage.
> If blender was more python-centric we could make sure the data was
> structured in a way that Numpy could read easily.
> But this can be limiting too if we switched to tile based ImLib,
> access to image buffers would not be contiguous breaking Numpy access.
> Further, we would need to agree on including Numpy with blender for it
> to be truly useful average blender-heads.
>
> Personally I prefer to focus on really great high level python api
> that can do things like record user actions and play them back to
> create custom tools.
>
> This way python can make use of functions written in C, rather then
> having to rewrite them in py like many scripts do now.
>
> If we want Numpy integration for compositor, this can be added as a
> irrespective of the 2.5 API, though I'd hold out for a GPU compositor
> :-)
>
> --
> - Campbell

Thanks for everyone's feedback,  I'll go ahead with the proposal...
* Complete auto generated API (rna, operators)
* Add necessary utility functions + python helpers + BPy2.4x API compat helpers
* Leave open the option to add the 2.4x API  back into 2.5, if things
don't work out with the new API.
* Leave open the option to use python 3.0
* No langauges other then python
* No support for faster/direct access (numpy)

- Campbell