[Bf-committers] Hypothetical take on client/server based addon/package repositories
Chad Fraleigh
chadf at triularity.org
Tue Feb 26 01:08:35 CET 2013
Here are some ideas/notes/comments/[delusions?] on what a
client/server for blender addon's could look like. Basically it is how
I would go about a basic implementation of such a system (give of take
some refinements along the way). Unfortunately, not being python
person, if I tried to write a prototype client, it would probably be
considered cruel and unusual punishment to the project. ;) I could
create a small demo repository using some of the existing addons, if
anyone thinks that would help. Otherwise, maybe some will read this
and point out any design flaws (that hopefully can be accounted for).
And, since I haven't seen the other repository addon manager demo, I
have no idea how this differs (or is redundant of) that one.
Anyway.. here it is (warning! it is a _little_ long)..
-Chad
Package Side Format
===================
Zip structure
-------------
myaddon.bap:
manifest.txt The package meta-data (required)
license.txt The full license text (optional)
icon.png An icon of the addon (optional).
python/ The base directory for python
(equivalent to each addon directories as-is now)
(required if python based)
lib32/freebsd/ Various platform specific directories for holding
lib32/linux/ binaries (i.e. DLL's/.so's) The os-mnemonic would
lib32/osx/ be a fixed defined value in blender for each OS.
lib32/windows/
lib64/freebsd/
lib64/linux/
lib64/osx/
lib64/windows/
Manifest Format (MIME style)
----------------------------
For general package manifest information, a simple MIME header style
should be adequate. This could be a modified form (no line
continuation, max line length ~200 characters maybe, UTF-8 allowed for
some field values).
Package-Version: 1
ID: {00000000-0000-0000-0000-000000000000}
Legacy-ID: myaddon
Name: My Add-on
Version: 1.0
Icon: icon.png
Type: addon/python
Creator: John Doe
Copyright: Copyright, 2013, John Doe Industries
License: GPL/3.0
Description: Do the things an add-on should do.
Description: A really long description is continued with multiple
Description: lines.
URL: http://myaddon.example.com/
Platform: linux/32
Platform: windows/32
Platform: linux/64
Platform: windows/64
Target-Application: blender:2.69-2.78
The breakout would be (some based on the firefox addon meta names):
Package-Version (required)
The spec version of all meta-fields in this group.
Ideally this should rarely need to change.
Suggested as first field, but not required as such.
ID (required)
The unique ID. This should be considered case insensitive.
For UUIDs, it would be enclosed in braces (i.e. {}).
Also could potentially use domain based identifier rules
(e.g. org.blender.addon.rigify), but this alone doesn't
account for domain ownership changes over time (without a
datestamp encoded with it, in a consistent format). Maybe
more like org.blender.addon.rigify at 201302, with the date
being a YYYYMM[DD] with-in the domain ownership period
(doesn't have to be the domain registration date like some
schemes use), as long as the same date is always used for
the same addon.
Legacy-ID (optional)
Used to match again the legacy name.
Maybe this wouldn't be needed, but was a thought.
Also could be called Friendly-ID (see below - far below)
Name (required/utf8)
The simple addon name.
Version (required)
The addon version (in "unbound" dot notation).
1.0
3.0.2
1.5.2.1
2.3.2013.02.18.1 (no, that's not a stardate!)
The versions parts would be compared as text (no assumption of
value ranges as integers). So 1.0.92800018178901018117191011.2
is a valid version. Missing [right side] parts are same as 0.
Possible a single letter suffix could be supported (which
would be treated as an addition .## notation of: X - 'a' + 1).
2.63 --> 2.63
2.63a --> 2.63.1
Icon (optional/multi?)
The package relative path of an icon file (always png?).
If multiple are allowed to be specified, then the "best fit"
would be used (each assumed to be a different resolution,
or maybe even format [like SVG over a bitmap]).
Type (required)
The [hierarchical] type mnemonic of what the package contains.
addon/python -- Standard addon (python)
addon/library -- Addon (shared object), if added
modifier/library -- Modifier plugin (shared object)
modifier/python -- Modifier plugin (python based)
render/library -- Render plugin (shared object)
Creator (optional?/utf8)
The author name.
Copyright (optional/multi?/utf8?)
The copyright line(s).
To include or imply the "Copyright, " prefix, I'm not sure.
License (required?)
License in <mnemonic>/<version> format.
Description (required?/multi/utf8)
The long description. Multiple entries will be concatenated
with a space separate. Maybe an empty field value could
indicate a paragraph break for easier to read long
descriptions.
URL (optional)
The addon homepage URL.
Platform (optional/multi)
Supported platform in <os-mnemonic>/<bit-size> format.
Multiple configurations use multiple meta entries.
The os-mnemonic is the same used for lib{32,64}/xxxxx/ names.
On no 'Platform' entries, assume platform independent or
maybe make field required and use special '*' value for that.
Target-Application (required/multi)
The application type and version range this addon is for
in <app-mnemonic>:<min-version>-<max-version>.
Multiple targets (or non-contiguous version ranges) can use
multiple entries for each.
blender -- The blender "editor" application
blender-player -- The blender player subset
In cases where the meta format changes (and thus would have a new
Package-Version), backwards compatibility could be supported by having
multiple meta chunks, each separated by a blank line (like signed jar
files use for each file entry):
Package-Version: 1
ID: {00000000-0000-0000-0000-000000000000}
.
.
.
Target-Application: blender:2.69-2.78
Package-Version: 2
ID: {00000000-0000-0000-0000-000000000000}
.
.
.
Target-Application: blender:2.69-2.78
The loading application would use the meta-data from the highest
format it supports. In cases where an addon _needs_ something in the
new meta format to work, then it obviously wouldn't include the older
format(s) and the application would just report an error that it is an
unsupported package. Also, field names are case insensitive (so
"Name:", "name:", and "NaMe:" would be the same).
As you may have noticed it has examples of "potential" things beyond
just python addons. If support for plugable modifiers, renders, or
other things eventually (maybe one day!) get added, blender might as
well use the same format for packaging all its "extensions".
If there was some real reason to ever need more complex meta formats,
like JSON, YAML, XML (RDF?), then a supporting [migrating] loaded
would search for manifest.txt, manifest.json, manifest.yml,
manifest.rdf to find a usable one. But I'm not sure that extra level
of complexity will every be needed in this scope. Other
purpose-specific meta files could be added for that (a file containing
package signing information, for example). And yes, to some, using the
MIME style probably seems kludgy (and it kinda is).. but it does have
the advantage of not needing a special parser lib/API, and being
easily processed by line-based utilities (like unixy tools -- grep,
awk, sed, perl, ...) and can be stream processed more easily (markup
formats typically require pulling the entire thing into memory before
processing, like an XML DOM-parser). For the individual package
manifests this probably isn't so bad.. but for master indexes (see
below), it can be more problematic for large repositories.
Repository Side Format
======================
The user should be able to add additional repositories for obtaining
packages, and not be locked in like many mobile device "app stores"
do. The repository(s) would be in the form of a base-URL an all
meta-data and package files would be relative to this base (in a
standard structure). The repository should be treated like a
collection of static files from the client's view, so a simple web
server serving static content could be used, or a dynamic server that
returns content from virtual files as-if they were static. While many
repositories will use HTTP[S], directory (file://) URLs could be use
also (since the referenced structure is valid as static files). This
would provide the flexibility of using a simple shared filesystem for
[importing and] distributing packages inhouse, or a DVD with a bundled
packages could be used without manually installing each one-by-one.
For the ability to co-exist with a normal web content, the structure
names should avoid naming conflicts with typical HTML filenames. This
will allow a URL like http://addons.example.com/ to be referenced by
normal browser and as a package repository base.
For per-user and/or per-package based access for clients (i.e. for
"paid" or members-only addon sites where not everything is free to
download), there is no guarantee of blender client support. Perhaps
[eventually] standard HTTP based authorization support could be added
to the blender package client for a basic members only control (of
some value for small free sites that can't handle the load of anyone
and everyone crawling/replicating/etc.. their content). But it would
be nice to allow the standard client (assuming python) to include
[reasonable] hook points so that a vendor "could" extend it with a
custom package client and add whatever repository extensions they need
to do such things (without forcing them to write their own clients
from scratch, which increases the risk of incompatibility/corruption
of standard client package data/handling if done badly).
While delivery of the repository content is limited to a simple static
content interface, management of its content is implementation choice
of each provider (be it REST, custom webapp scripts, cmdline
[re]generated, or manually updated). For blender, I assume having an
official package site would mean creation of a dynamic management
webapp/system, who's source would be available in SVN and that other
sites could use that, but should never be required to.
Upon connection to the repository the client would download the master
package index. And example name might be "/package.idx" (in reality,
using the extension ".idx" would probably be a bad idea).
This would be a summary of all packages in the repository.
Elements for each package:
These values from the package manifest (for each group) [probably
just any valid manifest field]:
Package-Version
ID
Name
Version
Type
Creator
Copyright
License
Description
URL
Platform
Target-Application
Index-Package
URL path to package file.
Index-Icon
URL path to icon file(s). (optional)
The Index-Package and Index-Icon URL's could either be simple paths
relative to the repository base, or full URL's. The use of full URL's
is to allow for master indexes that might cover multiple repositories,
or even sites that have a dynamic web server (for dynamic/scripted
package indexes) and static web server (that holds the package files).
For simplicity it would have nearly the same format as the package
manifest files. If using the MIME approach, each manifest group would
be a group in this file. The only exception is the addition of the
Index-Package and Index-Icon (the manifest Icon value could be
included, but should be ignored by the client as it is meaningless
without the package file itself). The Index-Icon field is independent
from the manifest one (but likely reference copies of the same file(s)
-- one set inside the package, the other(s) directly accessible by
URL). Also, if multiple Icon values are allowed in the manifest,
should each be extracted and mapped to an external copy for the index,
or should the index "pick one" when being generated.
In cases were the server supports a dynamic package index, some
[meaningful] query parameters could be passed (i.e.
package.idx?Platform=linux/32&Target-Application=blender:2.70).
Probably these would be limited to ID, Name, Type, Platform, and
Target-Application (and that would be in <app-mnemonic>:<version>
format as there is no need to query on a version range). If the index
is a static file, or the dynamic script wishes not to support some
fields, then it may ignore them. In all cases the client must do its
own filtering on the results, so no assumption of server support is
made (and allows for static sources). The dynamic filter option is
simply to reduce network bandwidth/download time for large indexes.
For [repository] co-located packages, a suggested directory hierarchy
might be one of the following:
/packages/
/packages/<package-ID>-<version>.bap
/packages/<package-ID>-<version>.png
/packages/<package-ID>-<version>.bap
/packages/<package-ID>-<version>.png
.
.
.
/packages/
/packages/<package-ID>/
/packages/<package-ID>/<package-ID>-<version>.bap
/packages/<package-ID>/<package-ID>-<version>.png
/packages/<package-ID>/<package-ID>-<version>.bap
/packages/<package-ID>/<package-ID>-<version>.png
/packages/<package-ID>/
/packages/<package-ID>/<package-ID>-<version>.bap
/packages/<package-ID>/<package-ID>-<version>.png
.
.
.
/packages/
/packages/<package-ID>/
/packages/<package-ID>/<version>/
/packages/<package-ID>/<version>/<package-ID>.bap
/packages/<package-ID>/<version>/<package-ID>.png
/packages/<package-ID>/<version>/
/packages/<package-ID>/<version>/<package-ID>.bap
/packages/<package-ID>/<version>/<package-ID>.png
/packages/<package-ID>/
/packages/<package-ID>/<version>/
/packages/<package-ID>/<version>/<package-ID>.bap
/packages/<package-ID>/<version>/<package-ID>.png
.
.
.
Where each package (by the unique ID) may exist for multiple versions.
In these examples the .bap file is the package, and .png is the icon
file. Also a user friendly name could be used for the .bap (and maybe
icon) to provide an easy to identify package if downloaded directly
using the browser (via a HTML based interface, which is [currently]
beyond this document's scope). It is suggested that structures that
use friendly name include an ID based parent, to avoid naming
collisions from different packages (if contributing users are the ones
defining the friendly name). Maybe the manifest field 'Legacy-ID'
could be named 'Friendly-ID' instead and used for this purpose.
In cases where co-existing with other content (like HTML) is not an
issue, the /packages/ prefix may be unnecessary (e.g. a local
directory repository where the only base entry would only be
packages.idx and packages/).
Of course it should go without saying, but I will anyway.. when taking
ID's that are provided from untrusted sources (and maybe even
"trusted" ones in case of typos, etc), ALWAYS validate that their
values match only documented encoding schemes and character sets,
_especially_ before using them as filenames.
Despite my earlier email suggesting the .bap (Blender Addon Package)
extension name.. if this system could (and hopefully will) expand to
other types of blender packages, a better suffix would probably be
more appropriate. Perhaps .bxp (Blender eXtension Package)?
Also there was a mention of addon dependencies. I assume this mean
that one addon requires another.. how is this done now (even if a
required addon is bundled, but a not enabled)? Other features (like
User Rating, stability status [aka Alpha/Beta/Experimental],
Evaluation state [Verified/Trusted [author]/As-Is]) could also be
added (as optional) meta. For installation, the zip package would be
extracted in the appropriate base directory using the ID (ideally
extracting as a temp directory name, and on successful extract,
renamed to the real name). While some files could potentially be used
in the compressed form (like java class files in jars), when shared
libs are included, the OS would need direct filesystem access to them.
Something else to point out (before anyone complains) is that even if
MIME style metadata is used for distribution files/repositories, how
the client stores its own structures need not use the same. In fact it
probably shouldn't as the client goal would be features/functionality
where the package file metadata has a goal of simplicity/portability.
More information about the Bf-committers
mailing list