[Soc-2010-dev] Status Report (testing-week 7)

[Leif Andersen]

Overall, I think this has been a good week.  I worked almost exclusively on
the image comparison diff tester this week.  I wrote it in python, and it
can actually work independent of blender (although it does assume that
blender is on your system, and in your path, although that can be changed
with the --blender-bin flag).  This means that the tests folder can be put
in a zip, and placed in the same location as where the current regression
tests (for blender 2.4x) are located on the website.

As I said last time, the general way to run the tests is by using python
run.py.  You can use the -v flag to see the output of blender as it's
rendering it.  Also, it might be possible to avoid even having to download
the images at all, by using the recently built --built-tests flag, which,
instead of rendering tests, will render the images and store them in the
known good folder.  (It uses the version of blender in your path).

Through the use of the --image flag, you can compare any blender image on
your harddrive (currently assuming it's in the right file structure), this
allowed me to make other image comparison tests, that compared
blend files built by pyunit based tests.  This (along with the hashcode
operator I started putting together last week), means that most of the tools
for building good unit tests (and regression tests in general), are almost
build (I think).  Meaning that now the only missing piece of the puzzle is
the one I was working on several weeks ago (without luck), which is:
A.  Getting the python API to work in background mode (which blender
currently isn't designed to do).
B.  Getting the python API to work properly in foreground mode (lots of
stuff in bpy.wm crash.  (At least it did as of a week ago, do to low
bandwidth caps here, I won't' merge until tomorrow evening (when I get
home), so they may have been fixed by now)).

Although Andrea only looked at my code to determine good structure, etc.   I
believe that most of the bugs have been removed.  (Although, this is python,
and bad code doesn't show up until someone tries to run it, and I haven't
made regression tests for my regression tests. ;) (at least not yet anyway,
and it's not on my radar in the near future)).

The comparator is also capable of producing animation tests.  However, this
code is slightly buggier.  Also, by default, I'm not running the animation
tests in the current regression suite, because it takes the running time up
from 2:30, to over an hour.  I have a --animation flag, but it doesn't work
at the moment.  (Although the underlying code works, so it's basically just
a matter of plugging the code into the flag).

Finally, because the animation images make the overall file size ~240 MB (as
apposed to 30 MB), they are not in the default build.  But rather, the user
has two choices, they can either A:  Render Good ones, or B.  Use a hash
code (and download images (or generate them) if something goes wrong).  The
hash code also work in images.  However, the code for this is only half
built.  The test case classes support it, but the surrounding structures
don't use it yet, this shouldn't be more than an hours worth of work to
implement it fully.

Finally, I didn't do as much work as I would have liked in the HTML output.
 Now, rather than using static CSS scripts from sphynx, I now build the CSS
in the script itself.  (With the exception of images, I'm not sure how to do
that as a python script...well, there is PIL, but I'd sooner put bamboo up
my fingernails than make the needed images that way).  Although the output
is starting to look like it's from blender's website.  (Although the table
doesn't quite fit).  Also, it will only generate images for single images
(well, that, and animations, but it only does the first frame).  Eventually,
the animations will be clickable, and you can view a separate HTML page for
the entire animation, but that's not built yet.  It shouldn't be too hard to
make, but I keep having to do one or the other thing, so the current
implementation is old, and would probably cause the script to crash if used.

~Leif Andersen

----------
Searching a DAG: http://xkcd.com/761/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.blender.org/pipermail/soc-2010-dev/attachments/20100709/180d39d3/attachment.htm