[Bf-blender-cvs] SVN commit: /data/svn/bf-blender [31362] branches/soc-2010-leifandersen/ tests/DesignDocuments.txt: Uploaded the Design Documents for the project.

Mon Aug 16 05:04:25 CEST 2010

Revision: 31362
          http://projects.blender.org/plugins/scmsvn/viewcvs.php?view=rev&root=bf-blender&revision=31362
Author:   leifandersen
Date:     2010-08-16 05:04:17 +0200 (Mon, 16 Aug 2010)

Log Message:
-----------
Uploaded the Design Documents for the project.

Added Paths:
-----------
    branches/soc-2010-leifandersen/tests/DesignDocuments.txt

Added: branches/soc-2010-leifandersen/tests/DesignDocuments.txt
===================================================================

--- branches/soc-2010-leifandersen/tests/DesignDocuments.txt	                        (rev 0)
+++ branches/soc-2010-leifandersen/tests/DesignDocuments.txt	2010-08-16 03:04:17 UTC (rev 31362)
@@ -0,0 +1,268 @@
+=Developer Documentation=
+
+The Blender Regression Tests Have several components which are used to test various functions of blender.  These parts include:
+
+1.  Tests Operators
+
+2.  A Test Python Module
+
+3.  A Render Test
+
+4.  PyUnit
+
+5.  Hashes
+
+6.  CTest
+
+7.  Gui
+
+==Test Operators==
+
+There are several test operators used for this test, these are mainly just the functions that are preformed oftain or would be usefull when testing blender while using it.  These operators are all written in python and can be found in:
+
+<pre>release/scripts/op/test_*.py</pre>
+
+Currently, there are two files
+
+a.  hash
+
+b.  render
+
+==hash==
+
+Hash is currently the most commonly used file, it contains the operator for generating a hashcode for each blend file, as well as the means for reading and writing the hashtable to a hashfile.
+
+(Note, I use the usage of the word hash in a slightly non-traditional sence.  The idea is to create a unique number for the contents of a blend file, this needs to be fairly quick, but not as instantaneous as a usual hash function needs to be, also, these hashes aren't actually used for storing data in a hashtable, rather, the blend names is used for that).
+
+===Hash Function/Operator==
+Because Python's hash() function is not system independent, the test uses python's hashlib, which returns a hexadecimal string, capable of uniquely identifying the blend file (based on it's contents).
+
+Because hashlib is used, every piece of data must be turned into a unique (but not system dependent) encoded unicode string, before it can be added to the total hash.
+
+Currently, the hash function only cares about data that has been associated with an object in a scene.  This was originally done because it seemed like that would be the only data that was cared about.  However, when porting several of the python tests to the Blender 2.5x API, it was determined that some tests may be useful if they cared about raw data, not just that associated to the scenes, as such, the hash function eventually should be changed accordingly.
+
+The largest possible issue in the hash function is how Blender calculates floating point numbers, which is system dependent.  Fortunately, in most cases the numbers are similar, and only differ in roundoff error.  This has been compensated for by simply rounding down the number before it's added to the hash.  However, there are a few cases where a number turns out to be fundamentally different (0.3851 vs 0.4153 for example).  A few lines that commonly caused this were commented out, but there are still a few tests that will fail on different computers.
+
+Once the hash is complete, the data will be stored in the tests module, in particular, tests.hashfile.last_hash (which by default is 'NotRun').   If the hash function somehow fails in the middle of the hash, than the resulting hash will be 'Working' (otherwise, it will change from 'Working' to the proper hash value when it's completed).
+
+===Import/Export Operators===
+
+There are two functions for saving/loading hashfiles.  (A hashfile is just a text file with each line in the form of: 
+<pre> (hash_nane) = (hash_value) </pre>
+This format was chosen to keep the data persistent after blender closes, the main hashfile can be found in tests/hashfile.txt).
+
+The hash operators will bring in the hashfile, and store it in a dictionary.  The dictionary will be placed in the tests module, in specific, tests.hashfile.data.
+
+==render==
+
+The render tests are currently not as complete as I would like them to be.  There are two reasons for this.
+
+1.  Limitations in the Blender API prevent using normal operator API calls to do needed functions (which would require some hacking to get around, aka, calling blender within blender)
+
+2.  PIL is currently not running on python 3.x, and it was outside of the scope of this project to port PIL to python 3.x.
+
+As such, the current solution is very horrible, and only their as a placeholder for when a better solution is made.  As put by elube, your using blender, to run a python 3 script, to run a python 2 script, which runs blender (good/testing), to render your immage and compare it.
+
+Basically, these tests simply have the use pick a blend file, and it calls the Image comparison test (talked about later), to compare the images.  Do to limitations of the API, blender currently assumes that the good version of blender it's testing against, is in the users path (a bad idea).  Optionally, the user can rely on the good images being pre-built, but it than has to conform to the file structure that the test imposes.
+
+The run status of the render tests can be found in the tests module, in specific tests.render
+
+==A Test Python Module==
+
+The Tests module can be found in:
+<pre>release/scripts/modules/tests/</pre>
+
+The Test module is by far the most clunky part of the current suite.  It was originally introduced back at the beginning of the project, before I learned how blender operators worked (or rather, thought in that type of mindset), as such, it originally stored many of the functions that are now as operators.  (It also stored several other functions, that are now deemed pointless, and no longer in the codebase).
+
+Now, the only reason the code exists is do to what appears to be a possible flaw (although it is an architectural design), of the way Blender operators work, which is to say the only data they can returned is pass/fail/not run/etc. data.  Rather, they are supposed to manipulate the context/data of the blend file.  As such, there was two options.  I could modify bContext to include tests, or simply keep the tests module I was about to throw out (on a code cleanup cycle).  Although modifying bContext is probably the preferred option, the time it would have taken to inject the tests portions would not have been worth it, as such, I opted for the tests module.  In time though, it would probably be better to remove it altogether.
+
+In other words, the tests module is nothing more than a giant global struct, that tests operators can dump values into and get values out of.
+
+==A Render Test==
+
+The Render tests are a suite of tests that can be run independently, or as part of a larger CTest suite, for comparing rendered images in Blender.  can be found in:
+<pre>tests/render</pre>
+
+Unlike the rest of the suite, the render test has been written in python 2.x.  This is entirely because PIL currently does not work with Python 3.x.  Two major redactors must be made before it will work with python 3.x:
+
+1.  the parentheses around print
+
+2.  string.replace(stringVar, old, new) -> stringVar.replace(old, new)
+
+other than that, it should work with python 3.x.
+
+The structure of these tests consist of an image test case (not associated with pyunit, or the XUnit framework at all), and an animation test case, which is extended from the image test case.  It also has some surrounding functions for generating HTML output, getting the local tests, and parsing parameters.
+
+The work flow for each test case is:
+
+1.  Initialize the test case.
+
+2.  Set it up for image comparisons (give it a good path, test path, and path for the blender file)
+
+3.  Render the test
+
+4.  Compare the rendered images
+
+5.  (optional) save a diff(s) of the images
+
+Steps 1 and 2 were initially done at the same time, however this was changed after the animation tests were made, because the cases were going to be set up with alternate routes to follow, mainly comparing hashes.  However, this became to complex and was later removed, although the structure is now probably sound enough to allow it to be placed back in, the functionality is not currently needed.  As such, the split steps remains as an artifact, in case hash tests ever do get put back in.
+
+The HTML output relies upon some CSS to format the table, this CSS is based upon the CSS from the Sphinx generated Blender API documentation.  At one point, the script generated the documentation itself, but it really killed the clarity of the script, so it simply assumes it's in the same folder as the created HTML file.
+
+The algorithm for comparing images is thus:
+for every pixel in the images:
+	find the absolute value of the difference between the pixels. (in the three color bands)
+	If the sum of the three differences is too large, add it to the failed pixel count
+	Add the sum of the three differences to the total difference some of the whole image
+if the different pixels are too large, the test failed
+if the total difference is too large, the test failed
+
+For animation tests, it preforms this algorithm for every rendered image, it than allows for a small margin of error in images failing (about 1%), before it fails the animation.
+
+Currently, the largest problem with this test is particles.  I'm not sure whether they should be considered as passed, but particle simulations are slightly different in different versions of Blender, leading to high image differences.
+
+In addition to working as an independent unit, these tests can also be plugged into the whole test suite.  See the CMakeLists.txt file in the tests/render folder to see how it is done.  Essentially, for every blend file in the folder, it runs the render test independently with that one blend file.  See the tests arguments (in the manual, or with the -h flag) for the arguments.
+
+==PyUnit==
+
+With the exception of the render tests, pyunit is really the skeleton of the rests of the tests.  Currently though, pyunit only does one main thing, which is to compare hashes.  There are currently two pyunit files in the build, they can be found at:
+
+<pre>tests/hash_compare.py</pre>
+
+and
+
+<pre>tests/python/hash_compare.py</pre>
+
+Both scripts are built with python 3.x, and designed to be run in blender.
+
+The first script simply compares the hash of the current blend file to the one on record (stored in hashfile,txt, put in the bin folder at build time), gives a pass or a fail, and exits blender.

@@ Diff output truncated at 10240 characters. @@


