[Bf-committers] GSoC Proposal: Unit Testing

Andrea Weikert andrea at aweikert.de
Sat May 8 18:23:02 CEST 2010


Hi,

> Hi,
>
> I think the direction of this project should also be discussed with
> your mentor, but among the summer of code mentors I think we agreed
> that it should be python based and not really fine grained unit tests
> for C code. Unit testing specifically is not necessarily what we think
> is the best idea, if it is the kind of thing that requires a lot of
> effort to write and maintain as code changes. We want something that
> allows us test as much as possible with as little effort as possible.
>
>   
I think there might have been a slight misunderstanding. The proposal 
was listed as Unit Tests
even on our suggestion page and while I have said that we will evaluate 
at which level and to what
extent they can be added these are definitely different from end-to-end 
(regression) tests.
We will not add unit tests to each and every function (way over the 
scope of a GSOC project anyway ;)
and certainly add tests only to parts where the code is currently not 
very volatile and the API is mostly stable.
> Instead you can think of things like:
> * Regression testing of renders compared to existing version (compare pixels)
> * Same for e.g. mesh tools (compare mesh topology)
> * Importers/exporters testing if the result is the same
> * Maybe even ecording user operations and then verifying if the result
> is the same?
> * Verify operators don't crash when run in different contexts
> * Testing that using API functions/properties don't crash with various inputs
> * Testing for NaN/Inf number results.
> * Testing for memory leak error prints.
>   
While I agree that end-to-end tests like comparing renders are useful 
too, I don't think there is enough
substance in them for a GSOC project, as it's more installing a few 
tools, creating some .blend files and
comparing the output with previous version.

Unit tests are useful *during* development, to make sure new code 
doesn't break existing functionality.
They help pointing out to exactly the function/part of the code that 
doesn't run as expected in contrast to
a more coarse end-to-end test that tests the whole pipeline. Unit tests 
are meant to run fast (it is definitely
a goal to keep the running time of *all* the tests definitely under a 
minute and single test suites under 10 secs or so)
and thus can be run constantly like to make sure nothing breaks before I 
commit.

The end-to-end tests also run *much* more slowly than unit tests (at 
least as soon as some kind of render
is involved), so I don't believe they will be picked up by developers 
unless maybe shortly before a release.

> In fact just based on renders you can test if most modifiers, nodes,
> constraints, particle options still work as before. Clearly this is
> not as useful as fine-grained unit testing as you usually do with
> toolkits like Google Test, but only a testing framework that is simple
> and requires little work to maintain will actually be picked up by
> developers. This leads a bit more to 'regression testing' maybe, but
> many of the things that break can't be expressed easily as assertions
> unless you write hundreds of thousands of lines of testing code.
>
> Brecht.
>   
I understand that for developers who are not familiar with unit testing 
the fact that there is code required to
write the unit tests and some effort to maintain them might seem too 
much at first. I do hope to show with this project
that the effort to maintain them isn't really that much, since if you 
think about it, the only time a test needs to change
is when the API of a function changes. And this is something that we are 
moving to anyway - to have much more stable
internal APIs that likely won't change *that* much and I wouldn't 
propose to test every single internal function,
but test through the API that is exposed to other parts of Blender. In 
this light, if the API changes,
you'd probably have to change a few places anyway and updating the tests 
is then not that much more
compared to the normal development work.

 From my experience, having unit tests in place leads to code that is 
more bug free and stable and greatly reduces time spent debugging.
So in the end the tests should *save* time.
I actually don't think writing a small test takes more time than to add 
debug printf in strategic places and remove them later,
which so far has been probably the most used debugging tool. The only 
difference is that the unit tests would remain
as a permanent safety net and also act as a documentation on how the 
original coder expected the code to be used.
The tests will be kept separate from the "productive code", so as to not 
clutter it, though we will work out how to structure this
exactly as part of the GSOC project.

To allow all developers to gain experience, I suggested to Leif to 
first  look into specific areas that would benefit most from testing,
such as mesh tools or modifiers and write a small number of tests 
initially to get the framework and automatic running of the
tests in place. We would also probably not add tests to areas that are 
currently under heavy development.
As tests are added, my hope is that more developers would see the 
benefits and add tests for their area of code.
Of course tests will be discussed with the maintainers for their area of 
code and reviewed. The only thing that would be asked
of the developers is to not break any tests in code they are not 
maintaining.
I also would count it as a huge success, if during the GSOC project we 
even get a test coverage of 10% of the code.

The second result I hope to get from the GSOC project is to find out how 
difficult it is to test the different areas of Blender and where we
could probably benefit from cleaning up the architecture and 
dependencies a bit.

Lastly Leif also planned for some code documentation towards the end of 
the project, which I think would
also benefit the whole development process and make life easier for new 
developers.

The reason why we are currently looking into C or C++ testing frameworks 
is that Leif is more familiar with C/C++ than with Python
and as far as operators or tools are concerned, the tests can 
theoretically test the same level of the code, the calls would just
not go through the Python API. As a mentor I would like to allow the 
student to choose his preferred language for this to make things easier.

I hope that I could clarifiy things a bit and alleviate some fears, I 
honestly don't think that it will be such a big issue to maintain the tests.

- Andrea -



More information about the Bf-committers mailing list