[Bf-committers] GSoC Proposal: Unit Testing
Andrea Weikert
andrea at aweikert.de
Sat May 8 18:23:02 CEST 2010
Hi,
> Hi,
>
> I think the direction of this project should also be discussed with
> your mentor, but among the summer of code mentors I think we agreed
> that it should be python based and not really fine grained unit tests
> for C code. Unit testing specifically is not necessarily what we think
> is the best idea, if it is the kind of thing that requires a lot of
> effort to write and maintain as code changes. We want something that
> allows us test as much as possible with as little effort as possible.
>
>
I think there might have been a slight misunderstanding. The proposal
was listed as Unit Tests
even on our suggestion page and while I have said that we will evaluate
at which level and to what
extent they can be added these are definitely different from end-to-end
(regression) tests.
We will not add unit tests to each and every function (way over the
scope of a GSOC project anyway ;)
and certainly add tests only to parts where the code is currently not
very volatile and the API is mostly stable.
> Instead you can think of things like:
> * Regression testing of renders compared to existing version (compare pixels)
> * Same for e.g. mesh tools (compare mesh topology)
> * Importers/exporters testing if the result is the same
> * Maybe even ecording user operations and then verifying if the result
> is the same?
> * Verify operators don't crash when run in different contexts
> * Testing that using API functions/properties don't crash with various inputs
> * Testing for NaN/Inf number results.
> * Testing for memory leak error prints.
>
While I agree that end-to-end tests like comparing renders are useful
too, I don't think there is enough
substance in them for a GSOC project, as it's more installing a few
tools, creating some .blend files and
comparing the output with previous version.
Unit tests are useful *during* development, to make sure new code
doesn't break existing functionality.
They help pointing out to exactly the function/part of the code that
doesn't run as expected in contrast to
a more coarse end-to-end test that tests the whole pipeline. Unit tests
are meant to run fast (it is definitely
a goal to keep the running time of *all* the tests definitely under a
minute and single test suites under 10 secs or so)
and thus can be run constantly like to make sure nothing breaks before I
commit.
The end-to-end tests also run *much* more slowly than unit tests (at
least as soon as some kind of render
is involved), so I don't believe they will be picked up by developers
unless maybe shortly before a release.
> In fact just based on renders you can test if most modifiers, nodes,
> constraints, particle options still work as before. Clearly this is
> not as useful as fine-grained unit testing as you usually do with
> toolkits like Google Test, but only a testing framework that is simple
> and requires little work to maintain will actually be picked up by
> developers. This leads a bit more to 'regression testing' maybe, but
> many of the things that break can't be expressed easily as assertions
> unless you write hundreds of thousands of lines of testing code.
>
> Brecht.
>
I understand that for developers who are not familiar with unit testing
the fact that there is code required to
write the unit tests and some effort to maintain them might seem too
much at first. I do hope to show with this project
that the effort to maintain them isn't really that much, since if you
think about it, the only time a test needs to change
is when the API of a function changes. And this is something that we are
moving to anyway - to have much more stable
internal APIs that likely won't change *that* much and I wouldn't
propose to test every single internal function,
but test through the API that is exposed to other parts of Blender. In
this light, if the API changes,
you'd probably have to change a few places anyway and updating the tests
is then not that much more
compared to the normal development work.
From my experience, having unit tests in place leads to code that is
more bug free and stable and greatly reduces time spent debugging.
So in the end the tests should *save* time.
I actually don't think writing a small test takes more time than to add
debug printf in strategic places and remove them later,
which so far has been probably the most used debugging tool. The only
difference is that the unit tests would remain
as a permanent safety net and also act as a documentation on how the
original coder expected the code to be used.
The tests will be kept separate from the "productive code", so as to not
clutter it, though we will work out how to structure this
exactly as part of the GSOC project.
To allow all developers to gain experience, I suggested to Leif to
first look into specific areas that would benefit most from testing,
such as mesh tools or modifiers and write a small number of tests
initially to get the framework and automatic running of the
tests in place. We would also probably not add tests to areas that are
currently under heavy development.
As tests are added, my hope is that more developers would see the
benefits and add tests for their area of code.
Of course tests will be discussed with the maintainers for their area of
code and reviewed. The only thing that would be asked
of the developers is to not break any tests in code they are not
maintaining.
I also would count it as a huge success, if during the GSOC project we
even get a test coverage of 10% of the code.
The second result I hope to get from the GSOC project is to find out how
difficult it is to test the different areas of Blender and where we
could probably benefit from cleaning up the architecture and
dependencies a bit.
Lastly Leif also planned for some code documentation towards the end of
the project, which I think would
also benefit the whole development process and make life easier for new
developers.
The reason why we are currently looking into C or C++ testing frameworks
is that Leif is more familiar with C/C++ than with Python
and as far as operators or tools are concerned, the tests can
theoretically test the same level of the code, the calls would just
not go through the Python API. As a mentor I would like to allow the
student to choose his preferred language for this to make things easier.
I hope that I could clarifiy things a bit and alleviate some fears, I
honestly don't think that it will be such a big issue to maintain the tests.
- Andrea -
More information about the Bf-committers
mailing list