[Bf-committers] Automated testing

[Howard Trickey]

I agree with the sentiments about the value of automated tests.
I see the value every day in my day job.

Blender has some automated tests, in spite of what that wiki link says.
See source/tests in the source tree, and the test target in the builds.

Unfortunately, these cover only a small part of functionality (mostly I/O)
and have so lacked attention that more of them are failing than not right
now.

I have started contributing tests for modeling here, but it would be great
if all of the developers could try to add tests and fix the failing ones.


On Wed, Mar 26, 2014 at 11:31 AM, Reed Law <reedlaw at gmail.com> wrote:

> I was curious about the state of Blender's source code so I checked it out
> and after looking about I was surprised not to find automated tests for
> much of the C/C++ code. Then I came across this FAQ
> item<
> http://wiki.blender.org/index.php/Dev%3aDoc/FAQ#Why_don.27t_you_run_automated_tests.3F
> >that
> spells out the reasons for not running automated tests. None of the
> "reasons" hold water (and they should be called "excuses").
>
>    - Tests that require user interaction are called "integration" or
>    "end-to-end" tests. If all of the underlying units are well-tested and
> the
>    UI is just a thin layer then this type of testing can be kept to a
> minimum.
>    - Tests that take too long to run indicate tightly coupled code.
>    - Tests taking too much effort to maintain is an illusion. I would bet
>    *not* having tests requires much more effort and leads to the problem
>    described in The Mythical Man
> Month<https://en.wikipedia.org/wiki/The_Mythical_Man-Month>:
>
>       - The fundamental problem with program maintenance is that fixing a
>       defect has a substantial (20-50 percent) chance of introducing
> another. So
>       the whole process is two steps forward and one step back.
>    - Tests covering areas that hardly ever change is a *good* thing. They
>    can help a new contributor understand what the code is doing by showing
>    examples of the interface for those classes.
>    - Having tests for areas that are immediately noticeable when broken is
>    also good because it saves that extra step of noticing the breakage, or
> of
>    getting a bug report from someone else.
>    - Tests that break but are not bugs should be updated to reflect the new
>    features/interface.
>    - Tests that break but are not a problem for the end user should also be
>    refactored. What is the use case being tested in that case?
>    - Although many Blender developers are volunteers, tests can be enforced
>    by only merging pull requests that have good test coverage and pass the
>    automated test suite run on a continuous integration server for each
> commit.
>
> I realize that the solution to these problems is found in the call for help
> at the bottom of the FAQ item. But there is only so much a single
> contributor can do, especially in light of a long history of untested code.
> Perhaps setting up a continuous integration server with a service such as
> Travis CI would be a first step.
> _______________________________________________
> Bf-committers mailing list
> Bf-committers at blender.org
> http://lists.blender.org/mailman/listinfo/bf-committers
>