[Bf-committers] GSoC Proposal: Unit Testing

Leif Andersen leif.a.andersen at gmail.com
Sun May 9 02:59:55 CEST 2010


Regarding the debate about what type of tests there will be, and the use of
C/C++/Python, I think that was covered sufficiently well on #blendercoders.
 Although, I'm not sure if #blendercoders has IRC logs.

 It's a one liner
>
> g++ -o example example.cc -lgtest

Well, that didn't work, nor did the 3 liner in the README file.

  g++ -I${GTEST_DIR}/include -I${GTEST_DIR} -c ${GTEST_DIR}/src/gtest-all.cc

  ar -rv libgtest.a gtest-all.o

   g++ -I${GTEST_DIR}/include path/to/your_test.cc libgtest.a -o your_test


After setting GTEST_DIR to be the folder where I downloaded the GTEST files
too, and replacing path/to/your_test.cc and your_test to the unit tests.
 Later, I also tried adding the classes that the tests relied upon, but that
didn't work.   However, using the makefile template, I finally was able to
get it to compile, and run.  Although I'm not entirely sure of what it's
doing. ;)  I guess I have more to learn.

Note that #include does not imply link. You need to include the right
> headers for the object file to compile (the .o) and then you need the right
> libraries specified with -lfoo when linking to get the final ELF binary.
>
> I suggest using the command line for a bit to make sure you understand how
> the compiler and linker are different and how they work together.
> Otherwise,
> trying to track down eclipse problems is really tough because there are
> added layers of abstraction that hide the issues.


Really?  #include doesn't imply a link, than what is it for?  (Or maybe it's
just me getting terminology wrong, again).   Anyway, can do, I am back in
the command line, and will wait until I think I've learned it before I go
back to an IDE to speed things up a bit.

Again, thank you everyone.

~Leif Andersen

----------
That was easy:
http://www.appbrain.com/app/net.leifandersen.mobile.android.easybutton


On Sat, May 8, 2010 at 10:23, Andrea Weikert <andrea at aweikert.de> wrote:

> Hi,
>
> > Hi,
> >
> > I think the direction of this project should also be discussed with
> > your mentor, but among the summer of code mentors I think we agreed
> > that it should be python based and not really fine grained unit tests
> > for C code. Unit testing specifically is not necessarily what we think
> > is the best idea, if it is the kind of thing that requires a lot of
> > effort to write and maintain as code changes. We want something that
> > allows us test as much as possible with as little effort as possible.
> >
> >
> I think there might have been a slight misunderstanding. The proposal
> was listed as Unit Tests
> even on our suggestion page and while I have said that we will evaluate
> at which level and to what
> extent they can be added these are definitely different from end-to-end
> (regression) tests.
> We will not add unit tests to each and every function (way over the
> scope of a GSOC project anyway ;)
> and certainly add tests only to parts where the code is currently not
> very volatile and the API is mostly stable.
> > Instead you can think of things like:
> > * Regression testing of renders compared to existing version (compare
> pixels)
> > * Same for e.g. mesh tools (compare mesh topology)
> > * Importers/exporters testing if the result is the same
> > * Maybe even ecording user operations and then verifying if the result
> > is the same?
> > * Verify operators don't crash when run in different contexts
> > * Testing that using API functions/properties don't crash with various
> inputs
> > * Testing for NaN/Inf number results.
> > * Testing for memory leak error prints.
> >
> While I agree that end-to-end tests like comparing renders are useful
> too, I don't think there is enough
> substance in them for a GSOC project, as it's more installing a few
> tools, creating some .blend files and
> comparing the output with previous version.
>
> Unit tests are useful *during* development, to make sure new code
> doesn't break existing functionality.
> They help pointing out to exactly the function/part of the code that
> doesn't run as expected in contrast to
> a more coarse end-to-end test that tests the whole pipeline. Unit tests
> are meant to run fast (it is definitely
> a goal to keep the running time of *all* the tests definitely under a
> minute and single test suites under 10 secs or so)
> and thus can be run constantly like to make sure nothing breaks before I
> commit.
>
> The end-to-end tests also run *much* more slowly than unit tests (at
> least as soon as some kind of render
> is involved), so I don't believe they will be picked up by developers
> unless maybe shortly before a release.
>
> > In fact just based on renders you can test if most modifiers, nodes,
> > constraints, particle options still work as before. Clearly this is
> > not as useful as fine-grained unit testing as you usually do with
> > toolkits like Google Test, but only a testing framework that is simple
> > and requires little work to maintain will actually be picked up by
> > developers. This leads a bit more to 'regression testing' maybe, but
> > many of the things that break can't be expressed easily as assertions
> > unless you write hundreds of thousands of lines of testing code.
> >
> > Brecht.
> >
> I understand that for developers who are not familiar with unit testing
> the fact that there is code required to
> write the unit tests and some effort to maintain them might seem too
> much at first. I do hope to show with this project
> that the effort to maintain them isn't really that much, since if you
> think about it, the only time a test needs to change
> is when the API of a function changes. And this is something that we are
> moving to anyway - to have much more stable
> internal APIs that likely won't change *that* much and I wouldn't
> propose to test every single internal function,
> but test through the API that is exposed to other parts of Blender. In
> this light, if the API changes,
> you'd probably have to change a few places anyway and updating the tests
> is then not that much more
> compared to the normal development work.
>
>  From my experience, having unit tests in place leads to code that is
> more bug free and stable and greatly reduces time spent debugging.
> So in the end the tests should *save* time.
> I actually don't think writing a small test takes more time than to add
> debug printf in strategic places and remove them later,
> which so far has been probably the most used debugging tool. The only
> difference is that the unit tests would remain
> as a permanent safety net and also act as a documentation on how the
> original coder expected the code to be used.
> The tests will be kept separate from the "productive code", so as to not
> clutter it, though we will work out how to structure this
> exactly as part of the GSOC project.
>
> To allow all developers to gain experience, I suggested to Leif to
> first  look into specific areas that would benefit most from testing,
> such as mesh tools or modifiers and write a small number of tests
> initially to get the framework and automatic running of the
> tests in place. We would also probably not add tests to areas that are
> currently under heavy development.
> As tests are added, my hope is that more developers would see the
> benefits and add tests for their area of code.
> Of course tests will be discussed with the maintainers for their area of
> code and reviewed. The only thing that would be asked
> of the developers is to not break any tests in code they are not
> maintaining.
> I also would count it as a huge success, if during the GSOC project we
> even get a test coverage of 10% of the code.
>
> The second result I hope to get from the GSOC project is to find out how
> difficult it is to test the different areas of Blender and where we
> could probably benefit from cleaning up the architecture and
> dependencies a bit.
>
> Lastly Leif also planned for some code documentation towards the end of
> the project, which I think would
> also benefit the whole development process and make life easier for new
> developers.
>
> The reason why we are currently looking into C or C++ testing frameworks
> is that Leif is more familiar with C/C++ than with Python
> and as far as operators or tools are concerned, the tests can
> theoretically test the same level of the code, the calls would just
> not go through the Python API. As a mentor I would like to allow the
> student to choose his preferred language for this to make things easier.
>
> I hope that I could clarifiy things a bit and alleviate some fears, I
> honestly don't think that it will be such a big issue to maintain the
> tests.
>
> - Andrea -
>
> _______________________________________________
> Bf-committers mailing list
> Bf-committers at blender.org
> http://lists.blender.org/mailman/listinfo/bf-committers
>


More information about the Bf-committers mailing list