[Bf-committers] Beta Testing for pre-release versions?

[Ton Roosendaal]

Hi,

Before accepting the useful suggestions for improving releases, It's =20
good to look at the exact circumstances of what went wrong. A couple of =20=

facts I'd like to straighten up.

There are 18 fixes mentioned in the 2.33a release notes. Only three of =20=

these were bugs related to features that worked in 2.32 and before:

1. the Windows installer
The whole installation topic isn't sorted out, there's confusion on the =20=

topic, and differences among platforms. The mistake is that testing the =20=

installer itself isn't part of the 'cvs freeze' and test cycle we do. =20=

No big issue, errors are allowed to make... once! :)

2. The audio sequencer crash
Thanks to the engine being returned, full audio was enabled in 2.33 by =20=

default. This showed on OSX a substantial slowdown at Blender startup. =20=

Doing the full audio init at startup was moved to the code where audio =20=

is actually used. Committed and tested at april 21, 9 days before =20
release. No error report came in until the very day after release!
It appeared that there were 2 independent audio systems in Blender, =20
where the sequencer didn't re-use code as expected. This was my error, =20=

a wrong assumption.

3. The error in UV unwrapping menu
Actually just a typo (according the commit) and caused by someone who =20=

finally took a lot of efforts improving UV editing in Blender. All of =20=

the new options worked and were tested, just one old option broke. Soit!

A decision to publish the 2.33a release update was because of the =20
previous mentioned three bugs. I consider that a normal quality =20
decision, and no reason at all to be afraid for Blender getting a bad =20=

name as buggy application at all. In contrary, in all of Blender's =20
public past (since january 1998), there's never before been so much =20
succesful attention for getting the quality at a higher level.

Actions for improving releases further:

-> further sort out the 'regression suite' with simple files that help =20=

to quickly verify the general fitness of Blender (a sequence audio test =20=

has been added btw!)

-> sort out the installation procedure, and how to test that

-> make sure developers closely work with users who are enthusiast =20
about their work, and who continiously test that exact area where's =20
being worked on.

-> (difficult) find a mechanism to have code reviews. Larstiq added a =20=

nice new feature to cvs commit mails, which now includes a link to a =20
diff. I expect at least everyone who's been assigned 'owner' of a piece =20=

of code should check on commits. And everyone who's familiar with parts =20=

of code should check on other peoples commits as well.

-> stricter release/test scheduling
Just comes back each time, and I don't even expect we'll ever fully =20
solve it. Deadlines just are the only means to get adrenaline pumped =20
into coders veins, and get them to work late nights fixing the last =20
details. :)

I don't know really how an 'official' betatester group could be =20
organized... but personally I'd welcome people who want to spend time =20=

on creating and maintaining a good regression suite, and be standby on =20=

forums and on this mailing list in the week(s) before release.

-Ton-



On Saturday, May 15, 2004, at 04:20 Europe/Amsterdam, Juan J. Pena Mena =20=

wrote:

> Don't you find annoying that each new version of Blender ships with =20=

> new bugs, and most of the time those bugs are related to things that =20=

> used to work fine on previous releases (while one would expect to find =
=20
> more bugs related to new features) <image.tiff> Some better quality =20=

> control measures should be put in practice.
>
> I know that there is a Test Release forum at BO, and it works really =20=

> fine when it comes to testing new/experimental features, but it falls =20=

> flat out when it comes to caching bugs non related to the spectacular =20=

> new feature of the moment. Most of the bugs showing on new releases =20=

> are quite obvious, but still managed to make it into the final =20
> release. See 2.33a: First day of release and already two QUITE obvious =
=20
> bugs related to the interface !!!
>
> I would suggest for the platform managers to post the final compiled =20=

> versions (built after the CVS is TOTALLY frozen) a fixed number of =20
> days on the forum before the releases are made official. i.e. The just =
=20
> released 2.33a would be posted today on the forum, have bugs reports =20=

> come in, and 7 days from today then 2.33a would be officially released =
=20
> if there isn't any fix to be made.
>
> Even further, I would like to see some sort of official beta testers =20=

> group, not just the casual forum passer-by.
>
> I know that by taking this approach the release cycle would be =20
> extended for a week each time (or even more, depending on the bugs), =20=

> but it is outrageous how blender has been shipping release after =20
> release with so many bugs.=A0 Certainly the bugs eventually get into =
the =20
> bug tracker and are sorted out, but isn't it better if the bugs where =20=

> detected BEFORE the release goes public.
>
> Shall the trend continue, and sooner than later Blender will be known =20=

> as =A8The free 3D app with an odd interface and always many bugs=A8, =
and I =20
> don't need to explain how hard is to erase those kind of negative =20
> images from the public's mind.
>
> Apollux.
>
> P.D.=A0 If the official beta-tester groups are formed, I volunteer for =
=20
> the Linux platform.
>
>
------------------------------------------------------------------------=20=

--
Ton Roosendaal  Blender Foundation ton@blender.org =20
http://www.blender.org