[Bf-committers] Beta Testing for pre-release versions?
Ton Roosendaal
bf-committers@blender.org
Sat, 15 May 2004 13:12:47 +0200
Hi,
Before accepting the useful suggestions for improving releases, It's =20
good to look at the exact circumstances of what went wrong. A couple of =20=
facts I'd like to straighten up.
There are 18 fixes mentioned in the 2.33a release notes. Only three of =20=
these were bugs related to features that worked in 2.32 and before:
1. the Windows installer
The whole installation topic isn't sorted out, there's confusion on the =20=
topic, and differences among platforms. The mistake is that testing the =20=
installer itself isn't part of the 'cvs freeze' and test cycle we do. =20=
No big issue, errors are allowed to make... once! :)
2. The audio sequencer crash
Thanks to the engine being returned, full audio was enabled in 2.33 by =20=
default. This showed on OSX a substantial slowdown at Blender startup. =20=
Doing the full audio init at startup was moved to the code where audio =20=
is actually used. Committed and tested at april 21, 9 days before =20
release. No error report came in until the very day after release!
It appeared that there were 2 independent audio systems in Blender, =20
where the sequencer didn't re-use code as expected. This was my error, =20=
a wrong assumption.
3. The error in UV unwrapping menu
Actually just a typo (according the commit) and caused by someone who =20=
finally took a lot of efforts improving UV editing in Blender. All of =20=
the new options worked and were tested, just one old option broke. Soit!
A decision to publish the 2.33a release update was because of the =20
previous mentioned three bugs. I consider that a normal quality =20
decision, and no reason at all to be afraid for Blender getting a bad =20=
name as buggy application at all. In contrary, in all of Blender's =20
public past (since january 1998), there's never before been so much =20
succesful attention for getting the quality at a higher level.
Actions for improving releases further:
-> further sort out the 'regression suite' with simple files that help =20=
to quickly verify the general fitness of Blender (a sequence audio test =20=
has been added btw!)
-> sort out the installation procedure, and how to test that
-> make sure developers closely work with users who are enthusiast =20
about their work, and who continiously test that exact area where's =20
being worked on.
-> (difficult) find a mechanism to have code reviews. Larstiq added a =20=
nice new feature to cvs commit mails, which now includes a link to a =20
diff. I expect at least everyone who's been assigned 'owner' of a piece =20=
of code should check on commits. And everyone who's familiar with parts =20=
of code should check on other peoples commits as well.
-> stricter release/test scheduling
Just comes back each time, and I don't even expect we'll ever fully =20
solve it. Deadlines just are the only means to get adrenaline pumped =20
into coders veins, and get them to work late nights fixing the last =20
details. :)
I don't know really how an 'official' betatester group could be =20
organized... but personally I'd welcome people who want to spend time =20=
on creating and maintaining a good regression suite, and be standby on =20=
forums and on this mailing list in the week(s) before release.
-Ton-
On Saturday, May 15, 2004, at 04:20 Europe/Amsterdam, Juan J. Pena Mena =20=
wrote:
> Don't you find annoying that each new version of Blender ships with =20=
> new bugs, and most of the time those bugs are related to things that =20=
> used to work fine on previous releases (while one would expect to find =
=20
> more bugs related to new features) <image.tiff> Some better quality =20=
> control measures should be put in practice.
>
> I know that there is a Test Release forum at BO, and it works really =20=
> fine when it comes to testing new/experimental features, but it falls =20=
> flat out when it comes to caching bugs non related to the spectacular =20=
> new feature of the moment. Most of the bugs showing on new releases =20=
> are quite obvious, but still managed to make it into the final =20
> release. See 2.33a: First day of release and already two QUITE obvious =
=20
> bugs related to the interface !!!
>
> I would suggest for the platform managers to post the final compiled =20=
> versions (built after the CVS is TOTALLY frozen) a fixed number of =20
> days on the forum before the releases are made official. i.e. The just =
=20
> released 2.33a would be posted today on the forum, have bugs reports =20=
> come in, and 7 days from today then 2.33a would be officially released =
=20
> if there isn't any fix to be made.
>
> Even further, I would like to see some sort of official beta testers =20=
> group, not just the casual forum passer-by.
>
> I know that by taking this approach the release cycle would be =20
> extended for a week each time (or even more, depending on the bugs), =20=
> but it is outrageous how blender has been shipping release after =20
> release with so many bugs.=A0 Certainly the bugs eventually get into =
the =20
> bug tracker and are sorted out, but isn't it better if the bugs where =20=
> detected BEFORE the release goes public.
>
> Shall the trend continue, and sooner than later Blender will be known =20=
> as =A8The free 3D app with an odd interface and always many bugs=A8, =
and I =20
> don't need to explain how hard is to erase those kind of negative =20
> images from the public's mind.
>
> Apollux.
>
> P.D.=A0 If the official beta-tester groups are formed, I volunteer for =
=20
> the Linux platform.
>
>
------------------------------------------------------------------------=20=
--
Ton Roosendaal Blender Foundation ton@blender.org =20
http://www.blender.org