How not to QA your Open Source project

I really did not want to write a post about this because I completely understand the how and why of what happened, but after 2 days of not having access to a service that I’ve come to rely on it is getting hard to be patient and quiet.

Two days ago the very popular and useful Identi.ca site was updated to version 0.9rc3 along with other Status.net sites. This is not the issue, Identi.ca has always been the proving ground for new features and as the open source arm of the project you have to expect some bugs.

What is the issue is that this release candidate (that is what the “rc” part of 0.9rc3 means) contained a lot of new code to deal with how the internal queues are managed.

Let me repeat that:

a release candidate contained new code that was not a simple bug fix

So now the entire XMPP stream has been taken offline to deal with the integration issues between the code and their XMPP server.

Come on guys, surely you have a QA and/or DevTest server? Why the hell are you continuing to introduce new code into release candidates.

Oh, and why don’t you just rollback the change?

And yes, before I get flamed and yelled at that “It is *free* so you have no right to complain” let me point out that if they offered a pay version, I would pay. Also since they have custom users and, I believe, even corporate installs, they should have better QA. It really is that simple.


This entry was posted on January 15th, 2010 and is filed under mutterings. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.

 
« hello 2010
What is a Release Candidate anyways »
 
 
  • Absolutely understood. We'll release this version as 'beta3' instead of 'rc3' to make it clear what level of volatility there is in the branch right now.

    As far as QA is concerned: fair point. Our production environment has tens of thousands of registered users with Jabber enabled, from hundreds of different servers. It's a difficult environment to replicate accurately in development testing. Our particular problems with ejabberd this time around have been painful; we're actively seeking another Jabber server that can handle the traffic profile we need.

    Thanks for the heads-up. It's good to have our core supporters keep us honest. We're going to try for a smoother transition with the upcoming 0.9.0 release.
  • Thanks for the change from 'rc' to 'beta' - that goes a long way to identifying the nature of the changes contained within the release.

    I'm struggled to make sure the tone of the post was not skewed to the hateful side of finger-waggling, but rather to the "hey, that was unpleasant" side. Testing in the environment you work in *is* a very challenging area and that is exactly where being an open source project can help - knowing that it was a beta change we (the XMPP crowd and community) would have been able to setup multiple test sites and banged on it for you.

    Ugh, ejabberd, say no more.
  • It seems to me the issue is more about not properly testing a service rather than how the project is managed an open source level. If they had caught the problem in testing, it really wouldn't have mattered that they put new code in something they labelled as an RC. I think it's a good practice to not "add code" to RCs, but perhaps they thought of it as a bugfix, and sometimes that is more of a murky issue than it seems.

    Perhaps a post with the *right* way to do things is in order. This is what you do for a living, and I'd love to see what you use as guidelines for determining whether something is a bug, a feature, a refactor, what. What qualifies as a release, an RC, a beta, and an alpha? I would use/consider such advice in my own projects.
  • I guess I did not provide a lot of context or other information so my reaction does seem out of sync. The particular issue I'm most irked about is the change to the back-end queue structure which did not exist in rc1 that I know of so I do not see how it could have been considered.

    It does become in the end a question of process and getting those "best practice" items firmly in place so doing the release process becomes purposeful instead of reactionary.

    I'm still not going to bring up specific examples for this instance because I don't want to talk about things that were discussed in the irc channel as I consider that to be a place where frank and open discussion should be done without worry that someone is going to pull it out of context and blog about it.

    Let me work on creating an outline on the interesting challenge you've given me and generate some posts :)
blog comments powered by Disqus

Creative Commons Attribution-ShareAlike 3.0 United States
This work by Mike Taylor is licensed under a Creative Commons Attribution-ShareAlike 3.0 United States.