Comments on: How not to QA your Open Source project

By: bear

bear — Sat, 16 Jan 2010 21:12:09 +0000

Thanks for the change from 'rc' to 'beta' – that goes a long way to identifying the nature of the changes contained within the release.

I'm struggled to make sure the tone of the post was not skewed to the hateful side of finger-waggling, but rather to the “hey, that was unpleasant” side. Testing in the environment you work in *is* a very challenging area and that is exactly where being an open source project can help – knowing that it was a beta change we (the XMPP crowd and community) would have been able to setup multiple test sites and banged on it for you.

Ugh, ejabberd, say no more.

By: Evan Prodromou

Evan Prodromou — Sat, 16 Jan 2010 21:01:03 +0000

Absolutely understood. We'll release this version as 'beta3' instead of 'rc3' to make it clear what level of volatility there is in the branch right now.

As far as QA is concerned: fair point. Our production environment has tens of thousands of registered users with Jabber enabled, from hundreds of different servers. It's a difficult environment to replicate accurately in development testing. Our particular problems with ejabberd this time around have been painful; we're actively seeking another Jabber server that can handle the traffic profile we need.

Thanks for the heads-up. It's good to have our core supporters keep us honest. We're going to try for a smoother transition with the upcoming 0.9.0 release.

By: bear

bear — Sat, 16 Jan 2010 02:30:59 +0000

I guess I did not provide a lot of context or other information so my reaction does seem out of sync. The particular issue I'm most irked about is the change to the back-end queue structure which did not exist in rc1 that I know of so I see how it could have been considered.

It does become in the end a question of process and getting those "best practice" items firmly in place so doing the release process becomes purposeful instead of reactionary.

I'm still not going to bring up specific examples for this instance because I don't want to talk about things that were discussed in the irc channel as I consider that to be a place where frank and open discussion should be done without worry that someone is going to pull it out of context and blog about it.

Let me work on creating an outline on the interesting challenge you've given me and generate some posts :)

By: Fritzy

Fritzy — Sat, 16 Jan 2010 02:20:51 +0000

It seems to me the issue is more about not properly testing a service rather than how the project is managed an open source level. If they had caught the problem in testing, it really wouldn't have mattered that they put new code in something they labelled as an RC. I think it's a good practice to not “add code” to RCs, but perhaps they thought of it as a bugfix, and sometimes that is more of a murky issue than it seems.

Perhaps a post with the *right* way to do things is in order. This is what you do for a living, and I'd love to see what you use as guidelines for determining whether something is a bug, a feature, a refactor, what. What qualifies as a release, an RC, a beta, and an alpha? I would use/consider such advice in my own projects.