\o/

This was an interesting task because it involved a little bit of each job in the RelEng world at Mozilla:

I had to update the puppet manifest which we use to distribute files and manage configurations for the build farm so the signing keystore and configuration file would get to all of the production build servers. I also had to write a small Python program to make the actual call to Jarsigner because if we just called it from within the Buildbot step, the output would include the keystore passwords and that appears on the build server’s web page – not good.

I also had to drill deep into the various parts of the Mozilla Buildbot configuration and custom code because the Android build was new and just different enough from the other mobile builds that it wasn’t just a cut-n-paste solution.

And all of this I was doing the Mozilla way for the first time and let’s just say they have some unique custom code :)

But today the first half of the task was completed and that allows the nightly signed builds to be made public. What remains is to remove some work-arounds (for the packaging and tests) and then to start building the steps required to generate a release version of Fennec for Android.

Build/Release starts from when the first line of code is created and continues all the way to when a user installs and runs your application or accesses some part of your server or service. Yep, it’s inter-woven into all aspects of a project simply because if you don’t have quality steps all thru that process you end up with crap. It may be pretty shiny crap, but it’s still crap.

I find one of the better ways to visualize Build/Release is to work backwards from the customer and outline what it would take to research and solve any problem the customer is having. This often is classified by developers as bugs, but I’ve gotten more accustomed to thinking of them as Issues – because sometimes it’s not a coding defect that is causing the Issue, but rather a process or UX problem.

Your project, let’s call it twerzle has just been installed on your customer’s laptop and bang – they have a problem. What happens next depends on how prepared you were during the Release part of the Build/Release process.

• Do you have a support link or area on your project page
• Is the version number or other project identifier easily discovered or located
• Do you have a way for the user to give you the information you need without them having to be a digital forensics major?

These items will change from release to release, so you need to have outlined what steps are required to update your web site and secondary documentation – you do have a README file or CHANGES file and that information is accessible right? So our twerzle user has found your support link on the page they downloaded the app and that page also has what the latest version is so they don’t have to guess – what is next?

• Is this link a simple email box? Is someone assigned to answer the emails? Do you have an auto-responder?
• Is the link to a support forum? Has someone been tasked to monitor the forums? Is there a sticky-post showing the README contents?
• Is the link to an IRC forum – better hope you have 24hr coverage because that’s a fast way to an irked user if no one is around to answer them 24×7

So, community support methods and practices aside (that’s a whole other subject that I don’t even begin to understand fully), you now have a support request from a user and for the sake of our example, it actually contains a version number and a complete description of the problem. This is when you find out if you have a real Build/Release process.

Given the version information for the issue, can you now go back and install a clean instance of that exact version? No, I am not talking about a developer having what they think is a clean checkout of the source. I am talking about being able to retrieve the exact distribution image and then being able to install it on a computer that has NO development environment. Yep, this step is where most projects fail as they depend on developers to reproduce issues. But that works only for coding defects – not for application usage or environment issues.

That is what the crux of Build/Release is, the ability to have all of the information that goes into a project’s distribution available to back-track and solve any issue a customer may have. What goes into this nebulous hand-waving bundle depends on a lot of details, but it almost always boils down to three items:

1. Reproducibility
2. Documentation
3. Deployment

Reproducibility means simply, do you have available all of the source code, the libraries, and the environment needed to build said source code available for each release you have made. If you do not, then you cannot truly reproduce a release and while you may be able to figure out what the problem is, you will not be able to fix it for that release. You will have to fix it in the trunk and hope that the code is the same and that the fix doesn’t break other things.

Documentation is all of the things that make up the how and why of the building and creation of a distribution. What versions of libraries are needed, what steps are to be taken, what tools and assumptions are being used and what the process is to bundle all of that into a distribution. Yes, here source code and scripts do count – as long as you include them in the distribution snapshot :)

Deployment is basically all of the above but from the point of view of non-developers. What is the process to deploy a new release for testing, how do you take the developer generated items and QA them. Once they are tested, what are the steps for deployment to the public. Do you have upgrade steps, are there extra tools and/or documentation that needs to be generated for upgrades versus new installs.

I realize I have only given a very short overview of what I consider a Build/Release process is but most of the time the above is what a lot of projects don’t do well for a lot of little reasons. I myself need to sit down and try and map out how I will communicate more details since it is so project specific.

Thanks for reading and I hope my bumbling attempt at communicating this process has come across in a useful manner.

What is a Release Candidate? The Apache Incubator site has a guide that defines it as

Release Candidate

Collection of artifacts to be tested and voted on in order to release them. Sometimes folks use this term to refer only to a candidate source package.

and while that’s a rather dry wording, it does outline the most crucial piece – that it is a collection of artifacts (aka source files, images, etc.) that are to be considered for the next release. Those items are generated from a “frozen” source tree and used to test that the release candidate is ready for distribution.

In all but the extreme edge cases, a Release Candidate is a very specific thing that can only come about because the Developers and the QA team have decided that the current set of bug-fixes and/or new features are stable enough to be ready for consideration as the next release. This does not mean you can take the latest generated output from your continuous integration tool (or at worse case the tarball the developer was using as his test install) and move that to the web site and stamp a v2.0 sticker on it.

Only for the most basic of projects can you consider the developer version of a distribution to be ready for release – the vast majority of projects all have secondary information to be included and also will require additional changes to any number of support-related documents or sites. At the very least you need to create your ChangeLog to outline what has been fixed and what is new.

So a release candidate starts with what the developers have created, gets updated and massaged so that it contains the extra information that makes a distribution and then that part is tested in a non-developer environment. Once those tests are done can you consider it ready to be installed or distributed to the public.

The flip side is what things can cause a release candidate to not be considered – that in my opinion, is a very short list and depends on how the project defines “show stopper” so it’s hard to generalize into a post. The one item for me that is always an immediate veto is if it is discovered that code that isn’t part of a bug fix or a feature that was scheduled for the release is found to have been introduced. The reason why is really simple – how can you properly test code if you don’t have the bug notes or feature notes to guide you in testing.

That is basically a very fast overview of what I consider a release candidate to be. I’m going to work on an outline for what I think are the core tenets for any open source project and start a blog post series covering them.

thanks!

Unfortunately it doesn’t have any author or history so I never knew who to thank :(

With the Leopard upgrade it suddenly stopped working. The HFS format step was failing and was just outputing it’s help text. Looking at the line just above in the script that generated that output, I noticed that the echo output was not correct:

...
created: /tmp/20096.dmg
Creating HFS partition Chandler_iosx_debug_0.7.3.dev-r15733 on /tmp/20096.dmg at

usage: newfs_hfs [-h | -w] [-N] [hfsplus-options] special-device
options:
...

The line that starts “Creating HFS partition…” should end with “at /dev/disk5s1″ – what drive the HFS formatting would have happened on.

The bash script segment responsible for that is:

DEVICES=`hdid -nomount $TMPFILE`
DEVMASTER=`echo $DEVICES| awk '{ print $1 }'`
DEVHFS=`echo $DEVICES| awk '{ print $5 }'`
echo Creating HFS partition $NAME on $TMPFILE at $DEVHFS
newfs_hfs -v "$NAME" $DEVHFS

The line of interest is the output of hdid – up until Leopard it would output something like:

oliver:~ bear$ hdid -nomount /tmp/bear.dmg
/dev/disk2          	Apple_partition_scheme
/dev/disk2s1        	Apple_partition_map
/dev/disk2s2        	Apple_HFS

Which shows the drive and two partitions, but under Leopard it now outputs:

imac3:~ bear$ hdid -nomount /tmp/bear.dmg
/dev/disk5          	GUID_partition_scheme
/dev/disk5s1        	Apple_HFS

When you pass that to an environment variable (which, remember, removes all line feeds and makes it into a long string) you now get:

DEVHFS=/dev/disk5  GUID_partition_scheme  /dev/disk5s1  Apple_HFS

Instead of

DEVHFS=/dev/disk2  Apple_partition_scheme  /dev/disk2s1  Apple_partition_map  /dev/disk2s2  Apple_HFS

Basically to try and get to an actual point :) – when

awk '{ print $5 }'

is run on the new output there isn’t a fifth item so you get nothing back.

The fix is to switch to a method that isn’t dependent on the layout of the columns *and* the lines:

DEVHFS=`hdid -nomount $TMPFILE | grep Apple_HFS | awk '{ print $1 }'`

Now we take the raw output, grep for the line we really want and grab the first column of that. With the extra bonus of it working now on all OS X’s I could get my hands on to test.

Anywho, just wanted to post this in case someone else runs into their makediskimage.sh script suddenly start to fail.

Rui Carmo posted a small bit about Chandler:

I used to think this would be able to stand up to the Outlook hegemony, but I find the current GUI hopelessly confusing. And what’s with the need for a separate Intel package?

While I can’t answer about the GUI thought I can about why we publish a separate package for Intel-based OS X Macs.

Basically the decision was made back when our choices when Panther and Tiger were both PPC and Intel was *very* new, was to go with two package that were smaller in size. Another concern was that the techniques for creating a universal binary was still new and un-proven in the Python and wxPython realm.

Thankfully that is not an issue now and one of the things I’m working on is a change to do exactly that. I still feel that we will probably have a huge download size but I guess we will find out.

update: of course after hitting send I realized I can speak also about the GUI issue: The Chandler GUI reflects/exposes a very powerful set of tools to help manage and deal with incoming items (mail, tasks, etc) and also the daily routine of deciding just what needs to be done when and how. So it *is* very busy and that can, I think, come off as confusing. What’s missing in our Preview is a road-map for completely new users.

So we went with Subversion and really haven’t had any regrets. Merges and branches were much saner and after the Subversion team released the FSFS (filesystem storage) the server side repository issues went away.

Recently tho people have been asking me about distributed version control systems like mercurial, bazaar and darcs.

The first two I’ve been following indirectly by keeping track of the work the Mozilla build/release team have done in trying to pick their next system. See J. Paul Reeds blog entry Version Control System Redux Redux and also their wiki page for details. The short-form result is they haven’t chosen one yet.

darcs I’ve been following more directly because I’ve started using it to try and help the Buildbot team with some patch and bug work. So far, while it’s been different, I haven’t had any issues but I also haven’t gotten much beyond the more basic operations. I plan on posting a more detailed look at it after I do some true distributed operations.

One of the newer programs (for me that is) that has hit the nets with some force is GIT but I think that is because recently Linus Torvalds recently gave a talk at Google about it and seems to be on a tear promoting it :)

But it really seems to be worth looking into. A recent article about it’s design, GIT for Computer Scientists, really showed some interesting internal concepts and I like what I see. So I need to figure out how to work it into my project space to give it a fair look.

There are a couple other distributed systems I haven’t talked about, Monotone and Arch, but that’s because I know next to nothing about them. If you want to find out details on them I would recommend reading the Version Control Blog for articles about them – it’s a great read.

First an outline of the problem and how Chandler uses batch files.

Chandler uses two batch files (and two bash scripts for OS X and Linux) to start Chandler and our own build of Python from the command line. The scripts make sure the environment is somewhat sane and preset the PATH and PYTHONPATH variables to avoid any local environment issues from “leaking” into the Chandler runtime environment.

To make sure that our values don’t leak back out into the user’s environment we use setlocal/endlocal to “wall off” our changes, but this comes with a price. Once the endlocal boundary is crossed you lose the value of ERRORLEVEL and any variable you may have put ERRORLEVEL into for safe keeping, for example:

set RC=
setlocal
somecommand.exe
set RC=%ERRORLEVEL%
echo %RC% %ERRORLEVEL%
endlocal
echo %RC% %ERRORLEVEL%

Assuming somecommand.exe returns 1 you will get this for your output:

1 1
0 0

Not exactly helpful is it? While working on the problem Andi from work discovered a possible solution that is pretty slick – take advantage of the command chaining of & and the delayed expansion of environment variables:

set RC=
setlocal
somecommand.exe
echo %ERRORLEVEL%
endlocal & set RC=%ERRORLEVEL%
echo %RC%

This will generate the proper result so combine that with exit /B and it generates the proper result.

endlocal & exit /B %ERRORLEVEL%

Except for one small problem :)

exit /B doesn’t “transfer” properly when the batch file is called from a Python program using subprocess. I even worked on variations that involved shell=True and other attempts. It just would not take the return code and pass it on.

After spinning my wheels for a while on this, dropping it for a couple days and then coming back and spinning more wheels I finally decided to look at some of the bug reports for Ant and Maven to see how they solved it. It turns out that they really don’t solve the problem of getting the exit code value but rather just get *any* value to be set to flag that something had happened. The reason for this is because they have a wider range of OS’s to support: windows 95 on up to the current version.

But while reading the bug comments I hit upon an idea that ended up working! Yea for Open Source because otherwise the info would have been buried behind some corporate firewall.

The variation I ended up with was this:

set RC=
setlocal
somecommand.exe
endlocal & set RC=%ERRORLEVEL%
goto omega

:returncode
exit /B %RC%

:omega
call :returncode %RC%

This combines the exit /B trick with the & trick and adds a pretty slick wrinkle (if I do say so myself ;) — by calling the subroutine I can use exit /B to set the errorlevel and because the call is the last command of the batch file that errorlevel is passed out to the caller just like you expect it to be.

So hopefully Google will find this and enable someone to spin their wheels for far less time than I ended up doing :)