Let’s face it, developing web applications is, in some ways, getting more and more complex as time goes on. Sure, there are frameworks like Django, Rails or Seam to help you get sites built in record time with minimal resources, but at the same time applications are becoming more complex and more demanding. Being able to support Ajax-heavy applications through multiple browsers across multiple operating systems is now a primary requirement, as is being able to scale to thousands (or, if you’re lucky, millions) of users.

The frameworks help primarily with the initial effort of building the functionality of your site, but they are less useful addressing the longer-term problems associated with application development. The “old school” maxim that you spend 90% of your time supporting (as opposed to writing) your application is still fairly accurate. This article looks at one way of cracking the problem of regression testing (retesting previously working parts of an application following a new build) a modern web application, using two superb open source projects: Hudson and Selenium.

What will you get out of this article? Our set-up achieves the following:

  • It checks our Subversion repository every hour to see if anyone has committed any changes.
  • If they have, it updates the project from Subversion and builds it.
  • It then creates a clean version of our application database, loads in reference data and deploys the application on our app server.
  • A job is triggered that runs through a series of tests in a remotely-controlled web browser on the fresh application.
  • Anything that deviates from the accepted norm is logged and screenshots of the web browser are taken.
  • Screenshots of the browser are also taken for key pages of the site for later checking by a human.
  • If any of the tests fail, the developers responsible for the changes are notified by email of the problems.
  • Our issue tracker is updated with any issues that were fixed in the build.

Here’s what the architecture of the setup looks like:

A figure showing the architecture of the Hudson testing setup

The best thing about our set up is that it doesn’t rely on a single shell script. Really! For those applications that do need some form of scripting, Hudson offers relevant hooks in the build process to trigger these easily.

If You Build It, They Will Come

The first piece of the puzzle is Hudson. Hudson is a build server designed for Java applications, but it will work quite happily (albeit with slightly reduced functionality) with any language you choose to throw at it. Hudson is responsible for checking if your application needs building, building it, triggering any testing that you have defined and, finally, reporting on the state of the build.

The first step is to download Hudson and install it. To create a new project you need to provide some details:

  1. The source code repository that you are using. SVN or CVS support are provided out of the box.
  2. Under “build triggers” choose Poll SCM and in the schedule enter something like “5 * * * *“. This is a CRON-like notation that basically means “every five minutes past the hour”: Hudson will check your repository every hour. If it has changed, a build will be triggered.
  3. Select a build step. If you are running a Java project using either Ant or Maven, you’re in luck, because it’s as simple as adding the details of the appropriate script and target here. If not, choose Execute Shell; you can enter shell commands here to carry out your build and deployment tasks.
  4. Add an Email Notification post-build action to enable Hudson to notify the developers who were responsible for breaking the build! If you’re running a Java app, you can also include all your JUnit test results in the application reports.

The Hudson configuration screen

The most important aspect of the build step is to get the source code updated and deployed on your test web server. If you aren’t writing a Java application, you can still achieve this relatively easily using the Execute Shell build process. At its simplest, it can just copy the files from the Hudson server to the development server web root.

If the build fails, you will receive an email stating what went wrong, with a link to the Hudson page giving further details. If Hudson can figure out who committed the offending code, it will even email them too. If the tests failed, you can get good insight into where in your code the failure occurred, helping you to diagnose exactly what went wrong.

Hudson testing results screen

Browser Testing Is Boring. Get Someone Else To Do It For You…

The second piece of the puzzle is Selenium. Selenium is a bunch of JavaScript craziness (well, they call it a “Web application testing system”) that is a real lifesaver. Basically, you give it a bunch of web tests, each consisting of a series of actions to perform on the site, and it replays the commands in a browser window automatically.

We’re going to use Selenium Remote Control, which allows you to set a machine aside as the browser testing box, and control it remotely from your Hudson build. As if by magic, the browser will start up and run through your recorded tests. Hudson will alert you if anything out of the ordinary appears.

Download Selenium Remote Control, and install it. Once you have Selenium Remote Control up and running, you will want to install the Selenium IDE, a plug-in for Firefox. The IDE allows you to go onto your site, hit the record button, and then browse around as if you were using the site normally. It records any user actions you make, like clicking on a button, or entering data into a form field. You can then save these base tests to a file ready for regression testing.

You can drive Selenium Remote Control with a number of different clients: Java, .NET, Perl, PHP, Python, Ruby and JavaScript are all supported. Export your Selenium IDE tests into the chosen language, and then hook it up with Hudson as part of the build process. Driving the Selenium server using the language of your choice means that you have a large amount of control over your testing. You can take the base tests you created using the IDE and then augment them with conditional logic, loops, whatever you like . As an example, the following method tests one part of our application (in Java):

    public void testSendMessage() throws Exception {
        try {
            selenium.type("text1", "testuser1@mailinator.com");
            selenium.type("name", "123456");
            // Add the second user as a contact
            for (int second = 0;; second++) {
                if (second >= 60) fail("timeout");
                try { if (selenium.isElementPresent("xpath=id('inboxToolbar')/li[1]/a")) break; } catch (Exception e) {}
            assertTrue(selenium.isTextPresent("Ben Rometsch"));
            selenium.select("recipientSelect", "label=Ben2 Rometsch2");
            selenium.type("subject", "This is the subject");
            selenium.type("message_body", "This is the body");
            assertTrue(selenium.isTextPresent("This is the subject"));
            assertTrue(selenium.isTextPresent("Ben2 Rometsch2"));
            selenium.click("xpath=id('header')/ul/li[2]/a"); // Log out
        } catch (Exception e) {
            super.takeScreenshot("error-" + this.getName());


You will also be able to replay your tests with the language IDE you use. If you make use of a debugger, you can also step through your tests to isolate any problems that might occur.

Selenium has a number of ways of testing that the application is working as intended:

  • Any HTTP errors (like 404 or 500) will be classified as a failure.
  • Assertions against text can be made. So, for example, if you are expecting a certain string to be present, you can make a simple assertion stating that it should be.
  • Assertions against the DOM structure of your page can also be made using XPath (a killer Firefox addon that can really help with this is XPath Checker)

Do you support IE for Mac?

Regression testing application functionality is one of the main benefits of Selenium and Hudson, but with some tweaking you can also make cross-browser testing far less painful than it normally is. You will still need to have a real live human being checking things over, but the process becomes much quicker.

Selenium Remote Control has a method available to save a screenshot of the current browser to the file system of the local machine. We can build on this functionality to have screenshots of the site generated for key pages across multiple browsers, all running from the same host client machine.

The final test we run simply saves key screenshots of the site across four different Selenium browser sessions (Firefox, Internet Explorer, Safari and Opera). The screenshots are stored onto a shared drive on our development server. The Hudson build number is passed into the Selenium session, and we make use of this to organize all the screenshots. This way we can wait for a Hudson build to complete successfully, then sanity-check the screenshots taken in multiple browsers to ensure that nothing has been broken.

Talking to JIRA

We use JIRA to do all our bug tracking. It’s really superb. If you’re working on an open source project, it’s also free. Hudson has a plug-in architecture that means it can be easily extended. One of the available plug-ins connects the two applications.

Once the connection between a Hudson project and a JIRA project is made, the typical bug tracking process becomes:

  1. A user finds a bug and raises an issue in JIRA.
  2. A developer fixes the bug and commits the code changes to SVN. Crucially, they include the unique JIRA bug ID (which might be something like “PIPELINE-157″) as part of the SVN commit message. So the commit message might read: “Fixed PIPELINE-157. Re-factored the email daemon to better handle multi-part messages”.
  3. Hudson recognizes that the new changes have been committed, and performs a build. It also notices the JIRA bug ID in the SVN message and includes a link to the JIRA issue as part of the build change-set.
  4. If the build completes successfully, and there are no failed tests, Hudson posts a comment to the JIRA issue concerned, with a link back to the Hudson build within which it is fixed.
  5. The user that raised the issue receives an email of the message added by Hudson and checks that the new build has fixed the problem. They then close the bug.

Note how much of this process is automated.

When performing Staging or Production builds, the JIRA integration can help generate a list of fixed issues since the last build. 

Below is a change-list from Hudson. The links to things like MTB-211 point back to the JIRA issue.

Hudson change list screen

And here is the comment from the JIRA issue itself.

JIRA comment screen

No Silver Bullet

Writing good Selenium tests is tricky. The core assertion you can make are that either text is present, or that it is not. Although that can get you a very long way, it is no silver bullet – your application can break in many ways!

Earlier versions of Selenium also had serious problems with the timing associated with browser-based testing, and required the tester to place a lot of “wait” commands within their tests to ensure that page elements had loaded completely before any assertions could be made. A lot of progress has been made in this area, but the problem rears its ugly head when testing applications that make lots of dynamic page calls. Writing tests that are not horribly brittle when working on an Ajax-heavy site can be testing (pun intended!). Sometimes, unfortunately, there is no replacement for a human pair of eyes.

All that being said, our time-sheet and invoicing application, Pipeline, has benefited greatly from the set-up described above. Getting the environment and associated tests set up took about five days’ worth of work, but the feeling you get when you trap your first bug, automatically, is really great. More importantly, you can make changes to your code safe in the knowledge that the computers whirring away in the background will be checking your work as you go.