Using your IT monitor for automated testing

An automated IT monitoring system is a must-have for any team these days; few things give me more pleasure (at work) than seeing a web page full of green boxes telling me that all our systems are running smoothly. I like Nagios because it's robust and easy to install / configure / extend, but this tool is showing its age a bit; you might want to look at Zenoss or Zabbix if you're starting from scratch. Nagios' inability to show trend graphs out of the box might be an annoyance - more on that in a future post.

Monitors are great for alerting you when a system goes down or runs out of resources, but I hadn't used them for QA until recently, when I realized that they're also ideal for smoke testing new product builds. If you're working with web applications, this is a piece of cake - all you need to do is:

• set up a cron or batch job to install your new build on a schedule;
  
• have your monitor tool check one or more URLs in your application.

Since monitoring tools can alert you by email or SMS when anything goes wrong, you can get timely feedback if something goes wrong - maybe last night's changes broke the app so that it doesn't even come up any more, or perhaps there's an unexpected change to the content returned by your test URL. If you do your reinstall overnight, the alerts can be waiting in your in box when you start work the next day (Nagios also has a nice Firefox plugin that turns red when there's an alert, in addition to setting off a loud siren that's hard to ignore !)

The monitor runs 24x7, so you also get the built-in bonus of being alerted when somebody inadvertently shuts down your test application; you'd also get warned if the app crashed after running out of resources, although you'd still have to go in and debug the cause in that case.

Here's how a typical test would look in Nagios:

check_command check_http!-p 8080 -u /test/url/ -s "Text Expected"

This uses Nagios' built-in check_http command to go to the URL /test/url on port 8080 and look for a specific string in the HTML output (the host details are specified elsewhere in the Nagios setup).

check_http has lots of other useful options such as checking for a regular expression, sending POST data, controlling the response to a redirected page, checking the age of the returned document or checking the validity of the site's SSL certificate.

check_http does have a couple of limitations:

• it will only handle basic web server authentication; you can get around this for web apps that use authentication libraries by using the Nagios plugin for WebInject, which will handle web pages with input fields. WebInject also lets you set up multi-step tests that navigate to more than one page; you can write some fairly sophisticated tests this way without having to resort to a full-blown web testing tool.
• because check_http does a simple HTTP GET, it won't return any page content that's rendered in the browser; this limits what you can check for if your pages include a lot of JavaScript. If you need to test pages with a lot of JavaScript, you'll need to use a more comprehensive Web testing tool like Selenium that can emulate a browser. In this case I'd recommend integrating the tests with your continuous build system rather than the IT monitor, although tools like Nagios have a simple plug-in architecture that lets you integrate additional tools fairly easily.
  

Take a look at this approach if you need to get some basic testing in place and are strapped for time, people and / or money - all the tools mentioned above are open source.

How I got here

I've had quite a varied career in high tech, starting off as a developer and doing spells in QA, customer management and project management (some of the transitions involved bouncing from the UK to the US twice, but that's another blog post).

For the last few years I've settled into technical operations, which I'd essentially define as all the things that keep an engineering team running smoothly. The job can cover a lot of ground: testing, customer support, project management, IT and, of course, the people management work involved in keeping all those things running. Recently I've been working with small startups, so I get to do a lot of hands-on work as well as steering the ship.

Tech ops is a big problem space with a lot of competing best practices (and I kind of like it that way), but I've found that there are a lot of commonalities, too; more on that in future posts.

I'm looking forward to sharing some of the knowledge I've picked up along the way and, hopefully, learning more from you all !