Migrating a Hudson instance

Quick post to hopefully help others with an error I got when moving our Hudson instance from Windows to its new home on a Linux server.

The basic migration is super easy - just zip up the Hudson home directory (default on Windows XP is C:\Documents and Settings\[username running Hudson]\.hudson) and restore it onto the new server. However, when you start your new Hudson instance you may see one of both of the following errors in the console output:

SEVERE: Timer task hudson.model.LoadStatistics$LoadStatisticsUpdater@74e8f8c5 failed
java.lang.AssertionError: class hudson.node_monitors.DiskSpaceMonitor is missing its descriptor
at hudson.model.Hudson.getDescriptorOrDie(Hudson.java:937)
at hudson.node_monitors.NodeMonitor.getDescriptor(NodeMonitor.java:83)
at hudson.node_monitors.NodeMonitor.getDescriptor(NodeMonitor.java:67)
at hudson.util.DescribableList.get(DescribableList.java:104)
at hudson.model.ComputerSet.(ComputerSet.java:356)
at hudson.model.LoadStatistics$LoadStatisticsUpdater.doRun(LoadStatistics.java:214)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)

Jul 29, 2010 1:47:39 PM hudson.triggers.SafeTimerTask run
SEVERE: Timer task hudson.model.LoadStatistics$LoadStatisticsUpdater@74e8f8c5 failed
java.lang.NoClassDefFoundError
at hudson.model.LoadStatistics$LoadStatisticsUpdater.doRun(LoadStatistics.java:214)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)

The fix is easy - just stop Hudson, delete the file nodeMonitors.xml in the main Hudson directory, and restart Hudson. It looks as if that file contains data that's specific to the old system; Hudson will recreate the file after restart.

Entropy at Work

Like anyone who studied physics at school or college I knew the concept of entropy, but I only found out this week that Linux systems have their own version of the phenomenon.

I was setting up an RPM repository and wanted to create a GPG key for signing the repository files. Near the end of the key setup, I got this message:

Not enough random bytes available. Please do some other work to give
the OS a chance to collect more entropy! (Need 276 more bytes)

Entropy, eh ? Well, according to the physics I remember, the entropy of the universe is continually increasing; however; it turns out that a Linux system doesn't classify as its own universe and its entropy can go up and down, as evidenced by:

watch cat /proc/sys/kernel/random/entropy_avail

I could see the entropy value merrily going up and down between near zero and a couple of thousand, seemingly at random. After watching in fascination for a while, I developed the theory that the universe must comprise all of the Linux machines anywhere, merrily swapping their entropy bytes in some vast, complex dance which mere mortals cannot contemplate.

Turns out that the reality is a bit more down to earth; Linux calculates entropy by transforming interrupt events - activity from input devices such as the keyboard and mouse, or file system activity.

If you're in front of your system you can just point on the keyboard to generate enough entropy bytes to make your GPG key, but I happened to be working on an Amazon EC2 instance that I was logged into remotely. I managed to get enough randomness in the end by downloading a few large files from external web sites, and running "ls -R /" a couple of times to run through the entire root file system. Even though the entropy_avail value kept going up and down while I was doing this, it turned out that the GPG key generation just kept taking what was available as new bytes, and after 10 minutes or so had enough bytes to finish:

++++++++++.++++++++++>+++++...............>+++++............<.+++++....>.+
++++<.+++++>.+++++..................+++++^^^^^^^^^

gpg: key 2BC5527E marked as ultimately trusted
public and secret key created and signed.

Hopefully this post will help you to avoid one of those "WTF" moments; thanks to Carson Reynolds for providing most of the source material.

Neat trick for getting relative dates in a Linux shell script

I wanted to get the SVN updates done in the last 24 hours for a nightly script that updates and restarts one of our applications. The command would look something like:

svn log -v -r {#Calculate yesterday's date somehow }:{#Calculate today's date in yyyy-mm-dd format}

Getting today's date is easy enough using the "+" parameter with a format string, but yesterday's date seemed more tricky; checking the man page for the date command didn't look very promising. It looked like I'd have to do the famous programming exercise of keeping an array of the number of days in each month, checking whether yesterday was still this month or last month, etc. etc.

However, using info date instead of man date gave a lot more information, including this very useful option:

-d DATESTR'
`--date=DATESTR'
Display the date and time specified in DATESTR instead of the
current date and time. DATESTR can be in almost any common
format.

DATESTR can be a word or phrase like "tomorrow" "yesterday", "last week", etc. So my problem was solved in a single line:

svn log -v -r {`date -d "yesterday" +"%Y-%m-%d"`}:{`date +"%Y-%m-%d"`}

Moral of the story - always look at the info page as well as the man page !

Forced education #1 - DNS and DHCP with dnsmasq

One of the, ahem, pleasures of working at a startup is that you often have to learn something new in a hurry when something goes wrong. This happened to me the other day when people on our office network suddenly stopped being able to get to anywhere else on the internet or our internal network.

After a bit of troubleshooting, I found out that we still had inbound and outbound connectivity, but DNS had intermittently stopped working - so, since normal people don't carry a long list of IP addresses in their heads, their access to anywhere via its host name was gone.

We had all our office network clients pick up their DNS server via DHCP from our LinkSys AV-082 router. I tried switching the DNS servers defined in the router to OpenDNS in case our ISP's DNS servers were having a problem, but the problem persisted.

Some internet digging and talking with a couple of folks who know a lot more about networking than me led me to the conclusion that the router's DNS and DHCP handling was acting up. Since it still seemed to be working fine as a regular router, I decided the best thing would be to turn off the router's DHCP server and run DNS and DHCP from one of the office servers instead. I set up the people in the office to point directly to OpenDNS to get them going while I was messing around, and got to work.

There's a nice, lightweight Linux package called dnsmasq that will handle both DNS and DHCP. I found it super easy to set up on one of our Fedora servers, thanks to Keith Fieldhouse's article on linux.com; the hardest part was typing in all the MAC addresses for servers that I wanted to have a fixed IP address, since the router's web UI wouldn't let me copy and paste the DHCP MAC to IP address mappings.

The only other tweak I had to make was to configure the network adapter on the dnsmasq machine to have a fixed IP address, rather than trying to get it from DHCP.

After a couple of hours of setup and careful testing, everything was up and running. An extra benefit of having a local DNS server is that it caches all its results, so that lookups to a host that it already knows about are nice and fast. In fact, I've now set up a local dnsmasq server just for DNS at home, since anything that makes web browsing faster has to be a good thing !