Jurassic Park, Uptime, And You!

January 11, 2011

Some of you may remember in the movie Jurassic Park where the park founder's granddaughter Lex, played by Ariana Richards, sits down at a computer terminal, gasps, and says "This is Unix. I know this!" That particular film moment has always resonated with me as a victory for realistic depiction of computer systems - the interface used in the movie is called fsn and was an actual Unix file manager - in an industry rife with horrific exaggerations; Swordfish, anyone? I'm sure there's an unwritten story as to how she (or her brother if you follow the book) gained her skills at a computer system that in 1993 was almost exclusively relegated to universities. However, I digress.

Shortly before that scene was another scene and catchphrase that should resound with familiarity to system administrators around the world. In the face of marauding dinosaurs and computer sabotage, the character John Arnold, played by Samuel L. Jackson, must sacrifice what I'm sure was an absurd amount of uptime by killing the power and rebooting the mainframe. Would the system come back up? Would everything load up as needed to get the park's systems back online? John's mantra was simple: "Hold on to your butts!"

Every day as a Systems Administrator I'm faced with a comparable (though far less exhilarating) situation. Linux is an extremely stable operating system, and I have logged into systems that have been online for quite literally years. Eventually, though, kernel updates or stray mounts necessitate a reboot. Will the server's filesystems need a check on reboot? Will the server even come back up? When a server's been online for that long, the only way to know is to "throw the switch" and cross your fingers.

One way to have a better idea of how your system will behave during reboots in a production environment is to take the time to update your kernel once a month or so and perform a reboot to make sure the update sticks. This allows routine file system checks to take place as necessary and keeps your system abreast of the latest kernel updates. It also familiarizes you with how long the process takes, what sort of caveats you may run into, and reduces the overall surface area of your server to outside attackers.

In the last year, I have seen at least two exploits that can give an attacker root access to a server running an outdated kernel using common toolkits that can attack commonly deployed Content Management Systems with trivial effort. Compromising an unprivileged user account gives an attacker even more leverage against unpatched systems. Google CVE-2009-2695 and CVE-2010-3081 if you don't believe me.

If you run a production system or even a backend system that is exposed to the big, bad Internet, it is absolutely essential to make sure that your kernel, software, and security measures are up to date. Today's Slashdot article is tomorrow's exploit.

What lesson can we learn from the unfortunate folks at Jurassic Park? Don't assume your server is safe and don't wait until there are velociraptors roaming your halls looking for a snack to perform proper maintenance on your system.