Last night while making my regular backup for my World of Warcraft configuration, I thought about the blog and I didn't remember seeing an article that went into more detail than "backups are good" about backing up and restoring data.
If you've been around the InnerLayer for a while you will have noticed that backing up of data comes up periodically. This happens because we frequently see customers whose world is turned upside down due to a mistyped command wiping out their data. If you just thought "that won't happen to me... I'm careful at a prompt"... well, how about a cracker getting in via an IIS zero day exploit? Kernel bug corrupting the filesystem? Hard drive failure? Data loss will happen to you, whatever the cause.
Data that is not backed up is data that isn't viewed as important by the server administrator. As the title of this blog mentioned, backing up isn't the end of the server administrator's responsibility. Consider the following points.
- Is the backup in a safe location? Backing up to the same hard drive which houses the live data is not a good practice.
- Is the backup valid? Did the commands to create it all run properly? Did they get all the information you need? Do you have enough copies?
- Can your backup restore a single file or directory? Do you know how to restore it? Simply put, a restore is getting data from a backup back into a working state on a system.
At a minimum backups should be stored on a separate hard drive from the data which the backup is protecting. Better would be a local copy of the backup on the machine in use and having a copy of the backup off the machine, perhaps in eVault, on a NAS which is _NOT_ always mounted, even on another server. Why? The local backup gives you quick access to the content while the off-machine copies give you the safety that if one of your employees does a secure wipe on the machine in question you haven't lost the data and the backup.
A backup is valid if it gets all the data you need to bring your workload back online in the event of a failure. This could be web pages, database data, config files (frequently forgotten) and notes on how things work together. Information systems get complicated and if you've got a Notepad file somewhere listing how Tab A goes into Slot B, that should be in your backups. Yes, you know how it works... great, you get hit by a bus, does your co-admin know how that system is put together? Don't forget dependencies. A forum website is pretty worthless if it is backed up but the database to which it looks is not. For me another mark of a valid backup is one which has some history. Do not backup today and delete yesterday. Leave a week or more of backups available. People don't always notice immediately that something has broken.
A good way to test a restore is get a 2nd server for a month configured the same as your primary then take the backup from the primary and restore it onto the secondary. See what happens. Maybe it will go great. Probably you will run into issues. Forget about a small operating system tweak made some morning at 4am? How about time? How long does it take to go from a clean OS install to a working system? If this time is too long, you might have too much going on one server and need to split up your workload among a few servers. As with everything else in maintaining a server, practicing your restores is not a one-time thing. Schedule yourself a couple of days once a quarter to do a disaster simulation.
For those who might be looking at this and saying "That is a lot of work". Yes, it is. It is part of running a server. I do this myself on a regular basis for a small server hosting e-mail and web data for some friends. I have a local "configbackup" directory on the server which has the mail configs, the server configs, the nameserver configs and the database data. In my case, I've told my users straight up that their home directories are their own responsibility. Maybe you can do that, maybe not. Weekly that configback data is copied to a file server here at my apartment. The fileserver itself is backed up periodically to USB drive which is kept at a friend's house.