Posts Tagged 'Recovery'

November 21, 2012

Risk Management: The Importance of Redundant Backups

You (should) know the importance of having regular backups of your important data, but to what extent does data need to be backed up to be safe? With a crowbar and shove, thieves broke into my apartment and stole the backups I've used for hundreds of gigabytes of home videos, photo files and archives of past computers. A Dobro RAID enclosure and an external drive used by Apple Time Machine were both stolen, and if I didn't have the originals on my laptop or a redundant offsite backup, I would have lost all of my data. My experience is not uncommon, and it's a perfect example of an often understated principle that everyone should understand: You need redundant backups.

It's pretty simple: You need to back up your data regularly. When you've set up that back up schedule, you should figure out a way to back up your data again. After you've got a couple current backups of your files, you should consider backing up your backups off-site. It seems silly to think of backing up backups, but if anything happens — failed drives, theft, fire, flood, etc. — those backups could be lost forever, and if you've ever lost a significant amount of data due to a hard drive failure or experience like mine, you know that backups are worth their weight in gold.

Admittedly, there is a point of diminishing return when it comes to how much redundancy is needed — it's not worth the time/effort/cost to back up your backups ad infinitum — so here are the best practices I've come up with over the course of my career in the information technology industry:

  • Plan and schedule regular backups to keep your archives current. If your laptop's hard drive dies, having backups from last June probably won't help you as much as backups from last night.
  • Make sure your data exists on three different mediums. It might seem unnecessary, but if you're already being intentional about backing up your information, take it one step further to replicate those backups at least one more time.
  • Something might happen to your easy onsite backups, so it's important to consider off-site backups as well. There are plenty of companies offering secure online backups for home users, and those are generally easy to use (even if they can be a little slow).
  • Check your backups regularly. Having a backup is useless if it's not configured to back up the correct data and running on the correct schedule.
  • RAID is not a backup solution. Yes, RAID can duplicate data across hard drives, but that doesn't mean the data is "backed up" ... If the RAID array fails, all of the hard drives (and all of the data) in the array fail with it.

It's important to note here that "off-site" is a pretty relative term when it comes to backups. Many SoftLayer customers back up a primary drive on their server to a secondary drive on the same server (duplicating the data away from the original drive), and while that's better than nothing, it's also a little risky because it's possible that the server could fail and corrupt both drives. Every backup product SoftLayer offers for customers is off-site relative to the server itself (though it might be in the same facility), so we also make it easy to have your backup in another city or on a different continent.

As I've mentioned already, once you set up your backups, you're not done. You need to check your backups regularly for failures and test them to confirm that you can recover your data quickly in the event of a disaster. Don't just view a file listing. Try extracting files or restore the whole backup archive. If you're able to run a full restore without the pressure of an actual emergency, it'll prove that you're ready for the unexpected ... Like a fire drill for your backups.

Setting up a backup plan doesn't have to be scary or costly. If you don't feel like you could recover quickly after losing your data, spend a little time evaluating ways to make a recovery like that easy. It's crazy, but a big part of "risk management," "disaster recovery" and "business continuity" is simply making sure your data is securely backed up regularly and available to you when you need it.

Plan, prepare, back up.

-Lyndell

February 28, 2012

14 Questions Every Business Should Ask About Backups

Unfortunately, having "book knowledge" (or in this case "blog knowledge") about backups and applying that knowledge faithfully and regularly are not necessarily one and the same. Regardless of how many times you hear it or read it, if you aren't actively protecting your data, YOU SHOULD BE.

Here are a few questions to help you determine whether your data is endangered:

  1. Is your data backed up?
  2. How often is your data backed up?
  3. How often do you test your backups?
  4. Is your data backed up externally from your server?
  5. Are your backups in another data center?
  6. Are your backups in another city?
  7. Are your backups stored with a different provider?
  8. Do you have local backups?
  9. Are your backups backed up?
  10. How many people in your organization know where your backups are and how to restore them?
  11. What's the greatest amount of data you might lose in the event of a server crash before your next backup?
  12. What is the business impact of that data being lost?
  13. If your server were to crash and the hard drives were unrecoverable, how long would it take you to restore all of your data?
  14. What is the business impact of your data being lost or inaccessible for the length of time you answered in the last question?

We can all agree that the idea of backups and data protection is a great one, but when it comes to investing in that idea, some folks change their tune. While each of the above questions has a "good" answer when it comes to keeping your data safe, your business might not need "good" answers to all of them for your data to be backed up sufficiently. You should understand the value of your data to your business and invest in its protection accordingly.

For example, a million-dollar business running on a single server will probably value its backups more highly than a hobbyist with a blog she contributes to once every year and a half. The million-dollar business needs more "good" answers than the hobbyist, so the business should invest more in the protection of its data than the hobbyist.

If you haven't taken time to quantify the business impact of losing your primary data (questions 11-14), sit down with a pencil and paper and take time to thoughtfully answer those questions for your business. Are any of those answers surprising to you? Do they make you want to reevaluate your approach to backups or your investment in protecting your data?

The funny thing about backups is that you don't need them until you NEED them, and when you NEED them, you'll usually want to kick yourself if you don't have them.

Don't end up kicking yourself.

-@khazard

P.S. SoftLayer has a ton of amazing backup solutions but in the interested of making this post accessible and sharable, I won't go crazy linking to them throughout the post. The latest product release that got me thinking about this topic was the SoftLayer Object Storage launch, and if you're concerned about your answers to any of the above questions, object storage may be an economical way to easily get some more "good" answers.

November 16, 2009

How Many Recovery Plans Do We Need?

Several of our bloggers have written about backups in The InnerLayer. This morning, I had an experience that makes me wonder how many recovery plans we need.

I walked out of the house to the driveway and saw that my left rear tire was flat. An enormous nail had punctured my tire right in the middle of the tread, and the slow leak deflated the tire overnight. To recover from this disaster, I needed to get my vehicle drivable and get to the Discount Tire location near my house so that they could fix the flat. Below is a log of how the recovery plans worked out.

Recovery Plan #1: Call roadside assistance. While waiting on them to change my tire, logon from home and get some work done before going to Discount Tire. I have leased four different brands of vehicles over the past 10 years, and roadside assistance was always included with the lease. So I call the 800 number and they tell me I don’t have roadside assistance. (Note to self: read the fine print on the next lease.) Result: FAIL

Recovery Plan #2: Inflate tire with can of Fix-a-Flat. I retrieved the can from my garage, followed the instructions, and when I depressed the button to fill the tire, the can was defective and the contents spewed from the top of the can rather than filling the tire. Result: FAIL

Recovery Plan #3: Use foot operated bicycle pump to inflate tire and drive to Discount Tire. I have actually done this successfully before with slow leaks like this one. It is third in priority because it is harder and more tiring than the first two options. So I go to my garage and look at where the pump is stored. It isn’t there. I scour the garage to find it. It is gone. Result: FAIL

Recovery Plan #4: Change out of office clothes into junky clothes, drag out the jack and spare and change the tire myself. This is number four in priority because it is the biggest hassle. I will spare you all the slapstick comedy of a finance guy jacking up a vehicle and changing the tire (finding the special key for the locking lug nuts was an interesting sub-plot to the whole story), so I’ll summarize and say RESULT: Success!

As a side note, I must give props to Discount Tire. Having bought tires there before, I was in their database as a customer and they fixed the flat and installed it on my vehicle for no charge. I recommend them!

All this got me to thinking about not only having backups, but having redundant recovery plans. Sure, you’ve got a recent copy of all your data – that’s great! Now, what’s your plan for restoring that data? If you have an experience like my flat tire recovery this morning, it might be a good idea to think through several ways to recover and restore the data. Our EVault offering will certainly be one good strategy.

May 15, 2009

Disaster Recovery Plan

A few days ago I was reading a news story about a man who just lost everything to a fire. One of the comments he made was that he had never thought to plan for something like this; it was the type of thing that happened to other people but never to me. I started thinking about how true that statement was. Many people just never think it will happen to them.

This type of situation happens every day in the IT field. There is some sort of disaster causing a server to crash or simply stop working all together, the drives on the server are completely corrupted and the data is just gone. The question is; when this happens to you, will you be prepared? Thankfully, there are steps each person can take to limit the pain and downtime a situation like this can cause. Like any other disaster recovery plan, the more you are willing to put into it, the more protection you will have when disaster strikes.

This is where SoftLayer comes in. Here at SoftLayer we understand the importants of providing our customers the means to create a good disaster recovery plan that meets their needs. We understand that a detailed disaster recovery plan will include things such as backups and replication. Our services such as NAS and EVault are perfect solutions for performing and managing the backups for you server. When looking into replication, we offer services such as iSCSI replication, Raids, local and global loadbalancing which will provide our customer with the tools to replicate not only their data across multiple locations but their servers as well. Above all, we provide our private network to securely transfer this data to the many locations without impacting the traffic on your public network.

We can only hope that on the day disaster strikes, everyone has some plan in place to deal with it. There is nothing more frustrating in this industry then the loss of crucial data that in many instances cannot be recovered.p

March 26, 2009

Use Caution when Outsourcing!

Outsource IT! I have been saying that for years now. But now I say; outsourcer beware!?!?! Really? How do you know if the company you are calling upon to keep your business up and running is safe and sound? Do they have certifications? Are they registered with the Better Business Bureau? Do they have scary fine print in the Terms of Service or User Agreement? Do you actually read those and understand them? How do you find out about all the questions above? Do you go to trade shows? Do you read about companies on the Hosting forum sites? Do you hear it from your friends? There are lots of ways to get that kind of information in today’s social internet jungle. Do you follow the company on Facebook, Twitter, MySpace, Linked-In, or all of the above? Should you? So many questions…

I am going to assume that you think this blog is going to be about how SoftLayer is a reputable, certified PCI compliant and SAS 70 datacenter, with competent and caring employees that can put themselves in the customer’s shoes and understand the frustrations that can go along with outsourcing your datacenter needs. Nah, that would be too easy and not very much fun.

This blog is about mud. Yes, I said mud. I was driving down a county road in Texas recently and we had a bit of rain in the days leading up to my trip. If you aren’t from Texas then you need a quick definition of “County Road”. A county can be paved, gravel or dirt topped and can be a great road or a horrible road, it just depends on the county that it is in, the tax base, and the abilities of the crews hired by the county to maintain them. I was travelling down a very wet gravel top county road, following along on my cell with GPS and Google maps and was about a mile from my destination. In what seemed the blink of an eye the road surface went from wet gravel to dirt and within about 10 feet my truck simply slid off the road into a nice 4 foot ditch filled with rain water. Looks harmless in the picture below doesn’t it?

Mud

It was a nice soft splash landing but my city slicker tires had no chance of getting me out of that ditch even with 4X4 engaged. So when water started coming under the door into the cab of the truck, I knew it was going to be a bad hour or so. It was time to outsource. I called the ranch to see if they had anything that could pull me out but they said that I was in a pretty tough spot and didn’t think they could help. So what would any techie do, I googled mud towing in the closet town. Of course I picked the first place on the list and gave them a call. They said they had a mud recovery truck and they would be out in about 45 minutes. Awesome, just 45 minutes! This was at 4:30PM and it was pretty cold and still raining and the ditch was filling up even further with water. Outsourcer beware, I was expecting a “Mud Recovery Truck!” I had visions of monster trucks dancing in my head. Fail!

Mud

Now I have to say that there weren’t ten forums about mud towing in Navarro county that I could visit, or customer references readily available so I just had to take that leap of faith and trust in the skills of my saviors. And I have to give credit where credit is due, that truck really is a monster! It did things a Transformer would love to be able to do. It got stuck at least 30 times in the 5 hours it took them to get me out of the ditch. Yes, I said 5 hours. Did I mention that monster trucks can do very bad things to city 4X4’s? Thank goodness I have an Echo to drive back and forth to work.

So I don’t want to leave you hanging but my truck is in the shop now and I am still waiting on an estimate. Things I know are wrong; front right A-arm damage from forcibly pulling the truck over a stump in the ditch, alignment issues, check engine light on, cruise control doesn’t work anymore, passenger side back door pushed up about half an inch including damage at bottom from the same stump, muffler caved in and exhaust pipe dragging the ground, front bumper air damn ripped off and metal bumper bent outward, yea you guessed it the pesky stump again and last but not least I need an entire new jack assembly because it is either broken or lost in the mud or both I should say (attempting to jack the truck over the stump).

The moral of this blog, if you have the tools available to research the company you are going to outsource to and they have references be sure to use them. They might save you a $300 mud recovery bill and a $1000 deductible somewhere down the road.

Subscribe to recovery