Posts Tagged ‘evault’

Everyone always says it’s a good idea to have a backup plan just in case your primary plan bites the dust. I couldn’t agree more. Recently my personal Xbox 360 failed and this has caused plenty of grief in my household. I used my Xbox to stream content from Windows Media Player on my desktop to the TV (via Media Center edition of Windows XP). This has worked great and has been able to provide me with a means to entertain my child. Of course, this going out has caused a screaming baby because now she can’t watch her “movies”.

Now, had I had a proper backup plan, this wouldn’t be an issue. See, I put all of my trust into a single device and/or single method to accomplish something. When this device failed, my operation came to a halt. I didn’t listen to the advice I’m always telling our customers… have a backup or backup plan. This is where our “extra services” come into play. Not only do we offer backup solutions (eVault, NAS…) but we also offer solutions that allow you access to high-availability configurations (Citrix XenServer, for example). With XenServer you can configure a cluster of systems and setup automatic failover. This would prevent any major outages of your website/services. If this isn’t something you think would work for you, utilizing eVault backups might. We now offer eVault Bare Metal Restore. Now, the problem is somehow applying these to my Xbox so my kiddo can go back to watching her movies… Long story short, don’t rely on a single solution. Always have a backup plan or system in place to prevent headaches in the future. You won’t regret it if you do.

Several of our bloggers have written about backups in The InnerLayer. This morning, I had an experience that makes me wonder how many recovery plans we need.

I walked out of the house to the driveway and saw that my left rear tire was flat. An enormous nail had punctured my tire right in the middle of the tread, and the slow leak deflated the tire overnight. To recover from this disaster, I needed to get my vehicle drivable and get to the Discount Tire location near my house so that they could fix the flat. Below is a log of how the recovery plans worked out.

Recovery Plan #1: Call roadside assistance. While waiting on them to change my tire, logon from home and get some work done before going to Discount Tire. I have leased four different brands of vehicles over the past 10 years, and roadside assistance was always included with the lease. So I call the 800 number and they tell me I don’t have roadside assistance. (Note to self: read the fine print on the next lease.) Result: FAIL

Recovery Plan #2: Inflate tire with can of Fix-a-Flat. I retrieved the can from my garage, followed the instructions, and when I depressed the button to fill the tire, the can was defective and the contents spewed from the top of the can rather than filling the tire. Result: FAIL

Recovery Plan #3: Use foot operated bicycle pump to inflate tire and drive to Discount Tire. I have actually done this successfully before with slow leaks like this one. It is third in priority because it is harder and more tiring than the first two options. So I go to my garage and look at where the pump is stored. It isn’t there. I scour the garage to find it. It is gone. Result: FAIL

Recovery Plan #4: Change out of office clothes into junky clothes, drag out the jack and spare and change the tire myself. This is number four in priority because it is the biggest hassle. I will spare you all the slapstick comedy of a finance guy jacking up a vehicle and changing the tire (finding the special key for the locking lug nuts was an interesting sub-plot to the whole story), so I’ll summarize and say RESULT: Success!

As a side note, I must give props to Discount Tire. Having bought tires there before, I was in their database as a customer and they fixed the flat and installed it on my vehicle for no charge. I recommend them!

All this got me to thinking about not only having backups, but having redundant recovery plans. Sure, you’ve got a recent copy of all your data – that’s great! Now, what’s your plan for restoring that data? If you have an experience like my flat tire recovery this morning, it might be a good idea to think through several ways to recover and restore the data. Our EVault offering will certainly be one good strategy.

A few days ago I was reading a news story about a man who just lost everything to a fire. One of the comments he made was that he had never thought to plan for something like this; it was the type of thing that happened to other people but never to me. I started thinking about how true that statement was. Many people just never think it will happen to them.

This type of situation happens every day in the IT field. There is some sort of disaster causing a server to crash or simply stop working all together, the drives on the server are completely corrupted and the data is just gone. The question is; when this happens to you, will you be prepared? Thankfully, there are steps each person can take to limit the pain and downtime a situation like this can cause. Like any other disaster recovery plan, the more you are willing to put into it, the more protection you will have when disaster strikes.

This is where SoftLayer comes in. Here at SoftLayer we understand the importants of providing our customers the means to create a good disaster recovery plan that meets their needs. We understand that a detailed disaster recovery plan will include things such as backups and replication. Our services such as NAS and EVault are perfect solutions for performing and managing the backups for you server. When looking into replication, we offer services such as iSCSI replication, Raids, local and global loadbalancing which will provide our customer with the tools to replicate not only their data across multiple locations but their servers as well. Above all, we provide our private network to securely transfer this data to the many locations without impacting the traffic on your public network.

We can only hope that on the day disaster strikes, everyone has some plan in place to deal with it. There is nothing more frustrating in this industry then the loss of crucial data that in many instances cannot be recovered.p

“ah – I don’t need backups.”
“Too busy to do backups – I’ll get to that later.”
“Backups? It costs too much.”
“I don’t need backups – MTBF of a Raptor is 1.2 Million hours.”
“Oops – I forgot about doing backups.”

Backups are one of the most commonly forgotten tasks of a system administrator. In some cases, they are never implemented. In other cases, they are implemented but not maintained. In other cases, they are implemented with a great backup and recovery plan – but the system usage or requirements change and the backups are not altered to compensate.

A hard drive really is a fairly reliable piece of IT equipment. The WD 150GB Raptor has a rating of 1.2 Million hours MTBF. With that kind of mean time between failures, you would think that you would never have to worry about a hard drive failing. How willing are you to take that chance? What if you double your odds by setting up two drives in a RAID 1 configuration? Now can you afford to take that chance? How willing are you to gamble with your data?

What if one of your system administrators accidentally deletes the wrong file? Maybe it’s your apache config file. Maybe it’s a piece of code you have been working on all day. Or, maybe your server gets compromised and you now have unknown trojans and back doors on your server. Now what do you do?

Working in a datacenter with thousands of servers, there are thousands and thousands of hard drives. When you see that many hard drives in production, you are naturally going to see some of them fail. I have seen small drives fail, large drives fail, and I have even seen RAID 1 mirrors completely fail beyond recovery. Is it bad hardware? Nope. Is it Murphy’s Law? Nope. It’s the laws of physics. Moving parts create heat and friction. Heat and friction cause failures. No piece of IT equipment is immune to failure.

That 1.2 million hours MTBF looks pretty impressive. For a round number, let’s say there are 15,000 drives in the SL datacenter. 1,200,000 hours / 15,000 drives = 80 hours. That means that every 80 hours, one hard drive in the SL datacenter could potentially fail. Now how impressive is that number?

Ultimately, regardless of the levels of redundancy you implement, there is always a chance of a failure – hardware or human – that results in data loss. The question is – how important is that data to you? In the event of a catastrophic failure, are you willing to just perform an OS reload and start from scratch? Or, if a file is deleted and unrecoverable, are you willing to start over on your project? And lastly, how much downtime can you afford to endure?

Regardless of how much redundancy you can build into your infrastructure with the likes of load balancers, RAID arrays, active/passive servers, hot spares, etc, you should always have a good plan for doing backups as well as checking and maintaining those backups.

Have you checked your backups lately?

-SamF