Posts Tagged 'Backups'

November 11, 2009

Viva Las Vegas!

I just got back in town from Las Vegas, Nevada. That town is filled with stories and you can really love it or hate it, depending on the hour (or if you are like me whether you are arriving into McCarran or departing). I had a great trip this last go around and actually made money on the tables. However, when they say that what happens in Vegas stays in Vegas they are really talking about your money. Never forget that the house always wins. Always. Even if you win money you’ll wind up spending it on stuff out there and perpetuating your own good time. There isn’t anything wrong with this at all. In fact I plan on coming up on the short side of the stick on both the tables and on simply spending cash when I go out that way.

I think the really interesting thing that happens when you go through “the Vegas experience” is the perceived value of a dollar. You can take it for granted that all of a sudden you are transplanted into this fantasy world that is reminiscent of Pleasure Island from the story of Pinocchio and you’ll find that you have anything and everything you could want to do, eat, drink, or experience right at your fingertips. As this begins to progress the value of a dollar plummets quickly. You start overpaying for things at a whim, tipping bigger, making bolder and even just dumber bets. I did this and I can admit that I doubled down on my 11 when the dealer was showing a 10 in blackjack. It was blind luck that I hit it and won every single time. It’s a bold and stupid bet to make, but when you are playing with house money the money doesn’t matter and it’s almost as if you are trying to give it all back. My game of choice is craps because it gives you the best odds and there is a lot of action. It’s good and bad as it can all come and go in a hurry.

I have only been to Las Vegas a handful of times, but each time there is a point where even for a second you can feel invincible – that you can’t lose. Or, that even if you do lose you won’t even care. The flight home is a completely different story. I call it the hangover flight. You may be literally hung over, but no matter what, you will start to deal with all of the actions that happened on your trip and how you will need to handle them. As soon as you touch down in your own home town things slowly start to become “real” again. Your own home can even feel somewhat foreign for a while, but you’ll quickly come to the realization that you had become a completely different person for a short time.

I have come to the conclusion that there is always risk in everything that we do. Exposing yourself to the tables of Las Vegas may carry more financial risk than your morning commute to work, but in both cases there are still risks. There are also risks that we take in setting and running a business. There are countless ways that you could be putting your business at risk without the right plan in place. From an IT perspective alone, you need to consider things like redundancy, failover, security, backups, growth, and even data loss. Knowing what is going to happen next for your business may be as likely as knowing what is going to come up on the next roll of the dice. If you know this for certain you can press your luck and come up big, but if you are not prepared you could lose everything you have on the table. It is better to be prepared.

I think of SoftLayer as the house, and remember as I said before, the house always wins. The good thing about this is that you are betting with the house. Even with this you need to bet on yourself and back up your own bet. If the bulk of your business is in your data then you need to have backups. If you absolutely need to have High Availability, then look into Clusters and Load Balancing. But remember, that you are betting with the house because SoftLayer gives you the capacity to do all of it and do it all at a very affordable price compared to trying to do it yourself and also do it without long term commitments. Long term commitments bring the most uncertainty in making moves that will positively affect your business. Imagine if a casino told you that you “had” to make 12 consecutive bets regardless of how well (or poorly) you were doing?

Coming home from Las Vegas to SoftLayer has been a very good thing and makes me thankful for where I am and what I have. There aren’t the levels of uncertainty here that are automatic with other datacenters or even other business models. SoftLayer is steady and it is very easy to get what you need here while cutting out the risk that you don’t want to deal with. SoftLayer is as much of a “sure thing” as any bet you can make!

October 28, 2009

Meet Virus Jack

I am Jack’s Vundo Virus. I cause Jack’s computer to have popup. I also disable Windows Automatic Updates, task manager, registry editor, and msconfig so Jack cannot boot to safe mode. I use Jack’s Norton AntiVirus to help spread my infection. I make Jack’s google searches to redirect to rogue antispyware sites. Jack got me by not keeping his system up to date. Now there are programs out there designed to remove me but the best way is for Jack to reformat. Let’s hope he has backups. Morale of this story is keep your computers up to date with the latest OS updates, AntiVirus definitions and program updates.

October 16, 2009

Raid 1 or Raid 0: which should I choose?

When considering these 2 raid options there are a few points you’ll want to consider before making your final choice.

The first to consider is your data, so ask yourself these questions:

  • Is it critical data that your data be recoverable?
  • Do you have backups of your data that can be restored if something happens?
  • Do you want some kind of redundancy and the ability to have a failed drive replaced without your data being destroyed?

If you have answered yes to most of these, you are going to want to look at a Raid 1 configuration. With a Raid 1 you have 2 drives of like size matched together in an array, which consists of an active drive and a mirror drive. Either of these drives can be replaced should one go bad without any loss of data and without taking the server offline. Of course, this assumes that the Raid card that you are using is up to date on it’s firmware and supports hot swapping.

If you answered no to most of these questions other than the backup question (you should always have backups), a Raid 0 set-up is probably sufficient. This is used mostly for disk access speeds and does not contain any form of redundancy or failover. If you have a drive failure while using a Raid 0 your data will be lost 99% of the time. This is an unsafe Raid method and should only be used when the data contained on the array is not critical in anyway. Unfortunately with this solution there is no other course of action that can be taken other than replacing the drives and rebuilding a fresh array.

I hope this helps to clear up some of the confusion regarding these 2 Raid options. There are several other levels of Raid which I would suggest fully researching before you consider using one of them.

May 15, 2009

Disaster Recovery Plan

A few days ago I was reading a news story about a man who just lost everything to a fire. One of the comments he made was that he had never thought to plan for something like this; it was the type of thing that happened to other people but never to me. I started thinking about how true that statement was. Many people just never think it will happen to them.

This type of situation happens every day in the IT field. There is some sort of disaster causing a server to crash or simply stop working all together, the drives on the server are completely corrupted and the data is just gone. The question is; when this happens to you, will you be prepared? Thankfully, there are steps each person can take to limit the pain and downtime a situation like this can cause. Like any other disaster recovery plan, the more you are willing to put into it, the more protection you will have when disaster strikes.

This is where SoftLayer comes in. Here at SoftLayer we understand the importants of providing our customers the means to create a good disaster recovery plan that meets their needs. We understand that a detailed disaster recovery plan will include things such as backups and replication. Our services such as NAS and EVault are perfect solutions for performing and managing the backups for you server. When looking into replication, we offer services such as iSCSI replication, Raids, local and global loadbalancing which will provide our customer with the tools to replicate not only their data across multiple locations but their servers as well. Above all, we provide our private network to securely transfer this data to the many locations without impacting the traffic on your public network.

We can only hope that on the day disaster strikes, everyone has some plan in place to deal with it. There is nothing more frustrating in this industry then the loss of crucial data that in many instances cannot be recovered.p

June 20, 2008

I Always Have a Backup Plan

It was the day of the big secret meeting. All my vice presidents were there except for the unix system administrator. He was a strange man, always wearing that robe, with the long beard and long hair. He considered himself some sort of wizard, and after the conflict last month when we decided to switch all our servers over to SoftLayer, I really didn’t want him involved in the meeting I called today. You see, I called it so I could announce my plan to switch our servers over to Windows. My goal was really to get rid of him; he’s the only one who ever managed to thwart my plans.

Just as I finished that thought, he burst through the door, trailing a long ribbon of old-fashioned printer paper behind him. “How dare you have a systems meeting without me!” he intoned, dropping his stack of papers on the conference table in front of me. A quick glance at the stack tells me that he has printed out operating statistics for every version of Unix and every version of Windows going back to 1985. I didn’t have time for this. Luckily, I always have a back up plan.

Turning away slightly, I quickly activated a program on my Blackberry. You see, yesterday I had written a few custom programs that utilize the SoftLayer API to control a variety of our services. Within moments, a confirmation had appeared on my screen. All of our web traffic had been redirected from our load balanced main servers to our tertiary backup server. In the middle of the work day, that means it was only a matter of minutes before our bandwidth would be exceeded on that server. I allowed the sysadmin to begin his presentation, confident that he would barely get past the 8086 before disaster stuck.

I was right! Within minutes, an email arrived notifying us that we were nearing the bandwidth cap on the hostname last_resort. Panicked, the sysadmin left the meeting. Quickly I summarized my plans to the other VPs, we all voted unanimously for Windows, and I retreated to my office. Shortly after sitting behind my desk, my door burst open. Framed in the light from the hallway, his long shadow washing over me, stood the sysadmin, slowly twirling his staff. “Do you think you can stop me with a simple change to our load balancer? I was configuring load balancers when you were still on dial-up! Now, you will listen, AOL user, and you will see why Unix is your only choice!” Of course, I had a backup plan for just such a situation.

I dove out the window next to my desk, landing nimbly next to my secretary’s bright pink LeBaron. I had made copies of all her keys months ago in order to utilize her unique vehicle for any necessary escapes. I quickly tapped out a text message to Michael in SoftLayer sales. We have a standing agreement that when he receives a message from me containing only the word DAWT, he is to send the best sale at his disposal to my sysadmin. As I drove past the front door of the building I saw him running toward the car. He pulled out his Blackberry in mid-stride and suddenly stopped dead. “Free double RAM AND double hard drives!? IMPOSSIBLE!” he screamed, and I managed to swerve around him and escape. As I drove away, I thought about my secretary. When she first started here, I had convinced her that if her car were ever stolen, the best plan of action would be to change the building security policies so that only my badge could open the doors. I hoped I didn’t need to make use of that plan, but the sysadmin has proved a worthy adversary.

Unbelievable! Even with my masterful backup plan, he was still following me. I saw his battered VW Bus merge into traffic behind me, his vulture-like shadow looming behind the wheel. I sped up until we were both racing down the road, weaving in and out of the other vehicles. Finally we passed a police car, and my next plan sprang into action. I knew that standard procedure was to radio in the vehicles you were pursuing, and I knew my friend Joe was on duty today. Joe knew that if he ever received a radio call about a business man in a pink LeBaron being chased down the highway by a wizard in a VW Bus, he was to call off the police and park a fire truck at a certain intersection. You see, I had hired an actor to pretend to be a corporate Psychiatrist, and learned that the Sysadmin had an irrational fear of fire trucks. Why? Because it always pays to have a backup plan.

I angled toward the intersection and managed to squeeze past the truck just as it pulled up to block the street. I heard the squeal of tires as the sysadmin slammed on his breaks and reversed wildly behind me. Now that I was free, however, I couldn’t return to the office. Luckily I was prepared for just such an eventuality. As I drove to my next location, I quickly used my Blackberry to shut down one of our production web servers. I knew that it would be 20 minutes before the monitoring system would officially declare the server “down,” so I had time.

I made it to my secret office above the video arcade not long after. Before leaving the car I collected the grappling hook and rope from a secret compartment in the door, then went inside. I walked in to the darkened room and immediately noticed something was wrong. My security system wasn’t beeping! The door slammed behind me and the sysadmin boomed out “NO PLAN CAN DEFEAT ME, MORTAL!”

“I’m ALWAYS prepared!” I shot back, and quickly glanced at my watch. It had been 19 minutes and 45 seconds since I shut down my server, the timing was perfect! The sysadmin walked toward me, twirling that staff. Just as he was about to reach me, his blackberry beeped. Pausing to check, he let out a stream of curses and then lunged at me, but I had already rappelled down the side of the building and made my escape.

As soon as I reached the car, my Blackberry alerted me that the server I shut down was back up. How!? The sysadmin must have his own API programs! I cringed as I activated my final backup plan: a program that constantly shut down all our servers. Let’s see him handle that! I took the direct route back to the office, past the still-idling fire truck. I threw Joe a wave, knowing that I’d owe him a big favor for this, and rocketed back to the office. I knew that he would be right behind me, but hopefully with all our servers offline he won’t beat me to my destination. Also, once I made it into the building, the security system wouldn’t allow anyone in behind me. I would be safe!

I raced into the building, looking frantically around for the sysadmin, but he was nowhere to be seen. Finally! I had defeated him! I walked calmly to my office and opened the door, only to see HIM, climbing in through my window. I had forgotten to close it when I escaped this morning! I quickly opened the secret panel in the wall next to the door and put my finger on the red button.

“WAIT!” cried the sysadmin. “We need to put our differences behind us. Our plans have almost destroyed our servers!”

“What do you mean?” I demanded. “They’re fine!”

“No, they’re not,” he said in a sad voice. “You see, I always have a backup plan, and I knew that eventually someone would attempt to power off our machines, so I wrote a script to constantly turn the machines on!”

“B-but…” I stammered, “but I wrote a script to constantly turn them OFF”

“I know” he said, “and the constant power cycling has corrupted our data base. We need to set aside this silly feud and fix it.”

“Don’t worry, dear end user” I proudly proclaimed, “I always have a backup-“

It was right then I realized that in all my planning, I had never actually created any backups.


June 18, 2008

Planning for Data Center Disasters Doesn’t Have to Cost a Lot of $$

One of the hot topics over the past couple of weeks in our growing industry has been how to minimize downtime should your (or your host’s) data center experience catastrophic failure leading to outages that could span multiple days.

Some will think that it is the host’s responsibility to essentially maintain a spare data center into which they can migrate customers in case of catastrophe. The reason we don’t do this is simple economics. To maintain this type of redundancy, we’d need to charge you at least double our current rates. Because costs begin jumping exponentially instead of linearly as extensive redundancy is added, we’d likely need to charge you more than double our current rates. You know what? Nobody would buy at that point. It would be above the “reservation price” of the market. Go check your old Econ 101 notes for more details.

Given this economic reality, we at SoftLayer provide the infrastructure and tools for you to recover quickly from a catastrophe with minimal cost and downtime. But, every customer must determine which tools to use and build a plan that suits the needs of the business.

One way to do this is to maintain a hot-synched copy of your server at a second of our three geographically diverse locations. Should catastrophe happen to the location of your server, you will stay up and have no downtime. Many of you do this already, even keeping servers at multiple hosts. According to our customer surveys, 61% of our customers use multiple providers for exactly that reason – to minimize business risk.

Now I know what you’re thinking – “why should I maintain double redundancy and double my costs if you won’t do it?” Believe me, I understand this - I realize that your profit margins may not be able to handle a doubling of your costs. That is why SoftLayer provides the infrastructure and tools to provide an affordable alternative to running double infrastructure in multiple locations in case of catastrophe.

SoftLayer’s eVault offering can be a great cost effective alternative to the cost of placing servers in multiple locations. Justin Scott has already blogged about the rich backup features of eVault and how his backup data is in Seattle while his server is in Dallas, so I won’t continue to restate what he has already said. I will add that eVault is available in each of our data centers, so no matter where your server is at SoftLayer, you can work with your sales rep to have your eVault backups in a different location. Thus, for prices that are WAY lower than an extra server (eVault starts at $20/month), you can keep near real-time backups of your server data off site. And because the data transfer between locations happens on SoftLayer’s private network, your data is secure and the transfer doesn’t count toward your bandwidth allotment.

So let’s say your server is in our new Washington DC data center and your eVault backups are kept in one of our Dallas data centers. A terrorist group decides to bomb data centers in the Washington DC area in an attempt to cripple US government infrastructure and our facility is affected and won’t be back up for several days. At this point, you can order a server in Dallas, and once it is provisioned in an hour or so, you restore the eVault backup of your choice, wait on DNS to propagate based on TTL, and you’re rolling again.

Granted, you do experience some downtime with this recovery strategy. But the tradeoff is that you are up and running smoothly after the brief downtime at a cost for this contingency that begins at only $20 per month. And when you factor in your SLA credit on the destroyed server, this offsets the cost of ordering a new server, so the cost of your eVault is the only cost of this recovery plan.

This is much less than doubling your costs with offsite servers to almost guarantee no downtime. The reason that I throw in the word “almost” is that if an asteroid storm takes out all of our locations and your other providers’ locations, you will experience downtime. Significant downtime.


June 14, 2008

In Memory of Dawn

Dawn was the best friend I’ve ever had, except for my little sister. Just yesterday I got home only to find out that Dawn had died silently in the night. No amount of resuscitation could bring her back. Needless to say, I was quite sad.

Dawn was my computer.*

The funny part of it all was just how much of my time involves a computer. I watch TV and Movies on my computer, I play games on my computer, I do my banking on my computer, I pay all my bills on my computer, I schedule my non-computer time on my computer, I use my computer as a jukebox.

In other words, I was completely lost. What made it worse, however, was that I had had yesterday scheduled to pay my bills. But where was my list of bills?

If you guessed “Dawn had all your bills”, then you are right.

What about paper bills? I’ve got the Internet and a computer! So, in most cases I’ve canceled paper bills. All paper bills I get are shredded forthwith. So I had no paper backup of bills.

Well, I made do. I kicked my roommate off his computer (a technique involving making annoying noises while he tries to concentrate playing Call of Duty 4) and used it to pay what bills I could remember. I kept track of the bills I was paying by entering them into a Google Document.

That’s when it hit me! Why wasn’t my bill spreadsheet on Google Documents? Along with my bill list? Along with all the other documents I work on every day? Cloud Computing For The Win! As soon as I get my next computer up and running (and I figure out a new naming algorithm) I’m going to put all my vital files on Google Docs. This ties in well with Justin Scott’s post; the key to not having your data disappear during a disaster is to have a backup copy. You want backups out there, far away from your potential point of failure. (I did have backups… but they’re all on CDs that I didn’t want to have to sort through to find just one file. And had the disaster been, say, a flood, I would have had no backups.)

Google Docs is a great example of Cloud Computing: Putting both the program and the file being worked on “in the cloud.” Having built internal applications for a few people, I would make the same recommendation: Since many business apps are moving to PHP anyway (thanks for the reminder, Daniel!), you might as well move the application AND the data out of the building and onto a secure server. And as Mr. Scott** mentioned, SoftLayer ALREADY has geographic diversity as well as a private network that will allow you to link your application and data servers together in real time through all datacenters… for free. Along with the added bonus of being able to access your application from any computer… should yours meet up with Misty, May, and Dawn at the Great Datacenter in the Sky.


* I had a system of naming my computers after the female protagonists from the Pokemon series. Dawn, however, is the last of that series…

** I’ve decided that since Justin is an Engineer, calling him Mr. Scott is funny.

June 4, 2008

Wait … Back up. I Missed Something!

I’ve been around computers all my life (OK, since 1977 but that’s almost all my life) and was lucky to get my first computer in 1983.

Over the summer of 1984, I was deeply embroiled in (up to that point) the largest programming project of my life, coding Z80 ASM on my trusty CP/M computer when I encountered the most dreaded of all BDOS errors, “BDOS ERROR ON B: BAD SECTOR”

In its most mild form, this cryptic message simply means “copy this data to another disk before this one fails.” However, in this specific instance, it represented the most severe case… “this disk is toast, kaputt, finito, your data is GONE!!!”

Via the School of Hard Knocks, I learned the value of keeping proper backups that day.
If you’ve been in this game for longer than about 10 milliseconds, it’s probable that you’ve experienced data loss in one form or another. Over the years, I’ve seen just about every kind of data loss imaginable, from the 1980’s accountant who tacked her data floppy to the filing cabinet with a magnet so she wouldn’t misplace it-- all the way to enterprise/mainframe class SAN equipment that pulverizes terabytes of critical data in less than a heartbeat due to operator error on the part of a contractor.

I’ve consulted with thousands of individuals and companies about their backup implementations and strategies, and am no longer surprised by administrators who believe they have a foolproof backup utilizing a secondary hard disk in their systems. I have witnessed disk controller failures which corrupt the contents of all attached disk drives, operator error and/or forgetfulness that leave gaping holes in so-called backup strategies and other random disasters. On the other side of the coin, I have personally experienced tragic media failure from “traditional backups” utilizing removable media such as tapes and/or CD/DVD/etc.

Your data is your life. I’ve waited up until this point to mention this, because it should be painfully obvious to every administrator, but in my experience the mentality is along the lines of “My data exists, therefore it is safe.” What happens when your data ceases to exist, and you become aware of the flaws in your backup plan? I’ll tell you – you go bankrupt, you go out of business, you get sued, you lose your job, you go homeless, and so-on. Sure, maybe those things won’t happen to you, but is your livelihood worth the gamble?

“But Justin… my data is safe because it’s stored on a RAID mirror!” I disagree. Your data is AVAILABLE, your data is FAULT TOLERANT, but it is not SAFE. RAID controllers fail. Disaster happens. Disgruntled or improperly trained personnel type ‘rm –rf /’ or accidentally select the wrong physical device when working with the Disk Manager in Windows. Mistakes happen. The unforeseeable, unavoidable, unthinkable happens.

Safe data is geographically diverse data. Safe data is up-to-date data. Safe data is readily retrievable data. Safe data is more than a single point-in-time instance.

Unsafe data is “all your eggs in one basket.” Unsafe data is “I’ll get around to doing that backup tomorrow.” Unsafe data is “I stored the backups at my house which is also underwater now.” Unsafe data is “I only have yesterday’s backup and last week’s backup, and this data disappeared two days ago.”

SoftLayer’s customers are privileged to have the option to build a truly safe data backup strategy by employing the Evault option on StorageLayer. This solution provides instantaneous off-site backups and efficiently utilizes tight compression and block-level delta technologies, is fully automated, has an extremely flexible retention policy system permitting multiple tiers of recovery points-in-time, is always online via our very sophisticated private network for speedy recovery, and most importantly—is incredibly economical for the value it provides. To really pour on the industry-speak acronym soup, it gives the customer the tools for their BCP to provide a DR scenario with the fastest RTO with the best RPO that any CAB would approve because of its obvious TCR (Total Cost of Recovery). Ok, so I made that last one up… but if you don’t recover from data loss, what does it cost you?

On my personal server, I utilize this offering to protect more than 22 GB of data. It backs up my entire server daily, keeping no less than seven daily copies representing at least one week of data. It backs up my databases hourly, keeping no less than 72 hourly copies representing at least three days of data. It does all this seamlessly, in the background, and emails me when it is successful or if there is an issue.

Most importantly, it keeps my data safe in Seattle, while my server is located in Dallas. Alternatively, if my server were located in Seattle, I could choose for my data to be stored in Dallas or our new Washington DC facility. Here’s the kicker, though. It provides me the ability to have this level of protection, with all the bells and whistles mentioned above, without overstepping the boundary of my 10 GB service. That’s right, I have 72 copies of my database and 7 copies of my server, of which the original data totals in excess of 22 GB, stored within 10 GB on the backup server.

That’s more than sufficient for my needs, but I could retain weekly data or monthly data without significant increase in storage requirements, due to the nature of my dataset.
This service costs a mere $20/mo, or $240/yr. How much would you expect to pay to be able to sleep at night, knowing your data is safe?

Are you missing something? Wait … Backup!


February 27, 2008


It’s a fact -- all software ends up relying on a piece of hardware at some point. And hardware can fail. But the secret is to create redundancy to minimize the impact if hardware does fail.
RAIDS, load balancers, redundant power supplies, cloud computing - the list goes on. And we support them all. Many of these options are not mandatory, but I wish they were! That’s where the customer comes in – it is critical to understand the value of the application and data sitting on the hardware and set a redundancy and recovery plan that fits.

Keep your DATA safe:

  • RAID - For starters *everyone* should have a RAID 1, 5, or 10. This keeps your server online in the event of a drive failure.

The best approach – RAID 10 all the way. You get the benefits of a RAID 0 (striping across 2 drives so you get the data almost twice as fast) and the security of RAID 1 (mirroring data on 2 separate drives) all rolled into one. I think every server should have this as a default.

  • Separate Backups – EVault Backup, ISCSI Storage, FTP/NAS Storage, your own NAS server or just a different server. Lose data just once (or have the ability to recover it painlessly) and these will pay for themselves. Remember, hardware is not the only way in which you can lose data -– hackers, software failures, and human error will always be a risk.

StorageLayer. Use it or lose it.

Going further:

  • Redundant servers in different locations – spread your servers out across different datacenters and use a load balancer. Nothing is safer than a duplicate server 1000’s of miles away. That’s why we have invested in a second data center – to keep your data and business safe.

Check 'em out in our Services > Network Services section.

The future:

  • Solid state drives – aww yeah baby. They are coming.

Solid state drives are just that – a drive with no moving parts. No more platters or read/write heads. I mean come on, hard drives are essentially using the same basics that old record players use. CD’s use this technology too. And you see where those went (can you say iPod? I prefer my iPod touch. I have never had an iPod until now so I skipped right to the new fancy pants model. Can you tell I just got it?).

Check out these comparison tests of solid state drives vs. conventional ones:

  • Faster, faster, faster! –- Processors, memory, drives, network -- everything is getting much faster. And in part by redundancy (dual and quad core processors, dual and quad processor motherboards). See? Redundancy is the way of the future!

We have 4 Intel Xeon Quadcore Tigertown processors on one motherboard. That’s 16 processors on one server! Shazam!

  • Robot DC patrol sharks – yep. Got the plans on my desk right now. But I can’t take all the credit, Josh R. suggested this one, I just make things happen.

I work to keep all of our hardware running in tip top condition. But I look at the bigger picture when it comes to hardware – how to completely eliminate the impact of any hardware issue. That’s why I suggest all the redundancies listed above. While I can reduce the probability of hardware issues with testing, monitoring of firmware updates, proper handling procedures, choosing quality components, etc., redundancy is the ultimate solution to invisible hardware.

Hardwhere?, if you will.


February 11, 2008

Spares at the Ready

In Steve's last post he talked about the logic of outsourcing. The rationale included the cost of redundant internet connections, the cost of the server, UPS, small AC, etc. He covers a lot of good reasons to get the server out of the broom closet and into a real datacenter. However, I would like to add one more often over looked component to that argument: the Spares Kit.

Let's say that you do purchase your own server and you set it up in the broom closet (or a real datacenter for that matter) and you get the necessary power, cooling and internet connectivity for it. What about spare parts?

If you lose a hard drive on that server, do you have a spare one available for replacement? Maybe so - that's a common part with mechanical features that is liable to fail - so you might have that covered. Not only do you have a spare drive, the server is configured with some level of RAID so you're probably well covered there.

What if that RAID card fails? It happens - and it happens with all different brands of cards.

What about RAM? Do you keep a spare RAM DIMM handy or if you see failures on one stick, do you just plan to remove it and run with less RAM until you can get more on site? The application might run slower because it's memory starved or because now your memory is not interleaved - but that might be a risk you are willing to take.

How about a power supply? Do you keep an extra one of those handy? Maybe you keep a spare. Or, you have dual power supplies. Are those power supplies plugged into separate power strips on separate circuits backed up by separate UPSs?

What if the NIC on the motherboard gets flaky or goes out completely? Do you keep a spare motherboard handy?

If you rely on out of band management of your server via an IPMI, Lights Out or DRAC card - what happens if that card goes bad while you're on vacation?

Even if you have all necessary spare parts for your server or you have multiple servers in a load balanced configuration inside the broom closet; what happens if you lose your switch or your load balancer or your router or your... What happens if that little AC you purchased shuts down on Friday night and the broom closet heats up all weekend until the server overheats? Do you have temperature sensors in the closet that are configured to send you an alert - so that now you have to drive back to the office to empty the water pail of the spot cooler?

You might think that some of these scenarios are a bit far fetched but I can certainly assure you that they're not. At SoftLayer, we have spares of everything. We maintain hundreds of servers in inventory at all times, we maintain a completely stocked inventory room full of critical components, and we staff it all 24/7 and back it all up with a 4 hour SLA.

Some people do have all of their bases covered. Some people are willing to take a chance, and even if you convince your employer that it's ok to take those chances, how do you think the boss will respond when something actually happens and critical services are offline?


Subscribe to backups