Posts Tagged 'Safety'

March 17, 2010

Redrum

How many of you when you were kids were scared to death of the movie The Shining? I know I was. I think it still scares me today. The movie even made a little kid scary; his voice is what pulled it off. I can still get in trouble with my wife for getting our 6 year old to say "redrum" in a scratchy, scary voice.*

What do The Shining and redrum have to do with SoftLayer? We're all about redrum but only when it comes to destroying left over customer data. What do I mean by destroying customer data you ask?

When you have a server that you spent Capex on and have it in front of you and can touch it and set coffee on it or use it for a plant stand, you know where your data is. When you replace that server or upgrade the hard drive you can then do what most people do with the old one and chunk it in the dumpster or be a little more secure and format the hard drive or even a little more secure and take the drive out and smash it into pieces. Now, that is secure.

So what do you do when you outsource your hardware to a provider like SoftLayer? You put your old data in our hands and we redrum the data and make Jack Nicholson seem like an angel.

It is a little more difficult for us to protect your old data because we are an on-demand provider. When you cancel a server we reuse that server for another customer. You probably don't want your data in that new customers hands so we have to do a little more than format the drive and we can't just take it outside and bash it into pieces because then we couldn't reuse the drive. So we use a little technology to make sure your old data is safe.

When you cancel a server, it sits in limbo for a while just to make sure we can't change your mind and have you keep it. After the waiting period we erase the data. This is a destructive process, so when you do cancel a server, make sure you have the data you still require somewhere else. Our system uses algorithms developed by the Department of Defense and several independent agencies that are considered military grade as defined by the DOD 5220.22-M (sounds official right?). Utilizing this process residual drive data is destroyed. This process is monitored and logged and we can track the history of any drive. Once complete the drive is ready to be redeployed to a new customer.

I know you are thinking, "That isn't redrum," but what do we do with a drive when it is at the end of its productive life? If it's too small, not fast enough, or dead and out of warranty? We redrum it for sure! We complete the steps above and then send them offsite to get destroyed and then get them back after they are destroyed for tracking and verification of redrum! Yes, we could get them shredded but then we would have no proof they were destroyed. Here is what they look like when they return:

hard drive 1
Note the hole in the center.
hard drive 2
This is looking down from the "top".
hard drive 3
And last but not least, a view from the bottom. Note the platters are bent and protruding through the board.

*Just in case you haven't seen The Shining (Spoiler Alert) a small boy in the movie mumbles "redrum" in an eerie voice in the beginning of the movie. He continues to say it more and more and finally he writes it on the bathroom door. When you see it reversed in the bathroom mirror you then understand what he is saying.

-@Skinman454

October 19, 2009

I have backups…Don’t I?

There is some confusion out there on what’s a good way to back up your data. In this article we will go over several options for good ways to backup and sore your backups along with a few ways that are not recommended.

There is some confusion out there on what’s a good way to back up your data. In this article we will go over several options for good ways to backup and sore your backups along with a few ways that are not recommended.

When it comes to backups storing them off site (off your server or on a secondary drive not running your system) is the best solution with storing them off site being the recommended course.

When raids come into consideration just because the drives are redundant (a lave mirror situation) there are several situations, which can cause a complete raid failure such as the raid controller failing, the array developing a bad stripe. Drive failure on more than one drive(this does happen though rarely) , out of date firmware on the drives and the raid card causing errors. Using a network storage device like our evault or a nas storege is also an excellent way to store backups off system. The last thing to consider is keeping your backups up to date. I suggest making a new back every week at minimum (if you have very active sites or data bases I would recommend a every other day backup or daily backup). It is up to you or your server administrator to keep up with your backups and make sure they are kept up to date. If you have a hardware failure and your backups are well out of date it’s almost like not having them at all.

In closing consider the service you provide and how your data is safe, secure, and recoverable. These things I key to running a successful server and website.

August 18, 2009

Backups Are Not the Whole Story

Last night while making my regular backup for my World of Warcraft configuration, I thought about the blog and I didn't remember seeing an article that went into more detail than "backups are good" about backing up and restoring data.

If you've been around the InnerLayer for a while you will have noticed that backing up of data comes up periodically.  This happens because we frequently see customers whose world is turned upside down due to a mistyped command wiping out their data.  If you just thought "that won't happen to me... I'm careful at a prompt"... well, how about a cracker getting in via an IIS zero day exploit?  Kernel bug corrupting the filesystem?  Hard drive failure?  Data loss will happen to you, whatever the cause.

Data that is not backed up is data that isn't viewed as important by the server administrator.  As the title of this blog mentioned, backing up isn't the end of the server administrator's responsibility.  Consider the following points.

  • Is the backup in a safe location?  Backing up to the same hard drive which houses the live data is not a good practice.
  • Is the backup valid?  Did the commands to create it all run properly?  Did they get all the information you need?  Do you have enough copies?
  • Can your backup restore a single file or directory?  Do you know how to restore it?  Simply put, a restore is getting data from a backup back into a working state on a system.

Backup Safety
At a minimum backups should be stored on a separate hard drive from the data which the backup is protecting.  Better would be a local copy of the backup on the machine in use and having a copy of the backup off the machine, perhaps in eVault, on a NAS which is _NOT_ always mounted, even on another server.  Why?  The local backup gives you quick access to the content while the off-machine copies give you the safety that if one of your employees does a secure wipe on the machine in question you haven't lost the data and the backup.

Validity
A backup is valid if it gets all the data you need to bring your workload back online in the event of a failure.  This could be web pages, database data, config files (frequently forgotten) and notes on how things work together.  Information systems get complicated and if you've got a Notepad file somewhere listing how Tab A goes into Slot B, that should be in your backups.  Yes, you know how it works... great, you get hit by a bus, does your co-admin know how that system is put together?  Don't forget dependencies.  A forum website is pretty worthless if it is backed up but the database to which it looks is not.  For me another mark of a valid backup is one which has some history.  Do not backup today and delete yesterday.  Leave a week or more of backups available.  People don't always notice immediately that something has broken.

Restores
A good way to test a restore is get a 2nd server for a month configured the same as your primary then take the backup from the primary and restore it onto the secondary.  See what happens.  Maybe it will go great.  Probably you will run into issues.  Forget about a small operating system tweak made some morning at 4am?  How about time?  How long does it take to go from a clean OS install to a working system?  If this time is too long, you might have too much going on one server and need to split up your workload among a few servers.  As with everything else in maintaining a server, practicing your restores is not a one-time thing.  Schedule yourself a couple of days once a quarter to do a disaster simulation.

For those who might be looking at this and saying "That is a lot of work".  Yes, it is.  It is part of running a server.  I do this myself on a regular basis for a small server hosting e-mail and web data for some friends.  I have a local "configbackup" directory on the server which has the mail configs, the server configs, the nameserver configs and the database data.  In my case, I've told my users straight up that their home directories are their own responsibility.  Maybe you can do that, maybe not.  Weekly that configback data is copied to a file server here at my apartment.  The fileserver itself is backed up periodically to USB drive which is kept at a friend's house.

Categories: 
July 6, 2009

The Cure for Irrelevance

I’ve been feeling rather irrelevant lately……yeah, yeah, I know . Watch out, because when lawyers feel irrelevant we sue!

Anyway, just thinking about the things going on in the rest of the world, like the wars in Iraq and Afghanistan. We have men and women over there who are still dying and losing their sanity on a daily basis, but for those of us who don’t have anyone close to us involved in the wars, it’s become a low hum…car bomb, soldier killed, hum, hum. But then I think of the daily terror that those who have loved ones go through – did she have to go out on patrol today? Did she get hurt? Did she get killed? Every day, every hour spent wondering if that person is safe and will come home again. That soldier is not irrelevant – he is making the greatest sacrifice so we can go on with our safe, secure lives over here.

My thoughts also turn to the people of Iran, and I find myself thinking: “If I lived in a repressive regime, would I be out on the streets in defiance of the government and particularly with the threat of being beaten, jailed or disappearing from the face of the earth?” I like to think I would, but I don’t know the answer, and that feeds my irrelevance. And come on Iran; give your people some credit. Make it at least look like the election wasn’t predetermined. You declare a landslide victory for Ahmadinejad (cut and paste, baby, cut and paste!) mere minutes after the polls close, yet the ballots are supposed to be hand counted. How can that work? I mean, wait an hour or so – pretend you counted. Please! Are the people of Iran irrelevant? No, they are making the greatest sacrifice in a battle for freedom, and in an uprising that may very well change the course of their history. The world is watching.

So how do I become relevant? (Assuming, that is, that a lawyer can ever be relevant). How do SLayers and SLackers become relevant? We go that extra mile. We’ve been dealing with cranky clients all day – keep the smile on the face and in the voice and treat them like they’ve just put in an order for 300+ servers a month. We don’t remain satisfied with the status quo – figure out how to make our system and services better, stronger, faster. We don’t rest on our laurels because we just had a major release, i.e., Cloudlayer instances. We look ahead and figure out or invent the next new thing our customers need or want. We scan the forums to keep a pulse on our clients (and it’s usually good for an eye-roll or two). We keep Lance and Mike out of trouble.

So am I like a U.S. soldier or the Iranian people? Not so much, but I can do things to stay relevant in my own little world.

Categories: 
Subscribe to safety