executive-blog

August 18, 2009

Backups Are Not the Whole Story

Last night while making my regular backup for my World of Warcraft configuration, I thought about the blog and I didn't remember seeing an article that went into more detail than "backups are good" about backing up and restoring data.

If you've been around the InnerLayer for a while you will have noticed that backing up of data comes up periodically.  This happens because we frequently see customers whose world is turned upside down due to a mistyped command wiping out their data.  If you just thought "that won't happen to me... I'm careful at a prompt"... well, how about a cracker getting in via an IIS zero day exploit?  Kernel bug corrupting the filesystem?  Hard drive failure?  Data loss will happen to you, whatever the cause.

Data that is not backed up is data that isn't viewed as important by the server administrator.  As the title of this blog mentioned, backing up isn't the end of the server administrator's responsibility.  Consider the following points.

  • Is the backup in a safe location?  Backing up to the same hard drive which houses the live data is not a good practice.
  • Is the backup valid?  Did the commands to create it all run properly?  Did they get all the information you need?  Do you have enough copies?
  • Can your backup restore a single file or directory?  Do you know how to restore it?  Simply put, a restore is getting data from a backup back into a working state on a system.

Backup Safety
At a minimum backups should be stored on a separate hard drive from the data which the backup is protecting.  Better would be a local copy of the backup on the machine in use and having a copy of the backup off the machine, perhaps in eVault, on a NAS which is _NOT_ always mounted, even on another server.  Why?  The local backup gives you quick access to the content while the off-machine copies give you the safety that if one of your employees does a secure wipe on the machine in question you haven't lost the data and the backup.

Validity
A backup is valid if it gets all the data you need to bring your workload back online in the event of a failure.  This could be web pages, database data, config files (frequently forgotten) and notes on how things work together.  Information systems get complicated and if you've got a Notepad file somewhere listing how Tab A goes into Slot B, that should be in your backups.  Yes, you know how it works... great, you get hit by a bus, does your co-admin know how that system is put together?  Don't forget dependencies.  A forum website is pretty worthless if it is backed up but the database to which it looks is not.  For me another mark of a valid backup is one which has some history.  Do not backup today and delete yesterday.  Leave a week or more of backups available.  People don't always notice immediately that something has broken.

Restores
A good way to test a restore is get a 2nd server for a month configured the same as your primary then take the backup from the primary and restore it onto the secondary.  See what happens.  Maybe it will go great.  Probably you will run into issues.  Forget about a small operating system tweak made some morning at 4am?  How about time?  How long does it take to go from a clean OS install to a working system?  If this time is too long, you might have too much going on one server and need to split up your workload among a few servers.  As with everything else in maintaining a server, practicing your restores is not a one-time thing.  Schedule yourself a couple of days once a quarter to do a disaster simulation.

For those who might be looking at this and saying "That is a lot of work".  Yes, it is.  It is part of running a server.  I do this myself on a regular basis for a small server hosting e-mail and web data for some friends.  I have a local "configbackup" directory on the server which has the mail configs, the server configs, the nameserver configs and the database data.  In my case, I've told my users straight up that their home directories are their own responsibility.  Maybe you can do that, maybe not.  Weekly that configback data is copied to a file server here at my apartment.  The fileserver itself is backed up periodically to USB drive which is kept at a friend's house.

Categories: 
August 10, 2009

Backups: It’s Good to Have Them!

Man, was this a weekend for me and backups! The first one I needed was for my second advance free fall course at skydive101.com on Friday (My Saturday). I jumped out of the plane a little early and one of my two instructors was not ready and did not jump on my call. Thank god I had a backup. Once my shoot deploys, being this is only my second jump out of a perfectly good airplane, I look up to check my slider and make sure my parachute is deployed and everything is correct I did not see my brake handles, I said some curse words I won’t put in this blog, and then was like it is OK I have a backup. I was still at 4000 ft at this time, still having another 1500 feet to decide if I want to release my main and pull the backup, luckily I found my brake handles pulled down twice and they worked and everything was OK.

So I get home after a day of skydiving and having fun in the sun (it is rather rare in Seattle, though not this summer) and I notice my trusty old Windows XP terminal has multiple errors on it. I do the first thing I always do with my personal Microsoft machines and reboot it. Ouch no operating system found, bad hard drive! Thank god I ghosted that machine 2 weeks ago as I figured the old IDE drives had been on their last legs. Saturday morning comes around, which is a big day for me, as I am hosting a party that night at a local night club. I notice I have a few (8900) email messages on my blackberry, so I decided it is about time to delete some. I tell it to delete, look back at it and 4 minutes later it says APP ERR. PLEASE RESTART. Needless to say, you guessed it. It wouldn’t boot back up and I had to force an OS onto it, and restore from my backup of a month ago, which reminded me that I need to start backup my blackberry more often.

So the moral of the story is, it is always good to have a backup, and we have plenty of backup options for you, so if you don’t have one, I would suggest contacting SLales. I would also suggest everyone trying to skydive at least once in their lifetime.

Categories: 
August 6, 2009

Punishing Success

Let’s say you worked for years to become a world class athlete. As a kid, you were in the gym while other athletes were at the movies. You were in the weight room on Saturday nights when no one else was there. You shunned pizza and soda in favor of grilled fish and fresh fruit. By the time Letterman hit the evening airwaves, you were well into restorative sleep. You were out the door for your morning runs while other athletes snoozed. As a result of all this, now you perform at an elite level and are very successful at your sport. Suddenly, you find that there are people who have a vested interest in helping you maximize your athletic potential. Your coaches, your managers, and companies who pay you to endorse their products all want to see you do your best. Why? Because doing your best helps them be more successful.

So, they provide you with all the things you need to maximize your potential. You get the best training gear and training regimens. You get the best nutrition. You get the right amount of rest. All these things help you maximize your potential. Thus the relationship is a nice symbiotic cycle – the more success you experience, the more success your coaches, managers, and endorsement companies experience. Win-win. Makes sense, right?

So, imagine the silliness if your coaches, managers, etc., made the decision that because you were so fortunate in your success that you had to “give back” almost half your resources to train the athletes who loafed, stayed out late, partied and gorged on pizza. Because you’re such a hard-working and smart athlete, you don’t need all those resources to participate adequately in your sport, they rationalize. Consequently, you don’t hit your potential, your coaches and managers don’t distinguish themselves, and endorsing companies don’t call you. You then feel that you’ve been punished for your hard work and success.

Sadly, much of our government policy falls under this flawed logic. The IRS just released their latest income tax stats for the year 2007. For that year the top 1% of earners paid 40.4% of all income taxes collected. We all know that right now we’re coming out of a recession and we really folks to invest in businesses and hire people to get the economy moving. So how do the 2007 numbers compare to, say, the 1980’s? During the ‘80’s, we managed to shake off the “stagflation” of the ‘70’s and get the economy rolling again. It was during this time that many technology juggernaut companies were spawned – Microsoft being a good example. So, how much of the income taxes in the ‘80’s were paid by the top 1% of earners? The average for the 10 years from 1980-1989 was 22.2%.

Let’s do some quick math. $1.116 trillion in income taxes was collected in 2007. Of that, $455.3 billion was paid by the top 1% of earners. If they paid 22.2% as in the ‘80’s, they would have paid $247.8 billion in taxes, and right now we’d have $207.5 billion MORE dollars invested in our economy. That would be quite a stimulus package! Our current policy punishes success and chokes off fuel from our economic engines while we’re trying to climb out of a worse recession than we had in the ‘70’s. Not smart.

Some may think that this would simply mean that our government deficit would be $207.5 billion higher. This is not the case at all. These folks that make up that top 1% didn’t get there by being lazy or not putting their money to work. I know some folks in that group, and they WANT to put their money to work! I know one gentleman who had to be told some legal docs for a deal could not be prepared over the weekend because Christmas was on that weekend. These folks are like the world class athlete I mentioned above – by and large they’re disciplined and hard-working. Their money will build new businesses and create more jobs, and the government will collect far more revenue from this new economic activity than it would give up in collections from these top 1% folks. Think about it – how many of us have ever been hired by a “poor” person? Instead of punishing economic success, we should encourage it!

Bottom line, if government policy were to make sense, it would encourage these folks to maximize their economic potential and find the correct balance of revenue to collect and yet still promote economic growth. What would we prefer? That the government collects 50% of $1 trillion or 30% of $2 trillion? Hint: 30% of $2 trillion is a WAY better deal.

At SoftLayer, we think very differently about things. We simply do not punish our customers for succeeding. We empower them to be more successful – why? Because if our customers succeed, we succeed. We get this.

Can we prove this? Perhaps a look at how customers vote with their feet is an indicator. For the past few months, SoftLayer has seen the lowest percentage of customers terminating business with us in our history. If we punished our customers for their success, they would go elsewhere.

August 5, 2009

SLales Motivation 101

We have a pretty good sales team here at SoftLayer, quite honestly I couldn’t ask for a better group than the one we have.  It’s pretty common for us to have various sales contests and awards to keep the team motivated and focused in the right direction, however last month I decided to try something a little different.  Lance and I set a pretty lofty sales goal for the team (one I didn’t really think they would achieve), and told them that if they reached the goal I would shave my head completely bald.

As the month progressed, I began to realize the genius of this contest.  The team was focused, driven by the desire to see their boss publically humiliated.  SLales worked extra hours, came in early, stayed late, made calls, sent emails, followed up every lead, they did everything possible to exceed the goal.  Looking back on it as their manager, it was a beautiful thing to watch and I’ve never been more proud of my team.

It ended up being close (down to the final hours on Friday actually).  However, SLales stepped up to the plate and met the challenge.  They worked together like a well oiled machine and overcame numerous obstacles, with the singular vision of seeing their boss embarrassed.  They succeeded as a team, and here is the payoff:

August 3, 2009

Education

Attending College Classes can be a daunting task. The hours of homework and studying (and the obligatory time spent actually in the classroom) can noticeably eat away at one’s free time (and at times, their sanity). While it can be painful to take on college, attending classes and working on top of it can be exponentially more difficult. Balancing your studies with your responsibilities at work can be tricky, even for those who are experts in time management. When all is said and done, though, the investment is well worth it. As I’ve stated before, Knowledge is power (yes, I know, shameless self promotion), and learning can occur at any opportunity.

I recently realized that with the exception for while sleeping (some days I can count the hours on one hand), I am always learning new things. While my progressing college education keeps me thinking, SoftLayer has taught me more than I ever thought I would learn in such a short amount of time. New operating systems (at least to me), and continual changes and improvements are synonymous with life at SL. Learning occurs at every customer request, every server build, and every operating system install. Certainly, employment here is not for the faint of heart. More so, no one can say that they didn’t leave their shift just a bit smarter than when they arrived.

Knowledge is important in this industry, as knowing the correct process to solve a problem can mean the difference between five hours and five minutes of downtime. While everyone has their strengths, the team that we have here supersedes any possible weaknesses, leading to one of the brightest group of individuals anyone could have the privilege of working with. I spend my shifts perpetually challenged, but never overwhelmed by the sheer magnitude of issues (read: learning opportunities) that present themselves every day. While I will concede that classes such as precalculus and humanities may not directly sharpen my troubleshooting skills, being able to think logically and follow procedures will certainly pay off in the long run.

July 27, 2009

Cool Tool: find

Have you ever gotten an e-mail from your server that a particular partition is filling up? Unfortunately, the e-mails don't usually tell you where the big files are hiding.

You can determine this and many other handy things by using the Unix utility 'find'. I use the 'find' command all the time in both my work at SoftLayer and also for running some sites that I manage outside of work. Being able to find the files owned by a particular user can be handy.

The 'find' command takes as arguments various tests to run on the files and directories that it scans. Just running 'find' with no arguments is going to list out the files and directories under your current location. Real power comes from using the different switches in various combinations.

find /some/path -name "myfile*" -perm 700

This format of the command will search for items within /some/path that have names starting with the string 'myfile' and also have the permission value of 700 (rwx------).

find /some/path -type f -size +50M

Find files that are larger than 50MB. The '-type f' argument tells find to only look for files.

find /some/path -type f -size +50M -ctime -7

Find files that are larger than 50MB and that have been created in the last seven days.

find /some/path -type f -size +50M -ctime -7 -exec ls -l {} \;

The -exec tells find to run some command against each match that it finds. In this case, it is going to run an 'ls -l'. Moves, removes and even custom full scripts are doable as well.

There are many, many more arguments that are possible for 'find'. Refer to the man pages for find on your particular flavor of Unix server to see all the different options for the command. As with all shell commands, know what you are running. Given the chance 'find' will wipe out anything it can ( via -exec rm {}, for example).

July 22, 2009

Turning Fantasy Into Reality

I remember when I first started here at SoftLayer it was quite exciting and nerve racking at the same time.  You see I came from the telecom industry, and I worked for a huge company that had 100,000+ employees.  Basically, I did the same thing everyday.  I learned a lot when I first joined this huge company, but I felt like my career had become stagnant and I needed a change.  I decided to look for a job at a small company and be challenged everyday.  Man, did I find the perfect job!!!  Anyway, back to the point.  At first, I didn't realize how advanced SoftLayer was till I began to look more into the company and the industry we are in.  The more I dug the more I was impressed and excited to be a part of something revolutionary.

I know we all have seen the movie "Charlie and the Chocolate Factory" (old school version) or at least most of us.  I know it sounds cheesy but  that is the best way to describe how I felt when I first joined.  I felt like Charlie.  I was just amazed to see what I saw inside the workings of SoftLayer.  I had no idea that some of the tools/services/automation SoftLayer had done was even possible.  The best way for me to describe the management  of SoftLayer is they are the Willie Wonka's of our industry.  Some people may think SoftLayer's ideas are radical or even impossible, but we don't.  All the great people here at SoftLayer work together to make the impossible possible.

While other companies try to mimic us, we are busy turning fantasies into realities.

July 20, 2009

Thankful

So here I sit cramped in a seat built for a 10 year old on American flight 1492 from New Orleans to Dallas. There isn’t enough room left for a marshmallow I bet. Yep, I am thankful I just left the site of the Microsoft WPC 2009 where I had to do a little booth duty and mingle with some folks that run the coolest companies out there. The show seemed a little different this year. Last year the recession was just getting going and gaining some steam. The big companies still had previously budgeted money to burn and were doing just that. They had very large booths and better swag and Microsoft rented Minute Maid Park in Houston and threw quite the party. This year was noticeably different. I would bet that over 50% of the people that came by our booth were international which tells me that US companies are still cutting back. One very large US Company wasn’t even at the show and they were a flagship last year. One of our much larger competitors was barely existent, a flyer here a business card there and most other companies had much smaller booths and the swag was just not quite as enticing. Thankfully my kids like anything so they will still be extremely happy with the 3 bags of stuff I was able to round up. I am thankful that SoftLayer was able to hand out just over 1000 cool SoftLayer Frisbees (boomers to our international friends) and 200 cooler bags. The boomers were a great hit at the show and as far as I know no one lost an eye in the process of handing them out. Microsoft still put on a great show and “The Party” at the House of Blues was really cool. I just wish one year they would actually send me the email so I could get the wristband instead of standing in the slacker line.

Enter the ton of bricks that hit me. I am extremely thankful and 99% percent sure, no wait, I am 100% sure that I work for the coolest and one of the fastest growing companies out there. Thankfully after a week of spreading the word about SoftLayer I am on this flight back home. Thankful the pilot knows how to fly and land this jet. Thankful the flight attendant just gave me some lukewarm orange juice. Thankful I remembered to take my laptop out of the bag I checked this time but still managed to leave my cell phone in the checked bag! I am off my leash!

Ok so back to my subject.

I am Thankful this is almost my 2 year anniversary at SoftLayer. Thankful the guys that started this place and the great minds they have added along the way are really running a top notch company. Thankful that I wake up every day excited about my company and my role, thankful that SoftLayer lives within our means, and thankful the things we spend money on are for one thing, to make our product better and our customers happy. Thankful we do not waste on the posh extras that some other companies brag about. Thankful that once a customer tries us out and understands what our system is capable of they rarely ever leave. Thankful SoftLayer has great punch and people drink it regularly with pride, both customers and employees alike. Thankful that our products have the ability to help struggling companies in this down economy and we continue to grow because of it. Thankful we are setting sales records and our churn rates are much lower than this time last year. Thankful word is spreading on how we can let a company hold on to their capital for other expenditures and simply pay monthly for their IT needs on demand. Thankful we have a plan and we stick to it. Thankful we know what we are great at and don’t try to be everything to everyone wasting countless hours complicating our business plan. Thankful I can sleep at night knowing I am at a stable company and I don’t have the worries that many people in our country have during this recession. I wish everyone affected by the recession a fruitful second half of the year and hope that everyone can start recovering from the current hardships.

Oh and I am thankful that the flight attendant has moved along to the folks behind me and is finally finished booty bumping me every 3 seconds and thankful we are 45 minutes from DFW!

And I know…… you are thankful……. that this blog is ending……. Thanks for reading…….thankfully……

July 17, 2009

Combinations

As you may have noticed from a previous blog a while back, we here at SoftLayer really like Whataburger quite a bit. For those of you not lucky enough to live in the southern part of the US, Whataburger has a chain of hamburger restaurants that started in Texas but now stretches across the south from Florida to Arizona. For the record, their breakfast taquitos and hamburgers are second to none.

Every so often Whataburger runs an ad that states there are over 34,000 different ways that you can order your Whataburger. Every time I see this ad it makes me wonder, how many possible ways can you order a server with SoftLayer? I’ve always said to myself “one day I will calculate this and write a blog about it”, but I never seemed to get around to it until now. My ultimate motivation to complete this calculation came while discussing our “perceived” limited configuration options with a customer due to having a set of “standard” solutions.

As I started to research the math involved in this calculation, I quickly realized that the total number of combinations for all servers and services we offer would be too much to comprehend or calculate. Therefore I limited the scope to just options for a Nehalem processor server. I also enlisted the help of Greg Kinman, one of our interns, to help me with the math to make sure we got it right. Those Kinmans are multiplying around here (almost as fast as the configuration options)…

Greg calculated that there are 96,631,664,476,185,600,000 different ways to order a Nehalem server. That’s 97 quintillion possibilities for just one server when you take into account the various hardware, software, and service options! Once you start to consider ALL the server options and cloud instances offered, it seems to me that unlimited would be a better description for the configurations SoftLayer offers.

Greg, I owe you lunch for helping me with the calculations today. How about Whataburger?

July 15, 2009

Subjecting Subjectivity To Math

I recently read an article about an endeavor that is currently being undertaken to develop a “Speech Analysis Algorithm Crafted to Detect and Help Dissatisfied Customers”. In short, a team of engineers are hoping to create software that will recognize when a caller is becoming stressed and immediately phone a manager to alert them of a developing situation. Wow! It is rare that you would see math and science applied to something that is so subjective. After all, math is used to quantify and measure things all based on a known or a baseline. In this particular effort, I would surmise that the team of engineer’s most difficult task will be to determine how to establish a unique baseline for each unique call and caller. Once upon a time as a student of Electrical Engineering, I took on my share of convolution integrals and that’s a path that I do not care to venture down again. I’ve also taken on my share of convoluted customer calls in a past life and witnessed our frontline assisting customers in complex situations here at SoftLayer.

Until there is such an application that can detect and address a conversation that may be heading in the wrong direction, we have to rely on good ole’ training and experience. With each call and query, the baseline is reset. I’d even go further to say that with each exchange; the baseline is reset as our Customer Service Agents seek information to get to the root of the issue. It’s not hard to imagine the frustration that can build in a back-and-forth conversation as two people look to come to a solution or an amiable conclusion just as it is understandable that sometimes, a customer may simply need to vent. How do you calculate and anticipate those scenarios?

I wish much success to the team involved in the customer service speech analysis program. And programmatically speaking, I see many CASE, SWITCH, FOR, WHILE, BREAK, CONTINUE, IF, ELSE, ELSE IF, NEXT statements in your future. Good Luck!

Pages

Subscribe to executive-blog