December 7, 2009

Availability with NetScaler VPX and Global Load Balancing

The concept Single Point of Failure refers to the fact that somewhere between your clients and your servers there is a single point that if it fails downtime happens. The SPoF can be the server, the network, or the power grid. The dragon Single Point of Failure is always going to be there stalking you; the idea is to push SPoF far enough out to where you have done the best you can with your ability and budget.

At the server level you could combat SPoF by using redundant power supplies and disks. You can also have redundant servers fronted by a load balancer. One of the benefits when using load balancer technology is that the traffic for an application is spread between multiple app servers. You have the ability to take an app server out of rotation for upgrades and maintenance. When you’re done you bring the server back online, the load balancer notices it UP on the next check and the server is back in service.

Using a NetScaler VPX you can even have two groups of servers—one group which generally answer your queries and another group which usually does something else—with the second group functioning as a backup against all of the primary servers for a service having to be taken down through the Backup Virtual Server function.

Result: no Single Point of Failure for the app servers.

What happens if you are load balancing and have to take the load balancer out of service for upgrades or maintenance? Right, now we’ve moved SPoF up a level. One way to handle this is by using the NetScaler VPX product we have at SoftLayer. A pair of VPX instances (NodeA/NodeB) can be teamed in a failover cluster so that if the primary VPX is taken down (either by human action or because the hardware failed) the secondary VPX will begin answering for the IPs within a few seconds and processing the actions. When you bring NodeA back online it slips into the role of secondary until such time as NodeB fails or is taken down. I will note here that VPX instances do have dependency on certain network resources and that dependency can take both VPX instances down.

Result: Loss of a single VPX is not a Single Point of Failure.

So what’s next? A wide-ranging power failure or general network failure of either the frontend or the backend network could render both of the NetScalers in a city unusable or even the entire facility unusable. This can be worked around by having resources in two cities which are able to process queries for your users and by using the Global Load Balancer product we offer. GLB load balances between the cities using DNS results. A power failure taking down Seattle just means your queries go to Dallas instead. Why not skip the VPX layer and just GLB to the app servers? You could, if you don’t have a need for the other functionalities from the VPX.

Result: no single point of failure at the datacenter level

Having redundant functionality between cities takes planning, it takes work, and it takes funding. You have to consider synchronization of content. The web content is easy. Run something like an rsync from time to time. Synching the database content between machines or across cities is a bit more complicated. I’ve seen some customers use the built-in replication capabilities of their database software while others will do a home-grown process such as having their application servers write to multiple database servers. You also have to consider issues of state for your application. Can your application handle bouncing between cities?

Redundancy planning is not always fun but it is required for serious businesses, even if the answer is ultimately to not do any redundancy. People, hardware and processes will fail. Whether a failure event is a nightmare or just an annoyance depends on your preparation.

December 3, 2009

Hey, I just got an email saying I won a million dollars! *Click* Wait, what just happened to my computer?

This is usually how it starts. Some shady person sends out spam telling people they have one a million dollars or a free laptop or mp3 player with a link a form they need to fill out to claim their prize. Only you don’t win an mp3 player or laptop. You win an infected computer that is now a drone in a much larger botnet. This botnet is either for direct malicious purposes (Denial-of-Service attacks) or indirect malicious purposes (spam, phishing, etc). How do you stop this from happening to you and you becoming “that guy”? Don’t click links in email unless you’re 100% sure who it’s from and what it’s for. That’s the basic rule to remember. Secondly, make sure you have an anti-virus program that’s capable of scanning email and keeping your system protected from malicious browser exploits. Thirdly, (and this should go without being said, but I’m saying it anyways) make sure your computer (and all software) is up-to-date. Sure, there’s the occasional bug and 0-day exploit on up-to-date systems, but there’s a whole slew of exploits and things that can be done to an un-patched system. Keep your systems up-to-date and you reduce the “known” exploits from literally thousands to maybe a few.

Think about this, 80% of the world’s email is considered spam. Of that 80%, the vast majority (more than 75%) is sent using infected computers (drones). If everyone would re-think blindly clicking links in emails and on webpages (social networking sites have a history of people trying to fool users into clicking bad links) then the spammers wouldn’t have drones available to them to send spam. Interesting thought, isn’t it? Let’s stop spam by being smart internet users and denying the “bad guys” the resources they need to send out the spam.

December 1, 2009

Holiday Seasons and Holiday Shopping

It’s that time of year again. No, not time to spread joy and cheer to your family (it can be that, too), but rather the time of waiting in long lines when it’s freezing outside hoping to get a good deal or the perfect gift. It’s the holiday shopping season. With the holiday shopping season comes the holiday “holy cow it’s freezing cold out here” or the holiday “hey! that person just cut in line” season as well. Of course, one thing people need to remember this time of year is that it’s not about the shopping and spending money (Money? What money?) but rather it’s about spending time with family and looking back at the year that has passed. Of course, retailers and scammers would like you to think otherwise. They’re trying their hardest to get you to part with your hard earned money. What does this mean to you? It means that you have to watch out who you’re buying from and what websites you are giving your personal information to. This is the perfect season for scammers to get your grandparents to give up their personal information (and with it everything in their bank accounts). Of course, this is easily avoided. Most reputable websites will have SSL certificates from a reputable vendor. Being a customer of SoftLayer you are already aware of a (fairly) new service we offer… A short while back SoftLayer began to offer VeriSign and GeoTrust SSL certificates (for more information: Simply making sure that the site you are buying from has a certificate installed like the ones we offer will help ensure that your information isn’t going to some thief hiding in his basement. What’s the moral of this story? (All Holiday stories have a moral…) Stay safe, be careful, and enjoy the holidays!

November 25, 2009

The Secret Mind of a SoftLayer Tech

I sit right in the middle of the NOC (Network Operations Center) here at SoftLayer. I hear all the tech calls, project discussion, and random banter from the techs on a daily basis. Most techs are also propeller heads on their own time. They have servers of their own, apps they like to run, preferences as to what hardware and software they like best, etc. Now, working in this field for most of my life I know that techs are not company loyal when it comes to their personal geeky funness (yes, that’s a word) I don’t care if spell check, Google and the rest of the world doesn’t think so (but I digress) they like what does the best job regardless of where it comes from.

I routinely hear techs talking about their personal servers, apps, etc. and referring back to SoftLayer with comments like, “I just host it on my server here at SoftLayer so I don’t have an issue.” With the issue being whatever the topic of conversation might have been. Network speed and stability, hardware and software reliability, ease of access (KVM over IP, the portal in general, multiple remote control options) cost, endless amount of add-ons, and the latest and greatest in everything!

I can relate.

I realized the potential of SoftLayer from the beginning and this place continues to exceed my expectations- and my expectations are always over the top! Simply put, after working in the corporate world and realizing what could be done with the right people and the right attitudes, I vowed only to work with a company that shared those views. And quite honestly I never thought I would see it happen. Then along came SoftLayer.

When techs constantly refer back to SoftLayer for their own fun computer projects as being the best solution, it just confirms what I already knew:

SoftLayer Rocks!

November 20, 2009

The Art of the Apology

I wrote a blog but it got ixnayed by legal. (That should be funny because I am “legal.” At this time I shall choose to remain cryptic, but as God is my witness, I’ll publish that blog someday after X, Y, and Z happens). Now, where was I – ah, yes, a new and different blog.

Today, boys and girls, we shall talk about the art of the apology. Since we were little, we’ve been taught to say “sorry.” (Well, most of us, but maybe not he whose names starts with J and ends in O-N-E-S, but I digress again). “Little Johnny, say sorry to your sister for bonking her on the head.” And Little Johnny will usually say sorry to avoid your wrath, rather than actually being sorry for the head bonking. This is the first lesson in the art of the apology – make sure it is sincere and that you mean it. Otherwise, it is really better if you say nothing at all. Maybe wait until it can become sincere, and if it can never become sincere, go back to step one and don’t say anything at all. The Boy often gets in trouble for head bonkings and other various and sundry misdeeds committed upon The Girl. He gets sent to time-out and then is supposed to apologize to The Girl. More often than not The Boy gets extremely defiant and grunts out a “sor-ry” as belligerently as he can. This only serves to piss The Mommy off and gets The Boy in even more trouble. (Can I use that word?) The takeaway on this is that The Boy needs to say sorry like he means it, or not bother getting out of time-out until he can do so. Another example of an apology that is better left unsaid is the disingenuous-apology-that-is-really-not-an-apology apology. Example: “I’m sorry you are an idiot, but….” Go back to time-out!!

Often a simple, sincere heart-felt apology can go a long way towards diffusing a situation that might otherwise result in hurt feelings, anger, and bitterness or, in my world, lawsuits. Maybe a manager loses his/her cool with an employee in one of the many stressful situations we face on a daily basis. When the manager calms down, an apology may be the cure to a situation that might later spiral out of control and explode. Maybe you have two feuding employees – an apology by one or both parties may be all it takes to turn a situation that may have resulted in a termination or two into one in which the working relationship is restored. This might involve a situation with your co-worker, your friend, your spouse or a client. Many times what happens is that we want to be right, rather than do what’s right. A meaningful apology to a client might save a $30,000/month account, but dad gummit, you are right and the client is wrong and they are an idiot and you are not. All of that may be true, but is it worth it? Is it really, really worth it? Is it worth that account? Is it worth that friendship? Is it worth your job? Is it worth that marriage?

Here, let me practice: “Mike, I am sorry you are mean and that I implied your upbringing was nothing less than stellar…..” Alright, alright – I’ll keep practicing.

*Note: This blog was inspired by the esteemed labor and employment lawyer Michael Maslanka and one of his recent blogs at, which I forwarded to our managers for their digestion.

I deeply and sincerely apologize in advance for any copyright infringement or any other legal no-no’s in my blog.

November 18, 2009


The show about nothing that took over NBC years ago is being lived out at SoftLayer. In case you haven’t been keeping up, SoftLayer has a team called STAT and without making you sit through the gory details we use ninja tactics in our efforts to keep the churn rate low. Much like the show Seinfeld which was about everything and nothing at the same time, the STAT group does everything and nothing as well.

It has been said that the team does simply enough to stay employed and we get a little grief from just about everyone in the company but I just blame the stealthy ninja tactics for all that. We haven’t built a bed under our desks just yet but a prototype is being designed as we speak. When the products, support, and culture are so cool why would customers want to leave?

The STAT group has been around since the dawn of time (2008 to be exact!) and have many years of tradition handed down which we must use daily to complete our mission. Some of the traditions have gone away over the long journey since our inception like a loud and proud bell ring when a customer was saved. It seemed to annoy some of our non SLeinfeld co-workers. Those crazy developers said “No bell for you!” There are other traditions that have gone away over time but we continue to make more as often as possible.

Our latest episode is a pretty cool one so we will not be “jumping the shark” just yet. It is one that the industry may have never seen. If so, it is very rare and this makes the STAT team very proud. In the on-demand virtual datacenter industry, churn is defined as, “when a customer doesn’t want your services anymore!” That being said we have designated churn as a bad thing (like Elaine dancing!). A higher churn percentage is not as good as a lower one. Get the picture? From this day forward let it be said that in October 2009 the STAT team and every other person involved with SoftLayer including every employee in every department and our resellers and customers have achieved a monumental goal! The year over year churn numbers are equal in raw numbers and LOWER in percentage for the month of October. When you incorporate the sales growth into that equation this is an impressive accomplishment because typically when you add servers month after month the churn rate grows due to sheer volume. So I say to everyone involved, take a few hours today and go hit some golf balls into the ocean (except Jones), you deserve a break!

Just know that the next time the hair on the back of your neck stands up and you feel like someone is watching you or their might be someone or something lurking in the shadows and Kramer doesn’t burst through your door, don’t be frightened it is most likely just a STAT team member waiting to help you in a time of need or maybe just goofing off in a relatively close proximity to you and creating yet another day in SLeinfeld land.

RIP Seinfeld!

November 16, 2009

How Many Recovery Plans Do We Need?

Several of our bloggers have written about backups in The InnerLayer. This morning, I had an experience that makes me wonder how many recovery plans we need.

I walked out of the house to the driveway and saw that my left rear tire was flat. An enormous nail had punctured my tire right in the middle of the tread, and the slow leak deflated the tire overnight. To recover from this disaster, I needed to get my vehicle drivable and get to the Discount Tire location near my house so that they could fix the flat. Below is a log of how the recovery plans worked out.

Recovery Plan #1: Call roadside assistance. While waiting on them to change my tire, logon from home and get some work done before going to Discount Tire. I have leased four different brands of vehicles over the past 10 years, and roadside assistance was always included with the lease. So I call the 800 number and they tell me I don’t have roadside assistance. (Note to self: read the fine print on the next lease.) Result: FAIL

Recovery Plan #2: Inflate tire with can of Fix-a-Flat. I retrieved the can from my garage, followed the instructions, and when I depressed the button to fill the tire, the can was defective and the contents spewed from the top of the can rather than filling the tire. Result: FAIL

Recovery Plan #3: Use foot operated bicycle pump to inflate tire and drive to Discount Tire. I have actually done this successfully before with slow leaks like this one. It is third in priority because it is harder and more tiring than the first two options. So I go to my garage and look at where the pump is stored. It isn’t there. I scour the garage to find it. It is gone. Result: FAIL

Recovery Plan #4: Change out of office clothes into junky clothes, drag out the jack and spare and change the tire myself. This is number four in priority because it is the biggest hassle. I will spare you all the slapstick comedy of a finance guy jacking up a vehicle and changing the tire (finding the special key for the locking lug nuts was an interesting sub-plot to the whole story), so I’ll summarize and say RESULT: Success!

As a side note, I must give props to Discount Tire. Having bought tires there before, I was in their database as a customer and they fixed the flat and installed it on my vehicle for no charge. I recommend them!

All this got me to thinking about not only having backups, but having redundant recovery plans. Sure, you’ve got a recent copy of all your data – that’s great! Now, what’s your plan for restoring that data? If you have an experience like my flat tire recovery this morning, it might be a good idea to think through several ways to recover and restore the data. Our EVault offering will certainly be one good strategy.

November 13, 2009

Buenos Dias

Growing up I would consider myself an average kid. I played football and basketball outside with my brothers; we’d come home every day from school and turn on cartoons. Depending on the day it may have been power rangers or the animaniacs, rarely would we ever dare tune into PBS for entertainment. I started thinking about this as my son of 17 months is beginning to use single words and overall starting to communicate more with me, and consequently starting to want to repeat everything he hears. We were watching cartoons last Saturday morning and I noticed something strange, every cartoon appeared to be teaching him way more then I remember the cartoons of my time teaching me.

Sure there were a few of the ones I expected, but the vast majority had a lot of learning. Even the commercials had learning games and exercises mixed in. With the amount of information younger generations have these days it makes me wonder just how much my son is picking up. Is it crazy to think by four or five he will know at least one hundred words of Chinese (Ni Hao, Kai Lan), and one hundred words of Spanish (Dora the Explorer), at this rate I don’t think that’s too crazy an accomplishment as he’s learning all of this while having fun in his eyes.

Wouldn’t it be embarrassing if your child came up to you and spoke a sentence in Chinese, and you had to tell him to hold on while you “googled” what he was trying to say. Before I had a child I always said, “I am not letting my child watch cartoons, that stuff will just make him less likely to enjoy learning and other activities.” Now I not only love the idea, but it’s fun and exciting for me too since I get to learn as well. My dad was always breaking and building computers when I was a kid so naturally I picked up on that and made it into what I do today. I’m not sure what effect if any these educational shows will have on him career or otherwise but I think overall shows today are making great progress in spurring children’s hunger to learn , which is great as I will be trying to teach him his ABC’s in the coming months. I just hope he doesn’t expect me to wear a funny hat and dance with stuffed animals in the process.

November 11, 2009

Viva Las Vegas!

I just got back in town from Las Vegas, Nevada. That town is filled with stories and you can really love it or hate it, depending on the hour (or if you are like me whether you are arriving into McCarran or departing). I had a great trip this last go around and actually made money on the tables. However, when they say that what happens in Vegas stays in Vegas they are really talking about your money. Never forget that the house always wins. Always. Even if you win money you’ll wind up spending it on stuff out there and perpetuating your own good time. There isn’t anything wrong with this at all. In fact I plan on coming up on the short side of the stick on both the tables and on simply spending cash when I go out that way.

I think the really interesting thing that happens when you go through “the Vegas experience” is the perceived value of a dollar. You can take it for granted that all of a sudden you are transplanted into this fantasy world that is reminiscent of Pleasure Island from the story of Pinocchio and you’ll find that you have anything and everything you could want to do, eat, drink, or experience right at your fingertips. As this begins to progress the value of a dollar plummets quickly. You start overpaying for things at a whim, tipping bigger, making bolder and even just dumber bets. I did this and I can admit that I doubled down on my 11 when the dealer was showing a 10 in blackjack. It was blind luck that I hit it and won every single time. It’s a bold and stupid bet to make, but when you are playing with house money the money doesn’t matter and it’s almost as if you are trying to give it all back. My game of choice is craps because it gives you the best odds and there is a lot of action. It’s good and bad as it can all come and go in a hurry.

I have only been to Las Vegas a handful of times, but each time there is a point where even for a second you can feel invincible – that you can’t lose. Or, that even if you do lose you won’t even care. The flight home is a completely different story. I call it the hangover flight. You may be literally hung over, but no matter what, you will start to deal with all of the actions that happened on your trip and how you will need to handle them. As soon as you touch down in your own home town things slowly start to become “real” again. Your own home can even feel somewhat foreign for a while, but you’ll quickly come to the realization that you had become a completely different person for a short time.

I have come to the conclusion that there is always risk in everything that we do. Exposing yourself to the tables of Las Vegas may carry more financial risk than your morning commute to work, but in both cases there are still risks. There are also risks that we take in setting and running a business. There are countless ways that you could be putting your business at risk without the right plan in place. From an IT perspective alone, you need to consider things like redundancy, failover, security, backups, growth, and even data loss. Knowing what is going to happen next for your business may be as likely as knowing what is going to come up on the next roll of the dice. If you know this for certain you can press your luck and come up big, but if you are not prepared you could lose everything you have on the table. It is better to be prepared.

I think of SoftLayer as the house, and remember as I said before, the house always wins. The good thing about this is that you are betting with the house. Even with this you need to bet on yourself and back up your own bet. If the bulk of your business is in your data then you need to have backups. If you absolutely need to have High Availability, then look into Clusters and Load Balancing. But remember, that you are betting with the house because SoftLayer gives you the capacity to do all of it and do it all at a very affordable price compared to trying to do it yourself and also do it without long term commitments. Long term commitments bring the most uncertainty in making moves that will positively affect your business. Imagine if a casino told you that you “had” to make 12 consecutive bets regardless of how well (or poorly) you were doing?

Coming home from Las Vegas to SoftLayer has been a very good thing and makes me thankful for where I am and what I have. There aren’t the levels of uncertainty here that are automatic with other datacenters or even other business models. SoftLayer is steady and it is very easy to get what you need here while cutting out the risk that you don’t want to deal with. SoftLayer is as much of a “sure thing” as any bet you can make!

November 9, 2009

Outstanding Tech Recognition: Destroyer Droid

A long time ago in a galaxy far, far away… our hero Romeo R., SoftLayer CSA was battling an endless sea of support tickets…

Ok so maybe it’s not that dramatic, but I was recently rewarded the Destroyer Droid award from our management team! In case you didn’t see the original post here it is:

I’ve written two other blogs on TheInnerLayer but there was one in particular that I think brings me to where I am today, it was entitled “What it’s like to be a Datacenter Technician”. Today I am a shift lead for our Dallas Support Staff. In my previous blog I mentioned how DC techs wore several different hats (Hardware Engineer, Network Support, and System Admin) at any given time and we have to always be on our toes. Now being a Shift Lead of course, I still get to do all the fun stuff a Datacenter Tech gets to do, but now I have more focus on how to get all of the above working together and more efficiently.

Enter Destroyer Droid:

Getting more into the management side of things is an entirely different monster; you have all of your previous duties plus the duties of setting up the flow of work for the day. It’s given me an entirely different mindset on how SoftLayer works and what it takes to be successful. It can be quite the handful on some occasions, but I enjoy coming to work every day because of the challenge and the people I get to work with. I think whenever someone in a Shift Lead type of position receives an award it is a direct reflection of how the team as a whole is performing, and it wouldn’t be possible unless every single tech was on their game. If you’re reading this and your boss/manager does something similar to our recognition awards let me know in the comments!

Now if I could only get the guys to stop calling me the destroyer…



Subscribe to introductions