funny

September 21, 2009

Hardwhere? - Part Deux: Softwhere (as in soft, fluffy clouds)

I won’t pretend to know the ins and outs of the cloud software we use (okay, maybe a little :),) but I know the gist of it as far as hardware is concerned- redundancy. Entire servers were the last piece of the puzzle needed to complete entire hardware redundancy. In my original article, Hardwhere?, (http://theinnerlayer.softlayer.com/2008/hardwhere/) I talked about using load balancers to spread the load to multiple servers (a service we already had at the time) and eluded to cloud computing.

Now cloud services are a reality.

This is a dream come true for me as the hardware manager. Hardware will always have failures and living in the cloud eliminates customer impact. Words cannot describe what it means to the customer. Never again will a downed server impact service.

Simply put, when you use a SoftLayer CloudLayer Computing Instance, your software is running on one or more servers. If one of these should fail, the load of your software is shifted to another server in the “cloud” seamlessly. We call this HA or High Availability.

If there is a sad part to all of this, it would be that I have spent considerable effort optimizing the hardware department to minimize customer downtime in the even on hardware failures. But I have a rather odd way of looking at my job. I believe the end game of any job I do is complete automation and/or elimination of the task altogether. (Can you say the opposite of job security?) I have a going joke where I say: “Until I have automated and/or proceduralized everything down to perfection with one big red button, there is still work to be done!”

Cloud computing eliminates the customer impact of hardware failures. Bam! Even though this has nothing to do with my hardware department planning, policies and procedures, I have no ego in the matter. If it solves the problem, I don’t care who did the work and was the genius behind it all, as long as it moves us forward with the best products and optimal customer satisfaction!

We have taken the worry out of hosting- no more deciding what RAID is best. No more worrying about how to keep your data available in the event of a hardware failure. CloudLayer does it for you and has all the same service options as a dedicated server and more! One more step to a big red button for the customer!

Now back to working on the DC patrol sharks (they keep eating the techs!) New project- tech redundancy!

September 18, 2009

Ninjas in the Datacenter

We tecchies are a weird bunch.  We equate everything to mythical figures and mysterious characters.  All around at SoftLayer, you can see and hear references to nerdy and mysterious things.  From Brad's incessant General Grievous-ish throat clearing, to FreeBSD's 'beastie' daemon:

Beastie
Copyright 1988 by Marshall Kirk McKusick.

Mythical figures surround us all the time.  IT guys tend to have a reputation for being a little, well, different, than the rest of the world.  Now that you're shaking your head, wondering what I'm rabbling about, allow me to introduce the one mythical figure that reigns supreme, especially here at SoftLayer.  That's right, it's the Ninja.

That's right, we've taken one of the most ridiculously awesome figures in modern mythology, and verbed it.  Not sure what verbing is?  Allow me to utilize one of my personal favorite comic strips as a visual:

Calvin
by Bill Watterson.

The ninja has a couple of meanings here at SoftLayer.  Allow me to give a few examples:

nin-ja [nin-juh]
-verb

  1. To Steal, as in a ticket that looked interesting or challenging: "Dude, you totally ninja'd that Network Question ticket from me!  I'm interested to know what you did to diagnose and fix it!"
  2. To fix an issue, against all probability that it is even fixable: "Wow, I thought that database was hosed.  He totally ninja'd that, and now it works like a charm."

The above are just two of the many examples of ninjas in our datacenter.  It's just one of the many ways we separate ourselves from the pack.  Our responsibilities are not only demanding, but unrelenting.  While we take these many responsibilities quite seriously (such as our commitment to the best support in the industry), we are always quick to lighten each other up.  As our big boss would say it:  "We are defining new standards and setting the tone for others to follow. Leading by example, pushing our luck, and having fun every step of the way."  Working at (and hosting at) SoftLayer is about kicking butt, leaving others in the dust, and relishing in every minute of it.

September 15, 2009

Managing Your Traffic in the Modern Era

Over the past 10 years, I’ve run or helped run all sizes of web sites and internet applications. I’ve seen everything from single-page brochure web sites to horizontally scaled interactive portals. And what I’ve learned is that it is all about the end-user experience.

I’m not a graphics specialist or a GUI designer. I just don’t have that in my DNA. I focus more on the technical side of things working on better ways to deliver content to the user. And in the purely technical area, the best thing to do to improve the user experience is to improve the delivery speed to the user.

There are a lot of tools out there that can be used to speed up delivery. CDN, for example, is an awesome way to get static content to an end user and is very scalable. But what about scaling out the application itself?

Traditionally, a simple Layer-4 Load Balancer has been a staple component of scalable applications. This type of Load Balancing can provide capacity during traffic peaks as well as increase availability. The application runs on several servers and the load balancer uses some simple methods (least connections, round robin, etc) to distribute the load. For a lot of applications this is sufficient to get content reliably and quickly to the end user. SoftLayer offers a relatively inexpensive load-balancing service for our customers that can provide this functionality.

There is another, more sophisticated, tool that can be used to manage internet application traffic. That is the “Application Delivery Controller” (obligatory Wikipedia link: http://en.wikipedia.org/wiki/Application_Delivery_Controller) or “Load Balancer on Steroids”. This class of traffic manager can act in Layer-7, the data layer. These devices can make decisions based on the actual content of the data packets, not just the source and destination.

And an ADC can do more than load balance. It can act as a Web Application Firewall to protect your data. It can speed up your application using SSL Offloading, Content Caching, TCP Optimization, and more. This type of device is very smart and very configurable and will help in the delivering the application to the end user.

At SoftLayer we have seen our customers achieve a lot of success with our Layer-4 Load Balancer product. But we are always looking for other tools to help our customers. We always have admired the advanced functionality in the appliance-based Application Delivery Controllers on the market. Finding a way to get this enterprise-grade technology to our customers in an affordable manner was problematic. When Citrix announced that they were going to create a version of their NetScaler product that didn’t require an appliance we were thrilled. With the announcement of the NetScaler VPX we finally thought we had found the right product that we could use to affordably provision this advanced technology on-demand to our customers.

SoftLayer is VERY excited to partner with Citrix to provide the NetScaler VPX Application Delivery Controller to our customers. Our customers can order a NetScaler VPX, and in a matter of minutes be managing the delivery of their online applications using one of the most sophisticated tools on the market. Citrix does a better job of promoting the product than I do, so here is the link to their site: http://citrix.com/English/ps2/products/product.asp?contentID=21679&ntref=hp_nav_US.

Remember, it’s all about the experience of the user at the other end of the wire. Find the right tools to manage that experience and you are most of the way there. Oh yeah, and find a good graphics designer too. That helps. So does good content.

-@nday91

September 9, 2009

Taking the Chance

I started working at the ripe’ole age of 16 and since then I’ve had 5 jobs including SoftLayer. I started off at “bullseye” which consisted of straightening merchandise shelves and onto being a cashier. For my second job, I moved on to harassing customers into purchasing leather from a well known mall leather supplier, but for some reason I was having extra-long chili cheese conies and bacon, egg and cheese toasters on my mind, so I made the move to “rollerskates” for job number three. These jobs gave me a decent income (for being 16-18 yrs old), but I knew I didn’t want to be hopping from job to job for the rest of my life… I needed to find a place to grow roots, a place that paid me what I was worth, and finally a place where growth within the company was available. So, I moved to the world of retail digital imaging (large format printing) with a local Dallas company. I started in the shipping and receiving department, slowly learning the whole production side of the company as I knew that is where I could grow. Four long years passed, roots in the company were set fairly deep, or so I thought and the opportunity was “kinda” there, but ultimately I was not happy.

The opportunity for me to work at SL came thru a friend and former colleague at the printing company, Shawna (thanks Shawna!) who left because she also saw the huge potential with SL. At the time I was so ready for something new, but I’ll have to admit, I was a bit reluctant to take the chance because of the four years I had invested, and my lack of knowledge in IT, and particularly SL, knowing that they were doing things that had never been done before (I did my homework). Needless to say I took the chance, and was hired on as an Infrastructure Engineer.

As an Infrastructure Engineer at SL responsibilities range from installing cage nuts, rails, filler panels all the way to installing Cisco switches and Ethernet cables. Basically making sure that the racks are ready to be populated with servers and sold to new or existing customers wanting to expand their business. I can only speak for myself, but there is a great sense of pride when you step back and look at all the live racks you just painstakingly set up, knowing the hard work you had invested was not only helping SL grow but knowing that I had taken a bigger step into starting a solid career.

Yes, I was hired on as an Infrastructure Engineer, but was not just limited to that position. I have been here for over 2 years already and having some of the best times of my life, the opportunities to advance are there, they look for it in their staff, they want me to succeed, knowing it only helps them to succeed as well. What’s next you ask? To be continued…

September 7, 2009

Local Phone for (Darn Near Almost) Free

I keep my ears perked for businesses that leverage Internet infrastructure – mainly because such businesses are potential customers for SoftLayer. Occasionally, I become a customer of the businesses that I hear about.

I took the plunge with one such company after loosely watching it for a year. In the summer of 2007, a friend of mine moved his home phone service to Ooma. Basically, it is local phone service with no monthly bill. Zip. Nada. $0.00 per month. To top that off, the quality of service is very high.

Now, it’s not totally free phone service because you have to have a high speed internet connection to run it. I suspect that if you are reading this, you do. By the way, I have fiber going to my house, and I have 20 Mbps download and 5 Mbps upload speed. I can get 50 down and 20 up should I ever need that much bandwidth. If I wanted local phone service from my local phone company, they would provide it through this fiber (not copper) at a price of about $45 per month plus taxes and fees. That means the monthly bill would be about $60 when it’s all said and done.

We yanked our landline when the fiber arrived 4 years ago since each family member at that point had a cell phone. Going all cellular has been pretty much fine except for a few minor hiccups. Sometimes, one of us has been unreachable at the house because of either a dead battery, phone set to silent mode, cellular network congestion, or the fact that the ringer just can’t be heard throughout the whole house, even at full volume. None of these, however, was worth an additional $60 per month to solve.

OK, back to Ooma. My friend has had it for a year with no problem. He loves it. It works perfectly with high quality. On top of that, Ooma is now sold at Costco for one-third lower than what he paid for it. You buy the device for a one-time fee up front and never have a phone bill. After three months (usually), you’ve made your money back in savings.

So a month ago, I bought it. It took 20 minutes to set up, and I’m a finance guy. If you’re a techie, I’ll bet you’re running in 10 minutes or less. It has worked flawlessly since. The sound quality is fantastic. There are more features and add-ons than I can mention here – go browse their website for more. The snarky ad video is worth the 45 seconds to watch it. In short, I highly recommend Ooma.

To keep things balanced, the ONLY advantage I see to a copper line is if there is a power outage and your broadband modem/router is down, the local phone is down. But if your home phone is cordless with a powered base unit, the copper line is down in that case too. And if the aliens from District 9 show up, the copper lines will be flooded too I’ll bet.

Ooma is just another example of how the Internet and its supporting infrastructure is not only here to stay, but to keep growing as traditional telecom infrastructure slowly dies. At SoftLayer, we’re here to make sure our innovation supports businesses that grow by leveraging Internet infrastructure.

September 4, 2009

First Grade

Some of you know that it is now “back to school” time. Those of you who don’t know are the ones who are still probably able to attend happy hours. (Our esteemed CFO may not know, but only because he’s had lots of birthdays. I’m not saying he’s addled. I’m just saying.) Nearly everyone has heard of, if not read, the book “All I Really Need to Know I Learned in Kindergarten” by Robert Fulghum. This blog is kind of a take on that, but since The Boy just started First Grade, these are things of relevance we’ve learned about the rules and such for First Grade and that can be applied to things here at the ‘Layer.

So The Boy has tickets in First Grade, and if he’s bad – he’s got to pull a ticket and put it in a jar. Sounds like, the fewer tickets, the better. Kind of like here at SoftLayer, from our standpoint you don’t want to have to have a bunch of tickets to plow through. From a client standpoint, you certainly don’t want to have a bunch of tickets, especially from abuse, or you might get your server pulled. The lesson to be learned is to do what you can to keep those tickets at a minimum.

This year in First Grade, there are a bunch of boys, including The Boy. At the parent meeting after the second day of school, the two first grade teachers were explaining that because of the more than 2:1 ratio of boys to girls, the restroom breaks could be a potential nightmare. The uniforms have belts and shorts with buttons and zippers. The teachers said it takes forever for them to go, and they wanted the parents to tell their Boy that he needs to be able to, ah, how do we say it? Well, he needs to be able to whip it out, put it away and go. No smacking other bottoms and goofing off and giggling. Here at SoftLayer, we often have crazy deadlines, for example in development, which requires us to whip out some new technology on an expeditious basis. Just like First Grade, whip it out, put it away, and go on to the next project. (Unlike First Grade, there is lots of goofing off here at SoftLayer, such as with 10,000 bouncy balls and such. Since I’m in legal, if there is any smacking of bottoms, neither I nor HR want to hear about it. Lalalalala, Lalalala…)

One of the guys wrote a blog earlier this summer pondering if the things you learned in college are applicable to the “real world.” His conclusion was yes. My blog further confirms that the things you learn in First Grade are applicable to the “real world.” Here’s hoping The Boy can go 15 days without getting his ticket pulled, because if he does he gets some Krispy Kreme “football” donuts. And if The Boy gets some “football” donuts, that means The Mommy gets some, too.

September 2, 2009

SSL Comes to SoftLayer

Those who keep a close eye on the menu options in the customer management portal will have noticed that recently there was added an option under Security where you can now order SSL certificates. For those not familiar with SSL, a certificate is used by an application to establish identity and provide encryption services. Naturally you do not have to order your SSL certificates through us. Certificates ordered other places will work just fine on your server here. Certificates ordered here will work fine elsewhere.

So why order your SSL through SoftLayer? To me, its a convenience and security thing. Ordering with us is convenient because you can place and manage the order via the portal just like you manage aspects of your account already. Management includes being able to see when your certificates are going to expire and the ability to renew them. If the certificate file itself is deleted by accident you can get a copy of it e-mailed via the portal. From a security point of view you already have a billing arrangement with us so why give your credit card information to another party?

I can see someone thinking "But is that safe.. what if I leave SoftLayer?" Yes, it is safe. The only information you have to provide to us in doing the ordering is the Certificate Signing Request and some billing verification. Both of these are things that would be provided to any SSL vendor. The private key, which is the core of SSL security, is not kept or handled by SoftLayer. The private key is generated and remains with your administration staff on your server.

Let us chat about the private key for a moment. The private key is meant to be known only by the server applications to which it is assigned on your server. If it is lost, corrupted, deleted, whatever it will require a new certificate. What this all means is that you should only allow people you really trust access to the private key and above all you must keep a good, safe backup of the file. SoftLayer support can perform quite a bit of server voodoo but recreating a lost private key isn't an option.

I'd invite anyone with a bit of time to experiment with the SSL functionality we offer. You might find something useful for your business.

August 26, 2009

Cool Tool: nslookup

If you've been around the Internet awhile you've probably heard of the Domain Name Service. DNS is what takes www.domain.com and turns it into the 1.2.3.4 IP address which your application actually uses to find the server hosting www.domain.com.

Fascinating, Michael, why do I care? Well if you ask that question you've never had DNS fail on you.

When name resolution goes on the blink one of the tools that support uses to see what is going is the command-line utility nslookup. In its most basic form nslookup is going to do an A record query for the string you supply as an argument and it'll send that query to your operating system's configured resolvers.


C:\>nslookup www.softlayer.com
Server: mydns.local
Address: 192.168.0.1

 


Non-authoritative answer:
Name: www.softlayer.com
Address: 66.228.118.51


What is the utility telling us? First off, it asked a resolver at 192.168.0.1 for the information. Non-authoritative answer means that the server which returned the answer (192.168.0.1) is not the nameserver which controls softlayer.com. It then gives the IP address or addresses which were found.



C:\>nslookup -q=mx softlayer.com ns1.softlayer.com
Server: ns1.softlayer.com
Address: 67.228.254.4


softlayer.com MX preference = 20, mail exchanger = mx02.softlayer.com
softlayer.com MX preference = 30, mail exchanger = mx03.softlayer.com
softlayer.com MX preference = 10, mail exchanger = mx01.softlayer.com
softlayer.com nameserver = ns2.softlayer.net
softlayer.com nameserver = ns1.softlayer.net

 


This is a slightly different query. Rather than asking my local resolver to do an A record query for www.softlayer.com I've sent an MX (mail exchanger) query for softlayer.com directly to the nameserver ns1.softlayer.com. Notice that the response does not have the non-authoritative tag. The server ns1.softlayer.com is one of the nameservers which is configured to respond with a definite answer to a question rather than just saying "well, this other guy said...".

One thing that both of these queries fail to do is show the TTL for the answer they give. Time to Live (TTL) is what generally controls how long a resolver will keep an answer in cache. While the TTL is valid the resolver will use that answer. Once the TTL expires, the resolver goes looking for a fresh answer. This is great for performance but it does have a dark side to it: because of TTL, changes to DNS records are not seen instantly by all clients. If ClientA hits your website often his resolver is going to have the query result cached (say www.domain.com -> 1.2.3.4). You change the record to www.domain.com -> 5.6.7.8 but ClientA's resolver is going to continue to respond with 1.2.3.4 until the TTL runs out. If ClientA controls their resolver they can flush its cache. Generally though it is controlled by their ISP and you just have to wait.

To see the TTL for an answer you can use the nslookup form below:



C:\>nslookup
Default Server: mydns.local
Address: 192.168.5.1

 


> set debug
> www.softlayer.com.
Server: mydns.local
Address: 192.168.5.1


------------
Got answer:
HEADER:
opcode = QUERY, id = 2, rcode = NOERROR
header flags: response, want recursion, recursion avail.
questions = 1, answers = 1, authority records = 2, additional = 0


QUESTIONS:
www.softlayer.com, type = A, class = IN
ANSWERS:
-> www.softlayer.com
internet address = 66.228.118.51
ttl = 86400 (1 day)
AUTHORITY RECORDS:
-> softlayer.com
nameserver = ns1.softlayer.net
ttl = 86400 (1 day)
-> softlayer.com
nameserver = ns2.softlayer.net
ttl = 86400 (1 day)


------------
Non-authoritative answer:
Name: www.softlayer.com
Address: 66.228.118.51


The key to this spew is 'set debug' which causes nslookup to display additional information about the response, including the TTL value of the answer. You'll notice that the TTL in the ANSWERS section is 86400 seconds, which is the number of seconds in one day. This is a common TTL value. If I run the query again though, I have the following answers section:



ANSWERS:
-> www.softlayer.com
internet address = 66.228.118.51
ttl = 85802 (23 hours 50 mins 2

 


Notice how the TTL is counting down. The resolver is going to continue responding with the answer 66.228.118.51 until that TTL hits zero. At zero, the resolver will go looking for a new answer. What this means for you as a domain operator is that if you know you're going to be changing a record you should adjust down the TTL for that record a couple of days in advance. For example when some friends and I moved our colo server from one provider to another we dropped the TTLs for our DNS records down to 30 minutes two days prior to the move. Once the move was complete we were able to put them back to prior values.

 

If you spend any time at all messing with DNS you should play around with nslookup.

If you're on a Unix system take a look at the command 'dig' as well.

Happy resolving.

August 18, 2009

Backups Are Not the Whole Story

Last night while making my regular backup for my World of Warcraft configuration, I thought about the blog and I didn't remember seeing an article that went into more detail than "backups are good" about backing up and restoring data.

If you've been around the InnerLayer for a while you will have noticed that backing up of data comes up periodically.  This happens because we frequently see customers whose world is turned upside down due to a mistyped command wiping out their data.  If you just thought "that won't happen to me... I'm careful at a prompt"... well, how about a cracker getting in via an IIS zero day exploit?  Kernel bug corrupting the filesystem?  Hard drive failure?  Data loss will happen to you, whatever the cause.

Data that is not backed up is data that isn't viewed as important by the server administrator.  As the title of this blog mentioned, backing up isn't the end of the server administrator's responsibility.  Consider the following points.

  • Is the backup in a safe location?  Backing up to the same hard drive which houses the live data is not a good practice.
  • Is the backup valid?  Did the commands to create it all run properly?  Did they get all the information you need?  Do you have enough copies?
  • Can your backup restore a single file or directory?  Do you know how to restore it?  Simply put, a restore is getting data from a backup back into a working state on a system.

Backup Safety
At a minimum backups should be stored on a separate hard drive from the data which the backup is protecting.  Better would be a local copy of the backup on the machine in use and having a copy of the backup off the machine, perhaps in eVault, on a NAS which is _NOT_ always mounted, even on another server.  Why?  The local backup gives you quick access to the content while the off-machine copies give you the safety that if one of your employees does a secure wipe on the machine in question you haven't lost the data and the backup.

Validity
A backup is valid if it gets all the data you need to bring your workload back online in the event of a failure.  This could be web pages, database data, config files (frequently forgotten) and notes on how things work together.  Information systems get complicated and if you've got a Notepad file somewhere listing how Tab A goes into Slot B, that should be in your backups.  Yes, you know how it works... great, you get hit by a bus, does your co-admin know how that system is put together?  Don't forget dependencies.  A forum website is pretty worthless if it is backed up but the database to which it looks is not.  For me another mark of a valid backup is one which has some history.  Do not backup today and delete yesterday.  Leave a week or more of backups available.  People don't always notice immediately that something has broken.

Restores
A good way to test a restore is get a 2nd server for a month configured the same as your primary then take the backup from the primary and restore it onto the secondary.  See what happens.  Maybe it will go great.  Probably you will run into issues.  Forget about a small operating system tweak made some morning at 4am?  How about time?  How long does it take to go from a clean OS install to a working system?  If this time is too long, you might have too much going on one server and need to split up your workload among a few servers.  As with everything else in maintaining a server, practicing your restores is not a one-time thing.  Schedule yourself a couple of days once a quarter to do a disaster simulation.

For those who might be looking at this and saying "That is a lot of work".  Yes, it is.  It is part of running a server.  I do this myself on a regular basis for a small server hosting e-mail and web data for some friends.  I have a local "configbackup" directory on the server which has the mail configs, the server configs, the nameserver configs and the database data.  In my case, I've told my users straight up that their home directories are their own responsibility.  Maybe you can do that, maybe not.  Weekly that configback data is copied to a file server here at my apartment.  The fileserver itself is backed up periodically to USB drive which is kept at a friend's house.

Categories: 
August 10, 2009

Backups: It’s Good to Have Them!

Man, was this a weekend for me and backups! The first one I needed was for my second advance free fall course at skydive101.com on Friday (My Saturday). I jumped out of the plane a little early and one of my two instructors was not ready and did not jump on my call. Thank god I had a backup. Once my shoot deploys, being this is only my second jump out of a perfectly good airplane, I look up to check my slider and make sure my parachute is deployed and everything is correct I did not see my brake handles, I said some curse words I won’t put in this blog, and then was like it is OK I have a backup. I was still at 4000 ft at this time, still having another 1500 feet to decide if I want to release my main and pull the backup, luckily I found my brake handles pulled down twice and they worked and everything was OK.

So I get home after a day of skydiving and having fun in the sun (it is rather rare in Seattle, though not this summer) and I notice my trusty old Windows XP terminal has multiple errors on it. I do the first thing I always do with my personal Microsoft machines and reboot it. Ouch no operating system found, bad hard drive! Thank god I ghosted that machine 2 weeks ago as I figured the old IDE drives had been on their last legs. Saturday morning comes around, which is a big day for me, as I am hosting a party that night at a local night club. I notice I have a few (8900) email messages on my blackberry, so I decided it is about time to delete some. I tell it to delete, look back at it and 4 minutes later it says APP ERR. PLEASE RESTART. Needless to say, you guessed it. It wouldn’t boot back up and I had to force an OS onto it, and restore from my backup of a month ago, which reminded me that I need to start backup my blackberry more often.

So the moral of the story is, it is always good to have a backup, and we have plenty of backup options for you, so if you don’t have one, I would suggest contacting SLales. I would also suggest everyone trying to skydive at least once in their lifetime.

Categories: 

Pages

Subscribe to funny