March 13, 2009

SSD. A Peek Into the Future?

Remember back in the day when Beta and VHS tapes were "it" for movies until those crazy discs called DVD's came out? Well some of you may not remember those days *rolls eyes*. Anyway, for me at least I couldn't imagine there being anything better until LCD TV's and Blu-rays became the norm. Now the question is what could possibly be better than Blu-ray? This next step could be movies written onto a chip something like a flash drive. After all we've seen flash drives go from 64MB to an astounding 64GB all packed into a small USB keychain in no time.

Your thinking, alright that's great and all, but what is SSD? It stands for Solid State Drive and could just very well be the beginning of the end for Disk Drives.

Here at SoftLayer we offer the best SSD's you can find on the market which are the Intel X25-E's (The E stands for Extreme Edition). Unlike the X25-M's which are multi-level cell (MLC) the Extreme Edition's are single-level cell (SLC). Usually you think that multi would be better than single. For space yes, but performance wise, it adds about 100MB/sec to the write speed. They both however, have identical 250MB/sec sustained read speeds.

Okay, so how can SSD's help your server?

Where SSD's scream is their I/O performance. According to benchmarks done by Tom's Hardware (TH) using IOMeter (Database Benchmarking), the X25-E's are pushing over 5,000 I/O operations per second falling to ~3000 dependant on the queue depth. Comparing this to a 15.5K Seagate Cheetah is like comparing a Cassette Tape to a CD, as the Cheetah maxes out ~500 I/O operations per second. TH did several other benchmarks including File Server, Web Server, and Work Station benchmarks, yielding similar results giving the X25-E a lead of about 10 fold over the Cheetah. Access times are also no competition at 0.1ms vs. 5.7ms. The TH review is definitely worth the read if you are looking into such an upgrade and can be found at:,2158.html

Going Green?

SSD consumes only a fraction of the power that a HDD uses. According to Intel the X25 uses only 0.06W at idle and peaks at 2.4W under load. Your typical Hard Disk Drive will use on average ~7.5W at idle and ~10W under load.

As well as using less power, since there are no moving parts that in turn means higher reliability as well as less noise generated. Less noise means us tech's won't have problems hearing you on the phone from the ringing in our ears from working in the datacenter *lol*.

Taking SSD to the extreme …

Samsung recently released a video where they strung together 24 Samsung 256GB MLC SSD's resulting in 6TB of storage and 2GB/sec of throughput. For your viewing pleasure:

Who knows what the future has in store for us, in ten years we may look at this video like we look at this one:

One thing is for certain, quieter, faster, more reliable, and more energy efficient datacenters.

March 11, 2009

Social Networking and Customer Relationships

Social networking websites are all the rage today. You are able to keep up with old friends, collegues, and now even business contacts. I believe this is truly the coolest part about these types of sites. While it is fun to constantly know what your first boyfriend's sister's cousin is up to at any given time, it is also nice to be able to make a business relationship a little more personal.

I started working in the web hosting industry back in 2003. While there were social networking websites out there, they weren't a huge thing. I certainly was not a member of one at the time. Therfore, my only way of communicating with my customers was via the regular methods: phone, chat, and email. I think I was successful at my job and did a good job of keeping up with customers. But now, I am friends with clients through everything from FaceBook to Linkedin to Twitter. Linkedin is great, because you are able to add referrals and comments on service, and really help to expand one another's business. It is also hilarious to get to know the more personal side of my clients by looking at silly pictures of them and seeing their comments on FaceBook. It definitely lightens the atmosphere and makes it even more enjoyable to work with one another when conducting business.

As you can see on SoftLayer's website, you can network with us as a company through FaceBook, Twitter, LinkedIn,, and GitHub. So please come and check us out for all of the latest updates on product offerings, improvements, and advancements that SoftLayer has to offer you! Oh, and I would be happy to be your personal friend too!

March 9, 2009

Spindle Theory

Spindle: the axis to which hard drive platters are attached.

I'm going to start this blog off with making a statement: 250 + 250 > 500. Some of you already know where this is going and you're sitting there thinking 'well, duh'. Other readers probably think I've just failed basic math. If you're in the first group, scan through and see if I bring up something you haven't thought of. If you're in the second group, this article could be useful to you.

Let us suppose you are running a medium popularity website. You have a forum, a database and you serve a bunch of video or image data. You knew that for content plus backups you'd need about 500GB. You have two hard drives, 2 x 250 GB. Content is on drive zero, and drive one is used for holding backups (you are routinely backing up, aren't you?). Being a good and noble system administrator you've kept an eye on your resource usage over time and lately you've noticed that your load average is going up.

Say you were to run a 'top' and get back something like this:


First a couple of caveats:

1) For the astute in the class... yes, I've made up the numbers above.. I don't have a distressed machine handy but the made up numbers come from what I've seen in actual cases.

2) A load of 15 may not be bad for your machine and workload. The point here is that if your load is normally 5 and lately its been more like 15 then something is going on. It is all about knowing what is normal for your particular system.

So, what is top saying? Its saying that on average you've got 14 or 15 things going on and wanting to run. You'll notice from the swap line that the machine isn't particularly hitting swap space so you're probably not having a RAM issue. Lets look closer at the CPU line.

Cpu(s): 10.3% us, 5.7% sy, 0.0% ni, 15.0% id, 80.3% wa, 0.0% hi, 0.0% si

10% user time, 5% system time.. doesn't seem so bad. Even have 15% idle time. But wait, what do we have here.. 80% wa? Who is wa and why is he hogging all the time? wa% is the percentage of time your system is spending waiting on an IO request to finish. Frequently this is time spent waiting on your hard drive to deliver data. If the processor can't work on something right now (say because it needs some data) that thing goes on the run stack. Well what you can end up with is that you have a bunch of processes on the run stack because the CPU is waiting on the hard drive to cough up some data for each one. The hard drive is doing its best but sometimes there is just too much going on.

So how do we dig down further? So glad you asked. Our next friend is going to be 'iostat'. The iostat command will show you which hard drive partitions are spending most of the time processing requests. Normally when I run it I'm not concerned with the 'right then' results or even the raw numbers... rather I'm looking for the on-going action trend so I'll run it as 'iostat 5' which tells it to re-poll every 5 seconds.


(again I've had to fudge the numbers)

So if you were running 'iostat 5' and over time you were seeing the above formation what this tells you is that disk0 (or hda, in the chart) is doing all the work. This makes sense.. previously we discussed that your setup is disk0 - content, disk1 - backups. What we know so far is that disk0 is spinning its little heart out servicing requests while disk1 is sitting back at the beach just watching and giggling. This is hardly fair.

How do you right this injustice? Well, that depends on your workloads and whether you want to (or can) go local or remote. The general idea is going to be to re-arrange the way you do your storage to spread the workload around to multiple spindles and you could see a nice gain in performance (make sure a thing and its backup is NOT on the same disk though). The exact best ideas for your situation is going to be a case-by-case thing. If you have multiple busy websites you might just start with putting the busiest one on disk1, then see how it goes. If you have only one really active site though then you have to look at how that workload is structured. Are there multiple parts to that site which could be spread between disks? The whole point of all this is that you want to eliminate bottlenecks on performance. A single hard drive can be just such a bottleneck.

If the workload is largely writes (most of the time files being created/uploaded) then you're looking at local solutions such as making better use of existing drives in the system or adding new drives to the system. Depending on what you are doing it might be possible to jump up to faster drives as well, say 15,000 RPM drives. I should include mention here of RAID0. RAID0 is striping without parity. What you have is multiple physical drives presented to the operating system as one single unit. What happens is that as a file is written, parts of it end up on different drives so you have multiple spindles going for each request. This can be wickedly fast. is also dangerous because if you lose one drive you potentially have lost the entire volume. Make no mistake, hard drives will fail and they'll find the most irritating time to do it. If you think you would want to use a RAID0 and you cannot afford the downtime when a drive fails taking the whole volume with it then you might look at a RAID10. Ten is a RAID0 that is mirrored against another RAID0. This provides some fault tolerance against a failed drive.

If the workload is mostly just reading, say you host images or videos, then you could do multiple drives in the system or the "Big Dog" of adding spindles for read-only workloads is something like our CDNLayer product where a global network of caches stores the image files and people get them from their nearby cache. This takes quite a bit of the I/O load off your system and also (subject to network routing goofiness) your visitors could be getting your content from a box that is hundreds or thousands of miles closer to them.

So, with apologies to my math teacher friend, there are times when 250 + 250 > 500 can be true. Until next time, have a good day and respect your spindles!

March 4, 2009

Web Site Optimization

There are many techniques to speed up the web page load time. This Yahoo developer network article sums most of them. If you own a web site with thousands of daily visitors, some of these tweaks will help you provide a better experience on your web site to visitors.

I’ve been on the CDNLayer development since I started working at SoftLayer. One day I wondered how much performance increase some of these techniques can bring. I chose to implement the techniques below.

  1. Using CDN
  2. Combining multiple JS and CSS files to a single JS and CSS
  3. Compressing JS and CSS
  4. Serving files from 2 different domains

I chose them because they are easy to implement and CDN has become very affordable nowadays. I copied the index page of and took 5 different steps to optimize the page. To make the page a bit larger, I added a JS and a CSS file to the index page. So the total file size was about 980 kilobytes.

  1. Step #1: “HTML + 2 JS + 2 CSS + images” served from my server
  2. Step #2: “HTML + 2 JS + 2 CSS + images” served from CDN
  3. Step #3: “HTML + 1 combined JS + 1 combined CSS + images” served from CDN
  4. Step #4: “HTML + 1 combined/compressed JS + 1 combined/compressed CSS + images” served from CDN
  5. Step #5: “HTML + 1 combined/compressed JS + 1 combined/compressed CSS” served from a single CDN host A + “images” from 2 different CDN hosts A and B

The page loads within 2 seconds in real life so I disabled both the disk and memory cache in Fire Fox to exaggerate the result. I requested each version 10 times and here is the average page load time in seconds.

  1. Step #1: 11.976
  2. Step #2: 9.602
  3. Step #3: 9.626
  4. Step #4: 9.123
  5. Step #5: 8.72

First, my test site is not on SoftLayer’s server, it is located somewhere in Pennsylvania. Second, using CDN, thus the files are served from Dallas POP, gave me a good 2 seconds decrease. Third, combining JS and CSS files did not give me any benefit. It was only 2 less trips to the server anyways and I’m using a high-speed Internet so I guessed this would not make much of difference in my case. However, if I had a dial-up, fewer trips to server, even it were only 2, will help the page load time. Fourth, gzip compression reduced the content size and it shortened page load by 0.5 seconds. It doesn’t seem like a big benefit as far as the page load time is concerned but keep in mind that the compression decreased the page size by more than 100 kilobytes. If you have a large amount of visitors, it will help you save lots of bandwidth. Finally, serving files from 2 different domains can reduce a significant amount of page load time. This is due to the limit within the browser itself. Most browsers are set to download 2 files from a domain at a time regardless of how fast your Internet connection is. So if you serve your files from multiple domains or sub domains, your visitors will be able to download more files simultaneously. If a visitor is using a high-speed connection, this trick will help the page load significantly.

I hope some of these techniques can help your sites get prepared for a large number of visitors. Web site optimization techniques not only will reduce the page load time but it will also help extend your web site’s capacity. Who knows, your web site may get Digged tomorrow!

March 2, 2009

Start Your Own Stimulus Plan 2009

In case you have been living under a rock, a stimulus plan was passed a few weeks ago here in the United States. There have been numerous blogs, news articles, videos posted on why it will and will not work. If you want to know more about it, you can get additional information from the government site You may be relieved to know I am not here to get into some big rant on the merits and flaws in the plan.

What I do want is to voice my opinion on the whole issue and what you can do about it. I think this is a very pivotal time for the entire world and sitting around waiting on the government is not in anyone’s best interest. At SoftLayer we began referring customers to server management companies a few months after we opened, via our forums. Since this has helped numerous companies listed in the forums, I think it is time to branch out and expand on this. I found a website not associated with SoftLayer called “Make a Referral Week”. I think it is best described on their website:

“As the talk of recession crowds the news and economic stimulus package debates rage in Washington DC, it’s time for small businesses to take the matter into their own hands. Therefore we hereby declare March 9-13, 2009 - Make a Referral Week.”

Make a Referral Week is an entrepreneurial approach to stimulating the small business economy one referred business at a time. The goal for the week is to generate 1000 referred leads to 1000 deserving small businesses in an effort to highlight the impact of a simple action that could blossom into millions of dollars in new business. Small business is the lifeblood and job-creating engine of the economy and merits the positive attention so often saved for corporate bailout stories.”

I want to encourage you to participate in the event, and if you stumble across this post after the event go ahead and try it out for a week. It’s a great way for you to help fellow business owners small and large and get involved in making a difference.

Click here for more information and to make a difference:

make a referral week

February 27, 2009

Different Views for the Same Event

Here at SoftLayer, we try to address customer issues as quickly as possible. Recently, I have been faced with a few more support tickets than usual, which got me thinking about an experience in my younger life. Back in the 90’s in South Korea, my buddies and I were in our early 20’s and it felt like we owned the world. We were full of energy and everything was fun. As Korea is a peninsula, the ocean was a big part of our lives. Every summer, we went to the beach together. There are many beaches as you can imagine, and they get pretty crowded during the summer.

Who wants to work or study during the hot summer days? Come on! Heading out for a day at the beach was a lot more fun.

It was one summer that we went to a nice beach (see picture). Some of us brought our girlfriends and some did not. On the way to the beach, one of my friends, Park, who was single and a decent looking young man, told us about his dream that he hoped would come true that summer. His idea was to rescue a girl from drowning and develop this coincidence into a romantic relationship. Yeah, right. The rest of us had trouble taking his dream seriously. Party’s on!

When we got to the beach, he was busy swimming right away. I suppose he should have been. His chances of being the first person to rescue a drowning woman were about as good as winning the lottery. Look at the picture. You get the idea.

We lost track of him for the most of the day and this is the story he later told us. He was chilling out in the water, on the lookout for damsels in distress. While he was dreaming about his coincidental rescue and romance, he heard a desperate woman’s voice just behind him.

“Help! I got a cramp in my leg.”

Instantly, he was in heart-pounding, breathtaking euphoria; thinking: “Wow, today is the day. Heck, yeah. I swam all day. God helps those who help themselves.”

So, off he goes. He turned himself toward the voice with his eyes strained, willing to risk anything to rescue that woman.

It immediately became apparent the girl was not quite his type at all. He muttered under his breath: “God is busy helping other people as usual”. He shouted loudly so she could hear him with no mistake.

“Sorry, I can’t swim very well!”

And he turned and began swimming back in the opposite direction.

He felt pretty bad after he got away from her; so, he looked back. Surprisingly, the lady had managed to swim to shore by herself. He had mixed emotions after swimming back to shore. I told him, “Don’t feel too bad. I think you helped her build up the mental strength and she found the inner strength to overcome the cramp.”

Clocking back to present, I’ve had quite a few support tickets over the past several days here at SoftLayer. It is not at all as bad as the girl’s situation but I’ve felt a little pressure since it isn’t normal and I’ve got other stuff to do.

Looking back on past experiences, I realize I can view the same event in two different ways, just like flipping a coin. It was a lot of pressure in the beginning to deal with the extra support issues; but, now I’ve learned so many things within such a short period of time. Not only have I learned more about our products, like CDNLayer, but I got a better idea of what customers want. So, drawing in tickets isn’t all bad; it strengthens me after all.

Hey, young single guys out there don’t just walk away from a girl in danger because she is not your type. I guess that’s my point, because I can tell you that my friend remains single to this day.

February 25, 2009

Greed is Good

I have always had a fascination with economics going back to my college days. Some of my classmates and I used to spend hours discussing economic theory and poking holes in our professors’ beliefs, particularly those who followed John Maynard Keynes (insert your own political commentary/sarcasm here). It was a fun time and formed the basis for many of my beliefs about how business and the economy should be run. I even passed some of that passion on to my daughter who has a degree in economics.

As a student of economics, I try to read articles from as many of great economists of the past and present as I can given the constraints of having to work, raise a family, etc. One of my favorites is Walter Williams, a professor at George Mason University in Virginia and frequent contributor to blogs and various network broadcasts. In one of his recent articles, he used the example of a supermarket and the miracle that 60,000 items can come together under one roof and operate smoothly without Congressional meddling (again, insert your own political commentary/sarcasm here). It got me to thinking about SoftLayer and what goes in to provisioning one server in our datacenter. Think about all the manufacturers building the component parts, assembling those parts and shipping those parts via a truck which has its own component parts, assembling etc. to be able to transport those servers. Add to that all the networking gear, software, bandwidth etc and I would be willing to bet that thousands, maybe even millions of inputs and people are required to bring one server up and keep it running in a datacenter. And the whole process runs relatively smoothly and efficiently from start to finish.

How does that happen? In his article, Williams quoted Adam Smith, the acknowledged father of modern economics who said:

"He (the businessman) generally, indeed, neither intends to promote the public interest, nor knows how much he is promoting it. ... He intends only his own security; and by directing that industry in such a manner as its produce may be of the greatest value, he intends only his own gain."…"He is in this, as in many other cases, led by an invisible hand to promote an end which was no part of his intention. ... By pursuing his own interest he frequently promotes that of the society more effectually than when he really intends to promote it." …"It is not from the benevolence of the butcher, the brewer, or the baker, that we expect our dinner, but from their regard to their own interest."

From my perspective what Smith is saying is that more often than not, by pursuing our own goals and being greedy, we end up creating more value to society than when we actually try to do something good. Those motivated by profit tend to produce more value than those in the non-profit world (i.e., academia and government). Before you attack me, with what parts of your own life are you most satisfied? As Williams asked at the end of his article, are you more satisfied with the profit-motivated Wal-Marts, ebays, and Amazons of the world or the non-profit motivated schools, postal service, social security or motor vehicle registration? Who would you rather have in charge of your datacenter, the government or SoftLayer?

While we may not say it out loud, we are all greedy. And contrary to what certain segments of today’s society will have you believe, that is not necessarily a bad thing.

February 23, 2009

Shouldn’t Service Industries Be About Service?

It’s a pretty fundamental idea that I am finding gets overlooked more and more these days. I recently ordered DirecTV for my home, and I haven’t had any real problems with it. After all, it’s just programming. It’s the same programming that we’ve had for years and take for granted all the time and just expect it to work, but thankfully it does work. If you have ever been a customer of the numerous companies that treat customers like cattle, then you may share my point of view and frustration in dealing with them.

I have spent hours on phone calls trying to sort out many different aspects of my services, but normally wind up being told that the issue is not part of my contract and that they cannot help me further. In most situations, I know that the person on the other end of the phone is simply doing their job. But, as a customer, any small effort and willingness to look at alternatives would go a long way towards eliminating a lot of the frustration. It is all too common that customer services reps work almost exclusively from a set of scripted answers as opposed to actually looking at the reason that one of their customers is calling in the first place. With the cynicism that gets built up, one begins to think if these companies care at all about their individual customers.

Ok, so enough for the doom-and-gloom about how many companies operate today and take advantage of these situations. It is time to stop focusing on the negative and start working positively on a sensible solution. One thing we can always control is our own actions, and I have vowed to never let a SoftLayer customer feel like we were unwilling to work with them in order to put their goals within reach. Think of the “golden rule” and make sure I am treating anyone I work with the same way that I wish companies I called would treat me.

It seems that in the global economy that we live and work in, you would expect many companies to strive for differentiation by not only providing a great product/service that people want, but also compliment their offerings by providing a high level of customer service that leaves the customer feeling fortunate to do business with them. If this were the case more often, contracts would not be necessary because the company would be able to count on its superior offerings and customers would not have any reason to leave based on the customer service.

In the vast world of IT and the seemingly limitless options our network infrastructure brings it can be nearly impossible to fit a standard model for everyone and their specific needs. Being flexible, understanding, and willing to work with our customers is what allows us to build better business relationships which promote more success for both our customers and us.

Simply put, the number one thing that any service industry business relies on most is its customers, and we must never forget that.

February 19, 2009

Virtualized Datacenters

It shouldn’t be any surprise to people who know SoftLayer that we follow the "Virtual Datacenter" discussions quite closely. In fact, it is awesome to see people discussing what sounds a lot like what SoftLayer already is.

The concept of Virtual Datacenter is that you have all the power of a datacenter at your command without having to worry about the details of actually running a datacenter. Chad Sakac from EMC wrote an excellent post in his personal blog about the transformation to a Virtual Datacenter.

One of the points Chad makes is the abstraction of the physical infrastructure. Quoting Chad:

"Every Layer of the physical infrastructure (CPU, Memory, Network, Storage) need to be transparent. Transparency means 'invisible'. This implies a lot, and implies that the glue in the middle, like a general purpose OS, needs to provide the "API models" for those hardware elements to be transparent. "

I latched on to this point because that is what we have been building at SoftLayer for the last few years. We realize that the abstraction of the physical infrastructure not only means that end-users don’t need to know how to manage the physical infrastructure, but that the abstraction can make more efficient use of resources (= money!).

Let’s talk about the advantages of virtualized infrastructure. Without virtualization, provisioning a web-facing server on the network would involve obtaining rack space, a server, licensing and loading an OS, finding a switch port, physically connecting a cable or three, setting up the switch port (I hope you know IOS), getting IP Addresses (hopefully you don’t have to go get more from ARIN), and adding a firewall and/or load balancer (more procurement, cabling, and configuration). Adding storage could be just as complex – also involving procurement, racking, cabling, and configuration. This doesn’t sound very efficient. In fact, it sounds a lot like creating a “circular device that is capable of rotating on its axis, facilitating movement or transportation whilst supporting a load”. It's been done before and I'll bet it’s been done better by people other than you.

Using virtualized infrastructure you should be able to perform the task with a few clicks of a mouse or a few API calls and have the functionality you need set up in a few minutes instead of days, weeks, or months. No worrying about procurement, physical constraints, or learning the specifics of network and storage devices from different vendors. All you should have to focus on is the running of your particular application. You shouldn’t have to worry about configuring servers, networking, and storage any more than you should have to worry about chillers, HVAC, generators, and UPS batteries.


February 13, 2009


Do you remember that song from Sesame Street? The lyrics were so catchy that very few people who grew up watching it have forgotten the song. You don’t even need to look up the lyrics, everyone knows them even if they’ve never watched the show.

1,2,3,4,5… 6,7,8,9,10… 11,12!

If you’re not aware, all UNIX and UNIX-like operating systems keep their time in a format known as “epoch time”. This is the number of seconds since January 1, 1970 00:00:00 GMT. Regardless of your timezone, your UNIX machine should show the same number of seconds as every other UNIX machine in the world. This clock is based off of GMT, and your local timezone settings simply interpret this epoch time based on your local timezone.

So what’s that have to do with the price of beans?

Well, today is an interesting day for the epoch timestamp. Friday, February 13 2009 at 23:31:30 GMT, the epoch timestamp will read 1234567890.

So how can you be sure that your UNIX (or windows machine) has accurate time? Well, if you have a SoftLayer server, you can simply point your ntp client to “”. This traffic then passes over the back-end private network, which has unlimited bandwidth, and you won’t consume your precious public-facing bandwidth to keep your server’s time accurate to within milliseconds. Just like every other NTP server on the internet, ours sync up constantly throughout the day with various atomic clocks around the world. You can’t get much more accurate than that, at least without having your own little chunk of the radioactive element cesium inside your computer. Incidentally, this is the same thing that makes your GPS system work. Hundreds of satellites overhead, which are basically nothing more than cesium clocks with transmitters that constantly broadcast the current time.

It’s just another one of those cool things that we do for our customers to help them get the most out of their server without having all the bare essentials stack up against their monthly bandwidth allocation.




Subscribe to cloud