Posts Tagged ‘hardware’

Quite often my friends who are not really that internet savvy ask me what I do at work, I think back to the time in the first grade when my teacher Mrs. Hyde told me: “ Bill you’re going to be a great problem solver when you get older, your problem solving skills are already at a fourth grade level.” Now you’re probably reading this wondering how problem solving problems in the first grade have anything to do with my job. It is, as she told me, all about how you think. She told me I was an outside the box thinker.

My co-workers and I deal with a network of 20,000+ servers, and 5500+ customers, in over 110 different countries, and support over 15 different operating systems. That leads to an almost infinite combination of language, hardware, and software options. When our customers submit an issue for us to work on, it is always different than the time before – whether that is a ticket from the same customer or a ticket on a similar topic. We have a very diverse range of customers using our servers for a number of things, so not every server in here is doing the same thing. In order to be good at supporting our customers, SoftLayer’s management, in my opinion, has hired some of the best problem solvers around the world to address all of our customer issues. So that is what I am: I am a problem solver! Otherwise known as a Customer Systems Administrator. We’re required to know a broad range of technologies and have the passion to learn the new ones as they come along. I think that is why I chose to work in the field that I work in, it is always changing. I tried moving over to telecommunications engineering a few years ago, but got bored with is as it was the same issues day in and day out on the equipment. Working here at SoftLayer is wonderful as there is never a dull moment.

The techs that build the servers here at SoftLayer are known as Server Build Engineers or SBE’s. These guys are on the front line of Operations. They are responsible for building out customer server orders, maintenances, fixing cranky provisions, and many other hardware related tasks.

One might think that a hardware tech is a simple job. Well, not the SBE position at SoftLayer. Not only are they responsible for time sensitive hardware builds and provision troubleshooting, but they work directly with all the other departments. We don’t have bazillion hardware techs like other companies might. We train ours up to be one man hardware machines.

Sometimes a provision might have a weird error that needs to be escalated to development. The SBE will work directly with the dev team to resolve the issue.

Sales might have questions about some hardware they are trying to sell. SBE’s answer the call.

SBE’s even jump in to help CSA’s (Customer Service Administrators) when the ticket load or phones get hectic.

SBE’s do numerous projects, too. From helping with large scale hardware compatibility testing to troubleshooting hardware, they are the jack of all trades at SoftLayer.

We have a pseudo paramilitary way of doing things in the hardware department. It’s all in fun, but we get down and dirty and have a “can do” and “yes sir” attitude. We pride ourselves in being able to tackle any problem. If we are asked to do it, we do it, regardless of whether or not it’s our job, we are too busy, or whatever the obstacle.

Be all you can be? Join the army. Be more than you expect you can be? Join the SoftLayer Hardware Team.

The few, the badass, the SBE’s!

When you think about all the things that have to go right all the time where all the time is millions of times per second for a user to get your content it can be a little… daunting. The software, the network, the hardware all have to work for this bit of magic we call the Internet to actually occur.

There are points of failure all over the place. Take a server for example: hard drives can fail, power supplies can fail, the OS could fail. The people running servers can fail.. maybe you try something new and it has unforeseen consequences. This is simply the way of things.

Mitigation comes in many forms. If your content is mostly images you could use something like a content delivery network to move your content into the “cloud” so that failure in one area might not take out everything. On the server itself you can do things like redundant power supplies and RAID arrays. Proper testing and staging of changes can help minimize the occurrence of software bugs and configuration errors impacting your production setup.

Even if nothing fails there will come a time when you have to shut down a service or reboot an entire server. Patches can’t always update files that are in use, for example. One way to work around this problem is to have multiple servers working together in a server cluster. Clustering can be done in various ways, using Unix machines, Windows machines and even a combination of operating systems.

Since I’ve recently setup a Windows 2008 cluster that is we’re going to discuss. First we need to discuss some terms. A node is a member of a cluster. Nodes are used to host resources, which are things that a cluster provides. When a node in a cluster fails another node takes over the job of offering that resource to the network. This can be done because resources (files, IPs, etc) are stored on the network using shared storage, which is typically a set of SAN drives to which multiple machines can connect.

Windows clusters come in a couple of conceptual forms. Active/Passive clusters have the resources hosted on one node and have another node just sitting idle waiting for the first to fail. Active/Active clusters on the other hand host some resources on each node. This puts each node to work. The key with clusters is that you need to size the nodes such that your workloads can still function even if there is node failure.

Ok, so you have multiple machines, a SAN between them, some IPs and something you wish to serve up in a highly available manner. How does this work? Once you create the cluster you then go about defining resources. In the case of the cluster I set up my resource was a file share. I wanted these files to be available on the network even if I had to reboot one of the servers. The resource was actually combination of an IP address that could be answered by either machine and the iSCSI drive mount which contained the actual files.

Once the resource was established it was hosted on NodeA. When I rebooted NodeA though the resource was automatically failed over to NodeB so that the total interruption in service was only a couple of seconds. NodeB took possession of the IP address and the iSCSI mount automatically once it determined that NodeA had gone away.

File serving is a really basic example but you can clustering with much more complicated things like the Microsoft Exchange e-mail server, Internet Information Server, Virtual Machines and even network services like DHCP/DNS/WINs.

Clusters are not the end of service failures. The shared storage can fail, the network can fail, the software configuration or the humans could fail. With a proper technical staff implementing and maintaining them, however, clusters can be a useful tool in the quest for high availability.

I love working at SoftLayer. I get to play with the newest hardware before anyone else. Intel, Adaptec, Supermicro… The list goes on. If they are going to release something new, we get to play with it first. I also like progression. Speed, size, performance, reliability; I like new products and technologies that make big jumps in these areas. I am always looking to push components and complete systems to the limits.

But alas, Thomas Norris stole my thunder! Check out his article “SSD: A Peek into the Future” for the complete skinny on the SSD’s we use. I seem to be a bit to concise for a nice long blog anyways. But not to worry, I’ve got some nifty numbers that will blow the jam out of your toes!

Solid State Drives (SSD) represent a large jump in drive performance. Not to mention smaller physical size, lower power consumption, and lower heat emissions. The majority of drive activity is random read/write. SSD drives have drastically improved in this area compared to mechanical drives. This results in a drastic overall performance increase for SSD drives.

This is a comparison of the Intel 32GB X25-E Extreme drive vs. other drives we carry. Note the massive jump in the random read/write speed of the SSD drive.

No more waiting on physical R/W heads to move around. How archaic!

Please note that no performance utility should be used to definitively judge a component or system. In the end, only real time usage is the final judge. But performance tests can give you a good idea of how a component or system compares to others.

Single drive performance increases directly translate into big improvements for RAID configurations as well. I have compared two of our fastest SATA and SAS four drive RAID 10 setups to a four drive SSD RAID 10 using an Adaptec 5405 Controller.

The Adaptec 5405 RAID controller certainly plays a part in the performance increase, on top on the simple speed doubling due to 2 drives being read simultaneously. (See my future blog on the basics or RAID levels, or check Wikipedia) .

Propeller heads read on:

The numbers indicate a multiplied increase if you take the base drive speed (Cheetah – 11.7mbps / X25-E – 64.8mbps) and double it (the theoretical increase a RAID 10 would give): 23.4mbps and 129.6mbps respectively. Actually performance tests show 27.3mbps and 208.1mbps. That means the Cheetahs are getting a 15% performance boost on random read/write and the X25-E a whopping 37% due to the RAID card. Hooray for math!

Once again, this is all performance tests and a bit of math speculation. The only real measure of performance, IMO, is how it performs the job you need it to do.

A couple days ago, I took my daughter to her favorite store. We picked up a fair amount and on the way to the car she asked a simple question, or so I thought. “Why did they only fill these bags half way”. Confused I looked at the bags and realized she was holding a bag which had a large stuffed bear in it and was looking at a bag less than half full of canned food.

Being the person I am, rather than attempt to explain this to her I wanted to let her try and figure it out for herself so she would understand it better. When we got home, I filled the rest of the bag with cans and had her try and pick it up, as I expected the bag broke in her hands. I explained to her that the cans were much heavier then the bag. She still doesn’t quite understand the concept that the bag has 2 limits, size and weight but she is starting to understand this concept.

I thought about this story this morning when I started working on a project of determining how many containers a Virtuozzo server could handle based on its system requirements. Just like the bag, a Virtuozzo system has multiple limitations that need to be observed, the size of the containers as well as their “weight”. In this situation “weight” would be the drain on overall system resources. When attempting to determine how many containers a system can handle, you need to take into account not only how many will fit size wise, but also how much of the overall system resources each container will require.

It turns out this question is much easier to ask then to answer. You can take a small server such as a dual core with 4GB of RAM and put 20 or even 30 containers onto the server and have it run flawlessly when those containers are small and do not require much in the way of system recourses. At the same time however I can take a quad proc quad core with 64GB of RAM and grind it to a halt with 1 or 2 containers.

At the end of the day, I have found that you can make just about anything work, but before you attempt to determine what hardware you will need to run a Virtuozzo server, it’s a good idea to have an estimate of what you expect the containers to be doing. What could be worse than spending hours configuring a server and getting it online only to watch it grind to a halt because there are just too many containers completely saturating your system resources?

-Mathew

‘Twas the month of Halloween, and all thru the halls,
Tech support and sales took all of the calls.
The servers were sitting safe in their racks,
Knowing that Hardware had their backs.

The developers were all snug at their desks,
Struggling with code that was behaving like pesks.
The workers were all decked out in SL black,
Just daring competitors to give them some flack.

When all of a sudden, there was a great THUD!
And everyone thought, “Uh-oh, we landed a dud.”
But alas, it was only another attorney,
To help and to aid in Softlayer’s journey.

“Now Trademark! And Patents!
And Copyrights, too!
Don’t worry, I’ll let you know
Just what to do!
Let’s litigate and obfuscate,
And watch Jonesie count beans!
Just kidding! I’m kidding-that’s not what I mean!”

With the economy diving, sputtering and tanking
She’ll help figure how to give competitors a spanking.
With caffeine in the morning, she’s happy as can be,
Ask her about “Dance, Dance Revolution” on the Wii.

As with all good things, this poem must end.
“Thank God!” you say, “It’s setting a bad trend.
So off to my contracts I will git.
But I’ll leave you with this last little bit:

“Happy hosting to all and to all a good day!”

-Suzy

IT is a very fast paced industry where you always need to fight just to keep up. Some companies go that extra mile in an attempt to stay ahead of the curve. In my time here at SoftLayer, I can say I have never once been bored. There is always something to keep your interest, some new innovation that needs to be looked at. Three of the aspects of this that have a drastic effect on the industry as a whole are the servers that we can provide, services to use on those servers and finally support for both.

We are always looking to provide the best possible hardware to our customers. Every time we evaluate a new piece of hardware, we run it through the gauntlet. “Will this hardware work with our system, what kind of driver support does it have, and most important is it going to be an improvement for customers?” Once it’s decided that a new piece of hardware would be an improvement the real testing begins. Everything is thrown at the hardware to ensure it meet our high standards. The latest and greatest isn’t rushed into production; we want to be sure that it really does perform to our standards in the real world and not just on paper.

Having all this hardware is great, but you need to have some innovative services for the customer to go along with it. SoftLayer really has shined in this area. There are so many different services available, there is no way I could even start to list them all. The portal alone offers more server control than anyone else in the industry. Our provisioning system can provide a server in a matter of hours and not days and with tools such as StorageLayer, RescueLayer and the Portal you can perform complete disaster recovery remotely! You can even reboot your server remotely via a cell phone while you’re on vacation without having to open a single ticket or talking to a single person.

When you have all these great services, you need to back them up with great support. Here at SoftLayer we have just that. On top of our ever growing KnowledgeLayer, we have fast and efficient ticket and phone support. Within minutes of opening a ticket, a technician has already read it and is already looking into your issue. We don’t have those pesky call queues you sit in for hours just to get a human voice, you call support and the phone is immediately picked up by a skilled technician ready to address your every issue. Unlike many places I have worked, the NOC is not a dead quiet room with people staring at computers screens. The NOC is constantly active. Our support team is just that, a team. They are constantly working together to find solutions to any number of problems.

I can’t even express how much of an honor it is to work at SoftLayer. Every day you come into the office wondering what great innovation we have next for our customers. At the end of the day, I can go home knowing that we are providing the best hardware, the best services and the best support to our customers.

-Mathew

As I turn on to the main road after leaving my Kumdo dojang (Korean fencing school), I glance at the rear view mirror down the street, in the direction of SoftLayer’s new east coast datacenter. The strangely cool, red light from the setting sun fills the mirror and signals the end of this long, hot day. My mind briefly escapes the fading heat by recalling the cool temperature and humidity regulated environs within the datacenter.

Ever wonder how to keep thousands of servers cool? In a word: CRAC – Computer Room Air Conditioning. These giants sit throughout the datacenter pumping cool air up through ventilated floors. The cool air blows up in front of the server racks, gets sucked in through the front of the servers, over the drives, past the CPU heat sinks and RAM, then out the back of the server. The warm air exits, rises, and returns to the CRACs where the humidity and temperature are adjusted, and the cycle continues. Just like you learned in science class.

So it must be a serene, sterile environment – like those IBM commercials? That would be nice, but the reality is : computers need fans. One or two fans wouldn’t bother anyone when they kick in on your gaming pc, but multiply 4 or 5 fans (do you like RAID arrays? You get extra fans!) by one thousand, or more and the decibels add up. Solid state hard drives – when they become available – might help with the noise (and also with power consumption), but it is mostly from the server fans. Liquid cooling works, but I think most people would prefer not to have fluid of any sort circulating over their motherboard. Zane (resident Linux guru) extols the benefits of passive cooling. Whatever cooling solutions arise in the future, you can be sure SoftLayer will be leading in technology implementation.

My attention returns to the road ahead and the pale blue of the evening sky. I hope to get a few hours of shut-eye before returning for my shift. Because SoftLayer doesn’t sleep. Always awake, cool and dry.

-Philip

More RAM. DDR2 must be going out of style, because Microcenter is selling a gig for $12.99. This time I don’t make it in time before they run out, and I settle for sour grapes: my home pc can’t use all the addressable memory, anyway. 4 Gigabytes. The maximum addressable memory for a 32-bit motherboard / OS. It used to be such a big deal to me- maxing out the 4 slots on my Dell, but not anymore. Why? Because now I work for SoftLayer. When you work with motherboards with up to 32 slots, 1 or 2 gig each; 4 measly slots just seems sad. I start nosing around for a video card that will fit the last expansion slot on my pc. No luck. I end up going home empty handed from Microcenter (outrageous!) and ordering the pci-e x1 video card from Newegg.

So, the hardware that customers can order at SoftLayer is impressive enough to jade the geek tech-lust of any home technician. And everything fits so nice and clean. Working on SoftLayer servers has really spoiled me for home pc’s. Open up the case on your home pc and what do you see? Fabulous shiny bits? No. Cables. Cables in the computer. Cables behind the computer. Cables everywhere! You get the nifty zips and loops from Radio Shack, spend 2 hours zipping and looping, and as you proudly call your wife over to take a look she says, “Can’t it all be wireless?”

The truth is neat cables take time, and SoftLayer engineers spend every spare minute making neat, organized, color-coordinated cables running to the servers. Cat 5e. Fiber. Special cables. Cables we can’t talk about. All very neat and aesthetic. If Mr. Crosby ever takes you for a walk through the dc (datacenter), it looks effortless and lovely. But it took hours. Hundreds of hours. Just on cabling (I think SoftLayer might have stock in zip ties). You can be sure your SoftLayer server is not lost in a sea of Cat5 and power cables. It has been gracefully bound to its slot, the formidable innards pumping away at your command, your data streaming straight and true from switch to switch into the Internet beyond…

-Philip

Dawn was the best friend I’ve ever had, except for my little sister. Just yesterday I got home only to find out that Dawn had died silently in the night. No amount of resuscitation could bring her back. Needless to say, I was quite sad.

Dawn was my computer.*

The funny part of it all was just how much of my time involves a computer. I watch TV and Movies on my computer, I play games on my computer, I do my banking on my computer, I pay all my bills on my computer, I schedule my non-computer time on my computer, I use my computer as a jukebox.

In other words, I was completely lost. What made it worse, however, was that I had had yesterday scheduled to pay my bills. But where was my list of bills?

If you guessed “Dawn had all your bills”, then you are right.

What about paper bills? I’ve got the Internet and a computer! So, in most cases I’ve canceled paper bills. All paper bills I get are shredded forthwith. So I had no paper backup of bills.

Well, I made do. I kicked my roommate off his computer (a technique involving making annoying noises while he tries to concentrate playing Call of Duty 4) and used it to pay what bills I could remember. I kept track of the bills I was paying by entering them into a Google Document.

That’s when it hit me! Why wasn’t my bill spreadsheet on Google Documents? Along with my bill list? Along with all the other documents I work on every day? Cloud Computing For The Win! As soon as I get my next computer up and running (and I figure out a new naming algorithm) I’m going to put all my vital files on Google Docs. This ties in well with Justin Scott’s post; the key to not having your data disappear during a disaster is to have a backup copy. You want backups out there, far away from your potential point of failure. (I did have backups… but they’re all on CDs that I didn’t want to have to sort through to find just one file. And had the disaster been, say, a flood, I would have had no backups.)

Google Docs is a great example of Cloud Computing: Putting both the program and the file being worked on “in the cloud.” Having built internal applications for a few people, I would make the same recommendation: Since many business apps are moving to PHP anyway (thanks for the reminder, Daniel!), you might as well move the application AND the data out of the building and onto a secure server. And as Mr. Scott** mentioned, SoftLayer ALREADY has geographic diversity as well as a private network that will allow you to link your application and data servers together in real time through all datacenters… for free. Along with the added bonus of being able to access your application from any computer… should yours meet up with Misty, May, and Dawn at the Great Datacenter in the Sky.

-Shawn

* I had a system of naming my computers after the female protagonists from the Pokemon series. Dawn, however, is the last of that series…

** I’ve decided that since Justin is an Engineer, calling him Mr. Scott is funny.