Posts Tagged 'Installation'

February 8, 2013

Data Center Power-Up: Installing a 2-Megawatt Generator

When I was a kid, my living room often served as a "job site" where I managed a fleet of construction vehicles. Scaled-down versions of cranes, dump trucks, bulldozers and tractor-trailers littered the floor, and I oversaw the construction (and subsequent destruction) of some pretty monumental projects. Fast-forward a few years (or decades), and not much has changed except that the "heavy machinery" has gotten a lot heavier, and I'm a lot less inclined to "destruct." As SoftLayer's vice president of facilities, part of my job is to coordinate the early logistics of our data center expansions, and as it turns out, that responsibility often involves overseeing some of the big rigs that my parents tripped over in my youth.

The video below documents the installation of a new Cummins two-megawatt diesel generator for a pod in our DAL05 data center. You see the crane prepare for the work by installing counter-balance weights, and work starts with the team placing a utility transformer on its pad outside our generator yard. A truck pulls up with the generator base in tow, and you watch the base get positioned and lowered into place. The base looks so large because it also serves as the generator's 4,000 gallon "belly" fuel tank. After the base is installed, the generator is trucked in, and it is delicately picked up, moved, lined up and lowered onto its base. The last step you see is the generator housing being installed over the generator to protect it from the elements. At this point, the actual "installation" is far from over — we need to hook everything up and test it — but those steps don't involve the nostalgia-inducing heavy machinery you probably came to this post to see:

When we talk about the "megawatt" capacity of a generator, we're talking about the bandwidth of power available for use when the generator is operating at full capacity. One megawatt is one million watts, so a two-megawatts generator could power 20,000 100-watt light bulbs at the same time. This power can be sustained for as long as the generator has fuel, and we have service level agreements to keep us at the front of the line to get more fuel when we need it. Here are a few other interesting use-cases that could be powered by a two-megawatt generator:

  • 1,000 Average Homes During Mild Weather
  • 400 Homes During Extreme Weather
  • 20 Fast Food Restaurants
  • 3 Large Retail Stores
  • 2.5 Grocery Stores
  • A SoftLayer Data Center Pod Full of Servers (Most Important Example!)

Every SoftLayer facility has an n+1 power architecture. If we need three generators to provide power for three data center pods in one location, we'll install four. This additional capacity allows us to balance the load on generators when they're in use, and we can take individual generators offline for maintenance without jeopardizing our ability to support the power load for all of the facility's data center pods.

Those of you who are in the fondly remember Tonka trucks and CAT crane toys are the true target audience for this post, but even if you weren't big into construction toys when you were growing up, you'll probably still appreciate the work we put into safeguarding our facilities from a power perspective. You don't often see the "outside the data center" work that goes into putting a new SoftLayer data center pod online, so I thought it'd give you a glimpse. Are there an topics from an operations or facilities perspectives that you also want to see?

-Robert

April 9, 2012

Scaling SoftLayer

SoftLayer is in the business of helping businesses scale. You need 1,000 cloud computing instances? We'll make sure our system can get them online in 10 minutes. You need to spin up some beefy dedicated servers loaded with dual 8-core Intel Xeon E5-2670 processors and high-capacity SSDs for a new application's I/O-intensive database? We'll get it online anywhere in the world in under four hours. Everywhere you look, you'll see examples of how we help our customers scale, but what you don't hear much about is how our operations team scales our infrastructure to ensure we can accommodate all of our customers' growth.

When we launch a new data center, there's usually a lot of fanfare. When AMS01 and SNG01 came online, we talked about the thousands of servers that are online and ready. We meet huge demand for servers on a daily basis, and that presents us with a challenge: What happens when the inventory of available servers starts dwindling?

Truck Day.

Truck Day not limited to a single day of the year (or even a single day in a given month) ... It's what we call any date our operations team sets for delivery and installation of new hardware. We communicate to all of our teams about the next Truck Day in each location so SLayers from every department can join the operations team in unboxing and preparing servers/racks for installation. The operations team gets more hands to speed up the unloading process, and every employee has an opportunity to get first-hand experience in how our data centers operate.

If you want a refresher course about what happens on a Truck Day, you can reference Sam Fleitman's "Truck Day Operations" blog, and if you want a peek into what it looks like, you can watch Truck Day at SR02.DAL05. I don't mean to make this post all about Truck Day, but Truck Day is instrumental in demonstrating the way SoftLayer scales our own infrastructure.

Let's say we install 1,000 servers to officially launch a new pod. Because each pod has slots for 5,000 servers, we have space/capacity for 3,000-4,000 more servers in the server room, so as soon as more server hardware becomes available, we'll order it and start preparing for our next Truck Day to supplement the pod's inventory. You'd be surprised how quickly 1,000 servers can be ordered, and because it's not very easy to overnight a pallet of servers, we have to take into account lead time and shipping speeds ... To accommodate our customers' growth, we have to stay one step ahead in our own growth.

This morning in a meeting, I saw a pretty phenomenal bullet that got me thinking about this topic:

Truck Day — 4/3 (All Sites): 2,673 Servers

In nine different data center facilities around the world, more than 2,500 servers were delivered, unboxed, racked and brought online. Last week. In one day.

Now I know the operations team wasn't looking for any kind of recognition ... They were just reporting that everything went as planned. Given the fact that an accomplishment like that is "just another day at SoftLayer" for those guys, they definitely deserve recognition for the amazing work they do. We host some of the most popular platforms, games and applications on the Internet, and the DC-Ops team plays a huge role in scaling SoftLayer so our customers can scale themselves.

-@gkdog

April 6, 2012

Of Cage Nuts and Customer Service

Sometimes it's the little hardships and annoyances that really mold you. How do you react? Do you manage to work through them, or do you let them eat away at you to the point that you're more paralyzed by them than you are a bigger problem?

As a new hire, I was required to take part in a Truck Day — an experience that helps everyone in the company understand (at a base level) what is involved with the actual products and services we sell. If you've ever had the fortune of working on one, there are certain activities that can leave you feeling weary. For me, that weariness-inducing activity was working with cage nuts.

For those of you unfamiliar with cage nuts, they're small pieces of metal that accommodate screw-in server rails on a rack meant for slide-in server rails. Installing them is one of the most frustrating things ever ... They have two little clips that fit inside the rack, and you have to bend them to get them in. Here's a great illustration of how they work from an Oracle Sun Rack user's guide:

Cage Nuts

I'd installed them before, but never more than eight or so at a time. After Truck Day, I now have nothing but the greatest respect for the amazing people working in the data centers who have to do them in massive volumes. I don't think I've ever received as many tiny cuts on my hand as I did in the few hours I spent installing the relatively small number I managed to complete.

As a Customer Support Administrator (CSA), I spend the majority of my time sitting at a computer, helping customers with their servers and doing my best to resolve issues as they are encountered. Physically installing cage nuts isn't part of my day-to-day responsibilities (until the next Truck Day), but I realized that my job has its own "cage nuts."

A customer wanting to lease a server from us isn't particularly worried about the fact that cage nuts have to be meticulously installed in the rack, and they also aren't paying any mind to the fact I might have worked with a dozen customers in my shift already — And, certainly, they shouldn't. They're paying for a great customer experience and helpful, friendly service, so they don't need to take into account the context of our operations when they're simply asking for us to help them with a server reboot to finish the installation of an OS patch upgrade.

SoftLayer, as a company, has amazed me in that everyone I've met is not only willing to deal with their "cage nuts," but they will also do so without losing the smile from their face (even if there's some good-natured grumbling every now and then). In many of the places I've worked, this sort of task would be met with protest, foot dragging and a tired resignation to doing the work. That simply isn't the case here.

I'm definitely a newbie around here, and I'm still getting a feel for the culture, catching up on the inside jokes, and learning the ins and outs of the company (and the people in it). The one thing that was abundantly clear to me from the very first night, though: SLayers are truly dedicated to what they do, and the resulting work environment is one that fosters and rewards that dedication.

So in my estimation, how have the little annoyances — the cage nuts of our lives — molded SoftLayer and the people who work here? I'd say that not only do we work through them, we do so enthusiastically in the company of friends, proud of the fact that these seemingly small things are part of what has made this all possible.

I hope all of you work in environments that enable you to deal with the small things you see every day without cursing under your breath and feeling stressed. If you don't, maybe you should look into finding a place that does. I hear we're hiring.

-Gregory

May 12, 2011

Follow 750 Servers from Truck to DC Rack

What do you call the day after you finish building a new data center server room and cabling the server racks in it? If you're an employee at SoftLayer, you call it Truck Day.

Last week, a few of the folks from marketing were invited to celebrate in the Truck Day festivities for Pod 2 in DAL05 (SR02.DAL05), and I jumped at the opportunity. I don't go anywhere without at least one camera on-hand to document and share what's going on with the SoftLayer community, and Truck Day wasn't an exception ... In fact, I had three different cameras going at all times!

The truck arrived at around 7 a.m. with a few dozen pallets of servers, and about forty employees from all around the company immediately jumped into action. As the pallets moved from the loading dock to the inventory room, people were unboxing servers and piling them on carts. When a cart was full, it was whisked to the data center and unloaded. The data center techs plugged in each of the servers to confirm its configuration and stacked it with matching configurations in designated areas around the data center. By the time one cart got back to the inventory room, another was on its way to the data center, so very little time was lost.

Back in 2007, SamF did a great job of explaining the process, so I won't reinvent the wheel. Instead, I'll let you see the activities as they were captured by the three cameras I toted along:

To give you an idea of how fast all of this was done, each the time lapse cameras set up in the data center and in the inventory room captured images every five seconds. When the video was compiled, the frame rate was set to 20 frames per second, so each second of time lapse video is the equivalent of 100 seconds of work. In a matter of just a few hours, we received, inventoried, racked, cabled and started selling around 750 servers in a brand new data center pod. Competitors: Be afraid. Be very afraid. :)

Pictures from DAL05 Pod 2 Truck Day have been posted on our Flickr Account: http://sftlyr.com/8g

In the past three weeks, we brought three different data center pods online in three different parts of the country: On April 25, it was our first server room in San Jose (SJC01); on May 2, the second server room in DAL05; and on May 10, our second server room in WDC01. As far as I know, we don't have a new pod planned for next month, but given how quickly the operations team has been building data center space, I wouldn't be surprised to get a call asking me to come in a little early to help unload servers in a new data center next week.

-@khazard

Music Credit: The background track in the video is "Your Coat" from SoftLayer's very own Chris Interrante. Keep an eye out for his soon-to-be released EP: OVERDRAFT.

March 23, 2011

SoftLayer Rocked SxSW 2011

South by Southwest 2011 is over. Phew. The chaotic buzz of sessions, trade show booths, concerts, happy hours and parties has subsided, so we can finally take a little time to look back at our experience in Austin last week. Our most talked about contributions to the 2011 SxSW community were our SxSL (South by SoftLayer) event at Iron Cactus on Monday and the SoftLayer Server Challenge on the trade show floor.

SxSL

If you've ever been to a SoftLayer soiree, you know that we know how to get down with our bad selves to throw a good party, and SxSL was no exception. The Cactus Room at Iron Cactus was a perfect venue to unwind after the first day on the trade show floor, and it proved to be a great setting for many interesting conversations about hosting, cloud computing and SoftLayer's plans for world domination ... err ... growth.

As you can see from a few of the pictures we took at the event, Snappy - HostGator's mascot - made an appearance, and he was quite the popular guy. He made so many friends at the party, he actually followed us back to the office in Dallas.

Server Challenge

If you followed the link to the Server Challenge at the top of this post, there's no need to reintroduce the competition, so we'll dive straight into how it went. Before I tell you what I think, listen to what @ipbrian had to say about it on Twitter: "Congrats to @SoftLayer for having the best contest and booth at #sxswi. I have NEVER wanted to repeat visit a vendor more."

Brian was a fierce competitor who pushed the limits of how fast our rack of servers could be reassembled, and prior to SxSW, he'd never heard of SoftLayer. As he hurried to reattach network cables, he experienced what we do in our data centers, and that experience is worth more than any piece of collateral we could have given him. That experience was our goal in designing the challenge, and based on our first show with it, we're confident that our goal is being met.

Some attendees saw the blazing times on our Server Challenge leader board as inspiration to complete an "Eye of the Tiger"-speed assembly while others - like the two squirrels from getacorn.com in the video below - knew they might not win the iPad 2 for being the fastest at SxSW but wanted to try anyway:

In addition to the official Server Challenge competition, we were happy to take part in Das IronGeek for the second year in a row. Das IronGeek put six press and bloggers through a series of five technology-related challenges to test their "geekiness," and the SoftLayer Server Challenge was the final "make it or break it" event to determine the champion. All of the competitors stared down the SoftLayer server rack and had a blast completing the challenge. Joshua Baer bested the other participants to become the 2011 Das IronGeek Champion, and if I were him, I'd be showing my kids the 2011 Das IronGeek wrap up for years to come.

As the trade show wound down on Thursday, our booth had a last surge of Server Challenge participants looking to reach the top of the leader board to with an iPad 2, but as you learned from Highlander, there can be only one. That "one" at South by Southwest was Erik Wagner from Netbiscuits with an amazing time of 1:08.8. When he recorded that time, we knew it would be tough to beat, so we had him complete it one more time on camera to show future generations of Server Challenge participants where the bar has been set. Even with the additional pressure of being on camera, he recorded a faster time than any other participant:

We have a few tweaks and improvements planned for our next Server Challenge competition, and we're excited to see how attendees at other shows respond.

As I write about SxSL and the Server Challenge, I'm reminded of stories about the popularity of the good ol' SoftLayer switch-ball and the hallway war we may or may not have been responsible for supplying with foam missile ammunition, but those stories will have to wait for another post. I'm still tired from SxSW sleep deprivation, and I need a nap.

-@khazard

January 11, 2008

I Need a Whataburger!!

Somebody...Anybody...I need a Whataburger!!

If you haven't been to a Whataburger, I'm sorry. It's an amazing fast food chain that sells not only the freshest made-to-order burgers, but they're also open 24-hours a day, and their breakfast is second to none (Chris Menard has a clinical addiction to their taquitos). The problem with this is that they only exist in the South. I'm in the North. In Seattle, Washington to be precise—accompanied by our go-live team to manage our newest datacenter and make sure the launch goes smoothly.

On the bright side (no pun intended, it hasn't stopped raining since we landed), it has. We have assembled an amazing team, the datacenter is absolutely spectacular, and the locals have been very friendly. Efficiencies we have built into our normal daily operations over the last two years have basically allowed us to "drag and drop" our datacenters as needed, where they are needed without having to reinvent the wheel every time we launch. Since the deployment is simple, we can focus on service upgrades—like the latest 40-Gigabit rack-level connections—while we roll out a new facility. Connectivity you could use…say…to look for a Whataburger near you http://www.whataburger.com/one_near_you.php (I look every day). We've already flown through our first historic Seattle Truck Day, and had a second one to boot. We're provisioning droves of machines for new and current customers who are taking advantage of our network architecture, tools, and StorageLayer to create their own custom solutions. In a nutshell, we have brought a new DC online and maintained the ability to provide our customers with the same cutting edge hardware and innovative utilities that they have come to expect in Dallas.

On the darker side, with everything is going so well, it leaves a lot of time to sit and think about a tasty Whataburger. With jalapenos. And bacon. Ugh.

-Joshua

Subscribe to installation