Posts Tagged 'Truck Day'

April 9, 2012

Scaling SoftLayer

SoftLayer is in the business of helping businesses scale. You need 1,000 cloud computing instances? We'll make sure our system can get them online in 10 minutes. You need to spin up some beefy dedicated servers loaded with dual 8-core Intel Xeon E5-2670 processors and high-capacity SSDs for a new application's I/O-intensive database? We'll get it online anywhere in the world in under four hours. Everywhere you look, you'll see examples of how we help our customers scale, but what you don't hear much about is how our operations team scales our infrastructure to ensure we can accommodate all of our customers' growth.

When we launch a new data center, there's usually a lot of fanfare. When AMS01 and SNG01 came online, we talked about the thousands of servers that are online and ready. We meet huge demand for servers on a daily basis, and that presents us with a challenge: What happens when the inventory of available servers starts dwindling?

Truck Day.

Truck Day not limited to a single day of the year (or even a single day in a given month) ... It's what we call any date our operations team sets for delivery and installation of new hardware. We communicate to all of our teams about the next Truck Day in each location so SLayers from every department can join the operations team in unboxing and preparing servers/racks for installation. The operations team gets more hands to speed up the unloading process, and every employee has an opportunity to get first-hand experience in how our data centers operate.

If you want a refresher course about what happens on a Truck Day, you can reference Sam Fleitman's "Truck Day Operations" blog, and if you want a peek into what it looks like, you can watch Truck Day at SR02.DAL05. I don't mean to make this post all about Truck Day, but Truck Day is instrumental in demonstrating the way SoftLayer scales our own infrastructure.

Let's say we install 1,000 servers to officially launch a new pod. Because each pod has slots for 5,000 servers, we have space/capacity for 3,000-4,000 more servers in the server room, so as soon as more server hardware becomes available, we'll order it and start preparing for our next Truck Day to supplement the pod's inventory. You'd be surprised how quickly 1,000 servers can be ordered, and because it's not very easy to overnight a pallet of servers, we have to take into account lead time and shipping speeds ... To accommodate our customers' growth, we have to stay one step ahead in our own growth.

This morning in a meeting, I saw a pretty phenomenal bullet that got me thinking about this topic:

Truck Day — 4/3 (All Sites): 2,673 Servers

In nine different data center facilities around the world, more than 2,500 servers were delivered, unboxed, racked and brought online. Last week. In one day.

Now I know the operations team wasn't looking for any kind of recognition ... They were just reporting that everything went as planned. Given the fact that an accomplishment like that is "just another day at SoftLayer" for those guys, they definitely deserve recognition for the amazing work they do. We host some of the most popular platforms, games and applications on the Internet, and the DC-Ops team plays a huge role in scaling SoftLayer so our customers can scale themselves.

-@gkdog

April 6, 2012

Of Cage Nuts and Customer Service

Sometimes it's the little hardships and annoyances that really mold you. How do you react? Do you manage to work through them, or do you let them eat away at you to the point that you're more paralyzed by them than you are a bigger problem?

As a new hire, I was required to take part in a Truck Day — an experience that helps everyone in the company understand (at a base level) what is involved with the actual products and services we sell. If you've ever had the fortune of working on one, there are certain activities that can leave you feeling weary. For me, that weariness-inducing activity was working with cage nuts.

For those of you unfamiliar with cage nuts, they're small pieces of metal that accommodate screw-in server rails on a rack meant for slide-in server rails. Installing them is one of the most frustrating things ever ... They have two little clips that fit inside the rack, and you have to bend them to get them in. Here's a great illustration of how they work from an Oracle Sun Rack user's guide:

Cage Nuts

I'd installed them before, but never more than eight or so at a time. After Truck Day, I now have nothing but the greatest respect for the amazing people working in the data centers who have to do them in massive volumes. I don't think I've ever received as many tiny cuts on my hand as I did in the few hours I spent installing the relatively small number I managed to complete.

As a Customer Support Administrator (CSA), I spend the majority of my time sitting at a computer, helping customers with their servers and doing my best to resolve issues as they are encountered. Physically installing cage nuts isn't part of my day-to-day responsibilities (until the next Truck Day), but I realized that my job has its own "cage nuts."

A customer wanting to lease a server from us isn't particularly worried about the fact that cage nuts have to be meticulously installed in the rack, and they also aren't paying any mind to the fact I might have worked with a dozen customers in my shift already — And, certainly, they shouldn't. They're paying for a great customer experience and helpful, friendly service, so they don't need to take into account the context of our operations when they're simply asking for us to help them with a server reboot to finish the installation of an OS patch upgrade.

SoftLayer, as a company, has amazed me in that everyone I've met is not only willing to deal with their "cage nuts," but they will also do so without losing the smile from their face (even if there's some good-natured grumbling every now and then). In many of the places I've worked, this sort of task would be met with protest, foot dragging and a tired resignation to doing the work. That simply isn't the case here.

I'm definitely a newbie around here, and I'm still getting a feel for the culture, catching up on the inside jokes, and learning the ins and outs of the company (and the people in it). The one thing that was abundantly clear to me from the very first night, though: SLayers are truly dedicated to what they do, and the resulting work environment is one that fosters and rewards that dedication.

So in my estimation, how have the little annoyances — the cage nuts of our lives — molded SoftLayer and the people who work here? I'd say that not only do we work through them, we do so enthusiastically in the company of friends, proud of the fact that these seemingly small things are part of what has made this all possible.

I hope all of you work in environments that enable you to deal with the small things you see every day without cursing under your breath and feeling stressed. If you don't, maybe you should look into finding a place that does. I hear we're hiring.

-Gregory

November 3, 2011

Global Expansion: Floating Like a Butterfly

Growing up, one of my heroes was Mohammad Ali. While I admired his athletic ability, with my scrappy build I was never going to be a boxer. What I liked the most about Ali was that he said whatever he wanted and backed up his words with action. That is what distinguished Ali from the others.

I'm sure you've been to job fairs and read companies' websites where they talk about how their company encourages teamwork, employee empowerment and innovation ... It's usually right next to a picture of someone skydiving or kite boarding, right? Well I've been with SoftLayer for about a month now, and as you saw from my 3 Bars 3 Questions interview, I spent my first two weeks on the job in Dallas.

I can tell you without hesitation (and with no need for a kite boarding picture) that when you walk around the office in Dallas, you can feel a buzz in the hallways ... An energy that only comes from from people who are passionate and work well together. When I made the trek back to Amsterdam, I knew the environment and culture our team in Europe would need to foster to earn our three bars.

Last week, we had our first Truck Day in the new Amsterdam data center, and it was a perfect opportunity to show off the SoftLayer spirit and work ethic to our newest AMS01 SLayers with the help of the Go Live Crew:

As soon as two large truckloads of servers were delivered, the team jumped into action. We unpacked, sorted, scanned and racked the servers in record time, and it was actually a lot fun. When I walked into the data center the next day, it felt like Christmas: new toys, flashing lights and Barbara Striesand.

It's safe to say that SoftLayer is the Mohammed Ali of hosting. We make bold statements and can back up them up!

If you're interested in joining the SoftLayer team in Amsterdam, we're hiring for several different positions right now, and we'd love to have you join us. When talking to prospective employees in interviews, I always tell the SoftLayer story with Ali-like pride, and moving forward, Truck Day is going to be a perfect example to share. Where else are you going to find a company culture where everyone in the company (even the CEO) celebrates the company's continued growth by helping to unpack and sort hardware?

Based on the conversations I've had since Truck Day, I can tell if they are right for the team simply by their reaction to that story. If you're ready to roll up your sleeves to help out your teammates and have fun doing it, call me.

-@jpwisler

May 12, 2011

Follow 750 Servers from Truck to DC Rack

What do you call the day after you finish building a new data center server room and cabling the server racks in it? If you're an employee at SoftLayer, you call it Truck Day.

Last week, a few of the folks from marketing were invited to celebrate in the Truck Day festivities for Pod 2 in DAL05 (SR02.DAL05), and I jumped at the opportunity. I don't go anywhere without at least one camera on-hand to document and share what's going on with the SoftLayer community, and Truck Day wasn't an exception ... In fact, I had three different cameras going at all times!

The truck arrived at around 7 a.m. with a few dozen pallets of servers, and about forty employees from all around the company immediately jumped into action. As the pallets moved from the loading dock to the inventory room, people were unboxing servers and piling them on carts. When a cart was full, it was whisked to the data center and unloaded. The data center techs plugged in each of the servers to confirm its configuration and stacked it with matching configurations in designated areas around the data center. By the time one cart got back to the inventory room, another was on its way to the data center, so very little time was lost.

Back in 2007, SamF did a great job of explaining the process, so I won't reinvent the wheel. Instead, I'll let you see the activities as they were captured by the three cameras I toted along:

To give you an idea of how fast all of this was done, each the time lapse cameras set up in the data center and in the inventory room captured images every five seconds. When the video was compiled, the frame rate was set to 20 frames per second, so each second of time lapse video is the equivalent of 100 seconds of work. In a matter of just a few hours, we received, inventoried, racked, cabled and started selling around 750 servers in a brand new data center pod. Competitors: Be afraid. Be very afraid. :)

Pictures from DAL05 Pod 2 Truck Day have been posted on our Flickr Account: http://sftlyr.com/8g

In the past three weeks, we brought three different data center pods online in three different parts of the country: On April 25, it was our first server room in San Jose (SJC01); on May 2, the second server room in DAL05; and on May 10, our second server room in WDC01. As far as I know, we don't have a new pod planned for next month, but given how quickly the operations team has been building data center space, I wouldn't be surprised to get a call asking me to come in a little early to help unload servers in a new data center next week.

-@khazard

Music Credit: The background track in the video is "Your Coat" from SoftLayer's very own Chris Interrante. Keep an eye out for his soon-to-be released EP: OVERDRAFT.

October 22, 2009

Weather is Here, Wish You Were Beautiful.

People just can't take a joke.

I'd like to start this tale of revenge by letting you know what a family affair Truck Day is around here. All month long, every month, SoftLayer's lightning fast infrastructure team works to build our high density racks from the ground up. They put in some serious hours to meet our fast paced deadlines, and at the end of the month everyone pitches in to fill those racks up with servers. Early in the morning all of our team members will pile in, sleepy eyed and jonesing for coffee (with the exception of the overnight "SLombies," who are jonesing for bed), to unload a semi truck full of servers.

Here's some pics of the event from way back.

Truck Day 1
The Unload
Truck Day 2
The Stack and Sort

True, it's not as fun as.. well, most things. But SoftLayer's Truck Day is an institution. A bonafide tradition! And I'd hate for anyone to think that I forgot one.

After scrimping and saving (and accepting generous handouts from family members) I finally got to cut loose and fly to Hawaii for a few days in the sun. The Mrs. and I made the most of it and headed to the beach as often as we could. While I was there I took this harmless photo and uploaded it to Facebook, to let everyone know that I was thinking of them on truck day:

Beach Day
Happy Truck Day

Apparently, it wasn't taken in the light hearted manner in which it was meant.

Desk 1
Desk 2
Desk 3

It's dangerous to go on vacation.

-Jeaves

June 6, 2008

VFB For Seattle

So it's been about 4 months now since Seattle went live. We have approximately 2000 servers active already! (That’s more than the last DC I worked at has and they’ve been selling servers in Seattle since August of 2005). Server room 1 has lots of cool servers with lots of blinky lights and we’ve been working hard on deploying Server room two around the clock to keep up with the demands of sales around here.

With the “Go-Live Team” back in Dallas loading up the truck to head out to Chantilly, everything is running smoothly up in the great northwest. We no longer have to hear about {insert random person from the Dallas Go-Live team} complain about not having a What-A-Burger for dinner. That’s fine because we have world famous Hot Dogs right up on the corner at Matt’s Famous Chilidogs to cure the hunger that strikes us in the middle of our shift.

With Server room two going live this last week we had another Seattle truck day (this was the first truck day that we completed with 100% staff from Seattle). We didn’t need those experts from Dallas to baby us through. No Brad Lewis’s to answer questions when we have them. :-)

I must say that everything this last week went great and to continue a new SoftLayer tradition, everyone in Seattle deserves a VFB!

So here is MY Seattle VFB to everyone!

- III!

Something cool is the guys from Operations decided to drop by late last Friday. They were more than pleased and impressed with everyone and the performance that we showed them with our preparation of SR02 this last week.

We’ve set sail guys, and are doing a great job, and we like being noticed by the guys down in Dallas and from what our customers have to say. We will keep up the great work.

It really rocks! I must say that we have it good at SL, but that must be because we got the best C.E.O. ever! (Wonder’s if I’ll see a extra $50 now.. hey Little Jones got it right!?! )

- III! to the guys in Seattle..

-Bill

January 11, 2008

I Need a Whataburger!!

Somebody...Anybody...I need a Whataburger!!

If you haven't been to a Whataburger, I'm sorry. It's an amazing fast food chain that sells not only the freshest made-to-order burgers, but they're also open 24-hours a day, and their breakfast is second to none (Chris Menard has a clinical addiction to their taquitos). The problem with this is that they only exist in the South. I'm in the North. In Seattle, Washington to be precise—accompanied by our go-live team to manage our newest datacenter and make sure the launch goes smoothly.

On the bright side (no pun intended, it hasn't stopped raining since we landed), it has. We have assembled an amazing team, the datacenter is absolutely spectacular, and the locals have been very friendly. Efficiencies we have built into our normal daily operations over the last two years have basically allowed us to "drag and drop" our datacenters as needed, where they are needed without having to reinvent the wheel every time we launch. Since the deployment is simple, we can focus on service upgrades—like the latest 40-Gigabit rack-level connections—while we roll out a new facility. Connectivity you could use…say…to look for a Whataburger near you http://www.whataburger.com/one_near_you.php (I look every day). We've already flown through our first historic Seattle Truck Day, and had a second one to boot. We're provisioning droves of machines for new and current customers who are taking advantage of our network architecture, tools, and StorageLayer to create their own custom solutions. In a nutshell, we have brought a new DC online and maintained the ability to provide our customers with the same cutting edge hardware and innovative utilities that they have come to expect in Dallas.

On the darker side, with everything is going so well, it leaves a lot of time to sit and think about a tasty Whataburger. With jalapenos. And bacon. Ugh.

-Joshua

Subscribe to truck-day