Truck Day Operations

July 11, 2007

How do you unload 1,000 servers and have them ready to go live in a datacenter in five hours? With lots and lots of planning. Every month we take in a shipment of servers to accommodate the next 30 days of sales. Preparation for each delivery starts several months in advance with forecasting models. You have to look far enough ahead in your models to continually adjust forecasts for sales, facilities and available resources. Some vendors need more lead time than others so you have to constantly update your forecasts, all the way up to final order placement.

Also, you don't just walk into a datacenter with a server and set it down. There's a lot of work that goes into physical prep for the datacenter as well. You have to plan the datacenter layout, order and assemble racks, add rails, power strips, switches, power cord bundles, network cable bundles, etc. Every rack we deploy has almost 400 cage nuts and just under 200 cables in it. We don't just string a bunch of cables up and call it a day. Every cable bundle is meticulously routed, combed and hung to make them look professional. With that much cabling, you have to make it right or you'll never be able to work around it.

With one week to go before the trucks arrive, all of the datacenter prep starts wrapping up. And with just a few days left, we have our last manager meeting to review server placement, personnel, timing and other delivery details.

Next is Truck Day - this is when the fun begins.

On Truck Day, we leave plenty of people behind to handle sales, support and accounting, but everyone else is expected at the loading dock. After all the pallets are pulled off the truck and accounted for, the team gets busy un-boxing. As servers are unboxed, all of the spare parts in the boxes - spare screws, riser cards, SATA cables, and various other pieces - are sorted into bins on the dock. The servers themselves are then placed in custom transport carts and moved to the datacenter.

From there, the teams inside the datacenter sort the servers according to type and perform a strict QA process that includes verifying the hardware configurations and verifying that the components are all seated properly.

Once sorted, the servers get scanned into the system and racked up. As all of the cables are plugged in, another QA process is completed to verify that all of the ports are correct. At that point, it's just a matter of turning each server on and watching them check in, get their bios flashed with the latest and greatest release and having the system update any component firmware that is needed. As the systems check themselves into inventory, they go through two more QA processes that include an inventory check and a burn-in process.

By the time the truck is empty, the last box is stashed and the final server is racked up, everyone is ready to get back to their day jobs. Months worth of planning - all wiped out in a matter of hours.

Mary is working on a great post about what Truck Day looks like from a Salesperson's perspective. It explains why we have everyone get involved in the process.

-SamF

Comments

July 11th, 2007 at 11:34am

Dunno about anyone else but I would love to see photos of truck day itself possibly with each stage photographed? :)

July 11th, 2007 at 11:41am

Photos of truck day? I know there are some on the Softlayer forums, but those are customer-accessible only...

I'd like to know how SL copes with the multiple options available on each server: they setup 1,000 new servers on a single day - but do you have to spend each day in the month then opening up each server to stick in an extra hard drive or extra RAM?

July 11th, 2007 at 2:26pm

Congrats on running such a smooth operation. Anyone who as racked and stacked servers knows the amount of planning required to get in and out of the datacenter smoothly.

Now to another point -- how absurd is this process? Think about it. You have all of this staff sorting, transporting, taking inventory and getting servers ready. Though I don't know if Sun's BlackBox Project (http://www.sun.com/emrkt/blackbox/index.jsp) will be the way of the future, I think there is significant strides to be made in deployment of servers.

Presumably, the server vendor they could send you modular racks to install. Pre-loaded with the servers, pre-wired and then use integrated connectors to tap into power, network and management infrastructure? How much time would that save you. Perhaps the number of clients needing such infrastructure has not (yet) reached critical mass.

When we get servers and take them out of the box, I just think how antiquated the entire process is. Processors, disks and networks have improved leaps and bounds over the past few years, but when it comes to the nuts and bolts of the servers, I've seen few improvements.

I would love to be order a pre-built 1/4, 1/2 or full rack of servers, neatly wired and ready to go. Yet, I see few large vendors supplying this type of service.

July 11th, 2007 at 6:55pm

RichyC.:

Good question about all of the different hardware combinations. We order several base server configs. Those orders are based on our most commonly requested server configurations. Beyond that, we analyze the various configuration statistics and then go out on the dc floor and pre-build some of the most common "one-off" configs. Since there is no way to pre-build *every* config for every customer order, we do have to do some on-demand builds, but we do what we can to keep that to a minimum.

rackaid:

Thanks for the props. There is a lot of planning and prep work that goes into each delivery and it's fun to see it all come together each month.

And in regards to "pre-configured racks" - I like the way you think. I've toured Sun's BlackBox product and while it is interesting, there isn't much practical use for it in *our* environment. They are very intriguing for specific uses - such as emergency relief management, off shore environments, etc. But, they are way too limited in all aspects for the high density environments in this industry. As an alternative, I would agree with what Lance described in one of his blog posts about "Your Datacenter is Obsolete" (http://theinnerlayer.softlayer.com/2007/your-datacenter-is-obsolete/). That's a good picture of the datacenter of the future.

As for pre-configured racks? We have had server vendors offer to provide such a service, we have had third party companies offer to provide the service and we have had individuals offer to come on site and do the work. The fact is, regardless of *who* does it, it still has to be done. After delving into pricing and all of the caveats that go into such services, it was determined that we can actually do it better and cheaper and more "on-demand" than any of the offers we have seen to date. The product is out there and available. It just doesn't fit our model at this time.

Besides - what would be the fun of Truck Day if everything was done for us?

Sam F

Leave a Reply

Filtered HTML

  • Web page addresses and e-mail addresses turn into links automatically.
  • You can enable syntax highlighting of source code with the following tags: <pre>, <blockcode>, <bash>, <c>, <cpp>, <drupal5>, <drupal6>, <java>, <javascript>, <php>, <python>, <ruby>. The supported tag styles are: <foo>, [foo].
  • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.

Comments

July 11th, 2007 at 11:34am

Dunno about anyone else but I would love to see photos of truck day itself possibly with each stage photographed? :)

July 11th, 2007 at 11:41am

Photos of truck day? I know there are some on the Softlayer forums, but those are customer-accessible only...

I'd like to know how SL copes with the multiple options available on each server: they setup 1,000 new servers on a single day - but do you have to spend each day in the month then opening up each server to stick in an extra hard drive or extra RAM?

July 11th, 2007 at 2:26pm

Congrats on running such a smooth operation. Anyone who as racked and stacked servers knows the amount of planning required to get in and out of the datacenter smoothly.

Now to another point -- how absurd is this process? Think about it. You have all of this staff sorting, transporting, taking inventory and getting servers ready. Though I don't know if Sun's BlackBox Project (http://www.sun.com/emrkt/blackbox/index.jsp) will be the way of the future, I think there is significant strides to be made in deployment of servers.

Presumably, the server vendor they could send you modular racks to install. Pre-loaded with the servers, pre-wired and then use integrated connectors to tap into power, network and management infrastructure? How much time would that save you. Perhaps the number of clients needing such infrastructure has not (yet) reached critical mass.

When we get servers and take them out of the box, I just think how antiquated the entire process is. Processors, disks and networks have improved leaps and bounds over the past few years, but when it comes to the nuts and bolts of the servers, I've seen few improvements.

I would love to be order a pre-built 1/4, 1/2 or full rack of servers, neatly wired and ready to go. Yet, I see few large vendors supplying this type of service.

July 11th, 2007 at 6:55pm

RichyC.:

Good question about all of the different hardware combinations. We order several base server configs. Those orders are based on our most commonly requested server configurations. Beyond that, we analyze the various configuration statistics and then go out on the dc floor and pre-build some of the most common "one-off" configs. Since there is no way to pre-build *every* config for every customer order, we do have to do some on-demand builds, but we do what we can to keep that to a minimum.

rackaid:

Thanks for the props. There is a lot of planning and prep work that goes into each delivery and it's fun to see it all come together each month.

And in regards to "pre-configured racks" - I like the way you think. I've toured Sun's BlackBox product and while it is interesting, there isn't much practical use for it in *our* environment. They are very intriguing for specific uses - such as emergency relief management, off shore environments, etc. But, they are way too limited in all aspects for the high density environments in this industry. As an alternative, I would agree with what Lance described in one of his blog posts about "Your Datacenter is Obsolete" (http://theinnerlayer.softlayer.com/2007/your-datacenter-is-obsolete/). That's a good picture of the datacenter of the future.

As for pre-configured racks? We have had server vendors offer to provide such a service, we have had third party companies offer to provide the service and we have had individuals offer to come on site and do the work. The fact is, regardless of *who* does it, it still has to be done. After delving into pricing and all of the caveats that go into such services, it was determined that we can actually do it better and cheaper and more "on-demand" than any of the offers we have seen to date. The product is out there and available. It just doesn't fit our model at this time.

Besides - what would be the fun of Truck Day if everything was done for us?

Sam F

Leave a Reply

Filtered HTML

  • Web page addresses and e-mail addresses turn into links automatically.
  • You can enable syntax highlighting of source code with the following tags: <pre>, <blockcode>, <bash>, <c>, <cpp>, <drupal5>, <drupal6>, <java>, <javascript>, <php>, <python>, <ruby>. The supported tag styles are: <foo>, [foo].
  • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.