The Human Element of SoftLayer - DAL05 DC Operations

July 19, 2012

One of the founding principles of SoftLayer is automation. Automation has enabled this company to provide our customers with a world class experience, and it enables employees to provide excellent service. It allows us to quickly deploy a variety of solutions at the click of a button, and it guarantees consistency in the products that we deliver. Automation isn't the whole story, though. The human element plays a huge role in SoftLayer's success.

As a Site Manager for the corporate facility, I thought I could share a unique perspective when it comes to what that human element looks like, specifically through the lens of the Server Build Team's responsibilities. You recently heard how my colleague, Broc Chalker, became an SBT, and so I wanted take it a step further by providing a high-level breakdown of how the Server Build Team enables SoftLayer to keep up with the operational demands of a rapidly growing, global infrastructure provider.

The Server Build Team is responsible for filling all of the beautiful data center environments you see in pictures and videos of SoftLayer facilities. Every day, they are in the DC, building out new rows for inventory. It sounds pretty simple, but it's actually a pretty involved process. When it comes to prepping new rows, our primary focus is redundancy (for power, cooling and network). Each rack is powered by dual power sources, four switches in a stacked configuration (two public network, two private network), and an additional switch that provides KVM access to the server. To make it possible to fill the rack with servers, we also have to make sure it's organized well, and that takes a lot of time. Just watch the video of the Go Live Crew cabling a server rack in SJC01, and you can see how time- and labor-intensive the process is. And if there are any mistakes or if the cables don't look clean, we'll cut all the ties and start over again.



 

In addition to preparing servers for new orders, SBTs also handle hardware-related requests. This can involve anything from changing out components for a build, performing upgrades / maintenance on active servers, or even troubleshooting servers. Any one of these requests has to be treated with significant urgency and detail.



 

The responsibilities do not end there. Server Build Technicians also perform a walk of the facility twice per shift. During this walk, technicians check for visual alerts on the servers and do a general facility check of all SoftLayer pods. Note: Each data center facility features one or more pods or "server rooms," each built to the same specifications to support up to 5,000 servers.



 

The DAL05 facility has a total of four pods, and at the end of the build-out, we should be running 18,000-20,000 servers in this facility alone. Over the past year, we completed the build out of SR02 and SR03 (pod 2 and 3, respectively), and we're finishing the final pod (SR04) right now. We've spent countless hours building servers and monitoring operating system provisions when new orders roll in, and as our server count increases, our team has grown to continue providing the support our existing customers expect and deserve when it comes to upgrade requests and hardware-related support tickets.



 

To be successful, we have to stay ahead of the game from an operations perspective. The DAL05 crew is working hard to build out this facility's last pod (SR04), but for the sake of this blog post, I pulled everyone together for a quick photo op to introduce you to the team.

DAL05 Day / Evening Team and SBT Interns (with the remaining racks to build out in DAL05):
DAL05 DC Ops

DAL05 Overnight Server Build Technician Team:
DAL05 DC Ops

Let us know if there's ever anything we can do to help you!

-Joshua