Posts Tagged 'Technicians'

March 12, 2015

Sydney’s a Go

Transforming an empty room into a fully operational data center in just three months: Some said it couldn’t be done, but we did it. In less than three months, actually.

Placing a small team on-site and turning an empty room into a data center is what SoftLayer refers to as a Go Live. Now, of course there is more to bringing a data center online than the just the transformation of an empty room. In the months leading up to the Go Live deployment, there are details to work out, contracts to sign, and the electrical fit out (EFO) of the room itself. During my time with SoftLayer I have been involved in building several of our data centers, or SoftLayer pods as we call them. Pods are designed to facilitate infrastructure scalability, and although they have evolved over the years as newer, faster equipment has become available, the original principles behind the design are still intact—so much so that a data center technician could travel to any SoftLayer data center in the world and start working without missing a beat. And the same holds true to building a pod from the ground up. This uniformity is what allows us to fast track the build out of a new SoftLayer pod. This is one of the reasons why the Sydney data center launch was such a success.

Rewind Three Months

When we landed in Sydney on December 11, 2014, we had an empty server room and about 125 pallets of gear and equipment that had been carefully packed and shipped by our inventory and logistics team. First order of business: breaking down the pallets, inspecting the equipment for any signs of damage and checking that we received everything needed for the build. It’s really quite impressive to know that everything from screwdrivers to our 25U routers to even earplugs had been logged and accounted for. When you are more than 8,500 miles away from your base of operations, it’s imperative that the Go Live team has everything it needs on hand from the start. Something seemingly inconsequential as not having the proper screws can lead to costly delays during the build. Once everything’s been checked off, the real fun begins.

(From Left) Jackie Vong, Dennis Vollmer, Jon Bowden, Chris Stelly, Antonio Gomez, Harpal Singh, Kneeling - Zachary Schacht, Peter Panagopoulos, and Marcelo Alba

Next we set up the internal equipment that powers the pod: four rows of equipment that encompass everything from networking gear to storage to the servers that run various internal systems. Racking the internal equipment is done according to pre-planned layouts and involves far too many cage nuts, the bane of every server build technician’s existence.

Once the internal rows are completed, it’s time to start focusing on the customer rows that will contain bare metal and virtual servers. Each customer rack contains a minimum of five switches—two for the private network, two for the public network, and one out-of-band management switch. Each row has two power strips and in the case of the Sydney data center, two electrical transfer switches at the bottom of the rack that provide true power redundancy by facilitating the transfer of power from one independent feed to another in the case of an outage. Network cables from the customer racks route back to the aggregate switch rack located at the center of each row.

Right around the time we start to wrap up the internal and customer rows, a team of network engineers arrive on-site to run the interconnects between the networking gear and the rest of the internal systems and to light up the fiber lines connecting our new pod to our internal network (as well as the rest of the world). This is a big day because not only do we finally get Wi-Fi up in the pod, but no longer are we isolated on an island. We are connected, and teams thousands of miles away can begin the process of remotely logging in to configure, deploy, and test systems. The networking team will start work on configuring the switches, load balancers, and firewalls for their specific purposes. The storage team will begin the process of bringing massive storage arrays online, and information systems will start work on deploying the systems that manage the automation each pod provides.

(From Left) Zach Robbins, Grayson Schmidt, Igor Gorbatok and Alex Abin

During this time, we start the process of onboarding the newest members of the team, the local Sydney techs, who in a few short months will be responsible for managing the data center independently. But before they fully take over, customer racks are prepped and are waiting to house the final piece of the puzzle: the servers. They arrive via truck day [check out DAL05 Pod 2 truck day]; Sydney’s was around the beginning of February. Given the amount of hardware we typically receive, truck days are an event unto themselves—more than 1,500 of the newest and fastest SuperMicro servers of various shapes and sizes that will serve as the bare metal and virtual servers for our customers. Through a combination of manpower and automation, these servers get unboxed, racked, checked in, and tested before they are sold to our customers.

Now departments involved in bringing the Sydney data center online wrap up and sign off. Then we go live.

Bringing a SoftLayer pod online and on time is a beautifully choreographed process and is one of my greatest professional accomplishments. The level of coordination and cohesion required to pull it off, not once, not twice but ten times all over the world in the last year alone can’t be overstated enough.


May 12, 2011

Follow 750 Servers from Truck to DC Rack

What do you call the day after you finish building a new data center server room and cabling the server racks in it? If you're an employee at SoftLayer, you call it Truck Day.

Last week, a few of the folks from marketing were invited to celebrate in the Truck Day festivities for Pod 2 in DAL05 (SR02.DAL05), and I jumped at the opportunity. I don't go anywhere without at least one camera on-hand to document and share what's going on with the SoftLayer community, and Truck Day wasn't an exception ... In fact, I had three different cameras going at all times!

The truck arrived at around 7 a.m. with a few dozen pallets of servers, and about forty employees from all around the company immediately jumped into action. As the pallets moved from the loading dock to the inventory room, people were unboxing servers and piling them on carts. When a cart was full, it was whisked to the data center and unloaded. The data center techs plugged in each of the servers to confirm its configuration and stacked it with matching configurations in designated areas around the data center. By the time one cart got back to the inventory room, another was on its way to the data center, so very little time was lost.

Back in 2007, SamF did a great job of explaining the process, so I won't reinvent the wheel. Instead, I'll let you see the activities as they were captured by the three cameras I toted along:

To give you an idea of how fast all of this was done, each the time lapse cameras set up in the data center and in the inventory room captured images every five seconds. When the video was compiled, the frame rate was set to 20 frames per second, so each second of time lapse video is the equivalent of 100 seconds of work. In a matter of just a few hours, we received, inventoried, racked, cabled and started selling around 750 servers in a brand new data center pod. Competitors: Be afraid. Be very afraid. :)

Pictures from DAL05 Pod 2 Truck Day have been posted on our Flickr Account:

In the past three weeks, we brought three different data center pods online in three different parts of the country: On April 25, it was our first server room in San Jose (SJC01); on May 2, the second server room in DAL05; and on May 10, our second server room in WDC01. As far as I know, we don't have a new pod planned for next month, but given how quickly the operations team has been building data center space, I wouldn't be surprised to get a call asking me to come in a little early to help unload servers in a new data center next week.


Music Credit: The background track in the video is "Your Coat" from SoftLayer's very own Chris Interrante. Keep an eye out for his soon-to-be released EP: OVERDRAFT.

April 22, 2009

The Tao of the Slayer

In the ever-changing world of IT, there are few times when a technician gets to relax. There are always new issues, new products/services, and long hours of investigation. However, once in awhile you find a moment of Zen in all the commotion: Rack Prep.

Recently, I had assigned myself to Rack Prep to allow my teammates to focus on their other duties. During this time, I was able to complete a large portion of the rack assembly process and release myself from the direct stresses of the IT environment in a busy NOC (network operations center).

The preparation of new racks in the datacenter is an arduous (and sometimes monotonous) task, but gives a technician time to reflect on his accomplishments and direction for his career. There are no distractions, other than the occasional dropped cage nut or screw. This allows the free mind to ponder the inner workings of itself and the body it inhabits.

I thought about the first time I had installed a rack rail. I had only been working in IT for a few months and was assigned to the task due to my lack of knowledge on the other portions of the project. I learned a lot that summer about architecture of hardware, networks, and even business.

I had time to think about how I had arrived at one of the fastest-growing host providers in the world. All the different places I had worked. I remembered the people who shared information – technical or otherwise – which had furthered my ability to solve issues – in servers and myself.

I remembered the managers and supervisors that I looked up to and hope to emulate in my current position in management. I was trying to remember all the wisdom which had been passed to me, leading me to reevaluate my approach and initiatives.

In short, the Rack Prep allowed me to reflect on all the things in my life. I was able to forget the current project while mindlessly pushing in cage nuts and look at my career from a wider perspective. Luckily, I can say that I am proud of how far I have come. Now, I have to install the cables which require much more thought. I better leave the Zen and continue with the task at hand.


February 21, 2008

What It's Like to be a Data Center Technician

As you may have guessed SoftLayer isn't just sales team members, data center managers and development team members. There is also a pretty important group of people who hideaway in their cubicles and can be seen running around our state of the art server rooms from time to time. I am of course talking about us DC Techs; you might know us from our ticket signature "SoftLayer CSA."

I had a question brought up for the first time while on a phone call with a customer, his question was,

"What is it like to be a data center technician?"

I could only laugh just a little bit as I looked around the office and saw several of my co-workers engaging in the organized chaos we call Datacenter Operations. You see, with datacenter operations there is no "daily routine" to follow, there isn't a "what to expect" sheet posted somewhere to prepare us for the day. We have to rely on experience and each other to keep our beloved customers happy. So would you like to know my answer to this customer?

"It depends on the ticket I'm working!"

I say that because this particular customer was calling about a networking issue. In this instance I was his "network engineer", helping him resolve an issue with secondary IP addresses. As I said before, not every issue is the same from one minute to the next so it keeps us on top of our game. One second I am a networking engineer, the next a hardware technician, the next a Systems Administrator. On some occasions we DC techs can be all three at once! It's because of this fact that I enjoy coming to work each and every day. I never know what problem will arise or what I will learn in the coming hours.

I decided to write this after a very long shift, because I think a lot of our customers and people who read this blog would like to know what exactly it's like. Of course there are good days and bad days, sometimes we make mistakes or take a little longer to reply to a ticket than we should. But for the vast majority of the time, our phone calls are ending with "Thanks so much!", and our tickets are ending with "Great Job, You guys are awesome!", and our customers are going to sleep knowing their server is in good hands.

Now what question do all of us DC Techs have? That's simple:

What is it like being a SoftLayer customer?

Judging by everything I have seen recently, with our company expanding to Seattle, building new datacenters, and shattering several of our own sales records, I think we're doing a pretty good job of putting everything you want from a dedicated hosting provider at your fingertips. There is always work to be done, and I speak for everyone here in the office when I say the most important thing to a DC Tech and the company as a whole are our bosses, and we currently have around 4,500 of you around the world and growing!

I’ll see you in the tickets soon!


Subscribe to technicians