Posts Tagged 'SBT'

August 22, 2013

Network Cabling Controversy: Zip Ties v. Hook & Loop Ties

More than 210,000 users have watched a YouTube video of our data center operations team cabling a row of server racks in San Jose. More than 95 percent of the ratings left on the video are positive, and more than 160 comments have been posted in response. To some, those numbers probably seem unbelievable, but to anyone who has ever cabled a data center rack or dealt with a poorly cabled data center rack, the time-lapse video is enthralling, and it seems to have catalyzed a healthy debate: At least a dozen comments on the video question/criticize how we organize and secure the cables on each of our server racks. It's high time we addressed this "zip ties v. hook & loop (Velcro®)" cable bundling controversy.

The most widely recognized standards for network cabling have been published by the Telecommunications Industry Association and Electronics Industries Alliance (TIA/EIA). Unfortunately, those standards don't specify the physical method to secure cables, but it's generally understood that if you tie cables too tight, the cable's geometry will be affected, possibly deforming the copper, modifying the twisted pairs or otherwise physically causing performance degradation. This understanding begs the question of whether zip ties are inherently inferior to hook & loop ties for network cabling applications.

As you might have observed in the "Cabling a Data Center Rack" video, SoftLayer uses nylon zip ties when we bundle and secure the network cables on our data center server racks. The decision to use zip ties rather than hook & loop ties was made during SoftLayer's infancy. Our team had a vision for an automated data center that wouldn't require much server/cable movement after a rack is installed, and zip ties were much stronger and more "permanent" than hook & loop ties. Zip ties allow us to tighten our cable bundles easily so those bundles are more structurally solid (and prettier). In short, zip ties were better for SoftLayer data centers than hook & loop ties.

That conclusion is contrary to the prevailing opinion in the world of networking that zip ties are evil and that hook & loop ties are among only a few acceptable materials for "good" network cabling. We hear audible gasps from some network engineers when they see those little strips of nylon bundling our Ethernet cables. We know exactly what they're thinking: Zip ties negatively impact network performance because they're easily over-tightened, and cables in zip-tied bundles are more difficult to replace. After they pick their jaws up off the floor, we debunk those myths.

The first myth (that zip ties can negatively impact network performance) is entirely valid, but its significance is much greater in theory than it is in practice. While I couldn't track down any scientific experiments that demonstrate the maximum tension a cable tie can exert on a bundle of cables before the traffic through those cables is affected, I have a good amount of empirical evidence to fall back on from SoftLayer data centers. Since 2006, SoftLayer has installed more than 400,000 patch cables in data centers around the world (using zip ties), and we've *never* encountered a fault in a network cable that was the result of a zip tie being over-tightened ... And we're not shy about tightening those ties.

The fact that nylon zip ties are cheaper than most (all?) of the other more "acceptable" options is a fringe benefit. By securing our cable bundles tightly, we keep our server racks clean and uniform:

SoftLayer Cabling

The second myth (that cables in zip-tied bundles are more difficult to replace) is also somewhat flawed when it comes to SoftLayer's use case. Every rack is pre-wired to deliver five Ethernet cables — two public, two private and one out-of-band management — to each "rack U," which provides enough connections to support a full rack of 1U servers. If larger servers are installed in a rack, we won't need all of the network cables wired to the rack, but if those servers are ever replaced with smaller servers, we don't have to re-run network cabling. Network cables aren't exposed to the tension, pressure or environmental changes of being moved around (even when servers are moved), so external forces don't cause much wear. The most common physical "failures" of network cables are typically associated with RJ45 jack crimp issues, and those RJ45 ends are easily replaced.

Let's say a cable does need to be replaced, though. Servers in SoftLayer data centers have redundant public and private network connections, but in this theoretical example, we'll assume network traffic can only travel over one network connection and a data center technician has to physically replace the cable connecting the server to the network switch. With all of those zip ties around those cable bundles, how long do you think it would take to bring that connection back online? (Hint: That's kind of a trick question.) See for yourself:

The answer in practice is "less than one minute" ... The "trick" in that trick question is that the zip ties around the cable bundles are irrelevant when it comes to physically replacing a network connection. Data center technicians use temporary cables to make a direct server-to-switch connection, and they schedule an appropriate time to perform a permanent replacement (which actually involves removing and replacing zip ties). In the video above, we show a temporary cable being installed in about 45 seconds, and we also demonstrate the process of creating, installing and bundling a permanent network cable replacement. Even with all of those villainous zip ties, everything is done in less than 18 minutes.

Many of the comments on YouTube bemoan the idea of having to replace a single cable in one of these zip-tied bundles, but as you can see, the process isn't very laborious, and it doesn't vary significantly from the amount of time it would take to perform the same maintenance with a Velcro®-secured cable bundle.

Zip ties are inferior to hook & loop ties for network cabling? Myth(s): Busted.

-@khazard

P.S. Shout-out to Elijah Fleites at DAL05 for expertly replacing the network cable on an internal server for the purposes of this video!

July 19, 2012

The Human Element of SoftLayer - DAL05 DC Operations

One of the founding principles of SoftLayer is automation. Automation has enabled this company to provide our customers with a world class experience, and it enables employees to provide excellent service. It allows us to quickly deploy a variety of solutions at the click of a button, and it guarantees consistency in the products that we deliver. Automation isn't the whole story, though. The human element plays a huge role in SoftLayer's success.

As a Site Manager for the corporate facility, I thought I could share a unique perspective when it comes to what that human element looks like, specifically through the lens of the Server Build Team's responsibilities. You recently heard how my colleague, Broc Chalker, became an SBT, and so I wanted take it a step further by providing a high-level breakdown of how the Server Build Team enables SoftLayer to keep up with the operational demands of a rapidly growing, global infrastructure provider.

The Server Build Team is responsible for filling all of the beautiful data center environments you see in pictures and videos of SoftLayer facilities. Every day, they are in the DC, building out new rows for inventory. It sounds pretty simple, but it's actually a pretty involved process. When it comes to prepping new rows, our primary focus is redundancy (for power, cooling and network). Each rack is powered by dual power sources, four switches in a stacked configuration (two public network, two private network), and an additional switch that provides KVM access to the server. To make it possible to fill the rack with servers, we also have to make sure it's organized well, and that takes a lot of time. Just watch the video of the Go Live Crew cabling a server rack in SJC01, and you can see how time- and labor-intensive the process is. And if there are any mistakes or if the cables don't look clean, we'll cut all the ties and start over again.



 

In addition to preparing servers for new orders, SBTs also handle hardware-related requests. This can involve anything from changing out components for a build, performing upgrades / maintenance on active servers, or even troubleshooting servers. Any one of these requests has to be treated with significant urgency and detail.



 

The responsibilities do not end there. Server Build Technicians also perform a walk of the facility twice per shift. During this walk, technicians check for visual alerts on the servers and do a general facility check of all SoftLayer pods. Note: Each data center facility features one or more pods or "server rooms," each built to the same specifications to support up to 5,000 servers.



 

The DAL05 facility has a total of four pods, and at the end of the build-out, we should be running 18,000-20,000 servers in this facility alone. Over the past year, we completed the build out of SR02 and SR03 (pod 2 and 3, respectively), and we're finishing the final pod (SR04) right now. We've spent countless hours building servers and monitoring operating system provisions when new orders roll in, and as our server count increases, our team has grown to continue providing the support our existing customers expect and deserve when it comes to upgrade requests and hardware-related support tickets.



 

To be successful, we have to stay ahead of the game from an operations perspective. The DAL05 crew is working hard to build out this facility's last pod (SR04), but for the sake of this blog post, I pulled everyone together for a quick photo op to introduce you to the team.

DAL05 Day / Evening Team and SBT Interns (with the remaining racks to build out in DAL05):
DAL05 DC Ops

DAL05 Overnight Server Build Technician Team:
DAL05 DC Ops

Let us know if there's ever anything we can do to help you!

-Joshua

May 4, 2012

From "Computer Guy" to SoftLayer Server Build Technician

As I sat down to brainstorm ideas for this blog, I began to think about where I was when I started a few years ago and where I am now. When I was hired, I knew next to nothing about the inner-workings of data center IT. I was just your average computer nerd, or "the computer guy" as I became known around the house and to my friends. I had plenty of experience with hardware, but I had no clue just how deep the IT rabbit hole went ... I jumped in anyway.

Before I give you an example of one of the challenges I had to tackle early on, I should back up and explain a pretty important observation I had about SoftLayer: Despite how cheesy it may sound, SoftLayer is a family. If you are willing to learn and have a good work ethic, SoftLayer will to take you under its wing, and the sky is the limit. I was willing to learn, and I'd like to think I have a good work ethic, so I took on a pretty ambitious task: Learn Linux.

As a Server Build Technician — the physical "hands and eyes" in a data center — you can't get by without an intimate knowledge of Linux. As it turns out, trying to learn everything there is to know about Linux is sort of like saying "Get to the end of the Internet." Even after a few years of working with Linux, I still learn new things almost daily, and I'm sure that I'll continue to learn as long as I'm surrounded by Linux servers and other brilliant technicians who can share their Linux expertise. I could probably write a whole series of blog posts about all of the crazy things I've seen Linux servers do, but I'll focus on this "intro" blog first. Since starting with SoftLayer, my tenuous grasp of Linux was strengthened and eventually validated by my Linux+ certification!

That's only one little example of the kind of environment SoftLayer creates, and I could share dozens more.

When SLayers are treated like individuals rather than "employees," the culture is different. Managers and supervisors LISTEN to your problems/frustrations and are quick to offer their help and advice. I can feel comfortable to express personal issues with anyone in management, and I've had a handful of heartfelt talks with higher-ups that I would never dreamed of having at previous jobs. As a result, I'm excited when I walk into work because I feel like I get to hang out and work with friends for eight hours every day.

My coworkers and I can joke around one minute, and the next minute, we can have a serious and thoughtful conversation about how we could improve our processes or serve customers better better. Not only does that experience make for a comfortable working environment, it also creates a net of trust among coworkers. You know without a doubt that you can rely on your coworkers for anything.

I know it sounds like I'm stretching the truth (and the blog word count), but to be honest, there isn't enough room on this page to describe exactly how awesome I think the people at SoftLayer are. I've made many, many friends and roughly zero enemies. That's a pretty good ratio if you ask me. If you are even a TINY bit interested in IT, there's no better place to get your career started (or continued) than SoftLayer. There are positions for every skill set and level, and it doesn't stop there ... You aren't locked into one position or department if you find yourself more passionately drawn to another area of the business. SoftLayer encourages you to branch out and explore your career options, and if you want to move up, you're encouraged and supported by management to put forth the effort.

TL;DR If you're interested in getting into anything IT related, SoftLayer has a place for you, and as a very happy employee, I'd highly recommend taking advantage of that opportunity.

-Broc

October 9, 2011

Getting Started as a Server Build Technician

When I was interviewed for a job as Server Build Technician (SBT) in Dallas, I was a little concerned that I was getting in over my head. I let my potential manager know that I had very little experience with Linux but that I was willing to learn. I tried to show that I'd be a quick study, and the interview must have gone well because by the end of the day, I was offered the job. I was really excited to know that SoftLayer was willing to take give me an opportunity to finally start pursuing a career path in technology (which is what I was looking for out of school).

As it turns out, I was the only female SBT in SoftLayer's Dallas-area data centers, so I felt a good amount of pressure to prove myself and step up my game. Luckily, my training took away a lot of those nerves, and it was also comforting to see that no matter where I was working (data center or office), I was welcomed by my coworkers. It didn't hurt that I met some really cool people in the process, too. From day one, I realized that I'd been given an amazing opportunity to learn from some really smart folks who know their stuff when it comes to everything related to technology.

I have been here for around six months, and I can't believe how much tech knowledge I've absorbed. I wouldn't claim to be an expert in Linux or a MySQL guru (yet), but if my experience here is any indication, it won't be too long before I know everything there is to know about every technology living in our data centers. When I run into a problem or a question I don't have the answer for, I can rely on my coworkers to have the solutions and break them down into terms I can understand if they're overly complex.

Would I recommend this job to others? Most definitely! This has been one of the best jobs that I've ever had. I've been able to take what I learned in school and actually apply it to my daily work life while continuing my real-world on-the-job education. If you have a server in DAL02 and need someone to check out the hardware or add some RAM, I might be the one jumping to get your request fulfilled quickly.

I'm proof that SoftLayer invests in its employees, so if you're interested in an amazing job for a company who values you, I want you to be a coworker! We have positions in all departments available in Dallas, Houston, Singapore, Amsterdam, San Jose, Seattle and Washington, D.C. (and probably more location in the near future), so keep an eye on the SoftLayer Careers page for the perfect opportunity to join our team.

-Rochelle

Subscribe to sbt