Posts Tagged 'Datacenter'

July 19, 2011

PHIL's DC: A Tour of the Facility

In the second episode of my self-made documentary series about the birth of a revolution in hosting, I explained how Lance and I mutually decided that a better course of action would be to build a data center for the future's future, and I sketched out the basics of effective data centering. Lance sent the keys to the new non-traditional facility, and I jumped at the chance to give a tour of the amazing digs.

Because I wanted to make sure to document as much of the process as I could for this documentary film (I'm coming for you, The Social Network), you're experiencing the tour as I explore the space for the first time, so I hope you find it as magical as I did. Note: I took the liberty of acquiring suitable transportation to give you the most professional "tour" experience.

You'll note that the facility features several important characteristics of the best data center environments:

  • Heightened Exterior Security
  • Data Center Operations Area
  • Weather Tracking Station
  • Tech Support Center
  • CEO Suite
  • Redundant Bandwidth Providers
  • Multi-phase Power
  • Power Generator
  • Built-in Cooling
  • Crash Cart Station
  • Vaulted Ceilings (for warm air circulation)

Now that I've got the lay of the land, it's just a matter of drawing up some plans for server racks, plugging in some servers and getting some customers to experience the newest wave of hosting innovation!

-PHIL

July 1, 2011

PHIL's DC: Fine-Tuning the Idea

When Lance opened the floor for SoftLayer employees to present their ideas for "innovative" approaches to the Internet, I put together a pretty ambitious proposal. As it turns out, the idea wasn't as fully baked as I may have wanted it to be, but I came to the decision to change gears a little and take a different approach.

Completely unrelated to that personal decision to adjust the direction of the project, I had a nice little chat with Lance on the phone. We decided that the world was underready for a revolution and that a more traditional nontraditional approach was in order:

The Internet needs data centers to hold all of your pictures. SoftLayer does a great job at being a data center, but I feel like there's still an opportunity for a revolution in data center design. I have a few ideas about how the world of web hosting can be completely redefined, and with the unique resources Lance has put at my disposal, I'm fairly confident that I'll be able to create a stellar hosting platform with an unbeatable discount price structure. PHIL's DC is the future of web hosting.

- PHIL

May 31, 2011

Bringing Home Data Center Security

Look at any time period in mankind's history, and you'll come to the undeniable conclusion that technology changes the daily lives people in any society. With the evolution of technology, our lives have gotten so much easier. Consider all the little luxuries and conveniences available now to get tasks done in the workplace and home. Unfortunately, our rapid technological advancements aren't necessarily exclusive to the "good guys" ... The "bad guys" are benefiting from new technologies as well. Crime and theft have become more sophisticated, and as a result, more technological advancement has to be pursued in security, and it's pretty remarkable to see some of the security measures and technologies put in place by companies like SoftLayer.

The day I started working here, I thought I was actually joining the CIA. I had to undergo several procedures to gain access to all the facilities: I had my photo taken and my fingerprints scanned before I registered for multiple key cards. The first job I had out of college only required its employees to have a single key card that allowed entrance through one door with access to all areas. Needless to say, it was a lot different to work in such a secure environment.

To give you an idea of what kinds of security we have at our data center, I'll walk you through my daily experience. I step into our lobby and am usually greeted by multiple security guards behind what appears to be bullet-proof glass. I have to pass a fingerprint scanner and numerous secured door checkpoints to get into the office. Every move is under the scrutiny of video cameras recording every square inch of the building. Big Brother is always watching, and for SoftLayer customers, that should be reassuring.

The facility's security reminds me of the movie Minority Report, and while those security measures may seem unnecessary or excessive, they're actually just visible evidence of SoftLayer's focus on the importance of security both online and in the physical realm.

Thinking about safety, I've also started considering heightening security at my home with a few security cameras. Some of my friends joke that this consideration is a sign of impending paranoia, but the "better safe than sorry" mantra should always be kept close to heart when it comes to protecting valuables. Apparently, I'm not alone in my home security research ... A day after writing a good portion of this article, I came to work and in the morning a coworker told me he'd recently bought a security camera with night vision for personal use. I didn't expect such a coincidence, and of course I enthusiastically replied to my coworker that I was thinking about making a similar purchase.

In closing, I'd like to ask you if you've entertained the idea of increasing security in your own home, and if so, do you have any suggestions about what equipment to purchase and features that prove useful? I doubt I'll go as far as hiring security guards and installing fingerprint scanners, but you never know!

-Danny

April 18, 2011

Before They Were SoftLayer Data Centers

Ever wonder what a SoftLayer data center looked like before it became a SoftLayer data center? Each one of our facilities is built from a "pod" concept: You can walk into any of our server rooms in any of our facilities around the country (soon to be "around the world"), and you'll see same basic layout, control infrastructure and servers. By building our data center space in this way, we're able to provide an unparalleled customer experience. Nearly every aspect of our business benefits from this practice, many in surprising ways.

From an operations perspective, our staff can work in any facility without having to be retrained and the data center construction process becomes a science that can be replicated quicker with each subsequent build-out. From a sales perspective, every product and technology can be made available from all of our locations. From a network perspective, the network architecture doesn't deviate significantly from place to place. From a finance perspective, if we're buying the same gear from the same vendors, we get better volume pricing. From a marketing perspective ... I guess we have a lot of really pretty data center space to show off.

We try to keep our customers in the loop when it comes to our growth and expansion plans by posting pictures and updates as we build new pods, and with our newest facility in San Jose, CA, we've been snapping photos throughout the construction progress. If you've been patiently reading this part of the blog before scrolling down to the pictures, you get bonus points ... If you looked at the pictures before coming back up to this content, you already know that I've included several snapshots that show some of the steps we take when outfitting new DC space.

SoftLayer San Jose Data Center Construction

The first look at our soon-to-be data center is not the flashiest, but it shows you how early we get involved in the build-out process. The San Jose facility is brand new, so we have a fresh canvas for our work of art. If I were to start talking your ear off about the specifics of the space, this post would probably go into next week, so I'll just show you some of the most obvious steps in the evolution of the space.

SoftLayer San Jose Data Center Construction

The time gap between the first picture and the second picture is pretty evident, but the drastic change is pretty impressive. Raised floor, marked aisles, PDUs ... But no racks.

SoftLayer San Jose Data Center Construction

Have no fear, the racks are being assembled.

SoftLayer San Jose Data Center Construction

They're not going to do much good sitting in the facility's office space, though. Something tells me the next picture will have them in a different setting.

SoftLayer San Jose Data Center Construction

Lucky guess, huh? You can see in this picture that the racks are installed in front of perforated tiles (on the cold aisle side) and on top of special tiles that allow for us to snake cabling from under the floor to the rack without leaving open space for the cold air to sneak out where it's not needed.

SoftLayer San Jose Data Center Construction

The next step in the process requires five very expensive network switches in each rack. Two of the switches are for public network traffic, two are for private network traffic and one is for out-of-band management network traffic.

SoftLayer San Jose Data Center Construction

Those switches won't do much good for the servers if the servers can't be easily connected to them, so the next step is to attach and bind all of the network cable from the switches to where the servers will be. As you'll see in the next pictures, the cabling and binding is done with extreme precision ... If any of the bundles aren't tightly wound, the zip ties are cut and the process has to be restarted.

SoftLayer San Jose Data Center Construction

While the cables are being installed, we also work to prepare our control row with servers, switches, routers and appliances that mirror the configurations we have in our other pods.

SoftLayer San Jose Data Center Construction

When the network cables are all installed, it's a pretty amazing sight. When the cables are plugged into the servers, it's even more impressive ... Each cable is pre-measured and ready to be attached to its server with enough length to get it to the port but not too much to leave much slack.

SoftLayer San Jose Data Center Construction

One of the last steps before we actually get the servers installed is to install the server rails (which make installing the server a piece of cake).

SoftLayer San Jose Data Center Construction

The servers tend to need power, so the power strips are installed on each rack, and each power strip is fed from the row's PDU.

SoftLayer San Jose Data Center Construction

Every network and power cable in the data center is labeled and positioned exactly where it needs to be. The numbers on the cables correspond with ports on our switches, spots in the rack and plugs on the power strip so we can immediately track down and replace any problem cables we find.

SoftLayer San Jose Data Center Construction

If you've hung around with me for this long, I want to introduce you to a few of the team members that have been working night and day to get this facility ready for you. While I'd like to say I could have done all of this stuff myself, that would be a tremendous lie, and without the tireless efforts of all of these amazing SoftLayer folks, this post would be a whole lot less interesting.

A funny realization you might come to is that in this entire "data center" post, there's not a single picture of a customer server ... Is it a data center if it doesn't have data yet?

-SamF

October 8, 2010

From Zero to Ten in 10

Our second Dallas data center went live 10 days ago and we are already pushing 10 GB of sustainable traffic out the door. I have spent some time in the DC with some of our ops guys, and the place is impressive.

A terrific amount of computing power sits in row after row of server racks, driving a diverse array of business to more than 110 countries. Each rack features powerful processors, lots of RAM and heaps of storage. There is very little that our customers are unable to do over Softlayer’s infrastructure. And if they need more, SoftLayer can add additional servers very quickly to meet this demand. I wish the rest of our business were as simple as this.

Despite the state of the art infrastructure that sits in the DC, it remains a challenge to meet the needs of our customers. Why? Network, that’s why. SoftLayer’s challenge will be to continuously stay ahead of our customers’ demands, primarily in the network. If the network is unable to support the traffic that is pushed across our DC, everything comes tumbling down.

To a degree, we are victims of our own success. As we add servers to racks, we are placing increasing demand on the network. The more successful we are, the more pressure we place on the network.

Consider the following statistics:

  • When SoftLayer went live five years ago, we used two carriers and pushed 20 Gbps out the door.
  • Four years ago, this had gone up to four carriers and eight 10 Gbps links.
  • In January 2009 we pushed about 70 Gbps of sustained traffic. And this doubled for President Obama’s inauguration.
  • Today we use over ten carriers, with over 1000 Gbps of capacity.
  • In addition to the needs that our customers drive, we cannot forget to consider DDOS attacks as DDOS attacks add significant load to the network. We consistently absorb and successfully defend attacks of 5 Gbps, 10 Gbps or more and the peaks have grown by a factor of ten since SoftLayer went live.

The trend revealed is significant – in five years the amount of traffic sustained over our network has increased by more than ten times. And it shows little signs of slowing down.

Suffice to say, we spend a significant amount of time designing our networks to ensure that we are able to handle the traffic loads that are generated – we have to. Aggressively overbuilding the network brings us some short term pain, but if we are going to stay ahead of demand it is simply good business (and it makes sure our customers are happy). The new DC in Dallas is a great example of how we stay ahead of the game.

Each server has 5 NICs – 2 x 1 Gbps (bonded) for the public network, 2 x 1 Gbps (bonded) for the private network and one for management. The net of this is that customers can push 2 Gbps to the internet assuming server processors can handle the load.

-@quigleymar

September 18, 2009

Ninjas in the Datacenter

We tecchies are a weird bunch.  We equate everything to mythical figures and mysterious characters.  All around at SoftLayer, you can see and hear references to nerdy and mysterious things.  From Brad's incessant General Grievous-ish throat clearing, to FreeBSD's 'beastie' daemon:

Beastie
Copyright 1988 by Marshall Kirk McKusick.

Mythical figures surround us all the time.  IT guys tend to have a reputation for being a little, well, different, than the rest of the world.  Now that you're shaking your head, wondering what I'm rabbling about, allow me to introduce the one mythical figure that reigns supreme, especially here at SoftLayer.  That's right, it's the Ninja.

That's right, we've taken one of the most ridiculously awesome figures in modern mythology, and verbed it.  Not sure what verbing is?  Allow me to utilize one of my personal favorite comic strips as a visual:

Calvin
by Bill Watterson.

The ninja has a couple of meanings here at SoftLayer.  Allow me to give a few examples:

nin-ja [nin-juh]
-verb

  1. To Steal, as in a ticket that looked interesting or challenging: "Dude, you totally ninja'd that Network Question ticket from me!  I'm interested to know what you did to diagnose and fix it!"
  2. To fix an issue, against all probability that it is even fixable: "Wow, I thought that database was hosed.  He totally ninja'd that, and now it works like a charm."

The above are just two of the many examples of ninjas in our datacenter.  It's just one of the many ways we separate ourselves from the pack.  Our responsibilities are not only demanding, but unrelenting.  While we take these many responsibilities quite seriously (such as our commitment to the best support in the industry), we are always quick to lighten each other up.  As our big boss would say it:  "We are defining new standards and setting the tone for others to follow. Leading by example, pushing our luck, and having fun every step of the way."  Working at (and hosting at) SoftLayer is about kicking butt, leaving others in the dust, and relishing in every minute of it.

July 13, 2009

What a View!

I can easily define myself as the crazy one up in the Seattle Datacenter. I like to ride dirt bikes, street bikes, go fast on the water, ride in small airplanes, I could go on and on how my co-workers (and friends/family) may think I am crazy when it comes to Adventures.

What can I say, I like a challenging experience.

One of those challenging experiences is working at SoftLayer, always preparing to be ahead of the rest in this industry, we're constantly learning new technologies and taking leaps and bounds. That is the reason why I love my job so much, we're always working with the latest and greatest, learning new stuff. Speaking of leaps and bounds, I finally did something the other day I have always wanted too. I signed up for an Advance Free Fall Skydive class and jumped out of a Cessna at 13,000 feet. Free falling at terminal speeds towards the earth, At 12,000 feet I mock pulled my parachute 3 times, so the instructors who where both holding onto me by their hands only could see I learned what to do. 6000 ft came, I locked onto my altimeter, 5500 I waived hands off to the instructors and they deployed below me, and I pulled my rip cord. That all happened in about 50 seconds after leaping out of the plane. For the next 8 minutes I saw the best view in Western Washington I have ever seen. One of the thoughts that came to my mind is the only thing to relate to how I have ever thought something was this nice, was the first time I walked into a SoftLayer Datacenter and admired how well thought out and nice it was.

I'm glad to say 19 months into this job and being part of the Operations team in Seattle, I walk in each day to the datacenter and can say the same thing day in and day out. Let's hope I can say the same thing about my second jump in a few weeks.

April 22, 2009

The Tao of the Slayer

In the ever-changing world of IT, there are few times when a technician gets to relax. There are always new issues, new products/services, and long hours of investigation. However, once in awhile you find a moment of Zen in all the commotion: Rack Prep.

Recently, I had assigned myself to Rack Prep to allow my teammates to focus on their other duties. During this time, I was able to complete a large portion of the rack assembly process and release myself from the direct stresses of the IT environment in a busy NOC (network operations center).

The preparation of new racks in the datacenter is an arduous (and sometimes monotonous) task, but gives a technician time to reflect on his accomplishments and direction for his career. There are no distractions, other than the occasional dropped cage nut or screw. This allows the free mind to ponder the inner workings of itself and the body it inhabits.

I thought about the first time I had installed a rack rail. I had only been working in IT for a few months and was assigned to the task due to my lack of knowledge on the other portions of the project. I learned a lot that summer about architecture of hardware, networks, and even business.

I had time to think about how I had arrived at one of the fastest-growing host providers in the world. All the different places I had worked. I remembered the people who shared information – technical or otherwise – which had furthered my ability to solve issues – in servers and myself.

I remembered the managers and supervisors that I looked up to and hope to emulate in my current position in management. I was trying to remember all the wisdom which had been passed to me, leading me to reevaluate my approach and initiatives.

In short, the Rack Prep allowed me to reflect on all the things in my life. I was able to forget the current project while mindlessly pushing in cage nuts and look at my career from a wider perspective. Luckily, I can say that I am proud of how far I have come. Now, I have to install the cables which require much more thought. I better leave the Zen and continue with the task at hand.

.IIIi

April 1, 2008

Chantilly Race

Here is the much anticipated Song of the Year. So to keep up with new DC openings and doing a blog about it, I felt I had to attempt to out do Sleepless in Seattle. I believe with a little help from Dave we have done just that! Enjoy!

Chantilly Race! (To the Tune of Chantilly Lace by The Big Bopper)
Lyrics by: Skinman
Produced by: Stringtapper Productions
Vocals by: Dave Huff

And for even more excitement, check out the music video.

-Skinman

February 11, 2008

Spares at the Ready

In Steve's last post he talked about the logic of outsourcing. The rationale included the cost of redundant internet connections, the cost of the server, UPS, small AC, etc. He covers a lot of good reasons to get the server out of the broom closet and into a real datacenter. However, I would like to add one more often over looked component to that argument: the Spares Kit.

Let's say that you do purchase your own server and you set it up in the broom closet (or a real datacenter for that matter) and you get the necessary power, cooling and internet connectivity for it. What about spare parts?

If you lose a hard drive on that server, do you have a spare one available for replacement? Maybe so - that's a common part with mechanical features that is liable to fail - so you might have that covered. Not only do you have a spare drive, the server is configured with some level of RAID so you're probably well covered there.

What if that RAID card fails? It happens - and it happens with all different brands of cards.

What about RAM? Do you keep a spare RAM DIMM handy or if you see failures on one stick, do you just plan to remove it and run with less RAM until you can get more on site? The application might run slower because it's memory starved or because now your memory is not interleaved - but that might be a risk you are willing to take.

How about a power supply? Do you keep an extra one of those handy? Maybe you keep a spare. Or, you have dual power supplies. Are those power supplies plugged into separate power strips on separate circuits backed up by separate UPSs?

What if the NIC on the motherboard gets flaky or goes out completely? Do you keep a spare motherboard handy?

If you rely on out of band management of your server via an IPMI, Lights Out or DRAC card - what happens if that card goes bad while you're on vacation?

Even if you have all necessary spare parts for your server or you have multiple servers in a load balanced configuration inside the broom closet; what happens if you lose your switch or your load balancer or your router or your... What happens if that little AC you purchased shuts down on Friday night and the broom closet heats up all weekend until the server overheats? Do you have temperature sensors in the closet that are configured to send you an alert - so that now you have to drive back to the office to empty the water pail of the spot cooler?

You might think that some of these scenarios are a bit far fetched but I can certainly assure you that they're not. At SoftLayer, we have spares of everything. We maintain hundreds of servers in inventory at all times, we maintain a completely stocked inventory room full of critical components, and we staff it all 24/7 and back it all up with a 4 hour SLA.

Some people do have all of their bases covered. Some people are willing to take a chance, and even if you convince your employer that it's ok to take those chances, how do you think the boss will respond when something actually happens and critical services are offline?

-SamF

Subscribe to datacenter