Posts Tagged 'Operations'

October 27, 2011

SoftLayer Features and Benefits - Data Centers

When we last talked, I broke down the differences between features and benefits. To recap: a feature is something prominent about a person, place or thing, while a benefit is a feature that is useful to you. In that blog, I discussed our customer portal and the automation within, so with this next installment, let's move into my favorite place: the data center ... Our pride and joy!

If you have not had a chance to visit a SoftLayer data center, you're missing out. The number one response I get when I begin a tour through any of our facilities is, "I have been through several data centers before, and they're pretty boring," or my favorite, "We don't have to go in, they all look the same." Then they get a glimpse at the SoftLayer facility through the window in our lobby:

Data Center Window

What makes a SoftLayer DC so different and unique?

We deploy data centers in a pod concept. A pod, or server room, is a designed to be an identical installation of balanced power, cooling and redundant best-in-class equipment in under 10,000 square feet. It will support just about 5,000 dedicated servers, and each pod is built to the same specifications as every other pod. We use the same hardware vendor for servers, the majority of our internal network is powered by Cisco gear and edge equipment is now powered by Juniper. Even the paint on the walls matches up from pod to pod, city to city and now country to country. That's standardization!

That all sounds great, but what does that mean for you? How do all these things benefit you as the end user?

First of all, setting standards improves our efficiency in support and operations. We can pluck any of our technicians in DAL05 and drop him into SJC01, and he'll feel right at home despite the outside world looking a bit different. No facility quirks, no learning curve. In fact, the Go Live Crews in Singapore and Amsterdam are all experienced SoftLayer technicians from our US facilities, so they help us make sure all of the details are exactly alike.

Beyond the support aspect, having data centers in multiple cities around the world is a benefit within itself: You have the option to host your solution as close or as far away from you as you wish. Taking that a step further, disaster recovery becomes much easier with our unique network-within-a-network topology.

The third biggest benefit customers get from SoftLayer's data centers is the quality of the server chassis. Because we standardize our SuperMicro chassis in every facility, we're able to troubleshoot and resolve issues faster when a customer contacts us. Let's say the mainboard is having a problem, and your Linux server is in kernel panic. Instead of taking time to try and fix the part, I can hot-swap all the drives into an identical chassis and use the portal to automatically move all of your IP addresses and network configurations to a new location in the DC. The server boots right up and is back in service with minimal downtime.

Try to do that with "similar" hardware (not "identical"), and see where that gets you.

The last obvious customer benefit we'll talk about here is the data center's internal network performance. Powered by Cisco internal switches and Juniper routers on the edge, we can provide unmatched bandwidth capacity to our data centers as well as low latency links between servers. In one rack on the data center floor, you can see 80Gbps of bandwidth. Our automated, high-speed network allows us to provision a server anywhere in a pod and an additional server anywhere else in the same pod, and they will perform as if they are sitting right next to each other. That means you don't need to reserve space in the same rack for a server that you think you'll need in the future, so when your business grows, your infrastructure can grow seamlessly with you.

In the last installment of this little "SoftLayer Features and Benefits" series, we'll talk about the global network and learn why no one in the industry can match it.

-Harold

October 17, 2011

New Data Centers, Barbara Streisand & "Da Bobby G"

Dealing with jet lag, unfamiliar surroundings, foreign currencies and different languages just begins to describe my hectic life over the past two months. We've been in overdrive, building out SoftLayer's Singapore and Amsterdam data centers in weeks (rather than months).

Our "Go Live Crew" of 16 dedicated SLayers has been working 'round the clock to make sure everything is up and running on time. The biggest challenge has been building out both data centers simultaneously ... With the "Go big or go home" mindset, when we decided to go international, we went all in. Our growing customer base of 23,000 won't stand still, so we need to deliver, whether it be through innovation or expansion. In less than 60 days we've been able to add 31,000+ servers to our network platform, bringing our unique cloud, dedicated and managed hosting solutions closer to our customers around the world.

This accomplishment has been something of a "miracle," and I really need to shout out to my team members on the GLC. Putting in 16-hour days and working weekends while still finding time to go out on the weekends (Jägermeister and Red Bull have been sampled at many a fine pub) has made us a pretty close-knit family. The old "work hard, play hard" saying is an understatement when it comes to the SoftLayer team.

If we're ever dragging a bit in the morning, we can always rely on Duck Sauce to get our pulses racing again by the time we get to the data center. With such a full work schedule, we become creatures of habit, and "Barbara Streisand" is only one example of a staple for the crew. Our daily consistency has even carried over into meal time: My favorite luncheon spot in Amsterdam even named a sandwich after me – Da Bobby G Meat Sandwich. Apparently the combination of meatballs, salami, ham and (a smothering of) ketchup on a bun is not a common order at this establishment, so my innovation needed to be recognized. Nutritional considerations aside, this is one fine sandwich:

Da Bobby G

I've been on the road for a while now, and these are just a few memories I'm taking with me. Jumping around between three continents has definitely had its challenges, but with a great team of focused SLayers, we've been getting the job done. I'm proud to have had a hand in making our international aspirations a reality, and I know that even though this has already been an unbelievable adventure, we're just getting started.

-Robert

October 9, 2011

Getting Started as a Server Build Technician

When I was interviewed for a job as Server Build Technician (SBT) in Dallas, I was a little concerned that I was getting in over my head. I let my potential manager know that I had very little experience with Linux but that I was willing to learn. I tried to show that I'd be a quick study, and the interview must have gone well because by the end of the day, I was offered the job. I was really excited to know that SoftLayer was willing to take give me an opportunity to finally start pursuing a career path in technology (which is what I was looking for out of school).

As it turns out, I was the only female SBT in SoftLayer's Dallas-area data centers, so I felt a good amount of pressure to prove myself and step up my game. Luckily, my training took away a lot of those nerves, and it was also comforting to see that no matter where I was working (data center or office), I was welcomed by my coworkers. It didn't hurt that I met some really cool people in the process, too. From day one, I realized that I'd been given an amazing opportunity to learn from some really smart folks who know their stuff when it comes to everything related to technology.

I have been here for around six months, and I can't believe how much tech knowledge I've absorbed. I wouldn't claim to be an expert in Linux or a MySQL guru (yet), but if my experience here is any indication, it won't be too long before I know everything there is to know about every technology living in our data centers. When I run into a problem or a question I don't have the answer for, I can rely on my coworkers to have the solutions and break them down into terms I can understand if they're overly complex.

Would I recommend this job to others? Most definitely! This has been one of the best jobs that I've ever had. I've been able to take what I learned in school and actually apply it to my daily work life while continuing my real-world on-the-job education. If you have a server in DAL02 and need someone to check out the hardware or add some RAM, I might be the one jumping to get your request fulfilled quickly.

I'm proof that SoftLayer invests in its employees, so if you're interested in an amazing job for a company who values you, I want you to be a coworker! We have positions in all departments available in Dallas, Houston, Singapore, Amsterdam, San Jose, Seattle and Washington, D.C. (and probably more location in the near future), so keep an eye on the SoftLayer Careers page for the perfect opportunity to join our team.

-Rochelle

March 17, 2011

Joining the SoftLayer Family

About a month ago, I joined SoftLayer, and I feel like I've been welcomed into a big new family. I live in the Dallas area, and when I saw a listing from SoftLayer for server build technicians on craigslist, I sent in my résumé and anxiously awaited a response. Two weeks later, I got that response in the form of a phone interview with my soon-to-be manager, and since you're reading this post on SoftLayer's blog, it was clearly a great fit.

I am a Server Build Technician (SBT), and I'm part of the team responsible for building servers for new customers and maintaining our current server fleet. It's a rewarding feeling to know that the work we do helps customers we've never met (and might not ever meet) run their businesses. I personally think we have one of the most important jobs in the company, and it's one you might not ever see if you're not looking for it.

One of the most important things I was looking for when I joined SoftLayer was a company that takes pride in its people. That attitude energizes me and my fellow employees, and it really makes it fun to come to work. Maybe that's why it feels like a family. SoftLayer invests in its employees, and once you're a part of the company, you have a bright future ahead of you. Whether you're looking to move up the career path in your department or apply for a position on a different team, Lance and the management team have made it clear that they want us to succeed.

When I became a part of the SoftLayer team, I felt like I found a home away from home. The people I work with are awesome, and if you're awesome, I want you to come work with us too. SoftLayer's customer base is growing like crazy (as Tom explained in his video interview last week), so our team is growing as well.

The last time I checked, we had more than 25 available positions in in at least six different departments ... One of them is probably a great fit for you. Visit our careers page to get a full list of opportunities: http://www.softlayer.com/about/careers/

Hopefully, I'll get to meet you and welcome you to the family. Rock on!

-Dayrum

January 26, 2011

Time for an Oil Change?

<Fade In>
Man driving into Jiffy Lube, car sputtering and smoking.
Attendant: "Looks like you need an oil change buddy."
Buddy: "Yep, I think so. I was here last week and I think they used the wrong oil!"
Attendant: "Nah, we wouldn't do that. In fact we only have one kind of oil here and that's SAS 70."
Buddy: "Well, that's odd; I am told that I need SSAE 16 for mine to work right."
<Mass Confusion>

Welcome to my world! We have SAS 70 today, but soon we will have the new synthetic, non abrasive, engine-cleaning SSAE 16. Sounds fun right? I sure hope so.

Why the change? Good question. When SAS 70 first appeared in the early 90s, the world's economies weren't quite as intertwined as they are today. It was much harder to do business globally than it is now. (I think the "fad" called the internet has a little something to do with that but I could be wrong!) Now that the oceans have shrunk to a more manageable size, there is a need for the standards that companies use worldwide to match more closely. The goal of the U.S. Statement on Standards for Attestation Engagements 16 (SSAE 16) is to meet a more uniform reporting standard.

What's the difference? It's an "attestation" not an "audit." Google and thefreedictionary.com define attestation as "To affirm to be correct, true, or genuine," and audit as "an inspection, correction, and verification of business accounts." Though they are closely related, they mean different things.

What stay's the same? The focus will still be on controls at service organizations when the controls are relevant to their user entities' internal control over financial reporting. (For some reason, servers tend to have quite a bit to do with that!) There will still be a Type 1 and Type 2 with similar scopes in format. The reports will look very similar but they should be a bit more descriptive. The report will still be used in the same methods and by the same type of user.

What Changes? SSAE 16 is now an attestation and not really an audit. The service auditor will still provide an opinion but it will align itself more closely with existing international attestation standards.

  • Written Management Assertion - Management will be required to provide an assertion, to be included in the report, stating the system is fairly represented, suitably designed and implemented and the related controls were suitably designed to achieve the stated control objectives, and that the controls operated effectively throughout the period. The report will reference that management is responsible for preparing the system description, providing the stated services, specifying the control objectives, identifying the risks, selecting the criteria and designing, implementing and documenting controls that are suitably designed and operating effectively. The auditor's opinion remains in the role of providing assurance, not as the entity responsible for the communication.
  • System Description - The more inclusive description must detail the services covered, classes of transactions, events other than transactions, report preparation processes, control objectives and related controls, complementary user controls and other relevant aspects of the organization's control environment, risk assessment process, information and communication systems, control activities and monitoring controls. (I think an accountant came up with all of that!)

There are quite a few other differences but I think these are the big headliners. SoftLayer is committed to making this change and having it available for our customers that require it. Our normal SAS 70 schedule is Nov. 1 – Oct. 31 but we will be accelerating the process to have the SSAE 16 in place as soon as possible.

We are continuously looking at other compliance, reporting, audits and certifications. If you have any that would help you and your business, let us know.

-Skinman

Categories: 
August 4, 2010

All in the Family

Any place you spend hours upon hours every week, you tend to get to know people pretty well. During my time on my shift, I’ve gotten to know my fellow SLayers quite well. You learn their favorite foods (I’ll tell you right away what almost everyone on my shift orders from our favorite Chinese deliver place – almost like clockwork), their choices in music, their favorite computer games (yes, WoW tops this list, seemingly a standard in the IT Industry), and even how they react in certain situations. While the operations team is a tight knit group, we also have our extended families in SLales, InfoSys, Dev, and the other departments.

As many know, they recently opened up our new HQ. Yes, it’s as awesome as every has said it is. Lance made it a point to acquire a Sonic-style ice machine, our facilities are state of the art, and the building is just simply awe-inspiring. I suppose you could consider it our new home. As a new home, we’ve recently brought all of our departments together under one roof. Operations can grab a quick break with SLales, Dev can bounce some questions off of the CSAs or SBEs simply by walking a few feet, and even better, the friendly trash talking and joking just got easier (we’re a family, of course we’re going to mess with each other every chance we get!).

I’m sure employees everywhere are akin to this metaphor, but here it seems to ring true. We have our family quarrels, we know each other like the backs of our hands, and when things get heavy, we’re there to watch each others’ backs to ensure the mission is accomplished, and we continue to dominate the hosting industry.

March 1, 2010

Don’t Run a Data Center – Run Your Business!

I have a friend who recently took a CTO position with a medium-sized company. The huge company that previously employed him moved their entire IT staff a long way outside of Texas to a rather unpleasant location as a cost cutting move. He and many others declined the relocation offer. I can’t say as I blame them.

The other day, he told me some of the interesting things he’s found at his new company. This company is not a technology company but a professional services company. Up to now, they have opted to be in the IT business by running their own data center. To keep this post to a reasonable length, I’ll just mention a few of the things he’s run into.

Keeping the room powered and cool – trust me, this is harder than it sounds. It involves things like redundant power, UPS devices, generators, CRAC units, dehumidifiers, fire suppression, etc. All this stuff must be tested and maintained constantly.

Ordering new servers – they have to go through an online configurator, and then wait to receive the shipment. Once it arrives, they have to unpack it, rack it, power it up, and install the software. The cycle time from ordering a new server to getting it into production can stretch from days to weeks.

Tracking assets – needless to say, he’s found several holes in the process here. Knowing how much RAM is supposed to be in each server vs. what’s really there is a struggle. Heck, even knowing what servers are supposed to be there is a challenge. It seems that as servers are moved, replaced, or disposed of that the asset tracking system and processes are not as solid as he would like. These loose operations also bring heat from accountants and auditors, especially if a server ‘s value is still on the balance sheet but it has actually been tossed out and they no longer own it.

Maintenance – they pay for a service agreement where a tech is guaranteed to be onsite in 4 hours to do anything up to a complete rip and replace to get them back in production. Once he asked why several servers, each north of $10,000 in value, were just laying around in a parts cage. He was told these were for spare parts in case of an emergency, just in case they couldn’t wait 4 hours.

Bankers and lessors routinely ask us who our biggest competitors are. We routinely tell them that they are not other hosting companies – they are companies like the one described above that insist on being in the data center business even though they are not IT infrastructure companies. Since these companies are our largest competition, let’s look at how SoftLayer beats the competition on the items listed above.

Keeping the room powered and cool – as a customer of SoftLayer, you simply don’t have to worry about all this. Not at all. This is a huge savings of time, effort, and money.

Ordering new servers – Once you either run through the configurator or call your SLales rep with your order, your new servers are immediately provisioned. The cloud products are up in minutes, and you can have a few HUNDRED dedicated servers ready for production in a few hours. Not in days or weeks or months.

Tracking assets – From the accounting side of things, you just don’t have to worry about tracking the assets at all as a SoftLayer customer. They are an operating expense to our customers, not a capital expenditure. As far as knowing what assets you have to work with, you have access to the best customer portal in the business where every detail about every server is kept up in real time, right down to the individual sticks of RAM and drive configurations of each server. If you need tighter integration, SoftLayer provides an API to put all this information seamlessly into your environment. Disposing of a server is a simple cancellation ticket. It couldn’t be easier.

Maintenance – this is also a simple ticket submission, which is resolved in an impressive turnaround time. This service is included in SoftLayer’s monthly fees. There is no need to stockpile parts or entire servers for emergencies.

Bottom line, if your business’s core competency is not IT infrastructure, you are being beaten in the IT infrastructure business by SoftLayer. You are spending way too much time, money, and attention to run something that isn’t a part of your business. Hey, if you can’t beat us, then join us!

By the way, my friend is proposing a major project for his company in 2010. That project is getting out of the business of running a datacenter. He faces a lot of resistance to change “the way we’ve always done it” from the other senior executives. From my point of view, it’s a no brainer. But I’m biased I guess. I’d just tell them, hey, don’t run a data center – run your business!

February 1, 2010

Fuel!

Fuel!

Ask anyone here on our staff, and they’ll tell you a few things about their position:

  1. It’s never boring
  2. It can be quite demanding
  3. We’re never technically “off duty”

That being said, we all need our fuel to keep us going at warp speed. Luckily at the NOC we’re lucky enough to have a fully stocked break room with all sorts of odds and ends to keep us going when the energy levels get low. Allow me to show a few of my personal favorites:

  1. Chocolate Covered Raisins
    These little buggers are great when you’re running like mad and just need a quick snack. You can scoop up a good cupful and keep them at the desk for the remainder of your shift. You can take a little detour to grab a couple while en route to your destination. You can also trick yourself into thinking that they’re healthy since they have raisins in them.
  2. Doritos
    These have made a reliable meal substitute on multiple occasions. A few bags of these can trick your hunger pains and quiet the ache for a few until you can grab an actual meal (not always a guarantee).
  3. Coffee
    Any fan of caffeine knows why I’m adding this. It’s often the first thing ingested at the start of the day, and is famous for its energy-inducing properties. Love it or hate it, you cannot deny the eye-opening effects of this one.
  4. Dr Pepper
    My second favorite carbonated beverage provided here. A quick drink and a quick pick me up.

And for my favorite:

RedBull!

Much like many of the techs here, I have a clinical addiction to caffeine. Caffeine is the lifeblood of the NOC and keeps us working at top speed and form. To date, I have found no quicker delivery of this than through the 8.2 ounce can of this elixir.

And there you have it. These are the snacks and beverages provided that keep me going. And while it’s no health food store, it certainly spikes the blood sugar or caffeine levels enough to sustain a happy and proficient technician through the long night hours.

January 29, 2010

Security and Plan B

Security is not a thing to be taken lightly. Think about the information that is stored on your server; think about how many months or years worth of data is stored in your databases. Your account information holds a master key to all of this data on your server. This is the very reason this information is protected so closely by the SoftLayer staff.

All companies work very hard to make sure that their products and services are as easy to use as possible. While on the other hand, security works as hard as possible to, seemingly, make the product or service difficult to use. While it is never our intention to make any service difficult to use, it is our intention to make them secure. This is the very reason why, when we are presented with any questions via phone that are sensitive to the operation of your server or account we ask the inconvenient questions to make sure the person on the other end of the phone line is authorized to make the requested changes to the account or the server.

Up to this point this article has not been as light hearted as I had originally intended, but it’s all about being prepared. The point is, everyone deserves a vacation at some point or another (or believes they do), and according to Murphy’s Law, something will inevitably occur that requires immediate attention. When you’re enjoying that time on the beach, your mind a million miles from bits and bytes, and you miles from anything that can be used to properly manage your server or your account an issue can occur.

While you are out, have you made proper provisions to ensure someone can manage your hardware in your place? Your staff may have the passwords for the servers, IP addresses, and may be able to drop your name; but, I assure you this is not enough information for the SoftLayer support staff to submit a ticket, reboot, or log into your server on your behalf. Have you made sure that in a panic situation someone will be able to provide us with the answers to the security questions on the phone? Are you sure whoever is left in charge has been given the proper permissions in our management portal? Making sure these points have been thoroughly covered prior to your vacation, or even leaving for the day, will help you minimize risk while maximizing your beach vacation.

December 24, 2009

The Power of Christmas

The Power of Christmas

Putting up Christmas lights this year was a serious beating. I kept blowing breakers due to the amount of lights I put up in response to my wife’s request for ‘more lights!’ It seems like every year things get bigger and bigger (like most things in America – trucks, combo meals, taxes, and the deficit). The problem is there is only so much power in convenient areas of my house and those locations don’t have enough power to run my lights because they are shared with things inside the house. My front porch outlet ties in with my garage outlets so every time we open up the garage door, the breaker blows and the Christmas lights on the front of the house go out. I got tired of resetting breakers and I ended up running 2x 20amp 110v dedicated feeds to my roof and to the front yard.

As I was putting the lights up, I found myself doing power calculations in my head. I multiplied the amount of lights I put up by the watts each bulb consumes to get the total watts. Then I took the total watts and put it into this conversion tool (http://www.mhi-inc.com/Converter/watt_calculator.htm) to calculate what they use in a Kilowatt hour. I have timers setup to turn on the lights from 6pm to 11pm (CST) so that is 5 hours a day. I plan to run them from December 8th through January 3rd which is 27 days totaling 135 hours of run time. Take the Kilowatt hour the lights generate times the hours of operation and you get the total Kilowatt hours used for the holiday season. I was then curious how much this was going to cost me (I am a cheap bastard) so I took out my electric bill (TXU, yes I am paying too much) and took what they charge me for a Kilowatt hour and got the dollar figure it costs to run the lights. I was surprised it is not as much as I thought considering how much light my house now generates. It lights up the neighborhood like the Griswold’s house in Christmas Vacation <http://www.imdb.com/title/tt0097958/> . I would not be surprised if you can now see my house from the space shuttle.

I don’t envy Softlayer’s operation guys because they do these types of power calculations (albeit on a much grander scale) on a daily basis. They have to figure out what types of servers with different components (CPU, drives, memory, raid cards) can go into a single rack to insure that power strips are not blown. Some people don’t understand that you can’t just fill a rack up with 44 1U (or 22 2U) servers and turn them on. You have to carefully plan down to the watt how many of each type of server can go into a rack without overloading circuits. You also have to take into account customer upgrades and make sure there is enough headroom for power spikes upon booting. The math involved in my yearly Christmas light escapade made my head hurt; I can’t imagine what Robert and Brad go though. Hats go off to them. My head would have exploded by now….

Here is the math (rounded):

15 ½ stands of C9 Christmas lights each with 25 bulbs = 385 bulbs
385 7 watt bulbs = 2695 watts
2695 watts = 2.695 Kilowatt hours (from http://www.mhi-inc.com/Converter/watt_calculator.htm)
2.695 Kilowatt hours multiplied by (5 hours a day for 27 days = 135) = 364 total Kilowatt hours
364 total Kilowatt hours times $0.12 = $44

So lighting my house for one month actually uses significantly more electricity than running a server in a SoftLayer data center for the same period of time.

Subscribe to operations