Posts Tagged ‘operations’

When I was interviewed for a job as Server Build Technician (SBT) in Dallas, I was a little concerned that I was getting in over my head. I let my potential manager know that I had very little experience with Linux but that I was willing to learn. I tried to show that I’d be a quick study, and the interview must have gone well because by the end of the day, I was offered the job. I was really excited to know that SoftLayer was willing to take give me an opportunity to finally start pursuing a career path in technology (which is what I was looking for out of school).

As it turns out, I was the only female SBT in SoftLayer’s Dallas-area data centers, so I felt a good amount of pressure to prove myself and step up my game. Luckily, my training took away a lot of those nerves, and it was also comforting to see that no matter where I was working (data center or office), I was welcomed by my coworkers. It didn’t hurt that I met some really cool people in the process, too. From day one, I realized that I’d been given an amazing opportunity to learn from some really smart folks who know their stuff when it comes to everything related to technology.

I have been here for around six months, and I can’t believe how much tech knowledge I’ve absorbed. I wouldn’t claim to be an expert in Linux or a MySQL guru (yet), but if my experience here is any indication, it won’t be too long before I know everything there is to know about every technology living in our data centers. When I run into a problem or a question I don’t have the answer for, I can rely on my coworkers to have the solutions and break them down into terms I can understand if they’re overly complex.

Would I recommend this job to others? Most definitely! This has been one of the best jobs that I’ve ever had. I’ve been able to take what I learned in school and actually apply it to my daily work life while continuing my real-world on-the-job education. If you have a server in DAL02 and need someone to check out the hardware or add some RAM, I might be the one jumping to get your request fulfilled quickly.

I’m proof that SoftLayer invests in its employees, so if you’re interested in an amazing job for a company who values you, I want you to be a coworker! We have positions in all departments available in Dallas, Houston, Singapore, Amsterdam, San Jose, Seattle and Washington, D.C. (and probably more location in the near future), so keep an eye on the SoftLayer Careers page for the perfect opportunity to join our team.

-Rochelle

About a month ago, I joined SoftLayer, and I feel like I’ve been welcomed into a big new family. I live in the Dallas area, and when I saw a listing from SoftLayer for server build technicians on craigslist, I sent in my résumé and anxiously awaited a response. Two weeks later, I got that response in the form of a phone interview with my soon-to-be manager, and since you’re reading this post on SoftLayer’s blog, it was clearly a great fit.

I am a Server Build Technician (SBT), and I’m part of the team responsible for building servers for new customers and maintaining our current server fleet. It’s a rewarding feeling to know that the work we do helps customers we’ve never met (and might not ever meet) run their businesses. I personally think we have one of the most important jobs in the company, and it’s one you might not ever see if you’re not looking for it.

One of the most important things I was looking for when I joined SoftLayer was a company that takes pride in its people. That attitude energizes me and my fellow employees, and it really makes it fun to come to work. Maybe that’s why it feels like a family. SoftLayer invests in its employees, and once you’re a part of the company, you have a bright future ahead of you. Whether you’re looking to move up the career path in your department or apply for a position on a different team, Lance and the management team have made it clear that they want us to succeed.

When I became a part of the SoftLayer team, I felt like I found a home away from home. The people I work with are awesome, and if you’re awesome, I want you to come work with us too. SoftLayer’s customer base is growing like crazy (as Tom explained in his video interview last week), so our team is growing as well.

The last time I checked, we had more than 25 available positions in in at least six different departments … One of them is probably a great fit for you. Visit our careers page to get a full list of opportunities: http://www.softlayer.com/about/careers/

Hopefully, I’ll get to meet you and welcome you to the family. Rock on!

-Dayrum

<Fade In>
Man driving into Jiffy Lube, car sputtering and smoking.
Attendant: “Looks like you need an oil change buddy.”
Buddy: “Yep, I think so. I was here last week and I think they used the wrong oil!”
Attendant: “Nah, we wouldn’t do that. In fact we only have one kind of oil here and that’s SAS 70.”
Buddy: “Well, that’s odd; I am told that I need SSAE 16 for mine to work right.”
<Mass Confusion>

Welcome to my world! We have SAS 70 today, but soon we will have the new synthetic, non abrasive, engine-cleaning SSAE 16. Sounds fun right? I sure hope so.

Why the change? Good question. When SAS 70 first appeared in the early 90s, the world’s economies weren’t quite as intertwined as they are today. It was much harder to do business globally than it is now. (I think the “fad” called the internet has a little something to do with that but I could be wrong!) Now that the oceans have shrunk to a more manageable size, there is a need for the standards that companies use worldwide to match more closely. The goal of the U.S. Statement on Standards for Attestation Engagements 16 (SSAE 16) is to meet a more uniform reporting standard.

What’s the difference? It’s an “attestation” not an “audit.” Google and thefreedictionary.com define attestation as “To affirm to be correct, true, or genuine,” and audit as “an inspection, correction, and verification of business accounts.” Though they are closely related, they mean different things.

What stay’s the same? The focus will still be on controls at service organizations when the controls are relevant to their user entities’ internal control over financial reporting. (For some reason, servers tend to have quite a bit to do with that!) There will still be a Type 1 and Type 2 with similar scopes in format. The reports will look very similar but they should be a bit more descriptive. The report will still be used in the same methods and by the same type of user.

What Changes? SSAE 16 is now an attestation and not really an audit. The service auditor will still provide an opinion but it will align itself more closely with existing international attestation standards.

• Written Management Assertion – Management will be required to provide an assertion, to be included in the report, stating the system is fairly represented, suitably designed and implemented and the related controls were suitably designed to achieve the stated control objectives, and that the controls operated effectively throughout the period. The report will reference that management is responsible for preparing the system description, providing the stated services, specifying the control objectives, identifying the risks, selecting the criteria and designing, implementing and documenting controls that are suitably designed and operating effectively. The auditor’s opinion remains in the role of providing assurance, not as the entity responsible for the communication.
• System Description – The more inclusive description must detail the services covered, classes of transactions, events other than transactions, report preparation processes, control objectives and related controls, complementary user controls and other relevant aspects of the organization’s control environment, risk assessment process, information and communication systems, control activities and monitoring controls. (I think an accountant came up with all of that!)

There are quite a few other differences but I think these are the big headliners. SoftLayer is committed to making this change and having it available for our customers that require it. Our normal SAS 70 schedule is Nov. 1 – Oct. 31 but we will be accelerating the process to have the SSAE 16 in place as soon as possible.

We are continuously looking at other compliance, reporting, audits and certifications. If you have any that would help you and your business, let us know.

-Skinman

Any place you spend hours upon hours every week, you tend to get to know people pretty well. During my time on my shift, I’ve gotten to know my fellow SLayers quite well. You learn their favorite foods (I’ll tell you right away what almost everyone on my shift orders from our favorite Chinese deliver place – almost like clockwork), their choices in music, their favorite computer games (yes, WoW tops this list, seemingly a standard in the IT Industry), and even how they react in certain situations. While the operations team is a tight knit group, we also have our extended families in SLales, InfoSys, Dev, and the other departments.

As many know, they recently opened up our new HQ. Yes, it’s as awesome as every has said it is. Lance made it a point to acquire a Sonic-style ice machine, our facilities are state of the art, and the building is just simply awe-inspiring. I suppose you could consider it our new home. As a new home, we’ve recently brought all of our departments together under one roof. Operations can grab a quick break with SLales, Dev can bounce some questions off of the CSAs or SBEs simply by walking a few feet, and even better, the friendly trash talking and joking just got easier (we’re a family, of course we’re going to mess with each other every chance we get!).

I’m sure employees everywhere are akin to this metaphor, but here it seems to ring true. We have our family quarrels, we know each other like the backs of our hands, and when things get heavy, we’re there to watch each others’ backs to ensure the mission is accomplished, and we continue to dominate the hosting industry.

I have a friend who recently took a CTO position with a medium-sized company. The huge company that previously employed him moved their entire IT staff a long way outside of Texas to a rather unpleasant location as a cost cutting move. He and many others declined the relocation offer. I can’t say as I blame them.

The other day, he told me some of the interesting things he’s found at his new company. This company is not a technology company but a professional services company. Up to now, they have opted to be in the IT business by running their own data center. To keep this post to a reasonable length, I’ll just mention a few of the things he’s run into.

Keeping the room powered and cool – trust me, this is harder than it sounds. It involves things like redundant power, UPS devices, generators, CRAC units, dehumidifiers, fire suppression, etc. All this stuff must be tested and maintained constantly.

Ordering new servers – they have to go through an online configurator, and then wait to receive the shipment. Once it arrives, they have to unpack it, rack it, power it up, and install the software. The cycle time from ordering a new server to getting it into production can stretch from days to weeks.

Tracking assets – needless to say, he’s found several holes in the process here. Knowing how much RAM is supposed to be in each server vs. what’s really there is a struggle. Heck, even knowing what servers are supposed to be there is a challenge. It seems that as servers are moved, replaced, or disposed of that the asset tracking system and processes are not as solid as he would like. These loose operations also bring heat from accountants and auditors, especially if a server ‘s value is still on the balance sheet but it has actually been tossed out and they no longer own it.

Maintenance – they pay for a service agreement where a tech is guaranteed to be onsite in 4 hours to do anything up to a complete rip and replace to get them back in production. Once he asked why several servers, each north of $10,000 in value, were just laying around in a parts cage. He was told these were for spare parts in case of an emergency, just in case they couldn’t wait 4 hours.

Bankers and lessors routinely ask us who our biggest competitors are. We routinely tell them that they are not other hosting companies – they are companies like the one described above that insist on being in the data center business even though they are not IT infrastructure companies. Since these companies are our largest competition, let’s look at how SoftLayer beats the competition on the items listed above.

Keeping the room powered and cool – as a customer of SoftLayer, you simply don’t have to worry about all this. Not at all. This is a huge savings of time, effort, and money.

Ordering new servers – Once you either run through the configurator or call your SLales rep with your order, your new servers are immediately provisioned. The cloud products are up in minutes, and you can have a few HUNDRED dedicated servers ready for production in a few hours. Not in days or weeks or months.

Tracking assets – From the accounting side of things, you just don’t have to worry about tracking the assets at all as a SoftLayer customer. They are an operating expense to our customers, not a capital expenditure. As far as knowing what assets you have to work with, you have access to the best customer portal in the business where every detail about every server is kept up in real time, right down to the individual sticks of RAM and drive configurations of each server. If you need tighter integration, SoftLayer provides an API to put all this information seamlessly into your environment. Disposing of a server is a simple cancellation ticket. It couldn’t be easier.

Maintenance – this is also a simple ticket submission, which is resolved in an impressive turnaround time. This service is included in SoftLayer’s monthly fees. There is no need to stockpile parts or entire servers for emergencies.

Bottom line, if your business’s core competency is not IT infrastructure, you are being beaten in the IT infrastructure business by SoftLayer. You are spending way too much time, money, and attention to run something that isn’t a part of your business. Hey, if you can’t beat us, then join us!

By the way, my friend is proposing a major project for his company in 2010. That project is getting out of the business of running a datacenter. He faces a lot of resistance to change “the way we’ve always done it” from the other senior executives. From my point of view, it’s a no brainer. But I’m biased I guess. I’d just tell them, hey, don’t run a data center – run your business!

Fuel!

Ask anyone here on our staff, and they’ll tell you a few things about their position:

1. It’s never boring
2. It can be quite demanding
3. We’re never technically “off duty”

That being said, we all need our fuel to keep us going at warp speed. Luckily at the NOC we’re lucky enough to have a fully stocked break room with all sorts of odds and ends to keep us going when the energy levels get low. Allow me to show a few of my personal favorites:

1. Chocolate Covered Raisins
   These little buggers are great when you’re running like mad and just need a quick snack. You can scoop up a good cupful and keep them at the desk for the remainder of your shift. You can take a little detour to grab a couple while en route to your destination. You can also trick yourself into thinking that they’re healthy since they have raisins in them.
2. Doritos
   These have made a reliable meal substitute on multiple occasions. A few bags of these can trick your hunger pains and quiet the ache for a few until you can grab an actual meal (not always a guarantee).
3. Coffee
   Any fan of caffeine knows why I’m adding this. It’s often the first thing ingested at the start of the day, and is famous for its energy-inducing properties. Love it or hate it, you cannot deny the eye-opening effects of this one.
4. Dr Pepper
   My second favorite carbonated beverage provided here. A quick drink and a quick pick me up.

And for my favorite:

RedBull!

Much like many of the techs here, I have a clinical addiction to caffeine. Caffeine is the lifeblood of the NOC and keeps us working at top speed and form. To date, I have found no quicker delivery of this than through the 8.2 ounce can of this elixir.

And there you have it. These are the snacks and beverages provided that keep me going. And while it’s no health food store, it certainly spikes the blood sugar or caffeine levels enough to sustain a happy and proficient technician through the long night hours.

Security is not a thing to be taken lightly. Think about the information that is stored on your server; think about how many months or years worth of data is stored in your databases. Your account information holds a master key to all of this data on your server. This is the very reason this information is protected so closely by the SoftLayer staff.

All companies work very hard to make sure that their products and services are as easy to use as possible. While on the other hand, security works as hard as possible to, seemingly, make the product or service difficult to use. While it is never our intention to make any service difficult to use, it is our intention to make them secure. This is the very reason why, when we are presented with any questions via phone that are sensitive to the operation of your server or account we ask the inconvenient questions to make sure the person on the other end of the phone line is authorized to make the requested changes to the account or the server.

Up to this point this article has not been as light hearted as I had originally intended, but it’s all about being prepared. The point is, everyone deserves a vacation at some point or another (or believes they do), and according to Murphy’s Law, something will inevitably occur that requires immediate attention. When you’re enjoying that time on the beach, your mind a million miles from bits and bytes, and you miles from anything that can be used to properly manage your server or your account an issue can occur.

While you are out, have you made proper provisions to ensure someone can manage your hardware in your place? Your staff may have the passwords for the servers, IP addresses, and may be able to drop your name; but, I assure you this is not enough information for the SoftLayer support staff to submit a ticket, reboot, or log into your server on your behalf. Have you made sure that in a panic situation someone will be able to provide us with the answers to the security questions on the phone? Are you sure whoever is left in charge has been given the proper permissions in our management portal? Making sure these points have been thoroughly covered prior to your vacation, or even leaving for the day, will help you minimize risk while maximizing your beach vacation.

Putting up Christmas lights this year was a serious beating. I kept blowing breakers due to the amount of lights I put up in response to my wife’s request for ‘more lights!’ It seems like every year things get bigger and bigger (like most things in America – trucks, combo meals, taxes, and the deficit). The problem is there is only so much power in convenient areas of my house and those locations don’t have enough power to run my lights because they are shared with things inside the house. My front porch outlet ties in with my garage outlets so every time we open up the garage door, the breaker blows and the Christmas lights on the front of the house go out. I got tired of resetting breakers and I ended up running 2x 20amp 110v dedicated feeds to my roof and to the front yard.

As I was putting the lights up, I found myself doing power calculations in my head. I multiplied the amount of lights I put up by the watts each bulb consumes to get the total watts. Then I took the total watts and put it into this conversion tool (http://www.mhi-inc.com/Converter/watt_calculator.htm) to calculate what they use in a Kilowatt hour. I have timers setup to turn on the lights from 6pm to 11pm (CST) so that is 5 hours a day. I plan to run them from December 8th through January 3rd which is 27 days totaling 135 hours of run time. Take the Kilowatt hour the lights generate times the hours of operation and you get the total Kilowatt hours used for the holiday season. I was then curious how much this was going to cost me (I am a cheap bastard) so I took out my electric bill (TXU, yes I am paying too much) and took what they charge me for a Kilowatt hour and got the dollar figure it costs to run the lights. I was surprised it is not as much as I thought considering how much light my house now generates. It lights up the neighborhood like the Griswold’s house in Christmas Vacation <http://www.imdb.com/title/tt0097958/> . I would not be surprised if you can now see my house from the space shuttle.

I don’t envy Softlayer’s operation guys because they do these types of power calculations (albeit on a much grander scale) on a daily basis. They have to figure out what types of servers with different components (CPU, drives, memory, raid cards) can go into a single rack to insure that power strips are not blown. Some people don’t understand that you can’t just fill a rack up with 44 1U (or 22 2U) servers and turn them on. You have to carefully plan down to the watt how many of each type of server can go into a rack without overloading circuits. You also have to take into account customer upgrades and make sure there is enough headroom for power spikes upon booting. The math involved in my yearly Christmas light escapade made my head hurt; I can’t imagine what Robert and Brad go though. Hats go off to them. My head would have exploded by now….

Here is the math (rounded):

15 ½ stands of C9 Christmas lights each with 25 bulbs = 385 bulbs
385 7 watt bulbs = 2695 watts
2695 watts = 2.695 Kilowatt hours (from http://www.mhi-inc.com/Converter/watt_calculator.htm)
2.695 Kilowatt hours multiplied by (5 hours a day for 27 days = 135) = 364 total Kilowatt hours
364 total Kilowatt hours times $0.12 = $44

So lighting my house for one month actually uses significantly more electricity than running a server in a SoftLayer data center for the same period of time.

It’s the holiday season, and that means everyone is getting busier. On top of all the existing responsibilities, millions of people are going shopping for gifts, decorating their houses, and navigating the bad weather. On top of all that, many people take their time off during the holiday season!

With this kind of time crunch, it’s best for your business to lie low until after the new year, right? Not so! With all this buying, selling, and giving going on, there’s a lot of extra retail data to process. Plus, it’s the end of the calendar year, many businesses have to get their finances in order too. ALSO, all these newly purchased electronic devices are soon going to be turned on and hooked up to the Internet, where they will almost surely put a new load on your servers.

Systems and network administrators need to be prepared for this influx of new traffic. Sometimes, this means purchasing new servers. However, it’s inefficient to buy the servers so far in advance when you don’t yet know what you will need. It’s best to wait until you’re sure you will need more servers and how many to order. At another hosting company, that would be a problem. People in our industry take the holidays off, too. Lowering the number of sales people and technicians and raising the number of new server requests would normally result in a disaster.

Luckily, SoftLayer does automatic provisioning. As soon as you order your server, it will be provisioned in two to four hours. Day or night, June 3rd or December 31st, if we have it, you can have control over it in two to four hours.

And therein lies the beauty of the SoftLayer system. You don’t have to wait for US to scale your business. If you need another server, get it. When it’s ready, it will automatically be added to your account’s private network and be available to you. You can even automate your server configuration and setup. Depending on the amount of data you need to transfer to a new server, you can have another server up and running your website less than 5 hours from the time you realized you needed it.

In fact, by using the SoftLayer API (and some clever configuration scripts on your servers) you can do live scaling on your website. Using the API, you can provision new servers exactly like the ones you already have. Once they’re available, a script can mirror the configurations from an existing machine to the new machine. Use the SoftLayer API once more to add the new servers to your load balancer rotation, and you’re in business! All without relying on any humans, even yourself! Treat yourself to some R&R this holiday season, while your website continues to get things done for you.

I just got back in town from Las Vegas, Nevada. That town is filled with stories and you can really love it or hate it, depending on the hour (or if you are like me whether you are arriving into McCarran or departing). I had a great trip this last go around and actually made money on the tables. However, when they say that what happens in Vegas stays in Vegas they are really talking about your money. Never forget that the house always wins. Always. Even if you win money you’ll wind up spending it on stuff out there and perpetuating your own good time. There isn’t anything wrong with this at all. In fact I plan on coming up on the short side of the stick on both the tables and on simply spending cash when I go out that way.

I think the really interesting thing that happens when you go through “the Vegas experience” is the perceived value of a dollar. You can take it for granted that all of a sudden you are transplanted into this fantasy world that is reminiscent of Pleasure Island from the story of Pinocchio and you’ll find that you have anything and everything you could want to do, eat, drink, or experience right at your fingertips. As this begins to progress the value of a dollar plummets quickly. You start overpaying for things at a whim, tipping bigger, making bolder and even just dumber bets. I did this and I can admit that I doubled down on my 11 when the dealer was showing a 10 in blackjack. It was blind luck that I hit it and won every single time. It’s a bold and stupid bet to make, but when you are playing with house money the money doesn’t matter and it’s almost as if you are trying to give it all back. My game of choice is craps because it gives you the best odds and there is a lot of action. It’s good and bad as it can all come and go in a hurry.

I have only been to Las Vegas a handful of times, but each time there is a point where even for a second you can feel invincible – that you can’t lose. Or, that even if you do lose you won’t even care. The flight home is a completely different story. I call it the hangover flight. You may be literally hung over, but no matter what, you will start to deal with all of the actions that happened on your trip and how you will need to handle them. As soon as you touch down in your own home town things slowly start to become “real” again. Your own home can even feel somewhat foreign for a while, but you’ll quickly come to the realization that you had become a completely different person for a short time.

I have come to the conclusion that there is always risk in everything that we do. Exposing yourself to the tables of Las Vegas may carry more financial risk than your morning commute to work, but in both cases there are still risks. There are also risks that we take in setting and running a business. There are countless ways that you could be putting your business at risk without the right plan in place. From an IT perspective alone, you need to consider things like redundancy, failover, security, backups, growth, and even data loss. Knowing what is going to happen next for your business may be as likely as knowing what is going to come up on the next roll of the dice. If you know this for certain you can press your luck and come up big, but if you are not prepared you could lose everything you have on the table. It is better to be prepared.

I think of SoftLayer as the house, and remember as I said before, the house always wins. The good thing about this is that you are betting with the house. Even with this you need to bet on yourself and back up your own bet. If the bulk of your business is in your data then you need to have backups. If you absolutely need to have High Availability, then look into Clusters and Load Balancing. But remember, that you are betting with the house because SoftLayer gives you the capacity to do all of it and do it all at a very affordable price compared to trying to do it yourself and also do it without long term commitments. Long term commitments bring the most uncertainty in making moves that will positively affect your business. Imagine if a casino told you that you “had” to make 12 consecutive bets regardless of how well (or poorly) you were doing?

Coming home from Las Vegas to SoftLayer has been a very good thing and makes me thankful for where I am and what I have. There aren’t the levels of uncertainty here that are automatic with other datacenters or even other business models. SoftLayer is steady and it is very easy to get what you need here while cutting out the risk that you don’t want to deal with. SoftLayer is as much of a “sure thing” as any bet you can make!