Posts Tagged 'Operations'

December 14, 2009

‘Tis the Season to Get Things Done

It’s the holiday season, and that means everyone is getting busier. On top of all the existing responsibilities, millions of people are going shopping for gifts, decorating their houses, and navigating the bad weather. On top of all that, many people take their time off during the holiday season!

With this kind of time crunch, it’s best for your business to lie low until after the new year, right? Not so! With all this buying, selling, and giving going on, there’s a lot of extra retail data to process. Plus, it’s the end of the calendar year, many businesses have to get their finances in order too. ALSO, all these newly purchased electronic devices are soon going to be turned on and hooked up to the Internet, where they will almost surely put a new load on your servers.

Systems and network administrators need to be prepared for this influx of new traffic. Sometimes, this means purchasing new servers. However, it’s inefficient to buy the servers so far in advance when you don’t yet know what you will need. It’s best to wait until you’re sure you will need more servers and how many to order. At another hosting company, that would be a problem. People in our industry take the holidays off, too. Lowering the number of sales people and technicians and raising the number of new server requests would normally result in a disaster.

Luckily, SoftLayer does automatic provisioning. As soon as you order your server, it will be provisioned in two to four hours. Day or night, June 3rd or December 31st, if we have it, you can have control over it in two to four hours.

And therein lies the beauty of the SoftLayer system. You don’t have to wait for US to scale your business. If you need another server, get it. When it’s ready, it will automatically be added to your account’s private network and be available to you. You can even automate your server configuration and setup. Depending on the amount of data you need to transfer to a new server, you can have another server up and running your website less than 5 hours from the time you realized you needed it.

In fact, by using the SoftLayer API (and some clever configuration scripts on your servers) you can do live scaling on your website. Using the API, you can provision new servers exactly like the ones you already have. Once they’re available, a script can mirror the configurations from an existing machine to the new machine. Use the SoftLayer API once more to add the new servers to your load balancer rotation, and you’re in business! All without relying on any humans, even yourself! Treat yourself to some R&R this holiday season, while your website continues to get things done for you.

November 11, 2009

Viva Las Vegas!

I just got back in town from Las Vegas, Nevada. That town is filled with stories and you can really love it or hate it, depending on the hour (or if you are like me whether you are arriving into McCarran or departing). I had a great trip this last go around and actually made money on the tables. However, when they say that what happens in Vegas stays in Vegas they are really talking about your money. Never forget that the house always wins. Always. Even if you win money you’ll wind up spending it on stuff out there and perpetuating your own good time. There isn’t anything wrong with this at all. In fact I plan on coming up on the short side of the stick on both the tables and on simply spending cash when I go out that way.

I think the really interesting thing that happens when you go through “the Vegas experience” is the perceived value of a dollar. You can take it for granted that all of a sudden you are transplanted into this fantasy world that is reminiscent of Pleasure Island from the story of Pinocchio and you’ll find that you have anything and everything you could want to do, eat, drink, or experience right at your fingertips. As this begins to progress the value of a dollar plummets quickly. You start overpaying for things at a whim, tipping bigger, making bolder and even just dumber bets. I did this and I can admit that I doubled down on my 11 when the dealer was showing a 10 in blackjack. It was blind luck that I hit it and won every single time. It’s a bold and stupid bet to make, but when you are playing with house money the money doesn’t matter and it’s almost as if you are trying to give it all back. My game of choice is craps because it gives you the best odds and there is a lot of action. It’s good and bad as it can all come and go in a hurry.

I have only been to Las Vegas a handful of times, but each time there is a point where even for a second you can feel invincible – that you can’t lose. Or, that even if you do lose you won’t even care. The flight home is a completely different story. I call it the hangover flight. You may be literally hung over, but no matter what, you will start to deal with all of the actions that happened on your trip and how you will need to handle them. As soon as you touch down in your own home town things slowly start to become “real” again. Your own home can even feel somewhat foreign for a while, but you’ll quickly come to the realization that you had become a completely different person for a short time.

I have come to the conclusion that there is always risk in everything that we do. Exposing yourself to the tables of Las Vegas may carry more financial risk than your morning commute to work, but in both cases there are still risks. There are also risks that we take in setting and running a business. There are countless ways that you could be putting your business at risk without the right plan in place. From an IT perspective alone, you need to consider things like redundancy, failover, security, backups, growth, and even data loss. Knowing what is going to happen next for your business may be as likely as knowing what is going to come up on the next roll of the dice. If you know this for certain you can press your luck and come up big, but if you are not prepared you could lose everything you have on the table. It is better to be prepared.

I think of SoftLayer as the house, and remember as I said before, the house always wins. The good thing about this is that you are betting with the house. Even with this you need to bet on yourself and back up your own bet. If the bulk of your business is in your data then you need to have backups. If you absolutely need to have High Availability, then look into Clusters and Load Balancing. But remember, that you are betting with the house because SoftLayer gives you the capacity to do all of it and do it all at a very affordable price compared to trying to do it yourself and also do it without long term commitments. Long term commitments bring the most uncertainty in making moves that will positively affect your business. Imagine if a casino told you that you “had” to make 12 consecutive bets regardless of how well (or poorly) you were doing?

Coming home from Las Vegas to SoftLayer has been a very good thing and makes me thankful for where I am and what I have. There aren’t the levels of uncertainty here that are automatic with other datacenters or even other business models. SoftLayer is steady and it is very easy to get what you need here while cutting out the risk that you don’t want to deal with. SoftLayer is as much of a “sure thing” as any bet you can make!

November 9, 2009

Outstanding Tech Recognition: Destroyer Droid

A long time ago in a galaxy far, far away… our hero Romeo R., SoftLayer CSA was battling an endless sea of support tickets…

Ok so maybe it’s not that dramatic, but I was recently rewarded the Destroyer Droid award from our management team! In case you didn’t see the original post here it is:

http://theinnerlayer.softlayer.com/2009/outstanding-tech-recognition-droid-awards/

I’ve written two other blogs on TheInnerLayer but there was one in particular that I think brings me to where I am today, it was entitled “What it’s like to be a Datacenter Technician”. Today I am a shift lead for our Dallas Support Staff. In my previous blog I mentioned how DC techs wore several different hats (Hardware Engineer, Network Support, and System Admin) at any given time and we have to always be on our toes. Now being a Shift Lead of course, I still get to do all the fun stuff a Datacenter Tech gets to do, but now I have more focus on how to get all of the above working together and more efficiently.

Enter Destroyer Droid:

Getting more into the management side of things is an entirely different monster; you have all of your previous duties plus the duties of setting up the flow of work for the day. It’s given me an entirely different mindset on how SoftLayer works and what it takes to be successful. It can be quite the handful on some occasions, but I enjoy coming to work every day because of the challenge and the people I get to work with. I think whenever someone in a Shift Lead type of position receives an award it is a direct reflection of how the team as a whole is performing, and it wouldn’t be possible unless every single tech was on their game. If you’re reading this and your boss/manager does something similar to our recognition awards let me know in the comments!

Now if I could only get the guys to stop calling me the destroyer…

Categories: 
September 28, 2009

Game Time

It’s Sunday morning and I’m leaving the NOC to make my morning rounds of the Washington, D.C. datacenter. Grumpy and tired I walk through the double doors into the fluorescent glare of the server room. In 30 paces the colorful eth bundles of our servers come into view and then I realize. I love the sound of server fans in the morning.

The past year and a half at SoftLayer’s newest datacenter have been incredibly stressful and rewarding. Those who endured have gained my respect. Personal differences have subsided and camaraderie has formed. Of course anyone would wonder how many tech nuts does it take to make a clan? And from the glue of hardship was born Team Orange DOW2.

You might wonder why people who work together so much (sometimes for 12+ hr shifts) want to spend more time with each other. I mean, haven’t you had enough already? The answer is that we already have so much in common and finding a few extra hours to hang out together online is a joy we can’t get enough of. Of course, the entertainment value of an innovative RTS like DOW2 is multiplied immensely when played with friends. Of the other SoftLayer members of Team Orange DOW2 I am the newest to multiplayer gaming and am impressed by how much tech goes into it. Numerous options for in-game chatting (Team Orange uses Mumble, which has the least lag and cleanest interface), hi-powered video cards (1.5GB onboard ram!), dual core procs running on Win7 RC, live-streamed replays with on-demand libraries, and much more.

Everyone has heard the theory that gaming has pushed the boundaries of computing, but I believe it is more likely that datacenters like SoftLayer have pushed the boundaries of networking and helped make advanced tech more affordable to the ravenous mass of online gamers. The number of mega-powered game servers hosted by SoftLayer is a testament to the unholy integration of gaming and networking, and to that all of us closet gamers must say, “moar please!”[sic]

September 4, 2009

First Grade

Some of you know that it is now “back to school” time. Those of you who don’t know are the ones who are still probably able to attend happy hours. (Our esteemed CFO may not know, but only because he’s had lots of birthdays. I’m not saying he’s addled. I’m just saying.) Nearly everyone has heard of, if not read, the book “All I Really Need to Know I Learned in Kindergarten” by Robert Fulghum. This blog is kind of a take on that, but since The Boy just started First Grade, these are things of relevance we’ve learned about the rules and such for First Grade and that can be applied to things here at the ‘Layer.

So The Boy has tickets in First Grade, and if he’s bad – he’s got to pull a ticket and put it in a jar. Sounds like, the fewer tickets, the better. Kind of like here at SoftLayer, from our standpoint you don’t want to have to have a bunch of tickets to plow through. From a client standpoint, you certainly don’t want to have a bunch of tickets, especially from abuse, or you might get your server pulled. The lesson to be learned is to do what you can to keep those tickets at a minimum.

This year in First Grade, there are a bunch of boys, including The Boy. At the parent meeting after the second day of school, the two first grade teachers were explaining that because of the more than 2:1 ratio of boys to girls, the restroom breaks could be a potential nightmare. The uniforms have belts and shorts with buttons and zippers. The teachers said it takes forever for them to go, and they wanted the parents to tell their Boy that he needs to be able to, ah, how do we say it? Well, he needs to be able to whip it out, put it away and go. No smacking other bottoms and goofing off and giggling. Here at SoftLayer, we often have crazy deadlines, for example in development, which requires us to whip out some new technology on an expeditious basis. Just like First Grade, whip it out, put it away, and go on to the next project. (Unlike First Grade, there is lots of goofing off here at SoftLayer, such as with 10,000 bouncy balls and such. Since I’m in legal, if there is any smacking of bottoms, neither I nor HR want to hear about it. Lalalalala, Lalalala…)

One of the guys wrote a blog earlier this summer pondering if the things you learned in college are applicable to the “real world.” His conclusion was yes. My blog further confirms that the things you learn in First Grade are applicable to the “real world.” Here’s hoping The Boy can go 15 days without getting his ticket pulled, because if he does he gets some Krispy Kreme “football” donuts. And if The Boy gets some “football” donuts, that means The Mommy gets some, too.

August 18, 2009

Backups Are Not the Whole Story

Last night while making my regular backup for my World of Warcraft configuration, I thought about the blog and I didn't remember seeing an article that went into more detail than "backups are good" about backing up and restoring data.

If you've been around the InnerLayer for a while you will have noticed that backing up of data comes up periodically.  This happens because we frequently see customers whose world is turned upside down due to a mistyped command wiping out their data.  If you just thought "that won't happen to me... I'm careful at a prompt"... well, how about a cracker getting in via an IIS zero day exploit?  Kernel bug corrupting the filesystem?  Hard drive failure?  Data loss will happen to you, whatever the cause.

Data that is not backed up is data that isn't viewed as important by the server administrator.  As the title of this blog mentioned, backing up isn't the end of the server administrator's responsibility.  Consider the following points.

  • Is the backup in a safe location?  Backing up to the same hard drive which houses the live data is not a good practice.
  • Is the backup valid?  Did the commands to create it all run properly?  Did they get all the information you need?  Do you have enough copies?
  • Can your backup restore a single file or directory?  Do you know how to restore it?  Simply put, a restore is getting data from a backup back into a working state on a system.

Backup Safety
At a minimum backups should be stored on a separate hard drive from the data which the backup is protecting.  Better would be a local copy of the backup on the machine in use and having a copy of the backup off the machine, perhaps in eVault, on a NAS which is _NOT_ always mounted, even on another server.  Why?  The local backup gives you quick access to the content while the off-machine copies give you the safety that if one of your employees does a secure wipe on the machine in question you haven't lost the data and the backup.

Validity
A backup is valid if it gets all the data you need to bring your workload back online in the event of a failure.  This could be web pages, database data, config files (frequently forgotten) and notes on how things work together.  Information systems get complicated and if you've got a Notepad file somewhere listing how Tab A goes into Slot B, that should be in your backups.  Yes, you know how it works... great, you get hit by a bus, does your co-admin know how that system is put together?  Don't forget dependencies.  A forum website is pretty worthless if it is backed up but the database to which it looks is not.  For me another mark of a valid backup is one which has some history.  Do not backup today and delete yesterday.  Leave a week or more of backups available.  People don't always notice immediately that something has broken.

Restores
A good way to test a restore is get a 2nd server for a month configured the same as your primary then take the backup from the primary and restore it onto the secondary.  See what happens.  Maybe it will go great.  Probably you will run into issues.  Forget about a small operating system tweak made some morning at 4am?  How about time?  How long does it take to go from a clean OS install to a working system?  If this time is too long, you might have too much going on one server and need to split up your workload among a few servers.  As with everything else in maintaining a server, practicing your restores is not a one-time thing.  Schedule yourself a couple of days once a quarter to do a disaster simulation.

For those who might be looking at this and saying "That is a lot of work".  Yes, it is.  It is part of running a server.  I do this myself on a regular basis for a small server hosting e-mail and web data for some friends.  I have a local "configbackup" directory on the server which has the mail configs, the server configs, the nameserver configs and the database data.  In my case, I've told my users straight up that their home directories are their own responsibility.  Maybe you can do that, maybe not.  Weekly that configback data is copied to a file server here at my apartment.  The fileserver itself is backed up periodically to USB drive which is kept at a friend's house.

Categories: 
July 13, 2009

What a View!

I can easily define myself as the crazy one up in the Seattle Datacenter. I like to ride dirt bikes, street bikes, go fast on the water, ride in small airplanes, I could go on and on how my co-workers (and friends/family) may think I am crazy when it comes to Adventures.

What can I say, I like a challenging experience.

One of those challenging experiences is working at SoftLayer, always preparing to be ahead of the rest in this industry, we're constantly learning new technologies and taking leaps and bounds. That is the reason why I love my job so much, we're always working with the latest and greatest, learning new stuff. Speaking of leaps and bounds, I finally did something the other day I have always wanted too. I signed up for an Advance Free Fall Skydive class and jumped out of a Cessna at 13,000 feet. Free falling at terminal speeds towards the earth, At 12,000 feet I mock pulled my parachute 3 times, so the instructors who where both holding onto me by their hands only could see I learned what to do. 6000 ft came, I locked onto my altimeter, 5500 I waived hands off to the instructors and they deployed below me, and I pulled my rip cord. That all happened in about 50 seconds after leaping out of the plane. For the next 8 minutes I saw the best view in Western Washington I have ever seen. One of the thoughts that came to my mind is the only thing to relate to how I have ever thought something was this nice, was the first time I walked into a SoftLayer Datacenter and admired how well thought out and nice it was.

I'm glad to say 19 months into this job and being part of the Operations team in Seattle, I walk in each day to the datacenter and can say the same thing day in and day out. Let's hope I can say the same thing about my second jump in a few weeks.

June 22, 2009

Really?

In catching up on some of my blog reading, I ran across this blog by Jill Eckhaus of AFCOM (a professional organization for data center managers). Yes, I realize that article is four months old, but like I said – I’m catching up.

One of the things that really concerns me with articles and blogs such is this one are the repetitive concerns about “data security” and “loss of control” of your infrastructure. Both of those points are easy to state because they prey on the natural fear of any system administrator or data center manager.

System administrators have long ago come to realize that, in the proper environment, there is no real downside to not being able to physically place their hands upon their servers. In the proper environment the system administrator can power on or off the server, can get instant KVM access to the server, can boot the server into a rescue kernel to try to salvage a corrupt file system, can control network port speeds and connectivity, can reload the operating system, can instantly add and manage services such as load balancers and firewalls, can manage software licenses and naturally, can control full access to the server with root or administrator level privileges. In other words, there is no “loss of control” and “data security” is still up to the system administrator.

The data center managers are understandably concerned about outsourcing because it can potentially impact their jobs. But let’s face it – in today’s economy, the capital outlay required to acquire new datacenter space or additional datacenter equipment is extremely difficult to justify. In those cases sometimes the only two options are to do nothing or to outsource to an available facility. Of course, another option is to jeopardize your existing facility by trying to cram even more services into an already overloaded data center. If a data center manager is trying to build a fiefdom of facilities and personnel, outsourcing is certainly going to be a concern. One interesting aspect of outsourcing is – datacenter management jobs are still there; they are just at consolidated and often times more efficient facilities.

In reality, “data security” and “loss of control” should be of no more or less concern if you are using your own data center versus if you are doing the proper research and selecting a viable outsourcing opportunity with a provider that can prove it has the processes, procedures and tools in place to handle the job for you.

(In the spirit of full disclosure; I am both a local and national AFCOM member and find the organization and the information they make available to be quite useful.)

-SamF

June 1, 2009

Proper Care

Over the years I have had many motorized toys, including boats, cars, trucks, dirt bikes, quads, riding lawn mowers and others. I got my first mini bike when I was about 6 years old. That thing was powerful - it had a 4HP Engine on it. One day I was riding it on our 100 acre homestead and the chain broke. Well I just popped the kick stand up and left it there waiting for Dad to get home. Upon my father arriving I let him know the chain broke, he explained to me the proper maintenance one must do in order to keep a chain working, proper oiling techniques, making sure it has the right tension and more. A few years later I got my first two stroke dirt bike. I loved that thing! I rode it all weekend long and then I mixed the gas too lean and blew the top head of it. That’s when I learned how to maintain a 2 cycle engine. My uncle helped me rebuild the bike engine (or shall I say I handed him the tools, and he rebuilt it) With all motorized engines they need proper care and maintenance. I now take my car for an oil change every 4000 miles (even though they say it can go 5,000) and get everything checked out.

The same thing can be said for internet servers. Quite often I talk to people and they think they can just install their operating system, upload the applications they want to run and/or data they want to serve, and walk away from that machine for the next 12 months. That is wrong! Computer software is always updating and you need to stay on top of updating your software. Security threats are found hourly, Viruses are written daily to attack the threats found yesterday. Proper maintenance is the only way to make sure your data is safe and secure. That is why SoftLayer has partnered with companies that offer extended server management. We call them SoftLayer Certified Management Companies. You can find them in our forums. These companies like rackaid.com, seeksadmin.com, Bitpusher, and many more have all been certified by SoftLayer to know our infrastructure and work closely with us and many of our clients. They provide the same great level of customer service that is standard at SoftLayer and do a lot of the advanced administration tasks for our customers. We have teamed up with these managed services partners in order to provide our customers with the proper maintenance of their infrastructure. So if you haven’t done a security audit on one of your machines in a few months, I would suggest taking it to the service center and contacting one of these companies, so you can insure your machine is safe and secure!

May 28, 2009

Hardware Heros

The techs that build the servers here at SoftLayer are known as Server Build Engineers or SBE’s. These guys are on the front line of Operations. They are responsible for building out customer server orders, maintenances, fixing cranky provisions, and many other hardware related tasks.

One might think that a hardware tech is a simple job. Well, not the SBE position at SoftLayer. Not only are they responsible for time sensitive hardware builds and provision troubleshooting, but they work directly with all the other departments. We don’t have bazillion hardware techs like other companies might. We train ours up to be one man hardware machines.

Sometimes a provision might have a weird error that needs to be escalated to development. The SBE will work directly with the dev team to resolve the issue.

Sales might have questions about some hardware they are trying to sell. SBE’s answer the call.

SBE’s even jump in to help CSA’s (Customer Service Administrators) when the ticket load or phones get hectic.

SBE’s do numerous projects, too. From helping with large scale hardware compatibility testing to troubleshooting hardware, they are the jack of all trades at SoftLayer.

We have a pseudo paramilitary way of doing things in the hardware department. It’s all in fun, but we get down and dirty and have a “can do” and “yes sir” attitude. We pride ourselves in being able to tackle any problem. If we are asked to do it, we do it, regardless of whether or not it’s our job, we are too busy, or whatever the obstacle.

Be all you can be? Join the army. Be more than you expect you can be? Join the SoftLayer Hardware Team.

The few, the badass, the SBE’s!

Subscribe to operations