Posts Tagged 'Operations'

May 20, 2009

Dealing with Customer Service

No – this isn’t one of those blogs or editorials ranting and railing about how no one out there is able to provide good customer service anymore. This isn’t about how no one in the service industry – from restaurants to retail and everything in between – seems to care about the customer anymore. People have been writing those stories for the past 50 years (about half as long as they have been writing about the coming demise of baseball). This is just a short little missive lamenting how the same people that complain about lack of service are often people that work in the service industry themselves.

I often find myself in a retail store wondering why I can’t get help locating an object. Or in a restaurant wondering where the wait staff is. Or trying to work my way through an automated phone help system. Part of me sympathizes with the wait staff knowing that they are probably just too busy to get to my table. Maybe the restaurant is understaffed or maybe they have an unexpected rush of customers. And part of me even realizes the operational value of the automated phone system. The ability to reduce head count and lower costs with an automated system seems like a great idea (and sometimes it is).

But when I find myself in those aggravating situations and my anger is just about to get the better of me, I generally come back to the fact that myself and everyone else that works at SoftLayer is in the customer service industry. Oh, I might complain to a manager or I might tip less or I might shop at that location less. But more important than that, I try to use that experience as a reminder of how important customer service is. I’m not talking about just the ability to provide the product the customer is looking for – I mean the ability to be able to answer questions in a timely manner, to answer the phone as quickly as possible, to handle outages as quickly and professionally as possible, to provide customers with frequent updates and most importantly, to treat every customer interaction with the level of urgency that the customer thinks it deserves.

And THAT’s the important part – not just solving the problem, but making sure that the customer’s expectations are met.

-SamF

May 18, 2009

Special Ops: The “SEALs” of SoftLayer

When you think about a Special Operations Unit, you probably think of TV shows like, “The Unit”, or maybe you have the Military Channel and have seen the reality TV show, “Navy SEALs: BUD/s training”, or maybe you are one of the 7 people that saw that 1980s movie starring Charlie Sheen…………naaahhh. Anyway, whether it is secretive missions in Iraq or taking out pirates, real Special Ops Teams are very well trained individually and as a team. It takes a desire on the part of the individual to be the best at what he does and a desire to be an intricate part of a highly skilled, successful team.

I have been at SoftLayer for over 2 years now, and I particularly enjoy how our support team has come together in much the same way as a military special ops team. No, most of us do not wear our hair “high and tight”, and, unlike Navy SEALs, there are various piercings about the face of several of my teammates adding to an already very “distinctive” style of dress (There is a very loose dress code in the support department). But, the focused hours of training put into being the best at our craft is very similar to a special ops team.

I remember an occasion during my time here at SoftLayer when we had a sudden outage in which a switch failed. Any major data center will have a piece of equipment fail eventually, but the difference comes in how it is handled. Monitoring alerts went off and the team jumped into action. The managers and shift leads were instantly organizing, although the rest of us already knew what to do as training had prepared us. Each of us took a group of servers and checked for network connectivity in order to localize the issue. We fielded phone calls while the switch was being replaced. Tickets were answered quickly and grouped according to information needed by the specific customer. Verbal, IM, and email communication was flying and everyone knew the status from moment to moment. The switch was replaced and the event was concluded. Customers received the information they needed to pass on to their customers and peace was restored. I am amazed by the speed and efficiency with which this situation was handled. And, our customers were very happy with our speed especially considering an outage is never good news.

We would all like for everything to always work perfectly, but knowing that a highly efficient, highly competent, well-trained, focused, “special ops” team is ready at a moment’s notice, whatever the mission, is very comforting to our customers when financial success is on the line. Secretly, I have always wanted to be part of a special ops team, I just never imagined it would happen at a technology company called SoftLayer.

May 8, 2009

Interview with the Printer

SL: Hey, The elevator was acting strangely this morning. I wanted your opinion on a few things.
Printer: *whir*

SL: Excellent, I’m glad to hear your enthusiasm. I’ve been doing a lot of thinking lately about how the web hosting industry seems to be weathering the storm of the global economic downturn quite well. It seems regardless of the bank bailouts, failed mortgages, and credit crises, there is still a high demand for social networking applications, online shopping and exchange of information via the many forums available. Furthermore, with the reliability of our redundant links, businesses are finding it more affordable to outsource their IT assets, and host with us.
Printer: PAPER_EMPTY

SL: I Get it. By hosting here, you can also begin the transition to a paperless business. Something to the effect of a tech taking an X-Ray, uploading it to an SL server through the private uplink, and instantly having it available to a doctor thousands of miles away. And that’s just one possibility out of the endless uses for a server here. We have a ton of space available, and by design, we use approximately a square foot per server in our Datacenter… How’s that for space efficiency?
Printer: WARMING_UP

SL: Yea, it gets pretty toasty in there, but luckily we have environmental controls in place to mitigate the heat put out by the thousands of servers in the pods. We also have monitoring in place to notify us of any possible situations. As uptime is vital in the web hosting industry, we have a number of features available both internally and externally. We have 24/7/365 monitoring, automatic reboots, and a highly intuitive customer portal. Not to mention the best technicians in the industry
Printer: *beep*

SL: Now that’s just rude. We have staff from every facet of IT working in our NOC every day of the year. With the highly skilled staff holding years upon years of experience, there are few issues that can’t be solved quickly and efficiently.
Printer: PAPER_JAM

SL: Doubtful. We continue to innovate in efficiency and features. Now you’re acting just like the elevator.
Printer: PC_LOAD_LETTER

SL: Ugh, you’re just as bad as the Elevator. What does that mean, anyway?

May 4, 2009

Paradigm Shift

From the beginning of my coming of age in the IT industry, It’s been one thing – Windows. As a system administrator in a highly mobile Windows environment, you learn a thing or two to make things tick, and to make them keep ticking. I had become quite proficient with the Active Directory environment, and was able to keep a domain going. While windows is a useful enterprise-grade server solution, it’s certainly not the only solution. Unfortunately when I made my departure from that particular environment, I hadn’t had much exposure to the plethora of options available to an administrator.

Then Along comes SoftLayer, and opens my eyes to an array of new (well, at least to me) operating systems. Now, I had begun my ‘new’ IT life, with exposure to the latest and greatest, to include Windows, as well as virtualization software such as Xen and Virtuozzo, and great open source operating systems such as CentOS, and FreeBSD. With the new exposure to all these high-speed technologies, I felt that maybe it was time for me to let the de-facto home operating system take a break, and kick the tires on a new installation.

I can say that while switching to open source was a bit nerve racking, it ended up being quick and painless, and I’m not looking back. I’ve lost a few hours of sleep here and there trying to dive in and learn a thing or two about the new operating system, as well as making some tweaks to get it just like I like it. The process was certainly a learning experience, and I’ve become much more familiar with an operating system that, at first, can seem rather intimidating. I went through a few different distributions till I settled on one that’s perfect for what I do (like reading the InnerLayer, and finishing the multitude of college papers).

The only problem with always reloading a PC is you have to sit there and watch it. It doesn’t hurt to have a TV and an MP3 player sitting around while you configure everything and get the reload going, but you still have to be around to make sure everything goes as planned. Imagine this… You click a button, and check back in a few. Sound Familiar? Yep, it would have been nice to have an automated reload system much like we have here at SoftLayer. Not to mention, if something goes awry, there’s the assurance that someone will be there to investigate and correct the issue. That way, I can open a cold one, and watch the game, or attend to other matters more important than telling my computer my time zone.

May 1, 2009

What A Cluster

When you think about all the things that have to go right all the time where all the time is millions of times per second for a user to get your content it can be a little... daunting. The software, the network, the hardware all have to work for this bit of magic we call the Internet to actually occur.

There are points of failure all over the place. Take a server for example: hard drives can fail, power supplies can fail, the OS could fail. The people running servers can fail.. maybe you try something new and it has unforeseen consequences. This is simply the way of things.

Mitigation comes in many forms. If your content is mostly images you could use something like a content delivery network to move your content into the "cloud" so that failure in one area might not take out everything. On the server itself you can do things like redundant power supplies and RAID arrays. Proper testing and staging of changes can help minimize the occurrence of software bugs and configuration errors impacting your production setup.

Even if nothing fails there will come a time when you have to shut down a service or reboot an entire server. Patches can't always update files that are in use, for example. One way to work around this problem is to have multiple servers working together in a server cluster. Clustering can be done in various ways, using Unix machines, Windows machines and even a combination of operating systems.

Since I've recently setup a Windows 2008 cluster that is we're going to discuss. First we need to discuss some terms. A node is a member of a cluster. Nodes are used to host resources, which are things that a cluster provides. When a node in a cluster fails another node takes over the job of offering that resource to the network. This can be done because resources (files, IPs, etc) are stored on the network using shared storage, which is typically a set of SAN drives to which multiple machines can connect.

Windows clusters come in a couple of conceptual forms. Active/Passive clusters have the resources hosted on one node and have another node just sitting idle waiting for the first to fail. Active/Active clusters on the other hand host some resources on each node. This puts each node to work. The key with clusters is that you need to size the nodes such that your workloads can still function even if there is node failure.

Ok, so you have multiple machines, a SAN between them, some IPs and something you wish to serve up in a highly available manner. How does this work? Once you create the cluster you then go about defining resources. In the case of the cluster I set up my resource was a file share. I wanted these files to be available on the network even if I had to reboot one of the servers. The resource was actually combination of an IP address that could be answered by either machine and the iSCSI drive mount which contained the actual files.

Once the resource was established it was hosted on NodeA. When I rebooted NodeA though the resource was automatically failed over to NodeB so that the total interruption in service was only a couple of seconds. NodeB took possession of the IP address and the iSCSI mount automatically once it determined that NodeA had gone away.

File serving is a really basic example but you can clustering with much more complicated things like the Microsoft Exchange e-mail server, Internet Information Server, Virtual Machines and even network services like DHCP/DNS/WINs.

Clusters are not the end of service failures. The shared storage can fail, the network can fail, the software configuration or the humans could fail. With a proper technical staff implementing and maintaining them, however, clusters can be a useful tool in the quest for high availability.

April 29, 2009

Musician's Mind

One thing I have noticed about my SoftLayer family is the number of musicians here. I spent nine years as a musician, working the bar/festival circuits all over the midwest. When I arrived at SoftLayer, people jokingly asked if I was joining the SoftLayer band due to my previous experience. Just looking at the Operations Management Team, most of us have written/performed and many continue to. The more I thought about it, I think that this is a good thing.

There have been a number of scientific research projects about the academic performance of children and teens who are involved in a music program at school. The mixed left/right brain activity of music lends itself to problem solving and critical thinking as well as creativity which are required by many jobs. A musician's mind is capable of working complex geometric patterns into physical movements in coordination with muscle memory. Once more advanced levels are reached, a musician is capable of not only composing music, but also improvisation. These aspects are similar to many necessary thought processes used in the office.

I'd like to think that these mental processes allow me to think better on my feet, deal with change, and have a global view of the projects that I am involved with. Maybe that's why our Inventory, Hardware, & 2/5 Datacenter Managers are musicians. Not only that, but our Director of Operations is an avid musician. If you count the actual system admins who are musical, then you would be adding another 3 people.

So if all the research is correct, this may have something to do with the quality of operations here at SoftLayer. I wonder if being a musician had anything to do with my interview and its result. I know that it would catch my attention knowing that an applicant had experience with music or any other analytical/creative endeavor, especially if it has been shown to improve overall performance or intelligence.

Maybe I should write a SoftLayer song - an anthem to our Datacenter or a love song about the management network and IPMI?

April 8, 2009

More “SLingo”: SLanket vs. Snuggie

I am not ashamed to say that I own both the Slanket and the Snuggie. I got the Snuggie as a gift last year, and I purchased the Slanket for myself a couple of months ago (only partly because of the cool SL name). The Slanket turned out to be a WAY better product. As I sat on the couch with my Slanket and my kitten Linux, I started drawing these hilarious and uncanny parallels between the Slanket and my company, SoftLayer. Once I got started, I just couldn’t stop:

  1. The Slanket is bigger, and thicker. The fabric of the Slanket is far more robust, and doesn’t start “pilling” immediately. It’s designed for constant and vigorous use, and will last me a very long time before wearing out.
  2. There are more buying options with the Slanket – they come in different sizes and many more colors (including a shade that’s *almost* SoftLayer Red), so there’s a Slanket for every need.
  3. The Slanket has a better name – Sleeves + Blanket = Slanket. I’m not likely to talk about my “Snuggie” to my peers, because … well, I’m not a 4-year old.
  4. While the Slanket is a bit more expensive, it is a far superior product. In the Blankets-With-Sleeves industry, you seem to get what you pay for.
  5. I used the Contact Us form on the Slanket website to talk to their SLales team, and got great, personalized service, and a discount! Btw, the owner’s email address is also available online, which shows that he is involved in the day to day operations, and wants to hear from you.

Does any of this sound familiar? These are the same things people love about SoftLayer. I am a very proud SLanket customer. A++ would buy again.

SLanket

March 26, 2009

Use Caution when Outsourcing!

Outsource IT! I have been saying that for years now. But now I say; outsourcer beware!?!?! Really? How do you know if the company you are calling upon to keep your business up and running is safe and sound? Do they have certifications? Are they registered with the Better Business Bureau? Do they have scary fine print in the Terms of Service or User Agreement? Do you actually read those and understand them? How do you find out about all the questions above? Do you go to trade shows? Do you read about companies on the Hosting forum sites? Do you hear it from your friends? There are lots of ways to get that kind of information in today’s social internet jungle. Do you follow the company on Facebook, Twitter, MySpace, Linked-In, or all of the above? Should you? So many questions…

I am going to assume that you think this blog is going to be about how SoftLayer is a reputable, certified PCI compliant and SAS 70 datacenter, with competent and caring employees that can put themselves in the customer’s shoes and understand the frustrations that can go along with outsourcing your datacenter needs. Nah, that would be too easy and not very much fun.

This blog is about mud. Yes, I said mud. I was driving down a county road in Texas recently and we had a bit of rain in the days leading up to my trip. If you aren’t from Texas then you need a quick definition of “County Road”. A county can be paved, gravel or dirt topped and can be a great road or a horrible road, it just depends on the county that it is in, the tax base, and the abilities of the crews hired by the county to maintain them. I was travelling down a very wet gravel top county road, following along on my cell with GPS and Google maps and was about a mile from my destination. In what seemed the blink of an eye the road surface went from wet gravel to dirt and within about 10 feet my truck simply slid off the road into a nice 4 foot ditch filled with rain water. Looks harmless in the picture below doesn’t it?

Mud

It was a nice soft splash landing but my city slicker tires had no chance of getting me out of that ditch even with 4X4 engaged. So when water started coming under the door into the cab of the truck, I knew it was going to be a bad hour or so. It was time to outsource. I called the ranch to see if they had anything that could pull me out but they said that I was in a pretty tough spot and didn’t think they could help. So what would any techie do, I googled mud towing in the closet town. Of course I picked the first place on the list and gave them a call. They said they had a mud recovery truck and they would be out in about 45 minutes. Awesome, just 45 minutes! This was at 4:30PM and it was pretty cold and still raining and the ditch was filling up even further with water. Outsourcer beware, I was expecting a “Mud Recovery Truck!” I had visions of monster trucks dancing in my head. Fail!

Mud

Now I have to say that there weren’t ten forums about mud towing in Navarro county that I could visit, or customer references readily available so I just had to take that leap of faith and trust in the skills of my saviors. And I have to give credit where credit is due, that truck really is a monster! It did things a Transformer would love to be able to do. It got stuck at least 30 times in the 5 hours it took them to get me out of the ditch. Yes, I said 5 hours. Did I mention that monster trucks can do very bad things to city 4X4’s? Thank goodness I have an Echo to drive back and forth to work.

So I don’t want to leave you hanging but my truck is in the shop now and I am still waiting on an estimate. Things I know are wrong; front right A-arm damage from forcibly pulling the truck over a stump in the ditch, alignment issues, check engine light on, cruise control doesn’t work anymore, passenger side back door pushed up about half an inch including damage at bottom from the same stump, muffler caved in and exhaust pipe dragging the ground, front bumper air damn ripped off and metal bumper bent outward, yea you guessed it the pesky stump again and last but not least I need an entire new jack assembly because it is either broken or lost in the mud or both I should say (attempting to jack the truck over the stump).

The moral of this blog, if you have the tools available to research the company you are going to outsource to and they have references be sure to use them. They might save you a $300 mud recovery bill and a $1000 deductible somewhere down the road.

June 24, 2008

A Little Philosophical Thought: The SoftLayer Family Tree

Somewhat picking up on the theme of the blog, “Here’s to you, that nerdy Sysadmin”, this is a sort of, “Here’s to you, our family tree of customers.”

I find it very interesting how everyone in the world has customers. Our customer’s, customer’s, customer’s, customer’s…and so on…depend on us here at SoftLayer. It is a sort of bottomless pit. Does it begin anywhere…or end anywhere. Does anyone NOT have customers?

For example:

Our customers have customers, who have customers, who have customer, who have customers, etc. We are the customer of INTEL, AMD, Seagate, etc. They are the customers of those that provide the material to make the hardware that they manufacture. The natural resources used to make the hardware are purchased from someone. That someone purchased the rights to those resources from someone, who purchased the rights from someone, who purchased the rights from someone, etc. I suppose somewhere upstream someone’s country went to war and took the rights to those resources from someone else. And, I suppose if you go far enough back, no one had “rights” to those resources before the land was “claimed”. As recent as 2005, the race was on to claim the land of the Arctic Circle. I suppose you could say that all customers began with a gift of land containing resources from God. And, SoftLayer gives free stuff to customers all the time…so…I guess God gave the land containing the resources to people…His customers!

Anyway, this necessity of life that we have termed “customers”, has been, and will always be, I suppose, the most important aspect of life in terms of survival. Without customers, basic necessities like food cannot be purchased. In other words, without customers, you cannot be a customer. And, if you are not a customer, you must, therefore, be dead. So, in terms on everyone being dependent on being a customer and having customers, we all depend on each other like a family of customers. And, we must take care of our family.

We know that our survival, here at SoftLayer, depends on our customers (our family) and that their survival depends on us. We take this responsibility very seriously and work very hard to provide for our family the way that we would like to be provided for.

In conclusion, I feel that we, here at SoftLayer, do a pretty good job of taking care of our “family”, and in turn, our family of customers do a great job of taking care of us. As we continue to grow together, our success will benefit each other for years and years to come.

*If you do not understand any of this, just write it off to the insane ramblings of a tired CSA at the end of a long, challenging, and yet satisfying day at Softlayer working for his family.

-David

Categories: 
June 12, 2008

Culture Shock?

After separating from the military about a year ago, I was sure I was in for a bit of a culture shock. As you may very well know, the military is very different from life in what we liked to call the “real world”. 24 hour duties, life on ship, the awful food, I was ready for a slight change of pace. Soon thereafter I stumbled upon a golden opportunity here at SoftLayer.

Little did I realize at the time that I would be on board with one of the fastest growing and most innovative companies in the industry. Looking back, I can see that one of the ingredients for our explosive growth is the culture that SL has… one as idiosyncratic as the military itself. Coincidently, the culture shock I prepared myself for seemed to be more of a shift in verbiage.

Allow to me to submit a few specifics:

  • Much like the Marine Corps loved to add “MC” before every single acronym, SoftLayer does much the same… (SLiki, SLales – you get the point).
  • There is a mindset here that the mission comes first. In this case, that mission is to provide our customers with the best possible hardware, support, and applications to accomplish THEIR mission.
  • The pride SL employees have of being part of the team is infectious, much like a close knit unit in the military. You can see this best when things go awry – fellow SL’ers rush to back up those in need. If you’ve read through the InnerLayer prior to reading my musings, you know that the team here takes great pride in being a part of SL’s success.
  • Much like the military, you’ve got your large mix of people, from all walks of life, each with something to add to the team. Those experiences conglomerate to further enhance our ability to innovate.

Needless to say, the things that made successful units and deployments while I was in, I now realize are the ingredients to a successful team anywhere, no matter whether you use MC or SL. I guess that culture shock I had prepared myself for wasn’t as bad or painful as I thought it would be. I’ve traded my rifle for a scan gun, and my camouflage uniforms for those SL Tees.

-Matthew

Categories: 
Subscribe to operations