Customer Service Posts

Giving a customer the ability to do something that they didn’t know they could (or even know was possible) can make for an exceptional customer experience.

I’ve had a season mini-pack of Dallas Mavericks tickets for a handful of years now and have always gotten the exact experience that I expected: The same seats every time, consistent food and drink, great entertainment, and a quality team on the court that wins considerably more often than not.

However, this year it’s been a little different. This year, they have thrown in several perks that cost them nothing or next to nothing but have made a huge difference in the overall experience.

One game in particular sticks out in my mind. A couple of weeks before a game against the Wizards, I got an email about a no-cost chance for me and one other person to stand in a high five line to give fives to the players as they came out for warmups. I had no idea fans actually got to do this, so I gladly signed up and took my 5 year-old son to the game. I had also received an invite from the sales rep to choose a date to spend the first half of a game in one of the suites, so I made it the same night.

That night, we joined a small group of people down by the tunnel before the game, and we got to give all the players, Mark Cuban, the Mavs Maniacs and even a few security guards high fives. My son was over-the-moon to “meet” his favorite players – Dirk, Kidd, and Jet – could hardly contain himself.

This game also happened to be the week before the Super Bowl. I only mention it because on the way to our suite, I was blinded by the biggest ring I had ever seen. It turned out to be a Super Bowl ring and the guy wearing it was James Harrison (the linebacker for Steelers that lost a bunch of money to fines for helmet to helmet hits last season), so I got to meet him and wish him luck for the big game.

Oh, and and I can’t forget to mention the free hats, shirts, and Roddy B. bobblehead.

Long story short, I probably couldn’t tell you who won the other ten games I went to this year, but I don’t think I’ll forget anything about this particular game.

The thing I took away from this experience is when you give a customer something above and beyond what is expected, however seemingly insignificant, you can monumentally improve their customer experience.

To bring it back around to SoftLayer, we give customers a great API – a REST API at that. We give them VPN, a private network, IPv6, and a fully provisioned server in a couple of hours. Each of these differentiators enables us to provide products and services that our competitors can only hope to imitate.

The first time the customer uses the API to automatically create a new Cloud Instance from their own program, it’ll be a Maverick-game experience. When they transfer data from Washington, D.C., to San Jose, CA, on our private network with zero bandwidth charge, they’ll feel like they’re high-fiving Dirk Nowitski. When they access their server over the free KVM over IP, they’re walking up to the suite and meeting a Super Bowl champion. And all of that is on top of a stable, speedy server environment!

What can we do to improve your customer experience?

-Brad

Since SoftLayer’s merger with The Planet in November, we have been working tirelessly to combine our legacy Orbit and SoftLayer customer portals, and we’ve got some great news: We’re ready to move all of our billing information and functionality onto the SoftLayer platform! The changes are designed to make managing your account quicker and easier. While change isn’t always welcome, when you see some of the new features and functionality in the SoftLayer billing portal, we’re sure you’ll be as excited as we are.

Once your Orbit account’s billing information is migrated to the SoftLayer portal, you will receive an email confirmation. As soon as you’re ready to start exploring the new system, you can log in at http://manage.softlayer.com with your master username and password. We recommend you use the master username to log in because some users may have access restrictions in the portal, and you need to be logged into a user that has accounting access. Once you are logged in, click on the “Administrative” tab near the top-left of your page. From the drop-down menu, you will choose “Accounting” to bring you to the billing-related section on your account.

Wait … Instead of just guiding you through the process via text, how about we walk you through a quick tour of the billing portal as a bit of show-and-tell?

In the Accounting section, you can retrieve invoices, check pricing and even see your next monthly invoice. As a legacy Orbit customer, you’ll also be happy to hear that when your billing information is moved to the new portal, PayPal is available as a payment method! Among other changes, you’ll also note that we have a One-Time Payment option to enable some flexibility in how your account is paid in a given month.

In the new system, you’ll also notice that order reconciliation is made much simpler. You can easily view invoices by type, date or status. You can even view invoices within a specified date range and save invoices in interactive PDF or Excel formats. Updates to your user and payment information are much more accessible, too.

Our interactive invoices make it much simpler to review your equipment and the costs on your account. The interactive PDF will give you a summary of all charges broken down by type and then by server. If you click on any one of your servers, you are instantly taken to the full pricing detail of that server by component. If you have any items not listed under a server on your invoice you can use our Associate Billing Orphans section to attach unassociated items to a server.

With these invoices, you can track your costs and equipment clearly to make sure the right gear is getting charged the right amount. You can even use our Show Next Invoice feature to project costs for the following month!

We hope you’ll be amazed at all the features you now have at your fingertips! Please give us your feedback so we can be sure all questions are answered!

-Nikki

Wow, I can’t believe it’s already 2011. I vividly remember what a big deal Y2K was and what I was doing that night. Note to self: It might seem like it was just yesterday, but it wasn’t, so you should probably stop telling people you’re still 29 years old.

Speaking of time flying, I’ve been at SoftLayer for three and a half years now. I was hired as Customer Service Manager and immediately started looking at ways to help our customers love us even more. I found some notes that I scribbled after my second week here and I notice some pretty interesting goals. Here are a few that I was able to decipher … some we knocked out of the park, some that are continuous efforts and some that we can still implement:

Completed:

• Implement ticket rating and survey to monitor and track support quality
• Build an on-boarding process to help new customers in their first 48 hours
• Streamline the cancellation process

Continuous Processes:

• Make sure current customers know they are more important than potential customers
• Teach our customers about our processes and procedures

To Be Completed:

• Include a link on the website and in the portal to get immediate feedback from customers
• Start using webinars to answer customer questions and share technical tutorials
• Create a customer advisory board to consult as we make business decisions about things like market expansions and new product releases

What does that mean? We still have ways to make our business even better for SoftLayer customers.

Enter my renewed focus: Customer Experience. In the next few weeks, I’ll be talking to internal groups and customers alike to find ways that we can improve our service, products, automation … and everything else for that matter. These discussions will involve every department in the company, so all is fair game.

Over the course of the next few months, we’ll share a few of the things we hear and what we think we can do to continue to improve the SoftLayer customer experience.

The main goal is to find the perfect way to incorporate the 4 areas above and others into our daily lives. I have a few ideas now like; the link on the website, ways to teach our customers, marketing and sales and the webinars, and we are well on our way to having customer advisory boards and user group meetings. If you have ideas, I will appreciate anything you have to offer.

-Skinman

“I, the undersigned, certify under penalty of perjury”, “We believe the following host has recently been compromised”, “I received the below unsolicited commercial e-mail”, are a few statements that we as The Softlayer Abuse Department receive on a routine basis. The responsibility of responding to these quite serious matters in of itself is what gives us our motivation and niche in the overall scheme of this company: the protection of our networks global reputation. Without a firm and diligent abuse department, many of our customers would experience extreme packet loss left and right. Some customers may be affected by another provider’s block on an entire subnet, due to a single server periodically attacking their network for a month. Others would assuredly have their IP addresses consistently listed in spam databases, and therefore restricting e-mail contact to most or all of their clients. So in order to help keep these things from happening; we need to ensure that any reported or detected abusive activities occurring on our network are thoroughly responded to. We do this by analyzing abuse reports, determining the nature of the issues, and if an issue is valid, a ticket is opened with the customer for further correspondence as we track the issues resolution. At the same time, we maintain communication with other organizations and providers to ensure that matters are quickly addressed.

While most issues are resolved, or are being resolved within 24 to 72 hours, some issues require a quicker response. One of these is Phishing sites, which need to be removed within a shorter time frame. Our procedures regarding these sites are due to the fact that they are one of the most dangerous and wide spread issues on the internet today. If you’re not familiar with, or just want to read up on some of the latest news regarding these sites, you can get everything you need to know at APWG’s (Anti-Phishing Working Group) website. Softlayer’s membership within APWG allows us access to the most recent industry level trends and activities for a range of abusive issues. This gives us a much greater insight and oversight to identify and resolve issues that are negatively affecting our network. I can’t speak too much publicly past the above general time frames; since most abuse work is to some degree like spam filters, immediate disclosure of detection methods and procedures would render them useless. However, I can say that we believe one of the most effective methods for combating phishing is consumer education. If users are familiar with how fraudulent operations work, they are more likely to recognize components of them when they see them and not become victims. In support of this concept, we encourage all of our customers to respond to phishing site ’take downs’ by replacing the phishing site with a redirect to the APWG’s phishing education landing page. This page is an informative document that explains to the user that they were about to become a victim of illegal activity, and goes on to explain phishing in more detail. Most people in today’s modern society won’t go too far out of their way to obtain new information regarding trends in cybercrime. As such, the moment in which someone is about to be the victim of a phishing scam is considered to the ‘teachable moment’. This is the moment that someone has clicked on a link that they believe goes to their banks’ website, but are redirected to an educational page about phishing instead. The page is also configured to work with a variety of different languages, based on the client browser settings. As more people encounter the APWG’s landing page instead of a phishing site, the faster phishing education will spread and the less number of potential victims will exist. You may find information on how to implement the redirect here.

One of the next most concerning matters that we address is, servers being used by unauthorized third parties to conduct some form of outbound attack. While each are in there own way malicious and need the same attention, here’s a few specifics on some of the general different types. Password Cracking/Brute Force – this is typically done by malicious content attacking multiple hosts simultaneously while attempting various username and password combinations, typically with a massive list of pre-defined words. One of the easiest ways to help protect a server against being effected is to change at least your SSH, FTP, RDP, to non standard ports and ensure that you have complex passwords. I would also advise enabling account lockouts after a certain number of failed login attempts. Another predominant type of malicious scanning is doing so on an entire netblock by checking each host within them to see if one or more ports are open per host, which is then reported back to a database for later use in the latter form of attack. Essentially anything that is in some way part of an intrusion attempt is a priority.

Next we move on to an area of abuse that has most likely affected all of us at some point in time – Malware. This is a very general term we use to describe any software that has been written with malicious intent. The possible functions and uses for malware are only limited by the imagination and the software platforms that they are built upon, assuming that the infection process doesn’t accidentally crash the server. Various forms of malware have been identified as responsible for every type of abuse issue noted in this article at some point in time. While at the same time, malware on your server is not the guaranteed reason it may be conducting outbound abusive activities. Most specific malware related tickets are in reference to a single or series of malicious files that are publicly accessible. These issues are often resolved quickly upon deletion of the file(s) in question. However, it is also equally as important to ensure that any security vulnerabilities that allowed these files to be uploaded are repaired, or you can almost guarantee that the problem will reoccur. Microsoft reported that during the 1st half of 2008, over 90% of system vulnerability and subsequent infections were attributable to ‘weak’ applications rather than malware targeting the operating system itself. – Microsoft S.I.R. Vulnerabilities within the application layer remained the predominant risk throughout the 2nd half of 2008 as well. Malware in general has remained a formidable electronic adversary through 2009 and on to the present. As such, it is very important to ensure that you are using the most current version of all installed applications, and that they were written by a trusted source in addition to the maintaining the operating system security.

One very common form of malware effecting servers is an IRC(Internet Relay Chat) bot. One bot alone can be responsible for the infections of countless other machines. This is commonly done by injecting malicious code into poorly written PHP scripts. However, the bigger problem with an IRC bot is the fact that it’s connected to an IRC Botnet Controller, which is capable of commanding massive amounts of infected hosts simultaneously. While these are typically used for spam or other similar illicit activities, there is still the potential for the infected servers to be involved with even worse situations. These are in effect: A virtual army that’s literally capable of taking small countries off of the internet grid. In June of 2007, the F.B.I. initiated operation ‘Bot Roast’ an ongoing investigation to locate the people behind the wires. But in the mean time, needless to say, these matters need to be addressed as soon as possible.

During our triaging of abuse reports, we also address the very common issue of Spam. The three major types listed in order of priority are: Phishing, General Fraudulence, and other infected hosts Spam. However, you may also be audited, if you will, with a Spam ticket regarding a mailing list one of your clients is operating. For additional information regarding email marketing and the industry’s best practices, spamhaus.org’s FAQ is a very useful resource.

Keeping the above in mind, there is also one last thing to consider; maintain a backup of all removed malicious content after it has been found. This evidence could prove invaluable to law enforcement, should the request for it be presented. We also encourage you to review your access logs to determine the source IP address(s) of any intruder or other malicious entity, such that you may report it to the appropriate organization. As it is with many other aspects of life, communication regarding these issues remains critical for timely and appropriate resolutions.

-Andrew Smith – Martinez

Mergers are a true test of a company’s mettle – the stress involved with working an integration process with the need to continue to drive the business is almost overwhelming at times (CBNO!). The activity that is going on around me at present is awe inspiring – everybody is pitching in to make this work. It is great to see that we are making it happen – just rewards for all the effort expended.

Mergers also mean change. And I don’t just refer to the fact that we are welcoming new colleagues to the SoftLayer team. I am also referring to the fact that we are changing how we do business. We are bringing together two different organizations and combining the best components of each to drive the most value for our customers.

This inevitably means that customers are going to see some change in how they interact with SoftLayer. For example, the portal is going to morph into something that is much better than what SoftLayer or the Planet were doing separately and new product additions will arrive by combining The Planet offers with SoftLayer’s automated (automagic from now on) implementation and service delivery. We think these things are big wins for the existing and new customers.

A tangible example that I can talk about now (we need time to get portals and new products to market!) revolves around SLAs. The new SLAs will work to the benefit of existing Softlayer and the Planet customers as well as new customers.

The SoftLayer SLA is improved upon by changing the SLA for hardware and hardware upgrades. The old world considered a 4 hour promise before credits started to accumulate; the new world moves that to 2 hours. In addition, service outage credits start accumulating after 30 minutes of down time versus 43 minutes under the old 99.9% uptime guarantee.

The Planet SLA is improved upon by introducing a hardware SLA across the board versus just for customers of a managed services product. In addition to the 100% uptime promise that we are keeping, we have erased the need for a customer to raise a ticket before the clock starts ticking. Once there is a problem, we start the clock running.

At the end of the day, the reason we have SLAs in place is simple. Service credits on next month’s invoice are of less value to you than the fact that the SLA is driving SoftLayer to deliver on a service promise. If it’s broke, we are going to fix it. Think of the service credits as a bonus – the real value is getting your stuff up and running again.

-Steven

Customers are the heart of any business. If you don’t have customers you will not have a business. Here at Softlayer we take the meaning of customer service to another level. Yes, we have a Customer Service department but we don’t view customer service as an event, but rather as a series of interrelated pieces of a process. Many departments within our organization in addition to Customer Service provide a degree of customer service whether it’s to an internal or external customer.

Softlayer employs the “best practice” approach to take the perspective of the customer in all that we do. Since our customers come in through our website, the Softlayer team is continually working to improve the online ordering experience with the customer usability in mind at all times. One of my first tasks here at Softlayer was to “act” like a customer and order a server. I then had to do a write up on the experience. I have to be honest; it required some thought as to what could be improved because this process has been fine tuned as it’s been done over and over again. I thought the website and ordering process was very well put together, intuitive and easy to use.

Another way a customer will experience Softlayer is through our portal. Now, working from the inside, the portal is quite overwhelming at first. As you spend more time on it you realize just how powerful it actually is. We have quite a few developers working on it daily and this is to ensure that the customer has every tool, option, and task at their fingertips so they can get the most from their servers. Being in the internet business we understand that a competitor is only one click away.

Also during my first week here, I spent several days working in the NOC. Aside from being thoroughly awed with our facilities and how neat and organized everything was, I was impressed as I listened to the techs on their calls with the customers. The calls are handled with customer satisfaction as the first priority. Yes, I even heard the challenging calls that were very difficult to handle. The techs remained calm and focused to solve the problem at hand. The quality of work depends on the quality of people in place so if you have any technical problems these are the folks you want to speak with.

According to Wikipedia, “Customer service is a series of activities designed to enhance the level of customer satisfaction – that is, the feeling that a product or service has met the customer expectation.”[i] Here at Softlayer we exceed the customer expectations and will continue to implement and improve best practice processes and initiatives to improve our customer’s experience.

So this is my first blog here at SoftLayer. I’ve worked here since February, but I am certainly very familiar with the industry. In a previous life I formed the sales department at one of our competitors and learned about the industry. Even though I worked at a competitor, I never heard anyone speak badly about SoftLayer, and in fact it was the ‘bar’ by which we measured ourselves.

Now that I work here, it is even more apparent how and why SoftLayer is the most respected name in the hosting industry. SoftLayer overall has the best reputation due to its people, innovations, dedication, and motivation of the entire team.

I work in the Customer Service department, and it is my responsibility to contact new clients to ensure that they are not running into any problems and to get some feedback on their experience thus far. I have heard virtually nothing but praise from any client I have spoken to (new or old) about their experiences here.

All in all, the only better thing than hosting at SoftLayer is to work at SoftLayer!

Working for a company like SoftLayer requires a lot of dedication. Our staff is on call 24/7, and is ready to react in any given circumstance. Our BlackBerry’s keep us in the know, and allow us to quickly communicate with each other via email, telephone, and SMS. It is this communication that allows us to rise above the competition.

One specific example of this occurred in the wee hours of the morning. As per usual, communication was established via email of a specific customer issue. Having updated the customer, and established the required email thread regarding the request, I proceeded with business as usual, being sure to keep this issue in the back of my mind to brief the inbound shift once they arrived. I was quite surprised as my phone rang shortly afterwards. It was one of the senior members of our management team. While I won’t name specifics, because it’s quite inconsequential, I was particularly impressed that at around 4:00 in the morning (on a Saturday, nonetheless), they were awake and reading their email. Occurrences such as this aren’t unusual. Our team, regardless of the level of responsibility is always ready to take ownership, or assist when necessary.

This says one thing very loudly, it doesn’t matter how high up in the food chain a member of the team is here. We all take customer concerns very seriously, and work diligently to ensure that customer requests are met in a timely fashion. This isn’t limited to the personnel in the NOC, or our management team. Our Sales team works endlessly to communicate with their customers. It’s not uncommon for a member of our sales team to sneak out while hanging out to answer a phone call or an email to ensure that our customers are taken care of. Our InfoSys and development team are frequently contacted in the middle of the night with issues that require their attention. They resolve those issues quickly, and allow us to continue on with keeping our customers up and running. Suffice to say, anyone who sports the three bars does so knowing that they’re the best in the industry. We all take pride in ownership of the issues, and making sure that they’re resolved quickly so that our customers can be successful.

I have a friend who recently took a CTO position with a medium-sized company. The huge company that previously employed him moved their entire IT staff a long way outside of Texas to a rather unpleasant location as a cost cutting move. He and many others declined the relocation offer. I can’t say as I blame them.

The other day, he told me some of the interesting things he’s found at his new company. This company is not a technology company but a professional services company. Up to now, they have opted to be in the IT business by running their own data center. To keep this post to a reasonable length, I’ll just mention a few of the things he’s run into.

Keeping the room powered and cool – trust me, this is harder than it sounds. It involves things like redundant power, UPS devices, generators, CRAC units, dehumidifiers, fire suppression, etc. All this stuff must be tested and maintained constantly.

Ordering new servers – they have to go through an online configurator, and then wait to receive the shipment. Once it arrives, they have to unpack it, rack it, power it up, and install the software. The cycle time from ordering a new server to getting it into production can stretch from days to weeks.

Tracking assets – needless to say, he’s found several holes in the process here. Knowing how much RAM is supposed to be in each server vs. what’s really there is a struggle. Heck, even knowing what servers are supposed to be there is a challenge. It seems that as servers are moved, replaced, or disposed of that the asset tracking system and processes are not as solid as he would like. These loose operations also bring heat from accountants and auditors, especially if a server ‘s value is still on the balance sheet but it has actually been tossed out and they no longer own it.

Maintenance – they pay for a service agreement where a tech is guaranteed to be onsite in 4 hours to do anything up to a complete rip and replace to get them back in production. Once he asked why several servers, each north of $10,000 in value, were just laying around in a parts cage. He was told these were for spare parts in case of an emergency, just in case they couldn’t wait 4 hours.

Bankers and lessors routinely ask us who our biggest competitors are. We routinely tell them that they are not other hosting companies – they are companies like the one described above that insist on being in the data center business even though they are not IT infrastructure companies. Since these companies are our largest competition, let’s look at how SoftLayer beats the competition on the items listed above.

Keeping the room powered and cool – as a customer of SoftLayer, you simply don’t have to worry about all this. Not at all. This is a huge savings of time, effort, and money.

Ordering new servers – Once you either run through the configurator or call your SLales rep with your order, your new servers are immediately provisioned. The cloud products are up in minutes, and you can have a few HUNDRED dedicated servers ready for production in a few hours. Not in days or weeks or months.

Tracking assets – From the accounting side of things, you just don’t have to worry about tracking the assets at all as a SoftLayer customer. They are an operating expense to our customers, not a capital expenditure. As far as knowing what assets you have to work with, you have access to the best customer portal in the business where every detail about every server is kept up in real time, right down to the individual sticks of RAM and drive configurations of each server. If you need tighter integration, SoftLayer provides an API to put all this information seamlessly into your environment. Disposing of a server is a simple cancellation ticket. It couldn’t be easier.

Maintenance – this is also a simple ticket submission, which is resolved in an impressive turnaround time. This service is included in SoftLayer’s monthly fees. There is no need to stockpile parts or entire servers for emergencies.

Bottom line, if your business’s core competency is not IT infrastructure, you are being beaten in the IT infrastructure business by SoftLayer. You are spending way too much time, money, and attention to run something that isn’t a part of your business. Hey, if you can’t beat us, then join us!

By the way, my friend is proposing a major project for his company in 2010. That project is getting out of the business of running a datacenter. He faces a lot of resistance to change “the way we’ve always done it” from the other senior executives. From my point of view, it’s a no brainer. But I’m biased I guess. I’d just tell them, hey, don’t run a data center – run your business!

Back in my younger days I had taken on a few new hobbies that strike fear into the hearts of most mortals; Auto Mechanics. While working on vehicles, especially your own, can be terrifying imagine placing a necessary part on incorrectly that would eventually lead to failure and possibly serious injury to yourself or others. Luckily I had some backup while working on my truck. The individuals at the auto shop, that I did most of my work at, were knowledgeable and always willing to help (when they could get to you).

One specific experience rings loudly in my head to this day (for more than one reason, as you will soon find out). While driving around San Diego, I noticed a rather unnerving grinding/squeaking noise emanating from one off my wheels. I quickly headed to the local parts store to pick up a new ball joint and proceeded to take on the attempt to fix it.

Long story short, after some nuts, bolts, cuts, and bruises, I finally got to where I needed to remove the part. One of the techs there suggested using a hammer to bang it out of the coupling (or whatever it’s seated in). Thirty minutes later, I was tired, frustrated, sore, and deaf. Come to find out there was a perfect tool at my disposal, which cut the job down to approximately five minutes. One of the individuals at the shop told me that “no one had been able to make it work”. Needless to say, I made a liar out of him.

Similarly, we provide the resources that allow our customers to quickly and effectively attack a possible situation and fix the problem. Our technicians are the best in the industry and our primary focus is to ensure that customer satisfaction is at its highest. We have several tools at our disposal and offer those tools, and guidance on using them. Also, much like the staff at the auto shop, we try and point our customers to the ‘tricks of the trade’. We have an extensive knowledge base that can assist with day to day problems.

Quite on the contrary, though, we never discourage using certain tools because we don’t think that they’ll work. As anyone in IT can attest to, just because something doesn’t work the first time doesn’t necessarily mean that it’s a lost cause. With the right instructions and a little bit of patience, a little can go a long way. The trick is to use the correct tools for the job. While an ICMP request will tell you that the server’s online, it won’t give you the reason for outages like output from /var/log/messages or event viewer would.

Luckily, I don’t think I’ll be changing a ball joint on my car any time soon but I’m always learning new tricks and tools that I can use to further enhance my proficiency and I’ll always be happy to share those tricks to you; our loyal customers!