Posts Tagged 'Tickets'

October 25, 2012

Tips from the Abuse Department: Save Your Sinking Ship

I often find that the easiest way to present a complex process is with a relatable analogy. By replacing esoteric technical details with a less intimidating real-world illustration, smart people don't have to be technically savvy to understand what's going on. When it comes to explaining abuse-related topics, I find analogies especially helpful. One that I'm particularly keen on in explaining Abuse tickets in the context of a sinking ship.

How many times have you received an Abuse ticket and responded to the issue by suspending what appears to be the culprit account? You provide an update in the ticket, letting our team know that you've "taken care of the problem," and you consider it resolved. A few moments later, the ticket is updated on our end, and an abuse administrator is asking follow-up questions: "How did the issue occur?" "What did you do to resolve the issue?" "What steps are being taken to secure the server in order to prevent further abuse?"

Who cares how the issue happened if it's resolved now, right? Didn't I respond quickly and address the problem in the ticket? What gives? Well, dear readers, it's analogy time:

You're sailing along in a boat filled with important goods, and the craft suddenly begins to take on water. It's not readily apparent where the water is coming from, but you have a trusty bucket that you fill with the water in the boat and toss over the side. When you toss out all the water onboard, is the problem fixed? Perhaps. Perhaps not.

You don't see evidence of the problem anymore, but as you continue along your way, your vessel might start riding lower and lower in the water — jeopardizing yourself and your shipment. If you were to search for the cause of the water intake and take steps to patch it, the boat would be in a much better condition to deliver you and your cargo safely to your destination.

In the same way that a hull breach can sink a ship, so too can a security hole on your server cause problems for your (and your clients') data. In the last installment of "Tips from the Abuse Department," Andrew explained some of the extremely common (and often overlooked) ways servers are compromised and used maliciously. As he mentioned in his post, Abuse tickets are, in many cases, the first notification for many of our customers that "something's wrong."

At a crucial point like this, it's important to get the water out of the boat AND prevent the vessel from taking on any more water. You won't be sailing smoothly unless both are done as quickly as possible.

Let's look at an example of what thorough response to an Abuse ticket might look like:

A long-time client of yours hosts their small business site on one of your servers. You are notified by Abuse that malware is being distributed from a random folder on their domain. You could suspend the domain and be "done" with the issue, but that long-time client (who's not in the business of malware distribution) would suffer. You decide to dig deeper.

After temporarily suspending the account to stop any further malware distribution, you log into the server and track down the file and what permissions it has. You look through access logs and discover that the file was uploaded via FTP just yesterday from an IP in another country. With this IP information, you search your logs and find several other instances where suspicious files were uploaded around the same time, and you see that several FTP brute force attempts were made against the server.

You know what happened: Someone (or something) scanned the server and attempted to break into the domain. When the server was breached, malware was uploaded to an obscure directory on the domain where the domain owners might not notice it.

With this information in hand, you can take steps to protect your clients and the server itself. The first step might be to implement a password policy that would make guessing passwords very difficult. Next, you might add a rule within your FTP configuration to block continued access after a certain number of failed logins. Finally, you would clean the malicious content from the server, reset the compromised passwords, and unsuspend the now-clean site.

While it's quite a bit more work than simply identifying the domain and account responsible for the abuse and suspending it, the extra time you spent investigating the cause of the issue will prevent the same issue from happening after your client "fixes" the problem by deleting the files/directories. Invariably, they'd get compromised again in the same way when the domain is restored, and you'd hear from the Abuse department again.

Server security goes hand in hand with systems administration, and even though it's not a very fun part of the job, it is a 24/7 responsibility that requires diligence and vigilance. By investing time and effort into securing your servers and fixing your hull breach rather than just bailing water overboard, your customers will see less downtime, you'll be using your server resources more efficiently, and (best of all) you won't have the Abuse team hounding you about more issues!

-Garrett

P.S. I came up with a brilliant analogy about DNS and the postal service, so that might be a topic for my next post ...

August 15, 2012

Managing Support Tickets: Email Subscriptions

This week, the development team rolled out some behind-the-scenes support functionality that I think a lot of our customers will want to take advantage of, so I put together this quick blog post to spread the word about it. With the new release, the support department is able to create "Ticket Email Subscriptions" for different ticket groups on every customer account. As a customer, you might not be jumping up and down with joy after reading that one-sentence description, but after you hear a little more about the functionality, if you're not clapping, I hope you'll at least give us a thumbs-up.

To understand the utility of the new ticket email subscription functionality, let's look at how normal tickets work in the SoftLayer portal without email subscriptions:

User Creates Ticket

  1. User A creates a ticket.
  2. User A becomes the owner of that ticket.
  3. When SoftLayer responds to the ticket, an email notification is sent to User A to let him/her know that the ticket has been updated.

SoftLayer Creates Ticket

  1. SoftLayer team creates a ticket on a customer's account.
  2. The primary customer contact on the account is notified of the new ticket.
  3. Customer logs into the portal and responds to ticket.
  4. Customer gets notifications of updates (as described above).

There's nothing wrong with the existing support notification process, but that doesn't mean there aren't ways to make the process better. What if User A creates an urgent ticket on his/her way out the door to go on vacation? User B and User C aren't notified when an update is posted on User A's ticket, so the other users aren't able to get to the ticket and respond as quickly as they would have if they received the notification. What if the primary customer contact on the account isn't the best person to receive a monitoring alert? The administrator who will investigate the monitoring alert has to see the new ticket on the account or hear about it from the primary contact (who got the notification).

Ticket email subscriptions allow for customers to set contact addresses to be notified when a ticket is created, edited or moved in a particular ticket group. Here are the ticket groups differentiated in our initial release:

  • Billing - Any ticket in our Billing department
  • Maintenance - Scheduled maintenance notifications for specific servers
  • Network Protection - DDoS mitigation and Null Routes
  • Monitoring - Host Down Alerts
  • CST, SysAdmin and Hardware - Any ticket in our support and data center departments
  • Managed Services - Tickets that relate to any managed services
  • Network Maintenance - Scheduled network maintenance

You'll notice that Abuse isn't included in this list, and the only reason it's omitted is because you've always been able to designate a contact on your account for abuse-related tickets ... Ticket subscriptions extend that functionality to other ticket groups.

Because only one email address can be "subscribed" to notifications in each ticket group, we recommend that customers use their own distribution lists as the email contacts. With a DL as the contact, you can enable multiple users in your organization to receive notifications, and you can add and remove users from each distribution list on your end quickly and easily.

When User A creates a ticket with the data center and goes on vacation, as soon as SoftLayer responds to the ticket, User A will be notified (as usual), and the supportsubscription@yourdomain.com distribution will get notified as well. When a network maintenance is ticket is created by SoftLayer, the netmaintsubscription@yourdomain.com distribution will be notified.

Ticket email subscriptions are additive to the current update notification structure, and they are optional. If you want to set up ticket email subscriptions on your account, create a ticket for the support department and provide us with the email addresses you'd like to subscribe to each of the ticket groups.

We hope this tool helps provide an even better customer experience for you ... If you don't mind, I'm going to head back to the lab to work with the dev team to cook up more ways to add flexibility and improvements into the customer experience.

-Chris

February 23, 2012

How to Get the Best Customer Service (Anywhere)

Shelves of books have been written about providing great customer support, but I haven't seen many written about how to get great customer support. Lance wrote a quick guide called "The 8 Keys to Successful Tickets" in May 2007, but because there have been over 730 blog posts between that post and this post, I thought I might take a shot at the topic again without stealing too many of his ideas. When you work with a service-based company, you're probably going to interact with customer support representatives regularly. During these interactions, your experience will not be defined by your question or the issue you have. Instead, it will be defined by how you present your issue.

It can be extremely frustrating when a server goes down or a script isn't working the way it should. When something like this happens, my gut reaction is to get upset and throw my keyboard. I've also noticed that when I am angry, I have a difficult time trying to explain my problem to technical support. I know I'm not alone in that regard, so I tried to pinpoint the most important points to remember when contacting customer support. While some of the explanations below are more SoftLayer-specific, each of the tips below can be used in any situation where you need customer support.

  • Remember there's a human on the other end. It doesn't matter where the customer support representative is; they're human, and their responsibility is to help you. I don't have any empirical data, but human nature tells me it's easier to be nice to someone who is nice to you. Once you realize there's a person on the other end of the phone trying to do his/her job, it's a little easier to thank them in advance for their help. It may seem insignificant, but if you thank me in advance for my help, I'll subconsciously work harder in an effort to deserve that gratitude.
  • Don't assume your request will be ignored. I'm surprised by the number of people who start or end their e-mail with, "No one will probably see this, but ..." or "Not that anyone cares, but ..." Don't assume that you'll be ignored. That assumption just creates overarching negative tone; it isn't a "reverse psychology" play. The support process can be defined by the expectations you set for it, so get started on the right foot and expect that your questions will be answered and issues will be resolved.
  • Don't start with a threat. "If you don't do this, I'm going to report this to my bank and other authorities," or "If you don't respond within 25 seconds, you'll be hearing from my lawyer." It's not uncommon to hear things like this in the first message in a ticket. It's much easier to help someone who seems easy to help. Invoking lawyers does not make your ticket seem easy to address. :-)
  • Provide useful, descriptive and relevant information. This tip can be tough since it's hard to understand what information is "relevant," but think about it before you send a support request. If you are having trouble logging in, then "I can't log in. Any ideas?" is not quite as clear as "Whenever I try to log in, the login screen just reloads without an error message. I know my username and password are correct. Any ideas? Thanks." That extra information will help considerably and will reduce the number of back-and-forth e-mails between you and the support representative.
  • Don't write overly detailed, wordy support requests. The longer your e-mail, the more difficult it is to read, diagnose and to respond. A representative has to read the entire ticket to find what's meaningful and figure out exactly what's wrong. Since they're trying to help you, you want to reduce their burden. You want to make it as easy as possible for them to help you. So, be clear, concise and brief. If you've got a couple different issues for support to look at, break them out into individual tickets. Different issues may need to be addressed by different departments, so multiple issues in a single ticket can lead to delays in responding to specific issues in the ticket.
  • More Tickets ≠ More Support. The flip-side of the above recommendation is that you shouldn't create multiple support tickets for a single issue. While it seems like you're drawing more attention to the issue and creating a sense of urgency, you're really slowing down the support process. Support representatives might be addressing the same issue in parallel or information might be lost between tickets, elongating the time to resolution.
  • Escalate your tickets smartly. If you think a ticket should be handled differently or if you would like a supervisor to look into a specific issue, you should always feel free to request escalation to a manager or a supervisor. The best way to make that request is to update your open ticket, initiate a live chat or place a call into the technical support phone line. If you aren't satisfied with your support experience, then we aren't either, so we want to hear from you.

As you can see, the prescription is not too complicated: Prepare yourself to receive the best support and help us provide the best support, and you're much more likely to receive it.

-@khazard

December 5, 2011

Quick Tip: Copy and Paste from the DOS Prompt

Having worked in SoftLayer's technical support department for a few years now, I can tell you that the more information you provide us, the faster we can get you to a resolution. If you can show us exactly the problem you're seeing with details from when you see it, it's much easier for us to troubleshoot, so I wanted to post a quick blog on the heels of Todd's "Global Network: The Proof is in the Traceroute" post to help you get information to us much more easily.

Document Format
Many people consider a Microsoft Word document the lowest common denominator when it comes to formatting an attachment or file while others prefer plain text for everything. I always advocate the use of plain text. Plain text is universally accessible, it doesn't require a third-party application to view, it doesn't add funky encoding, and it uses monospaced fonts that format the text like you'd see in a command prompt if you were sharing troubleshooting results from ping and traceroute commands. It's quite unnecessary to take a screen capture of a ping or traceroute when you run it, and it's doubly unnecessary to paste that screen capture into a Microsoft Word document.

Copying Your Ping/Traceroute
The problem many Windows users run into is that it's not very clear how to copy text from the command prompt ... The familiar keyboard shortcuts for copying (CTRL+C) and pasting (CTRL+V) don't work from the DOS Prompt, so the screen capture route is usually the easiest to execute. There is an easy way to copy, though.

Microsoft documented the instructions you need, and I wanted to share them with SoftLayer customers here:

  1. Open the command prompt. If you're unsure how to do this, open the Start Menu, click Run, enter "cmd" (without the quotes) and click OK.
  2. Execute your command. Use "tracert softlayer.com" to follow along with this test.
  3. Right-click the title bar of the command prompt window, point to Edit, and then click Mark.
  4. Click the beginning of the text you want to copy.
  5. Press and hold down the SHIFT key, and then click the end of the text you want to copy (or you can click and drag the cursor to select the text).
  6. Right-click the title bar, point to Edit, and then click Copy.

Now the text is in the clipboard. You can paste it anywhere, including the body of a ticket. To preserve layout, I usually paste the text in Notepad and attach that file to the ticket. If you don't want to go through the hassle of opening Notepad, just paste the results into the comment field below.

If you enjoy reading quick tips like this one that can make life easier, be sure to check out KnowledgeLayer.

-Lyndell

Bonus tip: If you want to submit your traceroute in a comment on this blog without losing the mono-spaced formatting, surround the pasted content with the <code> and </code> tags.

January 10, 2011

#SoftLayerMysteryEvent: For Those About to Rock

If you've been watching @softlayer and @softlayer_news for the past week, you've probably seen a few mentions of a SoftLayer Mystery Event. In the words of Winston Churchill, "It is a riddle, wrapped in a mystery, inside an enigma."

The hints got more and more revealing as the week progressed, and any geek worth his/her salt could probably have used some Google-fu around Hint 6 to figure out what was going down. If you consider that a challenge, here are the clues we posted. Try to figure it out without peeking past this image:

SoftLayer Mystery Event

Did you figure it out or did you just scroll down here to get the details on the event? Well tehe wait is over: SoftLayer is sponsoring NAMM JAM. Headliner: Megadeth.

This Friday, January 14, SoftLayer joins some of the biggest names in the music industry to co-present a face-melting 34th anniversary party for presenting sponsor Dean Guitars at The Grove of Anaheim.

NAMM JAM

If you live in Southern California and you're interested in having your socks rocked off on Friday night, stay tuned to @SoftLayer and @SoftLayer_News for your chance to score a pair of VIP tickets to the event. The tickets are going to go to the folks who want them the most, so get ready to prove your love for SoftLayer and metal!

-@khazard

UPDATE: Phil and I are featured in a YouTube video about the event:

March 22, 2010

Oatmeal is Good for YOU!

Have you seen the commercials for Quaker Oats oatmeal? In recent years they have changed their traditional marketing message to appeal to a specific customer profile. The ads new message is that by eating oatmeal every day for breakfast for 30 days, you will lower your blood cholesterol levels. Pretty slick! Eat our oatmeal and you drop your cholesterol and participate in a healthy lifestyle. Net result, you are healthier, live longer, better quality of life, Yada, Yada, Yada….All this from the simple, inexpensive miracle food… oatmeal. Hey, no need for that expensive prescription medication to control your HDL or LDL, just eat a bowl of oatmeal every day!!

Now, you’re probably asking, what the heck is the point to this blog? Well, glad you asked! I want to share a story with you. This past weekend, I went on a hunting excursion to Central Texas to hunt wild hogs. There are a number of interesting tales to share about the actual hunting, and I’ll post those at a later date! This story takes place in a small town I passed thru (or tried to anyway) on the way to the hunting lease. Flying down Hwy 29, we were passing thru a small, one stop light town named Bertram. Big signs all over town advertise the fact (Proudly) that Bertram is the Oatmeal Capital of Texas. They even have an Oatmeal Festival! My buddy was in a truck ahead of me, and made it thru the light, but I was caught and had to stop. My buddy really wanted to get to the lease and kept truckin’, leaving me to apply a heavy foot to the accelerator (thank God I don’t have a Toyota) to catch up. Next thing you know Jed’s a millionaire, and I have the bubble gum lights going off behind me on the local law enforcement vehicle (their one and only). For those of you not familiar with small town Texas law enforcement, Big Brother Bubba looooves to pull over city slickers from the big city. We represent a steady, easy revenue stream for the local coffers. To contest any citation, you are required to show up in person, usually in the middle of the week, usually late in the day or in the evening. Hence, most people will just pay the fine and go on down the road. I digress, back to my story! Well, Officer Bubba, looking just like Sheriff Buford T Justice from Smoky and the Bandit fame (short stature, big belly hanging over his gun belt, cowboy boots and straw hat) ambles up to the window and goes thru the standard drill. I think he was disappointed because I had pulled over immediately and had license and registration waiting for him! I quickly realized from his demeanor I had zero chance to talk my way out of the ticket, but gave it the old college try of “hey, I’m following my buddy, he made the light and blew ahead, and I’m just trying to catch up so I don’t get lost” explanation, but no good… Oh well! So, after a short wait, Officer Bubba ambles back up to the window and hands me my ticket with a big ol’ friendly country smile, that featured three missing top teeth, one barely hanging on by a slim part of the root, discolored by years of copious Redman, Skoal and or unfiltered cigarette use. Ugh!! But good news for Quaker Oats, I’m sending them an idea for a new ad… you got it... Officer Bubba in the Oatmeal Capital of Texas extolling the virtues of daily consumption of oatmeal to help “preserve” those few precious teeth that small town law enforcement officers are so fond of!!! Whadaya think?

Categories: 
January 18, 2010

Maintnenance FTW

I am a bit of an automotive enthusiast, so when I'm not working, I do spend a fair amount of time browsing automotive websites. I, like many people in the hosting industry, crave information. I like hearing about new design directions, emerging technologies, and past stories about others' experiences with their vehicles. While browsing, I came across some images of the guts of a BMW that had gone in excess of 60 thousand miles without an engine oil change. Needless to say, the internals were slathered with a gummy sludge and the engine was ruined.

Many technologies we use these days have become so common place and are operationally intuitive enough that we are often able to figure them out and use them without ever having to crack open an owner’s manual. I bring this up because, many technologies in the hosting industry follow suit. There are a number of developers who create software that is designed to make it easy to host websites. They are marketed as the only solution you ever need and, in some cases, imply that all you need to know is how to use a web browser to successfully host websites, not only for oneself but a plethora of other clients too! The servers run themselves, and you only need to spend a few minutes setting your clients up! It's like free money!

Unfortunately, as the owner of the previously mentioned BMW found out, this is not the case. There are a lot more things going on behind the scenes than just seats and a steering wheel, as are the same with servers. On occasion, we receive support tickets that just say "the site stopped working." In an attempt to gather more information, we will often ask the client a wide range of questions that help us find the problem faster and come up with the best possible solution. However, sometimes the answer from the customer is, "I haven't touched or logged into the server in days/ months/ (hopefully not) years." The more relevant metaphor for this is, "I haven't changed my BMW's oil in years!" Servers are like any other complex machine. They require constant maintenance. This includes: updating anti-virus definitions, monitoring bandwidth usage for anomalous spikes, rotating logs out if they are getting too large (provided some other rotation scheme is not already in place), keeping an eye on disk space usage, and creating a disaster recovery plan and backups. So take some time, get to know your server, and familiarize yourself with good preventative maintenance techniques. Your server, your clients, and your BMW (if applicable) will love you for it.

October 2, 2009

Is That a Real Computer?

Some mornings after work when the weather is nice I'll go to a local coffee shop on the way home to read or study for the CCNA exams. Sometimes I'll just end up pulling out the netbook and browse around online. There are times during these outings when I'll get asked the title question of this blog: is that a real computer? I guess the size that throws people but the answer is yes.

For those who are not familiar with the netbook class of systems here are the specs for mine:

  • 10.2 inch screen
  • 1 GB RAM
  • 1.6GHz Intel Atom processor
  • 160GB SATA hard drive
  • 3 USB ports
  • Card reader
  • Built-in Wifi
  • Built-in webcam
  • Windows XP (I've got plans for Windows 7)
  • 5 hour battery life
  • Light weight (I've got books that weigh more)

Netbooks are great for when you're just knocking around town and might want to do some light web work. This morning while at Starbucks I've checked e-mail several times, caught up on the daily news, and reviewed the game statistics from the Cowboys game I missed last night. Other mornings I've fired up a VPN connection into the office and been able to remotely help with tickets, work on documentation for our SSL product and tinker around with a NetScaler VPX Express virtual machine (an interesting bit of tech for a later article).

So how does this tie into server hosting?

You've probably had a time when your monitoring has indicated a service ceasing to respond on a server. If all you have is a cell phone the options are somewhat limited. With a fancy enough phone you might have an SSH or RDP client but do you really want to do anything on a PDA sized screen? I didn't think so. You can put in a ticket from your phone and our support can help out but the person best able to fix a service failure is still going to be you, the server administrator who knows where all the bodies are buried and how the bits tie together.

A small netbook can be a lightweight (and inexpensive) administration terminal for your servers hosted with us. Just find an Internet connection, connect up to the SoftLayer VPN and now you have complete access to work on your servers via a secure connection.

Through the wonders of the IPMI KVM this access even includes the console which opens up the possibility of doing a custom kernel build and install safely, while sitting under the stars, drinking a hot chocolate and watching the local nightlife.

Sounds like a pretty nice reality to me.

September 4, 2009

First Grade

Some of you know that it is now “back to school” time. Those of you who don’t know are the ones who are still probably able to attend happy hours. (Our esteemed CFO may not know, but only because he’s had lots of birthdays. I’m not saying he’s addled. I’m just saying.) Nearly everyone has heard of, if not read, the book “All I Really Need to Know I Learned in Kindergarten” by Robert Fulghum. This blog is kind of a take on that, but since The Boy just started First Grade, these are things of relevance we’ve learned about the rules and such for First Grade and that can be applied to things here at the ‘Layer.

So The Boy has tickets in First Grade, and if he’s bad – he’s got to pull a ticket and put it in a jar. Sounds like, the fewer tickets, the better. Kind of like here at SoftLayer, from our standpoint you don’t want to have to have a bunch of tickets to plow through. From a client standpoint, you certainly don’t want to have a bunch of tickets, especially from abuse, or you might get your server pulled. The lesson to be learned is to do what you can to keep those tickets at a minimum.

This year in First Grade, there are a bunch of boys, including The Boy. At the parent meeting after the second day of school, the two first grade teachers were explaining that because of the more than 2:1 ratio of boys to girls, the restroom breaks could be a potential nightmare. The uniforms have belts and shorts with buttons and zippers. The teachers said it takes forever for them to go, and they wanted the parents to tell their Boy that he needs to be able to, ah, how do we say it? Well, he needs to be able to whip it out, put it away and go. No smacking other bottoms and goofing off and giggling. Here at SoftLayer, we often have crazy deadlines, for example in development, which requires us to whip out some new technology on an expeditious basis. Just like First Grade, whip it out, put it away, and go on to the next project. (Unlike First Grade, there is lots of goofing off here at SoftLayer, such as with 10,000 bouncy balls and such. Since I’m in legal, if there is any smacking of bottoms, neither I nor HR want to hear about it. Lalalalala, Lalalala…)

One of the guys wrote a blog earlier this summer pondering if the things you learned in college are applicable to the “real world.” His conclusion was yes. My blog further confirms that the things you learn in First Grade are applicable to the “real world.” Here’s hoping The Boy can go 15 days without getting his ticket pulled, because if he does he gets some Krispy Kreme “football” donuts. And if The Boy gets some “football” donuts, that means The Mommy gets some, too.

June 17, 2009

Problem Solving

Quite often my friends who are not really that internet savvy ask me what I do at work, I think back to the time in the first grade when my teacher Mrs. Hyde told me: “ Bill you’re going to be a great problem solver when you get older, your problem solving skills are already at a fourth grade level.” Now you’re probably reading this wondering how problem solving problems in the first grade have anything to do with my job. It is, as she told me, all about how you think. She told me I was an outside the box thinker.

My co-workers and I deal with a network of 20,000+ servers, and 5500+ customers, in over 110 different countries, and support over 15 different operating systems. That leads to an almost infinite combination of language, hardware, and software options. When our customers submit an issue for us to work on, it is always different than the time before – whether that is a ticket from the same customer or a ticket on a similar topic. We have a very diverse range of customers using our servers for a number of things, so not every server in here is doing the same thing. In order to be good at supporting our customers, SoftLayer’s management, in my opinion, has hired some of the best problem solvers around the world to address all of our customer issues. So that is what I am: I am a problem solver! Otherwise known as a Customer Systems Administrator. We’re required to know a broad range of technologies and have the passion to learn the new ones as they come along. I think that is why I chose to work in the field that I work in, it is always changing. I tried moving over to telecommunications engineering a few years ago, but got bored with is as it was the same issues day in and day out on the equipment. Working here at SoftLayer is wonderful as there is never a dull moment.

Subscribe to tickets