Customer Service Posts

August 15, 2012

Managing Support Tickets: Email Subscriptions

This week, the development team rolled out some behind-the-scenes support functionality that I think a lot of our customers will want to take advantage of, so I put together this quick blog post to spread the word about it. With the new release, the support department is able to create "Ticket Email Subscriptions" for different ticket groups on every customer account. As a customer, you might not be jumping up and down with joy after reading that one-sentence description, but after you hear a little more about the functionality, if you're not clapping, I hope you'll at least give us a thumbs-up.

To understand the utility of the new ticket email subscription functionality, let's look at how normal tickets work in the SoftLayer portal without email subscriptions:

User Creates Ticket

  1. User A creates a ticket.
  2. User A becomes the owner of that ticket.
  3. When SoftLayer responds to the ticket, an email notification is sent to User A to let him/her know that the ticket has been updated.

SoftLayer Creates Ticket

  1. SoftLayer team creates a ticket on a customer's account.
  2. The primary customer contact on the account is notified of the new ticket.
  3. Customer logs into the portal and responds to ticket.
  4. Customer gets notifications of updates (as described above).

There's nothing wrong with the existing support notification process, but that doesn't mean there aren't ways to make the process better. What if User A creates an urgent ticket on his/her way out the door to go on vacation? User B and User C aren't notified when an update is posted on User A's ticket, so the other users aren't able to get to the ticket and respond as quickly as they would have if they received the notification. What if the primary customer contact on the account isn't the best person to receive a monitoring alert? The administrator who will investigate the monitoring alert has to see the new ticket on the account or hear about it from the primary contact (who got the notification).

Ticket email subscriptions allow for customers to set contact addresses to be notified when a ticket is created, edited or moved in a particular ticket group. Here are the ticket groups differentiated in our initial release:

  • Billing - Any ticket in our Billing department
  • Maintenance - Scheduled maintenance notifications for specific servers
  • Network Protection - DDoS mitigation and Null Routes
  • Monitoring - Host Down Alerts
  • CST, SysAdmin and Hardware - Any ticket in our support and data center departments
  • Managed Services - Tickets that relate to any managed services
  • Network Maintenance - Scheduled network maintenance

You'll notice that Abuse isn't included in this list, and the only reason it's omitted is because you've always been able to designate a contact on your account for abuse-related tickets ... Ticket subscriptions extend that functionality to other ticket groups.

Because only one email address can be "subscribed" to notifications in each ticket group, we recommend that customers use their own distribution lists as the email contacts. With a DL as the contact, you can enable multiple users in your organization to receive notifications, and you can add and remove users from each distribution list on your end quickly and easily.

When User A creates a ticket with the data center and goes on vacation, as soon as SoftLayer responds to the ticket, User A will be notified (as usual), and the supportsubscription@yourdomain.com distribution will get notified as well. When a network maintenance is ticket is created by SoftLayer, the netmaintsubscription@yourdomain.com distribution will be notified.

Ticket email subscriptions are additive to the current update notification structure, and they are optional. If you want to set up ticket email subscriptions on your account, create a ticket for the support department and provide us with the email addresses you'd like to subscribe to each of the ticket groups.

We hope this tool helps provide an even better customer experience for you ... If you don't mind, I'm going to head back to the lab to work with the dev team to cook up more ways to add flexibility and improvements into the customer experience.

-Chris

June 25, 2012

Tips from the Abuse Department: Part 2 - Responding to Abuse Reports

If you're a SoftLayer customer, you don't want to hear from the Abuse department. We know that. The unfortunate reality when it comes to hosting a server is that compromises can happen, mistakes can be made, and even the most scrupulous reseller can fall victim to a fraudulent sign-up or sly spammer. If someone reports abusive behavior originating from one of your servers on our network, it's important to be able to communicate effectively with the Abuse department and build a healthy working relationship.

Beyond our responsibility to enforce the law and our Acceptable Use Policy, the Abuse department is designed to be a valuable asset for our customers. We'll notify you of all valid complaints (and possibly highlight security vulnerabilities in the process), we'll assist you with blacklist removal, we can serve as a liaison between you and other providers if there are any problems, and if you operate an email-heavy platform or service, we can help you understand the steps you need to take to avoid activity that may be considered abuse.

At the end of the day, if the Abuse department can maintain a good rapport with our customers, both our jobs can be easier, so I thought this installment in the "Tips from the Abuse Department" series could focus on some best practices for corresponding with Abuse from a customer perspective.

Check Your Tickets

This is the easiest, most obvious recommendation I can give. You'd be surprised at how many service interruptions could be avoided if our customers were more proactive about keeping up with their open tickets. Our portal is a vital tool for your business, so make sure you are familiar with how to access and use it.

Keep Your Contact Information Current

Our ticket system will send notifications to the email address you have on file, so making sure this information is correct and current is absolutely crucial, especially if you aren't in the habit of checking the ticket system on a regular basis. You can even set a specific address for abuse notifications to be sent to, so make use of this option. The quicker you can respond to an abuse report, the quicker the complaint can be resolved, and by getting the complaint resolved quickly, you avoid any potential service interruption.

If we are unable to reach you by ticket, we may need to call you, so keep your current phone numbers on file as well.

Provide Frequent Updates

Stay in constant communication in the midst of responding to an abuse report, and adhere to the allotted timeline in the ticket. If we don't see updates that the abusive behavior is being addressed in the grace period we are able to offer, your server is at risk of disconnection. By keeping us posted about the action you're taking and the time you need to resolve the matter, we're able to be more flexible.

If a customer on your servers created a spamming script or a phishing account, taking immediate steps to mitigate the issue by suspending that customer is another great way to respond to the process while you're performing an investigation of how that activity was started. We'll still want a detailed resolution, but if the abuse is not actively ongoing we can work with you on deadlines.

Be Concise ... But Not Too Concise

One-word responses: bad. Page long responses: also not ideal. If given the option we would opt for the latter, but your goal should be to outline the cause and resolution of any reported abusive activity as clearly and succinctly as possible in order to ease communication and expedite closing of the ticket.

Responding to a ticket with, "Fixed," is not sufficient to for the Abuse department to consider the matter resolved, but we also don't need a dump of your entire log file. Before the Abuse team can close a ticket, we have to see details of how the complaint was resolved, so if you don't provide those details in your first response, you can bet we'll keep following up with you to get them. What details do we need?

Take a Comprehensive Approach

In addition to stopping the abusive activity we want to know:

  1. How/why the issue occurred
  2. What steps are being taken to prevent further issues of that nature

We understand that dealing with abuse issues can often feel like a game of Whack-A-Mole, but if you can show that you're digging a bit deeper and taking steps to avoid recurrence, that additional work is very much appreciated. Having the Abuse department consider you a proactive, ethical and responsible customer is a worthy goal.

Be Courteous

I'm ending on a similar note to my last blog post because it's just that important! We understand getting an abuse ticket is a hassle, but please remember that we're doing our best to protect our network, the Internet community and you.

Unplugging your server is a last resort for us, and we want to make sure everyone is on the same page to prevent us from getting to that last resort. In the unfortunate event that you do experience an abuse issue, please refer back to this blog — it just might save you some headaches and perhaps some unnecessary downtime.

-Jennifer

June 18, 2012

Tips from the Abuse Department: Part 1 - Reporting Abuse

SoftLayer has a dedicated team working around the clock to address complaints of abuse on our network. We receive these complaints via feedback loops from other providers, spam blacklisting services such as Spamcop and Spamhaus, various industry contacts and mailing lists. Some of the most valuable complaints we receive are from our users, though. We appreciate people taking the time to let us know about problems on our network, and we find these complaints particularly valuable as they are non-automated and direct from the source.

It stands to reason that the more efficient people are at reporting abuse, the more efficient we can be at shutting down the activity, so I've compiled some tips and resources to make this process easier. Enjoy!

Review our Legal Page

Not only does this page contain our contact details, there's a wealth of information on our policies including what we consider abuse and how we handle reported issues. For starters, you may want to review our AUP (Acceptable Use Policy) to get a feel for our stance on abuse and how we mitigate it.

Follow Proper Guidelines

In addition to our own policies, there are legal aspects we must consider. For example, a claim of copyright infringement must be submitted in the form of a properly formatted DMCA, pursuant to the Digital Millennium Copyright Act. Our legal page contains crucial information on what is required to make a copyright claim, as well as information on how to submit a subpoena or court order. We take abuse very seriously, but we must adhere to the law as well as our privacy policy in order to protect our customers' businesses and our company from liability.

Include Evidence

Evidence can take the form of any number of things. A few common examples:

  • A copy of the alleged spam message with full headers intact.
  • A snippet from your log file showing malicious activity.
  • The full URL of a phishing page.

Without evidence that clearly ties abusive activity to a server on our network, we are unable to relay a complaint to our customer. Keep in mind that the complaint must be in a format that allows us to verify it and pass it along, which typically means an email or hard copy. While our website does have contact numbers and addresses, email is your best bet for most types of complaints.

Use Keywords

We use a mail client specifically developed for abuse desks, and it is configured with a host of rules used for filtering and prioritization. Descriptive subject lines with keywords indicating the issue type are very useful. Including the words "Spam," "Phishing" or "Copyright" in your subject line helps make sure your email is sent to the correct queue and, if applicable, receives expedited processing. Including the domain name and IP address in the body of the email is also helpful.

Follow Up

We work hard to investigate and resolve all complaints received however, due to volume, we typically do not respond to complaining parties. That said, we often rely on user complaints to determine if an issue has resumed or is ongoing so feel free to send a new complaint if activity persists.

Be Respectful

The only portion of your complaint we are likely to relay to our customer is the evidence itself along with any useful notes, which means that paragraph of profanity is read only by hardworking SoftLayer employees. We understand the frustration of being on the receiving end of spam or a DDOS, but please be professional and try to understand our position. We are on your side!

Hopefully you've found some of this information useful. When in doubt, submit your complaint to abuse@softlayer.com and we can offer further guidance. Stay tuned for Part 2, where I'll offer suggestions for SoftLayer customers about how to facilitate better communication with our Abuse department to avoid service interruption if an abuse complaint is filed against you.

-Jennifer

April 6, 2012

Of Cage Nuts and Customer Service

Sometimes it's the little hardships and annoyances that really mold you. How do you react? Do you manage to work through them, or do you let them eat away at you to the point that you're more paralyzed by them than you are a bigger problem?

As a new hire, I was required to take part in a Truck Day — an experience that helps everyone in the company understand (at a base level) what is involved with the actual products and services we sell. If you've ever had the fortune of working on one, there are certain activities that can leave you feeling weary. For me, that weariness-inducing activity was working with cage nuts.

For those of you unfamiliar with cage nuts, they're small pieces of metal that accommodate screw-in server rails on a rack meant for slide-in server rails. Installing them is one of the most frustrating things ever ... They have two little clips that fit inside the rack, and you have to bend them to get them in. Here's a great illustration of how they work from an Oracle Sun Rack user's guide:

Cage Nuts

I'd installed them before, but never more than eight or so at a time. After Truck Day, I now have nothing but the greatest respect for the amazing people working in the data centers who have to do them in massive volumes. I don't think I've ever received as many tiny cuts on my hand as I did in the few hours I spent installing the relatively small number I managed to complete.

As a Customer Support Administrator (CSA), I spend the majority of my time sitting at a computer, helping customers with their servers and doing my best to resolve issues as they are encountered. Physically installing cage nuts isn't part of my day-to-day responsibilities (until the next Truck Day), but I realized that my job has its own "cage nuts."

A customer wanting to lease a server from us isn't particularly worried about the fact that cage nuts have to be meticulously installed in the rack, and they also aren't paying any mind to the fact I might have worked with a dozen customers in my shift already — And, certainly, they shouldn't. They're paying for a great customer experience and helpful, friendly service, so they don't need to take into account the context of our operations when they're simply asking for us to help them with a server reboot to finish the installation of an OS patch upgrade.

SoftLayer, as a company, has amazed me in that everyone I've met is not only willing to deal with their "cage nuts," but they will also do so without losing the smile from their face (even if there's some good-natured grumbling every now and then). In many of the places I've worked, this sort of task would be met with protest, foot dragging and a tired resignation to doing the work. That simply isn't the case here.

I'm definitely a newbie around here, and I'm still getting a feel for the culture, catching up on the inside jokes, and learning the ins and outs of the company (and the people in it). The one thing that was abundantly clear to me from the very first night, though: SLayers are truly dedicated to what they do, and the resulting work environment is one that fosters and rewards that dedication.

So in my estimation, how have the little annoyances — the cage nuts of our lives — molded SoftLayer and the people who work here? I'd say that not only do we work through them, we do so enthusiastically in the company of friends, proud of the fact that these seemingly small things are part of what has made this all possible.

I hope all of you work in environments that enable you to deal with the small things you see every day without cursing under your breath and feeling stressed. If you don't, maybe you should look into finding a place that does. I hear we're hiring.

-Gregory

April 3, 2012

Tips and Tricks - How to Use SFTP

Too often, new customers can get overwhelmed by a small administrative task on a Linux server. One of the more common questions I see in technical support is when a drive partition runs out of space. The website appears offline, and on of my coworkers advises you to just free-up some space. "Just?! Where can I find files that are deletable without affecting my website?"

Don't worry ... it's really quit simple. If you can use FTP (File Transfer Protocol), you can handle this bit of server management. Depending on the exact problem, we might instruct you to free up space by removing files in one of the following directories:

  • /var/log
  • /usr/local/cpanel
  • /usr/local/apache/logs
  • /usr/local/apache/domlogs

The reason these directories are usually overlooked is because they are not accessible by normal FTP users — users who only upload website content. When you upload website content to the server via FTP, the FTP user is limited to the directory structure for that website. Directories starting with "/var" and "/usr" cannot be accessed by these non-root users (The "root" user can access anything). And while root is a powerful user, for the sake of security, it is not normally allowed to log in over FTP because FTP is not secure ... That's where SFTP (Secure File Transfer Protocol) comes in.

Most FTP clients support SFTP, so you don't have to learn a new environment to securely access any file on the server. Every FTP client is different, but I'll illustrate with FileZilla because it's free and available on Mac, Windows and Linux. If you don't already have an FTP client, I highly recommend FileZilla. Because there are a few ways to use FileZilla to get an SFTP connection, I can share different options for you to try:

Quick Connect

The Quick Connect bar is the quickest way to connect to your server. Start FileZilla and look immediately under the toolbar for the Quick Connect bar:

SFTP Tutorial

Enter the hostname (IP address or domain name), “root” in the Username field, the root password in the Password field, and “22″ in the port field. Remember, port 22 is for SFTP, the same as SSH. Click the Quickconnect button to connect.

Using the Site Manager

The Site Manager lets you save your login details. Start FileZilla and you'll see the following:

SFTP Tutorial

To open the Site Manager, click the left-most icon in tool bar or go to File >> Site Manager in the menu.

SFTP Tutorial

Enter an IP address or domain name for your server in the Host field, and select "SFTP" as your protocol. You'll enter the root user's login information, and you're ready to connect by clicking the "Connect" button or you can click the "OK" button to save and close the dialog box.

If you just saved your settings and the Site Manager is not open, click the Site Manager icon again. From there, you can select the site under the "Select Entry" box, and you just have to click "Connect" to initiate the SFTP connection with your saved settings.

If you see a pop-up that warns of an "Unknown host key," clicking the "Always trust this host, add this key to the cache" option will prevent this interruption from showing in the future. Once you click "OK" to complete the connection, your FileZilla screen should look like this:

SFTP Tutorial

Notice the "Remote site" section on the middle right of the FileZilla screen:

SFTP Tutorial

This area in FileZilla is the directory and file listing of the server. Navigate the server's file structure here, and click "/" to access the top of the folder structure. You should see the "/usr" and "/var" directories, and you can explore the filesystem to delete the files technical support recommended to create space!

Message Log

If you have a problem connecting to your server by FTP or SFTP, the open area below the Quickconnect bar is the Message Log. If you can copy and paste this text into a ticket, you'll help technical support troubleshoot your connection problems. Below is an example log of a successful FTP session:

Status: Connecting to server.example.com...
Response:   fzSftp started
Command:    open "root@server.example.com" 22
Command:    Trust new Hostkey: Once
Command:    Pass: **********
Status: Connected to server.example.com
Status: Retrieving directory listing...
Command:    pwd
Response:   Current directory is: "/root"
Command:    ls
Status: Listing directory /root
Status: Calculating timezone offset of server...
Command:    mtime ".lesshst"
Response:   1326387703
Status: Timezone offsets: Server: -21600 seconds. Local: -21600 seconds. Difference: 0 seconds.
Status: Directory listing successful

And here's an example of a failed connection:

Status: Resolving address of example.com
Status: Connecting to 192.0.43.10:21...
Error:  Connection timed out
Error:  Could not connect to server
Status: Waiting to retry...
Status: Resolving address of example.com
Status: Connecting to 192.0.43.10:21...
Error:  Connection attempt interrupted by user

If you have any questions, leave them in a comment below. Enjoy your new-found SFTP powers!

-Lyndell

March 28, 2012

SoftLayer Mobile on WP7 - Live Tiles and Notifications

In the past couple of months we've added some really cool Windows Phone 7.1 (Mango) features to the Softlayer Mobile application, including Lives Tiles and Notifications. While a basic Live Tile implementation is relatively easy, there's a fair amount of coding and architecture requirements to facilitate cooler Live Tile functionality and Notifications ... And we're all about doing things cooler.

Live Tiles is a such great feature of Windows Phone 7 largely because it gives the developer much more control over the device's user experience when compared to other mobile OSes. Live Tile functionality in its simplest form can be just 'Pinning' the Tile to the Start Menu with a deep link to a specific location within the application so that when clicked the user is taken to that location within the app. This can save the user a lot of time in having to navigate deep into an app if they know where they want to go. More advanced features of Live Tiles include programmatically giving the Tile a custom background image and displaying a notification message on the background when the Tile flips.

Adding a Live Tile

To add a Live Tile, a user simply clicks and holds the module they'd like to pin to the start menu. When the context menu appears, the user can select 'pin as tile,' and he or she will be taken to the Start page where the new Tile is displayed:

SoftLayer on Windows Phone 7

The Magic Behind Sending Notifications

We really wanted to be able to notify a user when a notable event happens on his or her account (new ticket is created/updated, when a bill is overdue, etc.), and Windows Phone 7 provides some pretty phenomenal functionality in that area ... I wouldn't be surprised if other big mobile OSes copy Windows Phone 7's notifications in the future. When it comes to implementing notifications in SoftLayer Mobile, we needed to handle a few things:

  1. Get a Unique App+User Channel URI from Windows Push Notification Server
  2. Register URI & Channel Name with the Softlayer Registration Service (WCF we created)
  3. Store this URI, Channel Name and the user's Account in a DB
  4. Periodically poll for new tickets or updates (since we don't have a mechanism yet that can 'push' this alert when any notification event is triggered)
  5. Send Notification (whether it's a Toast or Tile notification) to device via the unique URI & Channel name.

I was going to include the architecture diagram here showing this relationship and process, but the designer sitting next to me told that nobody wants to see that.

What do the Numbers on the Tiles Mean?

We wanted to make our Tiles show information that the user would find useful, so we send the account's total unread ticket count to the main app's Tile, and we display the account's unread ticket update count on the "Ticket" Tile we pinned to the Start screen:

SoftLayer on Windows Phone 7

Why is the Tile Flipping?

We also have the ability to have the Tiles flip over and show an image or text on the TileBack, so we use that to explain the number shown on the Tile (so you don't have to remember):

SoftLayer on Windows Phone 7

What is a Toast Notification?

A Toast Notification is a message that pops up on the screen for 10 seconds. If the user clicks on it, he or she is taken to the application, but if the notification is not clicked, it will disappear. Here is the Toast Notification that is sent to a user when a ticket is updated if they subscribe to Toast Notifications:

SoftLayer on Windows Phone 7

How do I Enable Notifications in SoftLayer Mobile?

To enable Live Tiles, all you have to do is turn on the 'Use Push Notifications' option on the Settings view.

SoftLayer on Windows Phone 7

You'll be asked if you'd like to receive Toast Notifications, and if you click 'OK,' you'll start getting them:

SoftLayer on Windows Phone 7

We Love Feedback and Requests!

Now that you have Live Tiles & Notifications in Softlayer Mobile for WP7 (and coming soon for iPhone & Android), what else would you like to see in the mobile clients?

-Erik

February 23, 2012

How to Get the Best Customer Service (Anywhere)

Shelves of books have been written about providing great customer support, but I haven't seen many written about how to get great customer support. Lance wrote a quick guide called "The 8 Keys to Successful Tickets" in May 2007, but because there have been over 730 blog posts between that post and this post, I thought I might take a shot at the topic again without stealing too many of his ideas. When you work with a service-based company, you're probably going to interact with customer support representatives regularly. During these interactions, your experience will not be defined by your question or the issue you have. Instead, it will be defined by how you present your issue.

It can be extremely frustrating when a server goes down or a script isn't working the way it should. When something like this happens, my gut reaction is to get upset and throw my keyboard. I've also noticed that when I am angry, I have a difficult time trying to explain my problem to technical support. I know I'm not alone in that regard, so I tried to pinpoint the most important points to remember when contacting customer support. While some of the explanations below are more SoftLayer-specific, each of the tips below can be used in any situation where you need customer support.

  • Remember there's a human on the other end. It doesn't matter where the customer support representative is; they're human, and their responsibility is to help you. I don't have any empirical data, but human nature tells me it's easier to be nice to someone who is nice to you. Once you realize there's a person on the other end of the phone trying to do his/her job, it's a little easier to thank them in advance for their help. It may seem insignificant, but if you thank me in advance for my help, I'll subconsciously work harder in an effort to deserve that gratitude.
  • Don't assume your request will be ignored. I'm surprised by the number of people who start or end their e-mail with, "No one will probably see this, but ..." or "Not that anyone cares, but ..." Don't assume that you'll be ignored. That assumption just creates overarching negative tone; it isn't a "reverse psychology" play. The support process can be defined by the expectations you set for it, so get started on the right foot and expect that your questions will be answered and issues will be resolved.
  • Don't start with a threat. "If you don't do this, I'm going to report this to my bank and other authorities," or "If you don't respond within 25 seconds, you'll be hearing from my lawyer." It's not uncommon to hear things like this in the first message in a ticket. It's much easier to help someone who seems easy to help. Invoking lawyers does not make your ticket seem easy to address. :-)
  • Provide useful, descriptive and relevant information. This tip can be tough since it's hard to understand what information is "relevant," but think about it before you send a support request. If you are having trouble logging in, then "I can't log in. Any ideas?" is not quite as clear as "Whenever I try to log in, the login screen just reloads without an error message. I know my username and password are correct. Any ideas? Thanks." That extra information will help considerably and will reduce the number of back-and-forth e-mails between you and the support representative.
  • Don't write overly detailed, wordy support requests. The longer your e-mail, the more difficult it is to read, diagnose and to respond. A representative has to read the entire ticket to find what's meaningful and figure out exactly what's wrong. Since they're trying to help you, you want to reduce their burden. You want to make it as easy as possible for them to help you. So, be clear, concise and brief. If you've got a couple different issues for support to look at, break them out into individual tickets. Different issues may need to be addressed by different departments, so multiple issues in a single ticket can lead to delays in responding to specific issues in the ticket.
  • More Tickets ≠ More Support. The flip-side of the above recommendation is that you shouldn't create multiple support tickets for a single issue. While it seems like you're drawing more attention to the issue and creating a sense of urgency, you're really slowing down the support process. Support representatives might be addressing the same issue in parallel or information might be lost between tickets, elongating the time to resolution.
  • Escalate your tickets smartly. If you think a ticket should be handled differently or if you would like a supervisor to look into a specific issue, you should always feel free to request escalation to a manager or a supervisor. The best way to make that request is to update your open ticket, initiate a live chat or place a call into the technical support phone line. If you aren't satisfied with your support experience, then we aren't either, so we want to hear from you.

As you can see, the prescription is not too complicated: Prepare yourself to receive the best support and help us provide the best support, and you're much more likely to receive it.

-@khazard

January 27, 2012

Deciphering SoftLayer Acronyms

As a bit of an introduction, I began my career as a GSP and hosted LAMP sites with WHM for SMBs ... NBD. If you're not fluent in "Tech Geek Acronym," that sentence may as well be written in Greek. If I were to de-acronym it, I'd say, "I began my career as a Game Service Provider" and hosted Linux, Apache, MySQL and PHP sites with Web Host Managed for Small- and Medium-sized Businesses ... no big deal." For many, the humble acronym is a cornerstone of what it means to be a true techie. Stringing together dozens of three-letter abbreviations (TLAs) to compose semi-coherent sentences would seem to demonstrate your mastery of technology ... The problem is that if the reader of that sentence doesn't have the context you have, it's not very easy to easily get up to speed.

Every profession has their collection of acronyms. The little expressions serve as a verbal and written short hand for people who toil daily with the topics of their trade. I'm proud to confess that I've been using these minute medleys of letters for over twelve years. Given that I work on the Internet, I've been exposed to hundreds of acronyms in the fields of technology, business and management, and in my experience, I've had to break through several acronym "barriers" to get in the know. Because I happen to interact with customers every day as the manager of SoftLayer's technical support department, I've encountered a few "Can you tell means?" responses, so I thought I'd write a quick blog post to clarify some of the common acronyms you may see in the SoftLayer vernacular.

Within support we have our CSTs (customer support technicians) and CSAs (customer support admins) who, with the help of SBTs (server build technicians), manage our massive fleet of servers. SBTs are the hands and eyes of our data centers, working closely with the hardware to ensure your server is online and operating in peak condition. The CSTs and CSAs are focused on the software and services that power your websites and applications.

Beyond employee title acronyms, you'll probably see a collection of terms that describe the products and services that we manage. In support, we receive questions about accessing servers or CCIs (cloud computing instances) using KVM (Keyboard, Video and Mouse) or IPMI (Intelligent Platform Management Interface) through our VPN (Virtual Private Network). Once connected to our back-end network through a SSL (Secure Socket Layer), PPTP (Point-to-Point Tunnel Protocol) or IPSEC (Internet Protocol Security) VPN, you have access to services such as DNS (Domain Name Service), NAS (Network Attached Storage) or iSCSI (Internet Small Computer System Interface). Finally, while discussing our network, I often refer to http://www.softlayer.com/diagrams/pod-network-diagram/dal05 to show the difference between a VER (VPN Edge Router) and a BCS (Back-end Customer Switch).

If you run across an acronym you don't understand in a ticket, please let us know so we can share its full meaning ... By using these shortened terms, our team can provider faster service (and you can read their responses quicker). I know that seeing all the bold TLAs above may seem a little off-putting initially, but as you have a chance to read them in the context of some of the other acronyms you already know, I hope you have an "Aha!" moment ... Like finding the Rosetta Stone or the Code of Hammurabi. Given the quick glance at the terms above, if you want to learn more about one of the TLAs in particular, leave a comment below, and we'll respond in another comment with details.

CBNO

-Chris

January 16, 2012

What I've Learned About Leading

What does it take to be a good leader? What kind of leader do you want to follow? Throughout my life, I've gravitated toward leadership positions. Even when I was young, I tended to take charge of a group to achieve a goal or accomplish a mission ... though most of the "missions" in my younger days happened to be some sort of mischief. Having participated in the Boy Scouts and JROTC, I joined the Marine Corps where I served for seven years, and throughout my life, I've been fortunate to have more than my fair share of incredible mentors.

When my service in the Marine Corps concluded, I "enlisted" at SoftLayer as a data center technician. My primary responsibilities included building severs to order and installing software for our customers in four hours or less, and it was all pretty foreign to me. I had a lot to learn about the technical side of operating a data center, but based on my impression of the company, I was confident that I'd be in good hands.

Because I always find myself asking for more challenges and additional responsibility, I transitioned into a Customer Systems Administrator role. The CSA position required a lot more learning (at a breakneck pace), and in addition to the technical aspect of the job, I found myself learning just as much about the 'soft skills' required to provide the great customer service. Equipped with that knowledge (and a bit more experience), I pursued a leadership role on the team, and I made it my goal to use what I'd learned in the data center and in support to lead my team. I'd be lying if I said it wasn't a challenge, but I've never backed down from one before.

I don't mean to make this post all about me ... my goal in sharing a little of my background is to give a little context for what I've learned about leadership. It goes without saying that I've been fortunate, both in the Marine Corps as well as with SoftLayer, to have some of the most intelligent, talented and sometimes downright enigmatic leaders. I've managed to pick up advice, tips and tricks for handling all of the curveballs that are thrown on a daily basis, and over my years as a leader, I've developed a few philosophies (an amalgam of some of the key points I absorbed from all of those who have led me in the past) that I try to abide by daily:

Lead by Example.

I make it a point to never ask someone to do something that I've never done or something that I'd never be willing to do. This is the keen avoidance of the "do as I say, not as I do" mentality. Failing to do this hands-down one of the easiest ways to lose your team.

Employ Your Team to Their Abilities. Empower Them to do Their Best Work.

It's very important to know your team's strengths and weaknesses and use those to make everyone better. Using the strengths of one team member, I can push another outside of his or her comfort zone to improve his or her overall skill set. I've also seen amazing results from providing freedom for my teams to make decisions. Not only does that freedom build trust, it also gives some real "ownership" to every person, and with that sense of ownership, each team member does better work. I've been a little surprised to notice this empowerment coming from the coffee juggernaut Starbucks: Starbucks baristas and store employees are encouraged to make real-time decisions in the interest of taking care of their customers.1

Keep Learning.

Leading a team requires that you stay on top of what they're doing. Always ask questions. Continue to research so you can be a resource for your team. Find opportunities to learn and take advantage of every one of them.

Most Importantly: Learn how to Have Fun.
I see this cliché often, so when you see, "Have Fun," I wouldn't be surprised if you just rolled your eyes. I'd be lying if I said that things were great all of the time ... Realistically in any business, there's going to be a time or two when the *&#@ is going to hit the fan. It's important to find opportunities to cut loose and relax a bit. Cooking steaks for the group during a long overnight shift, grabbing a bite after work or a providing a happy hour once in a while builds a great deal of cohesion outside the office walls. Providing a relaxed environment does not prevent your team from doing stressful work ... It actually builds camaraderie, and it will help the team get through those tough times. The brutal honesty is that if people aren't enjoying where they're at, they'll look elsewhere – leaders have to help foster an environment that enables success.

At the end of the day, these tips may not work for everyone. There are a plethora of studies out there pertaining to the different leadership styles, the different types of leaders and how they influence teams. What's important is that leaders need to be intentional about bettering their teams (and ultimately bettering their businesses).

Along the lines of continuous education, I'd love to hear the leadership philosophies you've learned in your experience as a leader. Leave a comment on this article to share what you think has made you successful.

-Matthew

1This comes from both observation and talking to current and past employees. I've never worked for Starbucks, so I can't cite a specific company policy to back this up, but that evident organic culture is probably worth more than a million company policies that would try to create that culture.

December 29, 2011

Using iPerf to Troubleshoot Speed/Throughput Issues

Two of the most common network characteristics we look at when investigating network-related concerns in the NOC are speed and throughput. You may have experienced the following scenario yourself: You just provisioned a new bad-boy server with a gigabit connection in a data center on the opposite side of the globe. You begin to upload your data and to your shock, you see "Time Remaining: 10 Hours." "What's wrong with the network?" you wonder. The traceroute and MTR look fine, but where's the performance and bandwidth I'm paying for?

This issue is all too common and it has nothing to do with the network, but in fact, the culprits are none other than TCP and the laws of physics.

In data transmission, TCP sends a certain amount of data then pauses. To ensure proper delivery of data, it doesn't send more until it receives an acknowledgement from the remote host that all data was received. This is called the "TCP Window." Data travels at the speed of light, and typically, most hosts are fairly close together. This "windowing" happens so fast we don't even notice it. But as the distance between two hosts increases, the speed of light remains constant. Thus, the further away the two hosts, the longer it takes for the sender to receive the acknowledgement from the remote host, reducing overall throughput. This effect is called "Bandwidth Delay Product," or BDP.

We can overcome BDP to some degree by sending more data at a time. We do this by adjusting the "TCP Window" – telling TCP to send more data per flow than the default parameters. Each OS is different and the default values will vary, but most all operating systems allow tweaking of the TCP stack and/or using parallel data streams. So what is iPerf and how does it fit into all of this?

What is iPerf?

iPerf is simple, open-source, command-line, network diagnostic tool that can run on Linux, BSD, or Windows platforms which you install on two endpoints. One side runs in a 'server' mode listening for requests; the other end runs 'client' mode that sends data. When activated, it tries to send as much data down your pipe as it can, spitting out transfer statistics as it does. What's so cool about iPerf is you can test in real time any number of TCP window settings, even using parallel streams. There's even a Java based GUI you can install that runs on top of it called, JPerf (JPerf is beyond the scope of this article, but I recommend looking into it). What's even cooler is that because iPerf resides in memory, there are no files to clean up.

How do I use iPerf?

iPerf can be quickly downloaded from SourceForge to be installed. It uses port 5001 by default, and the bandwidth it displays is from the client to the server. Each test runs for 10 seconds by default, but virtually every setting is adjustable. Once installed, simply bring up the command line on both of the hosts and run these commands.

On the server side:
iperf -s

On the client side:
iperf -c [server_ip]

The output on the client side will look like this:

#iperf -c 10.10.10.5
------------------------------------------------------------
Client connecting to 10.10.10.5, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 0.0.0.0 port 46956 connected with 168.192.1.10 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 10.0 sec  10.0 MBytes  1.00 Mbits/sec

There are a lot of things we can do to make this output better with more meaningful data. For example, let's say we want the test to run for 20 seconds instead of 10 (-t 20), and we want to display transfer data every 2 seconds instead of the default of 10 (-i 2), and we want to test on port 8000 instead of 5001 (-p 8000). For the purposes of this exercise, let's use those customization as our baseline. This is what the command string would look like on both ends:

Client Side:

#iperf -c 10.10.10.5 -p 8000 -t 20 -i 2
------------------------------------------------------------
Client connecting to 10.10.10.5, TCP port 8000
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 10.10.10.10 port 46956 connected with 10.10.10.5 port 8000
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 2.0 sec  6.00 MBytes  25.2 Mbits/sec
[  3]  2.0- 4.0 sec  7.12 MBytes  29.9 Mbits/sec
[  3]  4.0- 6.0 sec  7.00 MBytes  29.4 Mbits/sec
[  3]  6.0- 8.0 sec  7.12 MBytes  29.9 Mbits/sec
[  3]  8.0-10.0 sec  7.25 MBytes  30.4 Mbits/sec
[  3] 10.0-12.0 sec  7.00 MBytes  29.4 Mbits/sec
[  3] 12.0-14.0 sec  7.12 MBytes  29.9 Mbits/sec
[  3] 14.0-16.0 sec  7.25 MBytes  30.4 Mbits/sec
[  3] 16.0-18.0 sec  6.88 MBytes  28.8 Mbits/sec
[  3] 18.0-20.0 sec  7.25 MBytes  30.4 Mbits/sec
[  3]  0.0-20.0 sec  70.1 MBytes  29.4 Mbits/sec

Server Side:

#iperf -s -p 8000 -i 2
------------------------------------------------------------
Server listening on TCP port 8000
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[852] local 10.10.10.5 port 8000 connected with 10.10.10.10 port 58316
[ ID] Interval Transfer Bandwidth
[  4]  0.0- 2.0 sec  6.05 MBytes  25.4 Mbits/sec
[  4]  2.0- 4.0 sec  7.19 MBytes  30.1 Mbits/sec
[  4]  4.0- 6.0 sec  6.94 MBytes  29.1 Mbits/sec
[  4]  6.0- 8.0 sec  7.19 MBytes  30.2 Mbits/sec
[  4]  8.0-10.0 sec  7.19 MBytes  30.1 Mbits/sec
[  4] 10.0-12.0 sec  6.95 MBytes  29.1 Mbits/sec
[  4] 12.0-14.0 sec  7.19 MBytes  30.2 Mbits/sec
[  4] 14.0-16.0 sec  7.19 MBytes  30.2 Mbits/sec
[  4] 16.0-18.0 sec  6.95 MBytes  29.1 Mbits/sec
[  4] 18.0-20.0 sec  7.19 MBytes  30.1 Mbits/sec
[  4]  0.0-20.0 sec  70.1 MBytes  29.4 Mbits/sec

There are many, many other parameters you can set that are beyond the scope of this article, but for our purposes, the main use is to prove out our bandwidth. This is where we'll use the TCP window options and parallel streams. To set a new TCP window you use the -w switch and you can set the parallel streams by using -P.

Increased TCP window commands:

Server side:
#iperf -s -w 1024k -i 2

Client side:
#iperf -i 2 -t 20 -c 10.10.10.5 -w 1024k

And here are the iperf results from two Softlayer file servers – one in Washington, D.C., acting as Client, the other in Seattle acting as Server:

Client Side:

# iperf -i 2 -t 20 -c 10.10.10.5 -p 8000 -w 1024k
------------------------------------------------------------
Client connecting to 10.10.10.5, TCP port 8000
TCP window size: 1.00 MByte (WARNING: requested 1.00 MByte)
------------------------------------------------------------
[  3] local 10.10.10.10 port 53903 connected with 10.10.10.5 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 2.0 sec  25.9 MBytes   109 Mbits/sec
[  3]  2.0- 4.0 sec  28.5 MBytes   120 Mbits/sec
[  3]  4.0- 6.0 sec  28.4 MBytes   119 Mbits/sec
[  3]  6.0- 8.0 sec  28.9 MBytes   121 Mbits/sec
[  3]  8.0-10.0 sec  28.0 MBytes   117 Mbits/sec
[  3] 10.0-12.0 sec  29.0 MBytes   122 Mbits/sec
[  3] 12.0-14.0 sec  28.0 MBytes   117 Mbits/sec
[  3] 14.0-16.0 sec  29.0 MBytes   122 Mbits/sec
[  3] 16.0-18.0 sec  27.9 MBytes   117 Mbits/sec
[  3] 18.0-20.0 sec  29.0 MBytes   122 Mbits/sec
[  3]  0.0-20.0 sec   283 MBytes   118 Mbits/sec

Server Side:

#iperf -s -w 1024k -i 2 -p 8000
------------------------------------------------------------
Server listening on TCP port 8000
TCP window size: 1.00 MByte
------------------------------------------------------------
[  4] local 10.10.10.5 port 8000 connected with 10.10.10.10 port 53903
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0- 2.0 sec  25.9 MBytes   109 Mbits/sec
[  4]  2.0- 4.0 sec  28.6 MBytes   120 Mbits/sec
[  4]  4.0- 6.0 sec  28.3 MBytes   119 Mbits/sec
[  4]  6.0- 8.0 sec  28.9 MBytes   121 Mbits/sec
[  4]  8.0-10.0 sec  28.0 MBytes   117 Mbits/sec
[  4] 10.0-12.0 sec  29.0 MBytes   121 Mbits/sec
[  4] 12.0-14.0 sec  28.0 MBytes   117 Mbits/sec
[  4] 14.0-16.0 sec  29.0 MBytes   122 Mbits/sec
[  4] 16.0-18.0 sec  28.0 MBytes   117 Mbits/sec
[  4] 18.0-20.0 sec  29.0 MBytes   121 Mbits/sec
[  4]  0.0-20.0 sec   283 MBytes   118 Mbits/sec

We can see here, that by increasing the TCP window from the default value to 1MB (1024k) we achieved around a 400% increase in throughput over our baseline. Unfortunately, this is the limit of this OS in terms of Window size. So what more can we do? Parallel streams! With multiple simultaneous streams we can fill the pipe close to its maximum usable amount.

Parallel Stream Command:
#iperf -i 2 -t 20 -c -p 8000 10.10.10.5 -w 1024k -P 7

Client Side:

#iperf -i 2 -t 20 -c -p 10.10.10.5 -w 1024k -P 7
------------------------------------------------------------
Client connecting to 10.10.10.5, TCP port 8000
TCP window size: 1.00 MByte (WARNING: requested 1.00 MByte)
------------------------------------------------------------
 [ ID] Interval       Transfer     Bandwidth
[  9]  0.0- 2.0 sec  24.9 MBytes   104 Mbits/sec
[  4]  0.0- 2.0 sec  24.9 MBytes   104 Mbits/sec
[  7]  0.0- 2.0 sec  25.6 MBytes   107 Mbits/sec
[  8]  0.0- 2.0 sec  24.9 MBytes   104 Mbits/sec
[  5]  0.0- 2.0 sec  25.8 MBytes   108 Mbits/sec
[  3]  0.0- 2.0 sec  25.9 MBytes   109 Mbits/sec
[  6]  0.0- 2.0 sec  25.9 MBytes   109 Mbits/sec
[SUM]  0.0- 2.0 sec   178 MBytes   746 Mbits/sec
 
(output omitted for brevity on server & client)
 
[  7] 18.0-20.0 sec  28.2 MBytes   118 Mbits/sec
[  8] 18.0-20.0 sec  28.8 MBytes   121 Mbits/sec
[  5] 18.0-20.0 sec  28.0 MBytes   117 Mbits/sec
[  4] 18.0-20.0 sec  28.0 MBytes   117 Mbits/sec
[  3] 18.0-20.0 sec  28.9 MBytes   121 Mbits/sec
[  9] 18.0-20.0 sec  28.8 MBytes   121 Mbits/sec
[  6] 18.0-20.0 sec  28.9 MBytes   121 Mbits/sec
[SUM] 18.0-20.0 sec   200 MBytes   837 Mbits/sec
[SUM]  0.0-20.0 sec  1.93 GBytes   826 Mbits/sec 

Server Side:

#iperf -s -w 1024k -i 2 -p 8000
------------------------------------------------------------
Server listening on TCP port 8000
TCP window size: 1.00 MByte
------------------------------------------------------------
[  4] local 10.10.10.10 port 8000 connected with 10.10.10.5 port 53903
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0- 2.0 sec  25.7 MBytes   108 Mbits/sec
[  8]  0.0- 2.0 sec  24.9 MBytes   104 Mbits/sec
[  4]  0.0- 2.0 sec  24.9 MBytes   104 Mbits/sec
[  9]  0.0- 2.0 sec  24.9 MBytes   104 Mbits/sec
[ 10]  0.0- 2.0 sec  25.9 MBytes   108 Mbits/sec
[  7]  0.0- 2.0 sec  25.9 MBytes   109 Mbits/sec
[  6]  0.0- 2.0 sec  25.9 MBytes   109 Mbits/sec
[SUM]  0.0- 2.0 sec   178 MBytes   747 Mbits/sec
 
[  4] 18.0-20.0 sec  28.8 MBytes   121 Mbits/sec
[  5] 18.0-20.0 sec  28.3 MBytes   119 Mbits/sec
[  7] 18.0-20.0 sec  28.8 MBytes   121 Mbits/sec
[ 10] 18.0-20.0 sec  28.1 MBytes   118 Mbits/sec
[  9] 18.0-20.0 sec  28.0 MBytes   118 Mbits/sec
[  8] 18.0-20.0 sec  28.8 MBytes   121 Mbits/sec
[  6] 18.0-20.0 sec  29.0 MBytes   121 Mbits/sec
[SUM] 18.0-20.0 sec   200 MBytes   838 Mbits/sec
[SUM]  0.0-20.1 sec  1.93 GBytes   825 Mbits/sec

As you can see from the tests above, we were able to increase throughput from 29Mb/s with a single stream and the default TCP Window to 824Mb/s using a higher window and parallel streams. On a Gigabit link, this about the maximum throughput one could hope to achieve before saturating the link and causing packet loss. The bottom line is, I was able to prove out the network and verify bandwidth capacity was not an issue. From that conclusion, I could focus on tweaking TCP to get the most out of my network.

I'd like to point out that we will never get 100% out of any link. Typically, 90% utilization is about the real world maximum anyone will achieve. If you get any more, you'll begin to saturate the link and incur packet loss. I should also point out that Softlayer doesn't directly support iPerf, so it's up to you install and play around with. It's such a versatile and easy to use little piece of software that it's become invaluable to me, and I think it will become invaluable to you as well!

-Andrew

Subscribe to customer-service