Posts Tagged 'Solutions'

October 16, 2013

Tips and Tricks: Troubleshooting Email Issues

Working in support, one of the most common issues we troubleshoot is a customer's ability to receive email. Depending on email server, this can be a headache and a half to figure out, but more often than not, we're able to fix the problem with one of only a few simple solutions. Because the SoftLayer Blog audience loves technical tips and tricks, I thought I'd share a few easy steps that make pinpointing the root cause of email issues much easier.

Before you gear up to go into battle, check the that server is not out of disk space on /var and that it is not in a read only state. That precursory step may seem silly, but Occam's Razor often holds true in technical troubleshooting. Once you verify that those two common problems aren't causing your email problems, the next step is to determine whether the email issues are server-wide or isolated to one mail account/domain. To do that, the first thing you need to do is make sure that the IMAP and POP services are responding.

Check IMAP and POP Services

The universal approach to checking IMAP and POP services is to use telnet:

telnet <serverip> 110
telnet <serverip> 143

If either of those commands fail, you're able to pinpoint which service to check on your server.

For most variants of Linux, you can check both services with a single command: netstat -plan|egrep -i "110|143". The resulting output will show if the services are listening and which process is doing the listening. In Windows, you can run a similar command from a command prompt: netstat -anb|find "LISTEN"| findstr "110 143".

If the ports are listening, and you're able to connect to them over telnet, your next stop should be your server's error logs.

Check Error Logs

You want to look for any mail errors that might clue you into the root cause of your email issues. In Linux, you can check /var/log/maillog, and in Windows, you can filter eventvwr.msc for mail only. If there are errors, a simple search will highlight them quickly.

If there are no errors, it's time to dig into the mail queue directly.

Check the Mail Queue

Depending on the mail server you use, the commands here are going to vary. Here are a few examples of how we'd investigate the most common mail servers we encounter:

QMail

Display the mail queue: /var/qmail/bin/qmail-qread
Display the number of messages in the queue: /var/qmail/bin/qmail-qstat
Reference article: Gaining Control Over the QMail Queue

Sendmail

Display the mail queue: sendmail -bp or mailq
Display the number of messages in the queue: mailq –OmaxQueueRunSize=1
Reference article: Quick Sendmail Cheatsheet

Exim

Display the mail queue: exim -bp
Display the number of messages in the queue: exim -bpc
Reference article: Exim cheatsheet

MailEnable

MailEnable users can can check to see that messages are moving by opening the mail directory:
Program Files\MailEnable\Queues\SMTP\Inbound\Messages
Reference article: How to diagnose inbound message delivery delays

With these commands, you can filter through the email queues to see whether any of them are for the users or domains you're having problems with. If nothing obvious presents itself at that point, it's time for some active testing.

Active Testing

Send an email to your mailserver from an external mailserver (anything will do as long as it's not on the same server). Watch for logging of the email as it's delivered:
tail -f maillog
On busy mailservers you might add |grep youremailid or simply look for a new message in the directory where the email will be stored.

The your primary goal in troubleshooting your email issues in this way is to isolate the root cause of your problem so that you can fix it more quickly. SoftLayer customers have direct access to our support team to help you through this process, but it's always nice to keep a quick reference like this in your back pocket to be able to pinpoint the problem yourself.

-Bill

October 8, 2012

Don't Let Your Success Bring You Down

Last week, I got an email from a huge technology conference about their new website, exciting new speaker line up and the availability of early-bird tickets. I clicked on a link from that email, and I find that their fancy new website was down. After giving up on getting my early-bird discount, I surfed over to Facebook, and I noticed a post from one of my favorite blogs, Dutch Cowboys, about another company's interesting new product release. I clicked the link to check out the product, and THAT site was down, too. It's painfully common for some of the world's most popular sites and applications buckle under the strain of their own success ... Just think back to when Diablo III was launched: Demand crushed their servers on release day, and the gamers who waited patiently to get online with their copy turned to the world of social media to express their visceral anger about not being able to play the game.

The question everyone asks is why this kind of thing still happens. To a certain extent, the reality is that most entrepreneurs don't know what they don't know. I spoke with an woman who was going to be featured on BBC's Dragons' Den, and she said that the traffic from the show's viewers crippled most (if not all) of the businesses that were presented on the program. She needed to safeguard from that happening to her site, and she didn't know how to do that.

Fortunately, it's pretty easy to keep sites and applications online with on-demand infrastructure and auto-scaling tools. Unfortunately, most business owners don't know how easy it is, so they don't take advantage of the resources available to them. Preparing a website, game or application for its own success doesn't have to be expensive or time consuming. With pay-for-what-you-use pricing and "off the shelf" cloud management solutions, traffic-caused outages do NOT have to happen.

First impressions are extremely valuable, and if I wasn't really interested in that conference or the new product Dutch Cowboys blogged about, I'd probably never go back to those sites. Most Internet visitors would not. I cringe to think about the potential customers lost.

Businesses spend a lot of time and energy on user experience and design, and they don't think to devote the same level of energy on their infrastructure. In the 90's, sites crashing or slowing was somewhat acceptable since the interwebs were exploding beyond available infrastructure's capabilities. Now, there's no excuse.

If you're launching a new site, product or application, how do you get started?

The first thing you need to do is understand what resources you need and where the potential bottlenecks are when hundreds, thousands or even millions of people want to what you're launching. You don't need to invest in infrastructure to accommodate all of that traffic, but you need to know how you can add that infrastructure when you need it.

One of the easiest ways to prepare for your own success without getting bogged down by the bits and bytes is to take advantage of resources from some of our technology partners (and friends). If you have a PHP, Ruby on Rails or Node.js applications, Engine Yard will help you deploy and manage a specialized hosting environment. When you need a little more flexibility, RightScale's cloud management product lets you easily manage your environment in "a single integrated solution for extreme efficiency, speed and control." If your biggest concern is your database's performance and scalability, Cloudant has an excellent cloud database management service.

Invest a little time in getting ready for your success, and you won't need to play catch-up when that success comes to you. Given how easy it is to prepare and protect your hosting environment these days, outages should go the way of the 8-track player.

-@jpwisler

March 3, 2011

Hosting != Glamorous

Infrastructure. Administration. Interface. Connectivity. Computer Room Air Conditioner. Data Center. Generator. Router.

I know what you're saying to yourself right now: "Hold onto your hat, self ... After that start, this post is going to be wild and crazy!" Actually, you might have literally yawned while reading those words. Why?

Hosting isn't very glamorous.

Go back to that first sentence and substitute exclamation points for each of the periods, then go back and make yourself yawn once between "Interface" and "Connectivity" and then again between "Generator" and "Router." Which one felt more natural? Unless you're on your fifth caffeinated beverage of the day or you happen to work for a hosting provider, the excited response is probably a lot less natural than the sleep-inducing one.

Don't get me wrong ... I'm not insulting hosting. I think it's just hamstrung by terms that reek of lameness to the uninitiated outsider. The closest we've got to an interesting term in the industry is "the cloud," and the industry was so happy about the positive response to that metaphor that everyone started calling everything "cloud" to engender fluffy, happy images in customers' heads. But as Lance said in his Parallels Summit keynote, it has just become a marketing term.

I propose that hosting doesn't need to be glamorous to be awesome. Hosting enables customers to make glamorous things.

In one day on the GDC 2011 expo hall floor, our team has talked to hundreds of attendees that have stopped by SoftLayer's booth (2116) to learn a little more about what SoftLayer does, and I love seeing someone "get it" for the first time. Nine times out of ten, if I'm talking to an attendee without a technical background, a glazed stare will slowly creep across his/her face as I explain a little about private networking and our nationwide MPLS network, but when I start talking about what our customers are doing with those tools, "Eureka!"

One of the most subtle explanations for SoftLayer's monumental growth is that our customers do some amazing things on our platform, and those successes, in turn, legitimize the platform and inspire other customers. Whether the story be about a phenomenally popular social gaming company or a reseller that enables tens of thousands of small businesses to get websites, being able to share a real world example takes our explanation out of the ether ... or maybe it takes the yawn-inducing ether out of our explanation.

SoftLayer's platform was meticulously designed to be spectacularly simple: Make sure everything works together, give customers as much control as possible, and get out of their way to - as Guy Kawasaki puts it - "let a hundred flowers blossom."

If you're at GDC this week and you haven't stopped by SoftLayer's booth yet, you need to listen to Natalie:

When you come by, we'll be happy to tell you anything you want to know about our hosting solutions, but we'll be downright excited to share with you the kinds of things our hosting solutions have done for our customers and could do for you.

-@khazard

P.S. If you're not in San Francisco this week, consider this an open invitation to take us up on the same offer at any of SoftLayer's events in the future.

P.P.S. We sincerely hope that this blog does not offend any "hosters" out there ... especially any that are Warlocks.

January 13, 2010

Always Have a Backup Plan...

Everyone always says it’s a good idea to have a backup plan just in case your primary plan bites the dust. I couldn’t agree more. Recently my personal Xbox 360 failed and this has caused plenty of grief in my household. I used my Xbox to stream content from Windows Media Player on my desktop to the TV (via Media Center edition of Windows XP). This has worked great and has been able to provide me with a means to entertain my child. Of course, this going out has caused a screaming baby because now she can’t watch her “movies”.

Now, had I had a proper backup plan, this wouldn’t be an issue. See, I put all of my trust into a single device and/or single method to accomplish something. When this device failed, my operation came to a halt. I didn’t listen to the advice I’m always telling our customers… have a backup or backup plan. This is where our “extra services” come into play. Not only do we offer backup solutions (eVault, NAS…) but we also offer solutions that allow you access to high-availability configurations (Citrix XenServer, for example). With XenServer you can configure a cluster of systems and setup automatic failover. This would prevent any major outages of your website/services. If this isn’t something you think would work for you, utilizing eVault backups might. We now offer eVault Bare Metal Restore. Now, the problem is somehow applying these to my Xbox so my kiddo can go back to watching her movies... Long story short, don’t rely on a single solution. Always have a backup plan or system in place to prevent headaches in the future. You won’t regret it if you do.

Categories: 
June 17, 2009

Problem Solving

Quite often my friends who are not really that internet savvy ask me what I do at work, I think back to the time in the first grade when my teacher Mrs. Hyde told me: “ Bill you’re going to be a great problem solver when you get older, your problem solving skills are already at a fourth grade level.” Now you’re probably reading this wondering how problem solving problems in the first grade have anything to do with my job. It is, as she told me, all about how you think. She told me I was an outside the box thinker.

My co-workers and I deal with a network of 20,000+ servers, and 5500+ customers, in over 110 different countries, and support over 15 different operating systems. That leads to an almost infinite combination of language, hardware, and software options. When our customers submit an issue for us to work on, it is always different than the time before – whether that is a ticket from the same customer or a ticket on a similar topic. We have a very diverse range of customers using our servers for a number of things, so not every server in here is doing the same thing. In order to be good at supporting our customers, SoftLayer’s management, in my opinion, has hired some of the best problem solvers around the world to address all of our customer issues. So that is what I am: I am a problem solver! Otherwise known as a Customer Systems Administrator. We’re required to know a broad range of technologies and have the passion to learn the new ones as they come along. I think that is why I chose to work in the field that I work in, it is always changing. I tried moving over to telecommunications engineering a few years ago, but got bored with is as it was the same issues day in and day out on the equipment. Working here at SoftLayer is wonderful as there is never a dull moment.

Subscribe to solutions