While the content of that article is less sensationalized than the title, there are still a few gaps to fill about when it comes to how SoftLayer is actually involved in the big picture (*SPOILER ALERT* We aren’t “helping to fuel the spike in big DDoS attacks”). The CloudFlare blog and the Ars Technica post presuppose that the presence of open recursive DNS resolvers is a sign of negligence on the part of the network provider at best and maliciousness at worst, and that’s not the case.

The majority of SoftLayer’s infrastructure is made up of self-managed dedicated and cloud servers. Customers who rent those servers on a monthly basis have unrestricted access to operate their servers in any way they’d like as long as that activity meets our acceptable use policy. Some of our largest customers are hosting resellers who provide that control to their customers who can then provide that control to their own customers. And if 23 million hostnames reside on the SoftLayer network, you can bet that we’ve got a lot of users hosting their DNS on SoftLayer infrastructure. Unfortunately, it’s easier for those customers and customers-of-customers and customers-of-customers-of-customers to use “defaults” instead of looking for, learning and implementing “best practices.”

It’s all too common to find those DNS resolvers open and ultimately vulnerable to DNS amplification attacks, and whenever our team is alerted to that vulnerability on our network, we make our customers aware of it. In turn, they may pass the word down the customer-of-customer chain to get to the DNS owner. It’s usually not a philosophical question about whether DNS resolvers should be open for the greater good of the Internet … It’s a question of whether the DNS owner has any idea that their “configuration” is vulnerable to be abused in this way.

SoftLayer’s network operations, abuse and support teams have tools that flag irregular and potentially abusive traffic coming from any server on our network, and we take immediate action when we find a problem or are alerted to one by someone who sends details to abuse@softlayer.com. The challenge we run into is that flagging obvious abusive behavior from an active DNS server is a bit of a cat-and-mouse game … Attackers cloak their activity in normal traffic. Instead of sending a huge amount of traffic from a single domain, they send a marginal amount of traffic from a large number of machines, and the “abusive” traffic is nearly impossible for even the DNS owner to differentiate from “regular” traffic.

CloudFlare effectively became a honeypot, and they caught a distributed DNS amplification DoS attack. The results they gathered are extremely valuable to teams like mine at SoftLayer, so if they go the next step to actively contact the abuse channel for each of the network providers in their list, I hope that each of the other providers will jump on that information as I know my team will.

If you have a DNS server on the SoftLayer network, and you’re not sure whether it’s configured to prevent it from being used for these types of attacks, our support team is happy to help you out. For those of you interested in doing a little DNS homework to learn more, Google’s Developer Network has an awesome overview of DNS security threats and mitigations which gives an overview of potential attacks and preventative measures you can take. If you’re just looking for an easy way to close an open recursor, scroll to the bottom of CloudFlare’s post, and follow their quick guide.

If, on the other hand, you have your own DNS server and you don’t want to worry about all of this configuration or administration, SoftLayer operates private DNS resolvers that are limited to our announced IP space. Feel free to use ours instead!

-Ryan

Beyond our responsibility to enforce the law and our Acceptable Use Policy, the Abuse department is designed to be a valuable asset for our customers. We’ll notify you of all valid complaints (and possibly highlight security vulnerabilities in the process), we’ll assist you with blacklist removal, we can serve as a liaison between you and other providers if there are any problems, and if you operate an email-heavy platform or service, we can help you understand the steps you need to take to avoid activity that may be considered abuse.

At the end of the day, if the Abuse department can maintain a good rapport with our customers, both our jobs can be easier, so I thought this installment in the “Tips from the Abuse Department” series could focus on some best practices for corresponding with Abuse from a customer perspective.

Check Your Tickets

This is the easiest, most obvious recommendation I can give. You’d be surprised at how many service interruptions could be avoided if our customers were more proactive about keeping up with their open tickets. Our portal is a vital tool for your business, so make sure you are familiar with how to access and use it.

Keep Your Contact Information Current

Our ticket system will send notifications to the email address you have on file, so making sure this information is correct and current is absolutely crucial, especially if you aren’t in the habit of checking the ticket system on a regular basis. You can even set a specific address for abuse notifications to be sent to, so make use of this option. The quicker you can respond to an abuse report, the quicker the complaint can be resolved, and by getting the complaint resolved quickly, you avoid any potential service interruption.

If we are unable to reach you by ticket, we may need to call you, so keep your current phone numbers on file as well.

Provide Frequent Updates

Stay in constant communication in the midst of responding to an abuse report, and adhere to the allotted timeline in the ticket. If we don’t see updates that the abusive behavior is being addressed in the grace period we are able to offer, your server is at risk of disconnection. By keeping us posted about the action you’re taking and the time you need to resolve the matter, we’re able to be more flexible.

If a customer on your servers created a spamming script or a phishing account, taking immediate steps to mitigate the issue by suspending that customer is another great way to respond to the process while you’re performing an investigation of how that activity was started. We’ll still want a detailed resolution, but if the abuse is not actively ongoing we can work with you on deadlines.

Be Concise … But Not Too Concise

One-word responses: bad. Page long responses: also not ideal. If given the option we would opt for the latter, but your goal should be to outline the cause and resolution of any reported abusive activity as clearly and succinctly as possible in order to ease communication and expedite closing of the ticket.

Responding to a ticket with, “Fixed,” is not sufficient to for the Abuse department to consider the matter resolved, but we also don’t need a dump of your entire log file. Before the Abuse team can close a ticket, we have to see details of how the complaint was resolved, so if you don’t provide those details in your first response, you can bet we’ll keep following up with you to get them. What details do we need?

Take a Comprehensive Approach

In addition to stopping the abusive activity we want to know:

1. How/why the issue occurred
2. What steps are being taken to prevent further issues of that nature

We understand that dealing with abuse issues can often feel like a game of Whack-A-Mole, but if you can show that you’re digging a bit deeper and taking steps to avoid recurrence, that additional work is very much appreciated. Having the Abuse department consider you a proactive, ethical and responsible customer is a worthy goal.

Be Courteous

I’m ending on a similar note to my last blog post because it’s just that important! We understand getting an abuse ticket is a hassle, but please remember that we’re doing our best to protect our network, the Internet community and you.

Unplugging your server is a last resort for us, and we want to make sure everyone is on the same page to prevent us from getting to that last resort. In the unfortunate event that you do experience an abuse issue, please refer back to this blog — it just might save you some headaches and perhaps some unnecessary downtime.

-Jennifer

It stands to reason that the more efficient people are at reporting abuse, the more efficient we can be at shutting down the activity, so I’ve compiled some tips and resources to make this process easier. Enjoy!

Review our Legal Page

Not only does this page contain our contact details, there’s a wealth of information on our policies including what we consider abuse and how we handle reported issues. For starters, you may want to review our AUP (Acceptable Use Policy) to get a feel for our stance on abuse and how we mitigate it.

Follow Proper Guidelines

In addition to our own policies, there are legal aspects we must consider. For example, a claim of copyright infringement must be submitted in the form of a properly formatted DMCA, pursuant to the Digital Millennium Copyright Act. Our legal page contains crucial information on what is required to make a copyright claim, as well as information on how to submit a subpoena or court order. We take abuse very seriously, but we must adhere to the law as well as our privacy policy in order to protect our customers’ businesses and our company from liability.

Include Evidence

Evidence can take the form of any number of things. A few common examples:

• A copy of the alleged spam message with full headers intact.
• A snippet from your log file showing malicious activity.
• The full URL of a phishing page.

Without evidence that clearly ties abusive activity to a server on our network, we are unable to relay a complaint to our customer. Keep in mind that the complaint must be in a format that allows us to verify it and pass it along, which typically means an email or hard copy. While our website does have contact numbers and addresses, email is your best bet for most types of complaints.

Use Keywords

We use a mail client specifically developed for abuse desks, and it is configured with a host of rules used for filtering and prioritization. Descriptive subject lines with keywords indicating the issue type are very useful. Including the words “Spam,” “Phishing” or “Copyright” in your subject line helps make sure your email is sent to the correct queue and, if applicable, receives expedited processing. Including the domain name and IP address in the body of the email is also helpful.

Follow Up

We work hard to investigate and resolve all complaints received however, due to volume, we typically do not respond to complaining parties. That said, we often rely on user complaints to determine if an issue has resumed or is ongoing so feel free to send a new complaint if activity persists.

Be Respectful

The only portion of your complaint we are likely to relay to our customer is the evidence itself along with any useful notes, which means that paragraph of profanity is read only by hardworking SoftLayer employees. We understand the frustration of being on the receiving end of spam or a DDOS, but please be professional and try to understand our position. We are on your side!

Hopefully you’ve found some of this information useful. When in doubt, submit your complaint to abuse@softlayer.com and we can offer further guidance. Stay tuned for Part 2, where I’ll offer suggestions for SoftLayer customers about how to facilitate better communication with our Abuse department to avoid service interruption if an abuse complaint is filed against you.

-Jennifer

DDoS Protection: Do you need it?

In the last couple of years, you’ve probably seen a references to denial-of-service attacks in the news and how the fallout from those attacks can leave businesses ‘picking up pieces’ for weeks or months after they occur. Think about the helplessness you’d feel if the business you poured your heart and soul into is shut down by some malicious person or group’s attack on your web presence. Worse yet, those attacks are usually for that person’s or group’s own monetary gain or to satisfy some ego-driven urge to punish you for being successful in either your business or a cause you believe in.

It happens all too often, and most people don’t realize that it can actually happen to them. On a weekly basis, I speak to at least one person that tells me, “We’re small, and we really don’t have any competitors … Our website is down. If we can’t stop this attack, I am going to have to send all of my employees home and close down!”

The truth is that denial-of-service protection providers normally sell “fear.” They do this because people don’t have answers to a few key questions about DDoS protection:

• What are the real statistics?
• What is the probability that my website will be hit with an attack?
• What is the real cost of impact?
• What about my data center? I’m sure they already have protection, right?
• We’ve never been hit before, so why should we consider it a priority?

Many of the causes for hesitation regarding the purchase of denial-of-service protection revolve around the lack of education and valid, statistical data. Most know about Distributed-Denial-of-Service (DDoS), but the details are hard to come by. Most people don’t have experience with attacks, so many assume it’s only the big companies or governments that need to worry.

The untold truth is that DDoS attacks occur on a daily basis, and as many as 2500 attacks occur every 24-hour period throughout the world. In the first 6 months of 2011, ServerOrigin saw 2.3 times the number of attacks we observed and mitigated over the course of the 2010 calendar year.

Let’s say you’re considering protection, and you want to ensure that you remain in control and your business continues operating even if you’re the target of extortion or a collective political or religious movement … and that’s assuming there is a reason behind the attack. Over 86% of attacks occur with no explanation! Considering that statistical tidbit, maybe fear isn’t just a manufactured, marketing gimmick to get you buying.

One of the biggest roadblocks in proactive DDoS mitigation in the past has been cost. The average cost for 12 months of a DDoS mitigation appliance or service to protect 1000Mbps is $78,942.00 – just for the equipment/service. Then you have to factor in the variable cost of the bandwidth USED during an attack.

ServerOrigin created ethProxy as a service that overlays your current server platform at an affordable price point. SoftLayer provides one of the best dedicated hosting environments, and we’ve built our reputation on DDoS protection, not hosting, so we bring our service to you.

How do I get protected? How time consuming is this process?
Contrary to the belief that DDoS mitigation is some mystical technology that is painful to implement, our ethProxy mitigation service works on the same premise as a Global Load Balancer or reverse-proxy. Setup is as simple as changing your website’s DNS record to a protected IP in our ethProxy filtering cloud, and once that is done, inbound connections are filtered so only clean traffic is sent to your web server. Transition to the ethProxy service is transparent for the end-user and requires no downtime to implement.

The average deployment time is less than 1-hour and the ethProxy protected IP becomes your public interface to the world. Not only does our service pass rigorous PCI certifications, it guarantees your hosted infrastructure is no longer vulnerable to attack, it allows for upgrading your bandwidth/protection in seconds, and it removes the need for additional web application firewalls or accelerators. On average, customers save around 71% by going with ethProxy when they compare us against the cost of traditional filtering methods.

Our ethProxy service is the combination of many different features or services that you may already pay for separately. This allows businesses to transition to our protection service by replacing one of their current providers which would be prove redundant with the ethProxy subscription. Why go through a budgeting process again when you can simply use a different provider that offers you DDoS protection in addition to the service you’re already paying for?

Included in Every ethProxy DDoS Mitigation Package

• Global Load Balancing
• DDoS Protected AnyCast DNS Services
• Multiple US locations / Complete Datacenter Redundancy
• Instant Scalability – Powered by ServerOrigin’s Cloud Network
• Global Content Delivery Network (CDN)

The options and overall value of these services provide protection that no website should be without, while saving you a ton of money … Especially when you consider that running all of these services separately could cost as much as $10,000/mo.

ServerOrigin Communications services more than 1,200,000 million domains worldwide. Our ethProxy service is the single largest globally deployed mitigation service worldwide, and we protect everyone from non-profit organizations to entire governments and some of Wall Street’s largest online providers (No, we’re not allowed to tell you which ones).

Let us show you why our expertise has saved hundreds of businesses and how we can ensure you never have to ‘pick up the pieces.’

- Kevin Hatfield, ServerOrigin Communications