The Trouble with Open DNS Resolvers

November 2, 2012

In the last couple of days, there's been a bit of buzz about "open DNS resolvers" and DNS amplification DDoS attacks, and SoftLayer's name has been brought up a few times. In a blog post on October 30, CloudFlare explained DNS Amplification DDoS attacks and reported the geographic and network sources of open DNS resolvers that were contributing to a 20Gbps attack on their network. SoftLayer's AS numbers (SOFTLAYER and the legacy THEPLANET-AS number) show up on the top ten "worst offenders" list, and Dan Goodin contacted us to get a comment for a follow-up piece on Ars Technica — Meet the network operators helping to fuel the spike in big DDoS attacks.

While the content of that article is less sensationalized than the title, there are still a few gaps to fill about when it comes to how SoftLayer is actually involved in the big picture (*SPOILER ALERT* We aren't "helping to fuel the spike in big DDoS attacks"). The CloudFlare blog and the Ars Technica post presuppose that the presence of open recursive DNS resolvers is a sign of negligence on the part of the network provider at best and maliciousness at worst, and that's not the case.

The majority of SoftLayer's infrastructure is made up of self-managed dedicated and cloud servers. Customers who rent those servers on a monthly basis have unrestricted access to operate their servers in any way they'd like as long as that activity meets our acceptable use policy. Some of our largest customers are hosting resellers who provide that control to their customers who can then provide that control to their own customers. And if 23 million hostnames reside on the SoftLayer network, you can bet that we've got a lot of users hosting their DNS on SoftLayer infrastructure. Unfortunately, it's easier for those customers and customers-of-customers and customers-of-customers-of-customers to use "defaults" instead of looking for, learning and implementing "best practices."

It's all too common to find those DNS resolvers open and ultimately vulnerable to DNS amplification attacks, and whenever our team is alerted to that vulnerability on our network, we make our customers aware of it. In turn, they may pass the word down the customer-of-customer chain to get to the DNS owner. It's usually not a philosophical question about whether DNS resolvers should be open for the greater good of the Internet ... It's a question of whether the DNS owner has any idea that their "configuration" is vulnerable to be abused in this way.

SoftLayer's network operations, abuse and support teams have tools that flag irregular and potentially abusive traffic coming from any server on our network, and we take immediate action when we find a problem or are alerted to one by someone who sends details to abuse@softlayer.com. The challenge we run into is that flagging obvious abusive behavior from an active DNS server is a bit of a cat-and-mouse game ... Attackers cloak their activity in normal traffic. Instead of sending a huge amount of traffic from a single domain, they send a marginal amount of traffic from a large number of machines, and the "abusive" traffic is nearly impossible for even the DNS owner to differentiate from "regular" traffic.

CloudFlare effectively became a honeypot, and they caught a distributed DNS amplification DoS attack. The results they gathered are extremely valuable to teams like mine at SoftLayer, so if they go the next step to actively contact the abuse channel for each of the network providers in their list, I hope that each of the other providers will jump on that information as I know my team will.

If you have a DNS server on the SoftLayer network, and you're not sure whether it's configured to prevent it from being used for these types of attacks, our support team is happy to help you out. For those of you interested in doing a little DNS homework to learn more, Google's Developer Network has an awesome overview of DNS security threats and mitigations which gives an overview of potential attacks and preventative measures you can take. If you're just looking for an easy way to close an open recursor, scroll to the bottom of CloudFlare's post, and follow their quick guide.

If, on the other hand, you have your own DNS server and you don't want to worry about all of this configuration or administration, SoftLayer operates private DNS resolvers that are limited to our announced IP space. Feel free to use ours instead!

-Ryan

Comments

November 2nd, 2012 at 11:43am

Thanks for clearing this up Ryan. This makes perfect sense due to the type of business that Softlayer runs.

Maybe it should be part of the AUP to properly configure DNS services; or if you're going to have open resolvers, provide a service that helps mitigate this type of attack. Sort of like what Google does :)

 

March 14th, 2013 at 8:43am

Hi Ryan,

Can't you actively scan for open DNS resolvers on your network? The bad guys certainly managed to find them, can't you?

March 14th, 2013 at 12:49pm

Finding open DNS resolvers on the network is not the challenge ... The challenge is getting customers who operate open DNS resolvers to understand what they are, why they're not advised and what they can do to avoid them. For self-managed dedicated and cloud services on SoftLayer's network, our team doesn't configure or change our customers' environments, so we don't have direct access to close those DNS resolvers.

March 14th, 2013 at 12:54pm

Wouldn't a link to a single well-written document explaining all that be enough for your customers to understand it and to deal with the problem?

I understand you can't simply suspend their servers, but now you're basically knowingly hosting abusable services, right?

March 14th, 2013 at 1:49pm

The majority of the open DNS resolvers are not being operated by SoftLayer's direct customers, so while we actively encourage our customers to advise their customers on better ways to handle their DNS resolvers, we don't communicate with the end users directly. As Ryan mentions in this blog, the users most likely to run open DNS resolvers are less-technical shared hosting customers who rent a piece of a server from a reseller who might still be disconnected from SoftLayer by layers of other customers. There are certainly people who believe that open DNS resolvers should be left open for the good of the Internet, but typically, open DNS resolvers are operated by users who don't want to lock themselves out of their DNS by misconfiguring access.

Any hosting service can be abused ... Every email address is a potential source of spam, and every server is a potential target for a hacker looking to break in and put up a phishing site to steal credit card information. It is specifically *because* we host "abusable" services that we have a full abuse team to investigate and take action if we notice or are contacted about any abuse from our network.

March 14th, 2013 at 2:14pm

Open (unprotected) DNS resolvers can and will be abused, right?
It kinda sounds like you're saying that's okay, sometimes, maybe.

Such abuse is NOT okay, even if such services are run by indirect customers.

March 15th, 2013 at 1:08pm

As I suggested in my previous comment, every aspect of a hosting environment *can* be abused. Email "can and will be abused" by hackers and spammers, but SoftLayer still allows our customers to use email services.

If open DNS resolvers were used exclusively for abuse, they would be forbidden on our network. Because open DNS resolvers have valid uses, we can advise customers to use private DNS resolvers (like the ones SoftLayer provides to our customers for free) but we don't force the open DNS resolvers to be closed simply because have potential for abuse.

Our terms of service state that any abusive behavior on the SoftLayer network is forbidden. Regardless of whether a customer is direct or indirect, if our abuse team is alerted to abusive activity coming from any server on our network, they take action to stop that activity.

March 16th, 2013 at 7:54am

How does one stop an open DNS resolver from being abused?
And what are valid uses of an open DNS resolver?

March 18th, 2013 at 11:29am

Are you asking the first question rhetorically? If it's a serious question, the way Google describes how they built Google Public DNS gives some insight into how they implemented several recommended solutions to help guarantee the authenticity of the responses Google Public DNS receives from other nameservers.

The valid use of an open DNS resolver is to allow users to resolve DNS over the public Internet.

March 18th, 2013 at 11:46am

I'm pretty sure Google's DNS services aren't as easy to abuse as the open DNS services run by your customers. What % of open DNS services of your customers is intended to be open and abusable?

I assume the majority isn't, which begs the question why they're still open and being abused (after 4 months).

Leave a Reply

Filtered HTML

  • Web page addresses and e-mail addresses turn into links automatically.
  • You can enable syntax highlighting of source code with the following tags: <pre>, <blockcode>, <bash>, <c>, <cpp>, <drupal5>, <drupal6>, <java>, <javascript>, <php>, <python>, <ruby>. The supported tag styles are: <foo>, [foo].
  • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
By submitting this form, you accept the Mollom privacy policy.

Comments

November 2nd, 2012 at 11:43am

Thanks for clearing this up Ryan. This makes perfect sense due to the type of business that Softlayer runs.

Maybe it should be part of the AUP to properly configure DNS services; or if you're going to have open resolvers, provide a service that helps mitigate this type of attack. Sort of like what Google does :)

 

March 14th, 2013 at 8:43am

Hi Ryan,

Can't you actively scan for open DNS resolvers on your network? The bad guys certainly managed to find them, can't you?

March 14th, 2013 at 12:49pm

Finding open DNS resolvers on the network is not the challenge ... The challenge is getting customers who operate open DNS resolvers to understand what they are, why they're not advised and what they can do to avoid them. For self-managed dedicated and cloud services on SoftLayer's network, our team doesn't configure or change our customers' environments, so we don't have direct access to close those DNS resolvers.

March 14th, 2013 at 12:54pm

Wouldn't a link to a single well-written document explaining all that be enough for your customers to understand it and to deal with the problem?

I understand you can't simply suspend their servers, but now you're basically knowingly hosting abusable services, right?

March 14th, 2013 at 1:49pm

The majority of the open DNS resolvers are not being operated by SoftLayer's direct customers, so while we actively encourage our customers to advise their customers on better ways to handle their DNS resolvers, we don't communicate with the end users directly. As Ryan mentions in this blog, the users most likely to run open DNS resolvers are less-technical shared hosting customers who rent a piece of a server from a reseller who might still be disconnected from SoftLayer by layers of other customers. There are certainly people who believe that open DNS resolvers should be left open for the good of the Internet, but typically, open DNS resolvers are operated by users who don't want to lock themselves out of their DNS by misconfiguring access.

Any hosting service can be abused ... Every email address is a potential source of spam, and every server is a potential target for a hacker looking to break in and put up a phishing site to steal credit card information. It is specifically *because* we host "abusable" services that we have a full abuse team to investigate and take action if we notice or are contacted about any abuse from our network.

March 14th, 2013 at 2:14pm

Open (unprotected) DNS resolvers can and will be abused, right?
It kinda sounds like you're saying that's okay, sometimes, maybe.

Such abuse is NOT okay, even if such services are run by indirect customers.

March 15th, 2013 at 1:08pm

As I suggested in my previous comment, every aspect of a hosting environment *can* be abused. Email "can and will be abused" by hackers and spammers, but SoftLayer still allows our customers to use email services.

If open DNS resolvers were used exclusively for abuse, they would be forbidden on our network. Because open DNS resolvers have valid uses, we can advise customers to use private DNS resolvers (like the ones SoftLayer provides to our customers for free) but we don't force the open DNS resolvers to be closed simply because have potential for abuse.

Our terms of service state that any abusive behavior on the SoftLayer network is forbidden. Regardless of whether a customer is direct or indirect, if our abuse team is alerted to abusive activity coming from any server on our network, they take action to stop that activity.

March 16th, 2013 at 7:54am

How does one stop an open DNS resolver from being abused?
And what are valid uses of an open DNS resolver?

March 18th, 2013 at 11:29am

Are you asking the first question rhetorically? If it's a serious question, the way Google describes how they built Google Public DNS gives some insight into how they implemented several recommended solutions to help guarantee the authenticity of the responses Google Public DNS receives from other nameservers.

The valid use of an open DNS resolver is to allow users to resolve DNS over the public Internet.

March 18th, 2013 at 11:46am

I'm pretty sure Google's DNS services aren't as easy to abuse as the open DNS services run by your customers. What % of open DNS services of your customers is intended to be open and abusable?

I assume the majority isn't, which begs the question why they're still open and being abused (after 4 months).

Leave a Reply

Filtered HTML

  • Web page addresses and e-mail addresses turn into links automatically.
  • You can enable syntax highlighting of source code with the following tags: <pre>, <blockcode>, <bash>, <c>, <cpp>, <drupal5>, <drupal6>, <java>, <javascript>, <php>, <python>, <ruby>. The supported tag styles are: <foo>, [foo].
  • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
By submitting this form, you accept the Mollom privacy policy.