While the content of that article is less sensationalized than the title, there are still a few gaps to fill about when it comes to how SoftLayer is actually involved in the big picture (*SPOILER ALERT* We aren’t “helping to fuel the spike in big DDoS attacks”). The CloudFlare blog and the Ars Technica post presuppose that the presence of open recursive DNS resolvers is a sign of negligence on the part of the network provider at best and maliciousness at worst, and that’s not the case.

The majority of SoftLayer’s infrastructure is made up of self-managed dedicated and cloud servers. Customers who rent those servers on a monthly basis have unrestricted access to operate their servers in any way they’d like as long as that activity meets our acceptable use policy. Some of our largest customers are hosting resellers who provide that control to their customers who can then provide that control to their own customers. And if 23 million hostnames reside on the SoftLayer network, you can bet that we’ve got a lot of users hosting their DNS on SoftLayer infrastructure. Unfortunately, it’s easier for those customers and customers-of-customers and customers-of-customers-of-customers to use “defaults” instead of looking for, learning and implementing “best practices.”

It’s all too common to find those DNS resolvers open and ultimately vulnerable to DNS amplification attacks, and whenever our team is alerted to that vulnerability on our network, we make our customers aware of it. In turn, they may pass the word down the customer-of-customer chain to get to the DNS owner. It’s usually not a philosophical question about whether DNS resolvers should be open for the greater good of the Internet … It’s a question of whether the DNS owner has any idea that their “configuration” is vulnerable to be abused in this way.

SoftLayer’s network operations, abuse and support teams have tools that flag irregular and potentially abusive traffic coming from any server on our network, and we take immediate action when we find a problem or are alerted to one by someone who sends details to abuse@softlayer.com. The challenge we run into is that flagging obvious abusive behavior from an active DNS server is a bit of a cat-and-mouse game … Attackers cloak their activity in normal traffic. Instead of sending a huge amount of traffic from a single domain, they send a marginal amount of traffic from a large number of machines, and the “abusive” traffic is nearly impossible for even the DNS owner to differentiate from “regular” traffic.

CloudFlare effectively became a honeypot, and they caught a distributed DNS amplification DoS attack. The results they gathered are extremely valuable to teams like mine at SoftLayer, so if they go the next step to actively contact the abuse channel for each of the network providers in their list, I hope that each of the other providers will jump on that information as I know my team will.

If you have a DNS server on the SoftLayer network, and you’re not sure whether it’s configured to prevent it from being used for these types of attacks, our support team is happy to help you out. For those of you interested in doing a little DNS homework to learn more, Google’s Developer Network has an awesome overview of DNS security threats and mitigations which gives an overview of potential attacks and preventative measures you can take. If you’re just looking for an easy way to close an open recursor, scroll to the bottom of CloudFlare’s post, and follow their quick guide.

If, on the other hand, you have your own DNS server and you don’t want to worry about all of this configuration or administration, SoftLayer operates private DNS resolvers that are limited to our announced IP space. Feel free to use ours instead!

-Ryan

Late Friday afternoon, Judiciary Committee Chairman Lamar Smith announced that the DNS-blocking provisions would be removed from SOPA, and on Saturday, The White House responded to in opposition to the the bills as they stand today. Shortly thereafter, SOPA was “shelved.”

The Internet was abuzz … but the Champagne wasn’t getting popped yet. After digging into the details, it was revealed that SOPA being “shelved” just meant that it is being temporarily put to sleep. Judiciary Committee Chairman Lamar Smith stood explained:

“To enact legislation that protects consumers, businesses and jobs from foreign thieves who steal America’s intellectual property, we will continue to bring together industry representatives and Members to find ways to combat online piracy.

Due to the Republican and Democratic retreats taking place over the next two weeks, markup of the Stop Online Piracy Act is expected to resume in February.”

I only mention this because it’s important not to forget that SOPA isn’t dead, and it’s still very dangerous. If you visit sites like reddit, Wikipedia, Mozilla and Boing Boing today (January 18, 2012), you experience the potential impact of the legislation.

The Internet’s outrage against SOPA has brought about real change in our nation’s capital: The House is reconsidering the bill, and they’ll hopefully dismiss it. With our collective momentum, we need to look at the PROTECT IP Act (PIPA, or Senate Bill 968) – a similar bill with similarly harmful implications that’s been sneaking around in SOPA’s shadow.

As it is defined today, PIPA has a stated goal of providing the US Government and copyright holders an additional arsenal of tools to aide in taking down ‘rogue websites dedicated to infringing or counterfeit goods.’ The Senate bill details that an “information location tool shall take technically feasible and reasonable measures, as expeditiously as possible, to remove or disable access to the Internet site associated with the domain name set forth in the order.” In addition, it must delete all hyperlinks to the offending “Internet site.”

Our opposition to PIPA is nearly identical to our opposition to SOPA. Both require a form of essentially breaking a core aspect of how the Internet functions – whether that breakage happens in DNS (as detailed in my last blog post) or in the required rearchitecture of the way any site that accepts user-generated content has to respond to PIPA-related complaints.

PIPA is scheduled for Senate vote on January 24, 2012. It is important that you voice your opinion with your government representatives and let them know about your opposition to both SOPA and PIPA. We want to help you get started down that path. Find your local representatives’ contact information:

[SOPA Concerns]: Contact your congressperson in the U.S. House of Representatives
[PIPA Concerns]: Contact your Senator in the U.S. Senate

Keep spreading the word, and make sure your voice is heard.

-@toddmitchell

Given how pervasive the Internet is in our daily lives, you shouldn’t need to be “a techie” to understand the basics of what makes the Internet work … And given the significance of the SOPA legislation, you should understand where the bill would “break” the process. Let’s take a high level look at how the Internet works, and from there, we can contrast how it would work if SOPA were to pass.

The Internet: How Sites Are Delivered

1. You access a device connected in some way to the Internet. This device can be a cell phone, a computer or even a refrigerator. You are connected to the Internet through an Internet Service Provider (ISP) which recognizes that you will be accessing various sites and services hosted remotely. Your ISP manages a network connected to the other networks around the globe (“inter” “network” … “Internet”).
2. You enter a domain name or click a URL (for this example, we’ll use http://www.softlayer.com since we’re biased to that site).

3. Your ISP will see that you want to access “www.softlayer.com” and will immediately try to find someone/something that knows what “www.softlayer.com” means … This search is known as an NS (name server) lookup. In this case, it will find that “www.softlayer.com” is associated with several name servers.

4. The first of these four name servers to respond with additional information about “softlayer.com” will be used. Domains are typically required to be associated with two or three name servers to ensure if one is unreachable, requests for that domain name can be processed by another.
5. The name server has Domain Name System (DNS) information that maps “www.softlayer.com” to an Internet Protocol (IP) address. When a domain name is purchased and provisioned, the owner will associate that domain name with an authoritative DNS name server, and a DNS record will be created with that name server linking the domain to a specific IP address. Think of DNS as a phone book that translates a name into a phone number for you.

6. When the IP address you reach sees that you requested “www.softlayer.com,” it will find the files/content associated with that request. Multiple domains can be hosted on the same IP address, just as multiple people can live at the same street address and answer the phone. Each IP address only exists in a single place at a given time. (There are some complex network tricks that can negate that statement, but in the interest of simplicity, we’ll ignore them.)
7. When the requested content is located (and generated by other servers if necessary), it is returned to your browser. Depending on what content you are accessing, the response from the server can be very simple or very complex. In some cases, the request will return a single HTML document. In other cases, the content you access may require additional information from other servers (database servers, storage servers, etc.) before the request can be completely fulfilled. In this case, we get HTML code in return.

8. Your browser takes that code and translates the formatting and content to be displayed on your screen. Often, formatting and styling of pages will be generated from a Cascading Style Sheet (CSS) referenced in the HTML code. The purpose of the style sheet is to streamline a given page’s code and consolidate the formatting to be used and referenced by multiple pages of a given website.

9. The HTML code will reference sources for media that may be hosted on other servers, so the browser will perform the necessary additional requests to get all of the media the website is trying to show. In this case, the most noticeable image that will get pulled is the SoftLayer logo from this location: http://static2.softlayer.com/images/layout/logo.jpg

10. When the HTML is rendered and the media is loaded, your browser will probably note that it is “Done,” and you will have successfully navigated to SoftLayer’s homepage.

If SOPA were to pass, the process would look like this:

The Internet: Post-SOPA

1. You access a device connected in some way to the Internet.
2. You enter a domain name or click a URL (for this example, we’ll use http://www.softlayer.com since we’re biased to that site).

*The Change*

3. Before your ISP runs an NS lookup, it would have to determine whether the site you’re trying to access has been reported as an “infringing site.” If http://www.softlayer.com was reported (either legitimately or illegitimately) as an infringing site, your ISP would not process your request, and you’d proceed to an error page. If your ISP can’t find any reference to the domain an infringing site, it would start looking for the name server to deliver the IP address.
4. SOPA would also enforce filtering from all authoritative DNS provider. If an ISP sends a request for an infringing site to the name server for that site, the provider of that name server would be forced to prevent the IP address from being returned.
5. One additional method of screening domains would happen at the level of the operator of the domain’s gTLD. gTLDs (generic top-level domains) are the “.____” at the end of the domain (.com, .net, .biz, etc.). Each gTLD is managed by a large registry organization, and a gTLD’s operator would be required to prevent an infringing site’s domain from functioning properly.
6. If the gTLD registry operator, your ISP and the domain’s authoritative name server provider agree that the site you’re accessing has not been reported as an infringing site, the process would resume the pre-SOPA process.

*Back to the Pre-SOPA Process*

7. The domain’s name server responds.
8. The domain’s IP address is returned.
9. The IP address is reached to get the content for http://www.softlayer.com.
10. HTML is returned.
11. Your browser translates the HTML into a visual format.
12. External file references from the HTML are returned.
13. The site is loaded.

The proponents of SOPA are basically saying, “It’s difficult for us to keep up with and shut down all of the instances of counterfeiting and copyright infringement online, but it would be much easier to target the larger sites/providers ‘enabling’ users to access that (possible) infringement.” Right now, the DMCA process requires a formal copyright complaint to be filed for every instance of infringement, and the providers who are hosting the content on their network are responsible for having that content removed. That’s what our abuse team does full-time. It’s a relatively complex process, but it’s a process that guarantees us the ability to investigate claims for legitimacy and to hear from our customers (who hear from their customers) in response to the claims.

SOPA does not allow for due process to investigate concerns. If a site is reported to be an infringing site, service providers have to do everything in their power to prevent users from getting there.

-@toddmitchell

10. SoftLayer terminates 40Gbps to every single rack!! 20Gbps to the public internet and 20Gbps to the private network.
9. SoftLayer offers three types of VPN services for out-of-band connectivity (SSL, PPTP, IPSEC)
8. SoftLayer manages its own nationwide MPLS network with 10 PoPs and over 1000Gbps of transit and peer connectivity
7. SoftLayer offers free enterprise grade DNS services through our DNS farms located in all 10 PoPs in North America
6. SoftLayer has over 1600 APIs for custom integration, a full service control panel for ease of use and a private label option for resellers
5. Every single server in every datacenter is a rackmount, hotswap, tool-less chassis offering enterprise grade hardware with ultra-fast modifications
4. SoftLayer has downloadable iPhone, Android and Blackberry apps in addition to our mobile phone friendly .mobi site for complete control.
3. Only hybrid solution available – dedicated, virtualized, and cloud instances operating in a single environment and control thru a single interface or API
2. Private Network – connect any server to any other server in any datacenter with a click of a button
1. Fastest service delivery

• Over 1000 servers in stock
• Dedicated servers – 4 hours or less
• Servers with virtualization – 2 hours or less
• Cloud instances & storage – 5 to 15 minutes
• Firewalls, Load Balancers, SAN Storage – added real time w/ no downtime

-@lavosby

On the end-user side, I just read an article about how public DNS providers like OpenDNS and Google are breaking the internet. OK, maybe not breaking the internet, but the public DNS providers are confusing CDN location-based algorithms. The article is here: http://www.sajalkayan.com/in-a-cdnd-world-opendns-is-the-enemy.html and I recommend strongly that both content providers and content consumers read it.

The summary is that some CDN algorithms use the ip address (and location) of the DNS server making the request and if that DNS server is nowhere near the end-user on the internet, the end-user will get served content from farther away and will get that content slower than desired. The conclusion is that an end-user should always use a DNS server located as close as possible network-wise, usually that ends up being a DNS server of the network provider.

That is good advice for the end-user, but what about the content provider? If you flip this around and come at DNS from the content provider’s point of view who doesn’t use CDN, you want to make sure that when a DNS request is made, that your authoritative DNS server gets the ip address as fast and reliably as possible back to the end-user.

SoftLayer has built out authoritative DNS farms in all our Datacenters and Network POPs and anycasted the ip addresses for the name servers. What that means is that SoftLayer customers – who get to use our DNS for free – can have their authoritative domain services hosted at all 10 points in North America and through the routing optimization inherent in the internet, the name to number conversion for those domains will happen as close as possible to the end-user and the results will be delivered as quickly as possible.

One very important goal of every content provider is to get the end-user the best experience as possible. Understanding how the internet works from the end-user and well as the server-side is critical. It doesn’t matter how good your content or app is if the end-user has a poor experience.

-@nday91

Fascinating, Michael, why do I care? Well if you ask that question you’ve never had DNS fail on you.

When name resolution goes on the blink one of the tools that support uses to see what is going is the command-line utility nslookup. In its most basic form nslookup is going to do an A record query for the string you supply as an argument and it’ll send that query to your operating system’s configured resolvers.

C:\>nslookup www.softlayer.com
Server: mydns.local
Address: 192.168.0.1



Non-authoritative answer:
Name: www.softlayer.com
Address: 66.228.118.51

What is the utility telling us? First off, it asked a resolver at 192.168.0.1 for the information. Non-authoritative answer means that the server which returned the answer (192.168.0.1) is not the nameserver which controls softlayer.com. It then gives the IP address or addresses which were found.


C:\>nslookup -q=mx softlayer.com ns1.softlayer.com
Server: ns1.softlayer.com
Address: 67.228.254.4


softlayer.com MX preference = 20, mail exchanger = mx02.softlayer.com
softlayer.com MX preference = 30, mail exchanger = mx03.softlayer.com
softlayer.com MX preference = 10, mail exchanger = mx01.softlayer.com
softlayer.com nameserver = ns2.softlayer.net
softlayer.com nameserver = ns1.softlayer.net


This is a slightly different query. Rather than asking my local resolver to do an A record query for www.softlayer.com I’ve sent an MX (mail exchanger) query for softlayer.com directly to the nameserver ns1.softlayer.com. Notice that the response does not have the non-authoritative tag. The server ns1.softlayer.com is one of the nameservers which is configured to respond with a definite answer to a question rather than just saying “well, this other guy said…”.

One thing that both of these queries fail to do is show the TTL for the answer they give. Time to Live (TTL) is what generally controls how long a resolver will keep an answer in cache. While the TTL is valid the resolver will use that answer. Once the TTL expires, the resolver goes looking for a fresh answer. This is great for performance but it does have a dark side to it: because of TTL, changes to DNS records are not seen instantly by all clients. If ClientA hits your website often his resolver is going to have the query result cached (say www.domain.com -> 1.2.3.4). You change the record to www.domain.com -> 5.6.7.8 but ClientA’s resolver is going to continue to respond with 1.2.3.4 until the TTL runs out. If ClientA controls their resolver they can flush its cache. Generally though it is controlled by their ISP and you just have to wait.

To see the TTL for an answer you can use the nslookup form below:


C:\>nslookup
Default Server: mydns.local
Address: 192.168.5.1



> set debug
> www.softlayer.com.
Server: mydns.local
Address: 192.168.5.1



------------
Got answer:
HEADER:
opcode = QUERY, id = 2, rcode = NOERROR
header flags: response, want recursion, recursion avail.
questions = 1, answers = 1, authority records = 2, additional = 0



QUESTIONS:
www.softlayer.com, type = A, class = IN
ANSWERS:
-> www.softlayer.com
internet address = 66.228.118.51
ttl = 86400 (1 day)
AUTHORITY RECORDS:
-> softlayer.com
nameserver = ns1.softlayer.net
ttl = 86400 (1 day)
-> softlayer.com
nameserver = ns2.softlayer.net
ttl = 86400 (1 day)



------------
Non-authoritative answer:
Name: www.softlayer.com
Address: 66.228.118.51


The key to this spew is ‘set debug’ which causes nslookup to display additional information about the response, including the TTL value of the answer. You’ll notice that the TTL in the ANSWERS section is 86400 seconds, which is the number of seconds in one day. This is a common TTL value. If I run the query again though, I have the following answers section:


ANSWERS:
-> www.softlayer.com
internet address = 66.228.118.51
ttl = 85802 (23 hours 50 mins 2


Notice how the TTL is counting down. The resolver is going to continue responding with the answer 66.228.118.51 until that TTL hits zero. At zero, the resolver will go looking for a new answer. What this means for you as a domain operator is that if you know you’re going to be changing a record you should adjust down the TTL for that record a couple of days in advance. For example when some friends and I moved our colo server from one provider to another we dropped the TTLs for our DNS records down to 30 minutes two days prior to the move. Once the move was complete we were able to put them back to prior values.

If you spend any time at all messing with DNS you should play around with nslookup.

If you’re on a Unix system take a look at the command ‘dig’ as well.

Happy resolving.

The first thing that has to happen is for the viewer to make a request by typing in a site name or clicking on a link in a web browser. That request usually has a text-based name as part of the request (like “www.softlayer.com“). Each name has a domain (“softlayer.com“) and each domain has an authoritative nameserver to translate the name into a numerical address. That numerical address is used by the internet infrastructure to make sure the request gets to the right place. Phone numbers work the same way, so just think of an IP address (and domain name) serving the same purpose as a traditional phone number which defines the location of the “owner” of the number (at least in the landline world) based on country, region, and city.

If the nameserver for a name is slow or down, then the request will be delayed, or even worse, fail because the nameserver was not available to translate the name into an address. And if the translation fails, the viewer will not get the content he or she requested.

So, if you are running a website, you want your nameservers to be highly available and service the request as quickly as possible. Here is where I get to brag about SoftLayer a little. We provide nameserver service to our customers. Our customers can use our web portal or a sophisticated programming interface (the SoftLayer API) to manage the numerical addresses for their names. We have located nameservers at several locations and we keep the data synchronized between the sites. Our nameservers themselves have the same addresses using a technology called anycast (http://en.wikipedia.org/wiki/Anycast).

What all this means is that our customers get to have their name to number translation hosted at multiple sites. This results in faster translation times and in the case of a disaster at one site, the other nameservers will still be working.

In other words, SoftLayer has very cool nameservers.

-@nday91

Let’s say you have a domain-something awesome like awesomedomain.wow–and awesomedomain.wow has a TTL of 24 hours. When I go to visit awesomedomain.wow as a new visitor (and you know I would, because it sounds awesome) I’m going to receive a record translating awesomedomain.wow to an IP address that will be valid for 24 hours. Any other time I visit that domain in the next 24 hours, I’m going to use that cached address because the record hasn’t expired yet. In 24 hours regardless of if awesomedomain.wow has moved IPs, I’m going to trash that old DNS record I’ve cached locally and go look it up again. The new record will then be referred to by me for the next 24 hours, at which time I’ll do it all over again.

But what happens when you have to change your IP, but you want your visitors to see the smallest amount of downtime possible? My first suggestion is to mirror your sites on both IPs, but that is a different discussion entirely. The second is to manipulate your TTL. First lower it to something smaller-from a day to an hour perhaps. Then give that new record with that new TTL at least 24 hours to propagate. Now you can be certain that at the 25th hour, all of your visitors now have a record that will expire in one hour. Next, change your IP for awesomedomain.wow, the record that your visitors have cached locally will expire in an hour, and then they will have your new record with your new IP. Feel free to bump your TTL back up to what it was originally in this step, since they have the new IP. Now your visitors have only had an old record for an hour rather than 24, and they probably missed that hour it was inaccessible while they were posing for a painting or having their top hats heightened. Because all of your visitors are terribly classy.

-Joshua