Posts Tagged ‘abuse’

May 7, 2013

Tips from the Abuse Department: DMCA Takedown Notices

By in Customer Service, SoftLayer

If you are in the web hosting business or you provide users with access to store content on your servers, chances are that you’re familiar with the Digital Millennium Copyright Act (DMCA). If you aren’t familiar with it, you certainly should be. All it takes is one client plagiarizing an article or using a filesharing program unscrupulously, and you could find yourself the recipient of a scary DMCA notice from a copyright holder. We’ve talked before about how to file a DMCA complaint with SoftLayer, but we haven’t talked in detail about SoftLayer’s role in processing DMCA complaints or what you should do if you find yourself on the receiving end of a copyright infringement notification.

The most important thing to understand when it comes to the way the abuse team handles DMCA complaints is that our procedures aren’t just SoftLayer policy — they are the law. Our role in processing copyright complaints is essentially that of a middleman. In order to protect our Safe Harbor status under the Online Copyright Infringement Liability Limitation Act (OCILLA), we must enforce any complaint that meets the legal requirements of a takedown notice. That DMCA complaint must contain specific elements and be properly formatted in order to be considered valid.

Responding to a DMCA Complaint

When we receive a complaint that meets the legal requirements of a DMCA takedown notice, we must relay the complaint to our direct customer and enforce a deadline for removal of the violating material. We are obligated to remove access to infringing content when we are notified about it, and we aren’t able to make a determination about the validity of a claim beyond confirming that all DMCA requirements are met.

The law states that SoftLayer must act expeditiously, so if you receive notification of a DMCA complaint, it’s important that you acknowledge the ticket that the abuse department opened on your account and let us know your intended course of action. Sometimes that action is as simple as removing an infringing URL. Sometimes you may need to contact your client and instruct them to take the material down. Whatever the case may be, it’s important to be responsive and to expressly confirm when you have complied and removed the material. Failure to acknowledge an abuse ticket can result in disconnection of service, and in the case of copyright infringement, SoftLayer has a legal obligation to remove access to the material or we face serious liability.

DMCA Counter Notifications

Most DMCA complaints are resolved without issue, but what happens if you disagree with the complaint? What if you own the material and a disgruntled former business partner is trying to get revenge? What if you wrote the content and the complaining party is copying your website? Thankfully there are penalties for filing a false DMCA complaint, but you also have recourse in the form of a counter notification. Keep in mind that while it may be tempting to plead your case to the abuse department, our role is not to play judge or jury but to allow the process to work as it was designed.

In some cases, you may be able to work out a resolution with the complaining party directly (misunderstandings happen, licenses lapse, etc.) and have them send a retraction, but most of the time your best course of action is to submit a counter notification.

Just as a takedown notice must be crafted in a specific way, counter notifications have their own set of requirements. Once you have disabled the material identified in the original complaint, we can provide your valid, properly formatted counter notification to the complaining party. Unless we receive a court order from the complaining party within the legally mandated time frame the material can be re-enabled and the case is closed for the time being.

While it might sound complicated, it’s actually pretty straightforward, but we urge you to do your research and make sure you know what to do in the event a client of yours is hit with a DMCA takedown notice. Just as we are unable to make judgment calls when it comes to takedown notices or counter notifications, we are also unable to offer any legal advice for you if you need help. Hopefully this post cleared up a few questions and misconceptions about how the abuse department handles copyright complaints. In short:

Do take DMCA notifications seriously. You are at risk for service interruption and possible legal liability.
Do respond to the abuse department letting them know the material has been disabled and, if applicable, if you plan to file a counter notification.
Don’t refuse to disable the material. Even if you believe the claim is false and you wish to file a counter notification, the material must be disabled within the time period allotted by the abuse department or we have to block access to it.
Don’t expect the abuse department to take sides.

As with any abuse issue, communication and responsiveness is important. Disconnecting your server is a last resort, but we have ethical and legal obligations to uphold. The DMCA process certainly has its weaknesses and it leaves a bit to be desired, but at the end of the day, it’s the law, and we have to operate inside of our legal obligation to it.

-Jennifer

November 2, 2012

The Trouble with Open DNS Resolvers

By in SoftLayer, Technology, Tips and Tricks

In the last couple of days, there’s been a bit of buzz about “open DNS resolvers” and DNS amplification DDoS attacks, and SoftLayer’s name has been brought up a few times. In a blog post on October 30, CloudFlare explained DNS Amplification DDoS attacks and reported the geographic and network sources of open DNS resolvers that were contributing to a 20Gbps attack on their network. SoftLayer’s AS numbers (SOFTLAYER and the legacy THEPLANET-AS number) show up on the top ten “worst offenders” list, and Dan Goodin contacted us to get a comment for a follow-up piece on Ars Technica — Meet the network operators helping to fuel the spike in big DDoS attacks.

While the content of that article is less sensationalized than the title, there are still a few gaps to fill about when it comes to how SoftLayer is actually involved in the big picture (*SPOILER ALERT* We aren’t “helping to fuel the spike in big DDoS attacks”). The CloudFlare blog and the Ars Technica post presuppose that the presence of open recursive DNS resolvers is a sign of negligence on the part of the network provider at best and maliciousness at worst, and that’s not the case.

The majority of SoftLayer’s infrastructure is made up of self-managed dedicated and cloud servers. Customers who rent those servers on a monthly basis have unrestricted access to operate their servers in any way they’d like as long as that activity meets our acceptable use policy. Some of our largest customers are hosting resellers who provide that control to their customers who can then provide that control to their own customers. And if 23 million hostnames reside on the SoftLayer network, you can bet that we’ve got a lot of users hosting their DNS on SoftLayer infrastructure. Unfortunately, it’s easier for those customers and customers-of-customers and customers-of-customers-of-customers to use “defaults” instead of looking for, learning and implementing “best practices.”

It’s all too common to find those DNS resolvers open and ultimately vulnerable to DNS amplification attacks, and whenever our team is alerted to that vulnerability on our network, we make our customers aware of it. In turn, they may pass the word down the customer-of-customer chain to get to the DNS owner. It’s usually not a philosophical question about whether DNS resolvers should be open for the greater good of the Internet … It’s a question of whether the DNS owner has any idea that their “configuration” is vulnerable to be abused in this way.

SoftLayer’s network operations, abuse and support teams have tools that flag irregular and potentially abusive traffic coming from any server on our network, and we take immediate action when we find a problem or are alerted to one by someone who sends details to abuse@softlayer.com. The challenge we run into is that flagging obvious abusive behavior from an active DNS server is a bit of a cat-and-mouse game … Attackers cloak their activity in normal traffic. Instead of sending a huge amount of traffic from a single domain, they send a marginal amount of traffic from a large number of machines, and the “abusive” traffic is nearly impossible for even the DNS owner to differentiate from “regular” traffic.

CloudFlare effectively became a honeypot, and they caught a distributed DNS amplification DoS attack. The results they gathered are extremely valuable to teams like mine at SoftLayer, so if they go the next step to actively contact the abuse channel for each of the network providers in their list, I hope that each of the other providers will jump on that information as I know my team will.

If you have a DNS server on the SoftLayer network, and you’re not sure whether it’s configured to prevent it from being used for these types of attacks, our support team is happy to help you out. For those of you interested in doing a little DNS homework to learn more, Google’s Developer Network has an awesome overview of DNS security threats and mitigations which gives an overview of potential attacks and preventative measures you can take. If you’re just looking for an easy way to close an open recursor, scroll to the bottom of CloudFlare’s post, and follow their quick guide.

If, on the other hand, you have your own DNS server and you don’t want to worry about all of this configuration or administration, SoftLayer operates private DNS resolvers that are limited to our announced IP space. Feel free to use ours instead!

-Ryan

October 25, 2012

Tips from the Abuse Department: Save Your Sinking Ship

By in Customer Service, SoftLayer, Tips and Tricks

I often find that the easiest way to present a complex process is with a relatable analogy. By replacing esoteric technical details with a less intimidating real-world illustration, smart people don’t have to be technically savvy to understand what’s going on. When it comes to explaining abuse-related topics, I find analogies especially helpful. One that I’m particularly keen on in explaining Abuse tickets in the context of a sinking ship.

How many times have you received an Abuse ticket and responded to the issue by suspending what appears to be the culprit account? You provide an update in the ticket, letting our team know that you’ve “taken care of the problem,” and you consider it resolved. A few moments later, the ticket is updated on our end, and an abuse administrator is asking follow-up questions: “How did the issue occur?” “What did you do to resolve the issue?” “What steps are being taken to secure the server in order to prevent further abuse?”

Who cares how the issue happened if it’s resolved now, right? Didn’t I respond quickly and address the problem in the ticket? What gives? Well, dear readers, it’s analogy time:

You’re sailing along in a boat filled with important goods, and the craft suddenly begins to take on water. It’s not readily apparent where the water is coming from, but you have a trusty bucket that you fill with the water in the boat and toss over the side. When you toss out all the water onboard, is the problem fixed? Perhaps. Perhaps not.

You don’t see evidence of the problem anymore, but as you continue along your way, your vessel might start riding lower and lower in the water — jeopardizing yourself and your shipment. If you were to search for the cause of the water intake and take steps to patch it, the boat would be in a much better condition to deliver you and your cargo safely to your destination.

In the same way that a hull breach can sink a ship, so too can a security hole on your server cause problems for your (and your clients’) data. In the last installment of “Tips from the Abuse Department,” Andrew explained some of the extremely common (and often overlooked) ways servers are compromised and used maliciously. As he mentioned in his post, Abuse tickets are, in many cases, the first notification for many of our customers that “something’s wrong.”

At a crucial point like this, it’s important to get the water out of the boat AND prevent the vessel from taking on any more water. You won’t be sailing smoothly unless both are done as quickly as possible.

Let’s look at an example of what thorough response to an Abuse ticket might look like:

A long-time client of yours hosts their small business site on one of your servers. You are notified by Abuse that malware is being distributed from a random folder on their domain. You could suspend the domain and be “done” with the issue, but that long-time client (who’s not in the business of malware distribution) would suffer. You decide to dig deeper.

After temporarily suspending the account to stop any further malware distribution, you log into the server and track down the file and what permissions it has. You look through access logs and discover that the file was uploaded via FTP just yesterday from an IP in another country. With this IP information, you search your logs and find several other instances where suspicious files were uploaded around the same time, and you see that several FTP brute force attempts were made against the server.

You know what happened: Someone (or something) scanned the server and attempted to break into the domain. When the server was breached, malware was uploaded to an obscure directory on the domain where the domain owners might not notice it.

With this information in hand, you can take steps to protect your clients and the server itself. The first step might be to implement a password policy that would make guessing passwords very difficult. Next, you might add a rule within your FTP configuration to block continued access after a certain number of failed logins. Finally, you would clean the malicious content from the server, reset the compromised passwords, and unsuspend the now-clean site.

While it’s quite a bit more work than simply identifying the domain and account responsible for the abuse and suspending it, the extra time you spent investigating the cause of the issue will prevent the same issue from happening after your client “fixes” the problem by deleting the files/directories. Invariably, they’d get compromised again in the same way when the domain is restored, and you’d hear from the Abuse department again.

Server security goes hand in hand with systems administration, and even though it’s not a very fun part of the job, it is a 24/7 responsibility that requires diligence and vigilance. By investing time and effort into securing your servers and fixing your hull breach rather than just bailing water overboard, your customers will see less downtime, you’ll be using your server resources more efficiently, and (best of all) you won’t have the Abuse team hounding you about more issues!

-Garrett

P.S. I came up with a brilliant analogy about DNS and the postal service, so that might be a topic for my next post …

October 23, 2012

Tips from the Abuse Department: Know Spam. Stop Spam.

By in Infrastructure, Technology, Tips and Tricks

As an abuse administrator, I’m surrounded by spam on a daily basis. When someone sends an abuse-related complaint to our abuse@softlayer.com contact address, it gets added to our ticket queue, and our Abuse SLayers take time to investigate and follow up with the customers whose servers violate our acceptable use policy. The majority of those abuse-related submissions are reporting spam coming from our network, and in my interaction with customers, I’ve noticed that spam (and the source of spam) is widely misunderstood.

Most spam tickets we create on customer accounts pinpoint spam sent from a compromised or exploited server. Our direct customer didn’t send the phishing email, malware distribution, pharmacy advertisement or pornographic spam, but that activity came from their account. While they’re accountable for the abusive behavior coming from their server, in many cases, they don’t know that there’s a problem until we post an abuse ticket on their account. These servers are targeted and compromised by common techniques and exploits that could have been easily avoided, but they aren’t very well known outside the world of abuse.

To protect yourself from a spammer, you need to think like a spammer. You need to understand how someone might try to exploit your environment so that you can prevent them from doing so. As you’re looking at ways to secure your server proactively, make sure you target these five exploits in particular:

1. User Auth Login

This is by far the most common exploit to used to send spam. This method involves a person or script using the credentials of a user to send spam through a domain’s mail server. The majority of these incidences are caused by malware on a client PC that obtains the login and password for a domain user and uses that information to log on and send mail from the client PC through the server. Often, these spam messages are sent through a botnet command structure.

When an account is compromised, simply changing the password for the compromised user on the server usually won’t stop the abuse. We see quite a few accounts that continue to send spam after an initial abuse ticket results in a password change. Most servers that are sending spam with this method are found to only be sending a small amount of spam at any given time to avoid detection. The low volume of spam that is being sent per server is made up for by the fact that there are thousands of servers being used for the same spamming campaigns.

In order to stop the User Auth Login exploit, a customer needs to clean all of the malicious software (malware) from their environments. To prevent future User Auth Login compromises, users should be made aware of the potential dangers of untrusted software, and if they believe their machines are infected, they need to know what to do.

2. Tell-a-friend Exploitation

The User Auth Login technique is the most common method employed by spammers, but the “tell-a-friend” script exploitation isn’t far behind when it comes to volume of affected servers. This spamming method find websites that use scripts to invite users to refer friends to a page or product. Spammers will use the ‘Your Message’ field in one of these scripts to input their own content and links, and they’ll push the actual page referral link to the bottom of the message. When these site scripts aren’t secure, the spammer will use them to send hundreds or thousands of messages.

To avoid having your website fall victim to this type of spam, be very wary of any widget or script you add. If you need to add Facebook, Twitter and email “share” functionality to your site, make sure you incorporate a tell-a-friend script that does not allow for customizable messages or does not accept input of more than one email address. Also, users won’t need the “cc” or “bcc” fields, so you can be sure those are axed as well. If you can’t find a good “share” script that you’re comfortable with from a security perspective, it might be a good idea to remove that functionality to avoid exploitation.

3. Uploaded Mailers

Spam sent via an uploaded third party mailer can sometimes prove difficult for admins to locate. An uploaded third party mailer could be capable of creating it’s own outbound SMTP connection, and that would allow a program to bypass the existing MTA on the server and render any legitimate mail logs useless for investigation. Another challenge is that a php mailer can be uploaded to a location within a user’s web content, and that mailer is run by the user ‘nobody’ (the default Apache user).

We strongly suggest configuring your server to have the mail headers show the script’s user (that’s not the Apache default user) and the location the script is running from on the server. Many times, these kinds of mailers are maliciously uploaded after a user’s FTP password is been compromised, so be sure your FTP login information is secure.

4. Software Exploits

The “software exploits” category casts a huge shadow. Every piece of software on a server — from mail servers, content management systems and control panels to the operating system itself — can be targeted by hackers. They probe servers to find security vulnerabilities and weak coding, and when they find a vulnerability, they take control.

The hacker who found the software vulnerability might not actually take advantage of the exploit immediately. That user may sell access to other entities for their use, and that use often ends up being spam. In addition to having strong firewall rules and access restrictions, you should update and maintain the current stable versions of all software on your servers.

5. WordPress Exploits

WordPress exploits would technically fall under the “Software Exploits” category, but I’m breaking it out into its own category simply due to the volume of spam issues that are the result of exploiting this particular piece of software. The first step to protecting against spam being sent through this source is to make sure you have the latest version of WordPress installed. With that done, be sure to research the latest security plugins for that version and install any that are applicable to your environment.

These five techniques are not the only ones used by spammers to take advantage of your environment, but they are some of the most common. To protect yourself from becoming a source of spam, make your servers a more difficult target to exploit. To stop spam, you need to know spam. Now that you know spam, it’s time to stop it. Ask questions, test your environment regularly and watch your logs for any unexplained usage.

-Andrew

June 25, 2012

Tips from the Abuse Department: Part 2 – Responding to Abuse Reports

By in Customer Service, SoftLayer, Tips and Tricks

If you’re a SoftLayer customer, you don’t want to hear from the Abuse department. We know that. The unfortunate reality when it comes to hosting a server is that compromises can happen, mistakes can be made, and even the most scrupulous reseller can fall victim to a fraudulent sign-up or sly spammer. If someone reports abusive behavior originating from one of your servers on our network, it’s important to be able to communicate effectively with the Abuse department and build a healthy working relationship.

Beyond our responsibility to enforce the law and our Acceptable Use Policy, the Abuse department is designed to be a valuable asset for our customers. We’ll notify you of all valid complaints (and possibly highlight security vulnerabilities in the process), we’ll assist you with blacklist removal, we can serve as a liaison between you and other providers if there are any problems, and if you operate an email-heavy platform or service, we can help you understand the steps you need to take to avoid activity that may be considered abuse.

At the end of the day, if the Abuse department can maintain a good rapport with our customers, both our jobs can be easier, so I thought this installment in the “Tips from the Abuse Department” series could focus on some best practices for corresponding with Abuse from a customer perspective.

Check Your Tickets

This is the easiest, most obvious recommendation I can give. You’d be surprised at how many service interruptions could be avoided if our customers were more proactive about keeping up with their open tickets. Our portal is a vital tool for your business, so make sure you are familiar with how to access and use it.

Keep Your Contact Information Current

Our ticket system will send notifications to the email address you have on file, so making sure this information is correct and current is absolutely crucial, especially if you aren’t in the habit of checking the ticket system on a regular basis. You can even set a specific address for abuse notifications to be sent to, so make use of this option. The quicker you can respond to an abuse report, the quicker the complaint can be resolved, and by getting the complaint resolved quickly, you avoid any potential service interruption.

If we are unable to reach you by ticket, we may need to call you, so keep your current phone numbers on file as well.

Provide Frequent Updates

Stay in constant communication in the midst of responding to an abuse report, and adhere to the allotted timeline in the ticket. If we don’t see updates that the abusive behavior is being addressed in the grace period we are able to offer, your server is at risk of disconnection. By keeping us posted about the action you’re taking and the time you need to resolve the matter, we’re able to be more flexible.

If a customer on your servers created a spamming script or a phishing account, taking immediate steps to mitigate the issue by suspending that customer is another great way to respond to the process while you’re performing an investigation of how that activity was started. We’ll still want a detailed resolution, but if the abuse is not actively ongoing we can work with you on deadlines.

Be Concise … But Not Too Concise

One-word responses: bad. Page long responses: also not ideal. If given the option we would opt for the latter, but your goal should be to outline the cause and resolution of any reported abusive activity as clearly and succinctly as possible in order to ease communication and expedite closing of the ticket.

Responding to a ticket with, “Fixed,” is not sufficient to for the Abuse department to consider the matter resolved, but we also don’t need a dump of your entire log file. Before the Abuse team can close a ticket, we have to see details of how the complaint was resolved, so if you don’t provide those details in your first response, you can bet we’ll keep following up with you to get them. What details do we need?

Take a Comprehensive Approach

In addition to stopping the abusive activity we want to know:

  1. How/why the issue occurred
  2. What steps are being taken to prevent further issues of that nature

We understand that dealing with abuse issues can often feel like a game of Whack-A-Mole, but if you can show that you’re digging a bit deeper and taking steps to avoid recurrence, that additional work is very much appreciated. Having the Abuse department consider you a proactive, ethical and responsible customer is a worthy goal.

Be Courteous

I’m ending on a similar note to my last blog post because it’s just that important! We understand getting an abuse ticket is a hassle, but please remember that we’re doing our best to protect our network, the Internet community and you.

Unplugging your server is a last resort for us, and we want to make sure everyone is on the same page to prevent us from getting to that last resort. In the unfortunate event that you do experience an abuse issue, please refer back to this blog — it just might save you some headaches and perhaps some unnecessary downtime.

-Jennifer

June 18, 2012

Tips from the Abuse Department: Part 1 – Reporting Abuse

By in Customer Service, SoftLayer, Tips and Tricks

SoftLayer has a dedicated team working around the clock to address complaints of abuse on our network. We receive these complaints via feedback loops from other providers, spam blacklisting services such as Spamcop and Spamhaus, various industry contacts and mailing lists. Some of the most valuable complaints we receive are from our users, though. We appreciate people taking the time to let us know about problems on our network, and we find these complaints particularly valuable as they are non-automated and direct from the source.

It stands to reason that the more efficient people are at reporting abuse, the more efficient we can be at shutting down the activity, so I’ve compiled some tips and resources to make this process easier. Enjoy!

Review our Legal Page

Not only does this page contain our contact details, there’s a wealth of information on our policies including what we consider abuse and how we handle reported issues. For starters, you may want to review our AUP (Acceptable Use Policy) to get a feel for our stance on abuse and how we mitigate it.

Follow Proper Guidelines

In addition to our own policies, there are legal aspects we must consider. For example, a claim of copyright infringement must be submitted in the form of a properly formatted DMCA, pursuant to the Digital Millennium Copyright Act. Our legal page contains crucial information on what is required to make a copyright claim, as well as information on how to submit a subpoena or court order. We take abuse very seriously, but we must adhere to the law as well as our privacy policy in order to protect our customers’ businesses and our company from liability.

Include Evidence

Evidence can take the form of any number of things. A few common examples:

  • A copy of the alleged spam message with full headers intact.
  • A snippet from your log file showing malicious activity.
  • The full URL of a phishing page.

Without evidence that clearly ties abusive activity to a server on our network, we are unable to relay a complaint to our customer. Keep in mind that the complaint must be in a format that allows us to verify it and pass it along, which typically means an email or hard copy. While our website does have contact numbers and addresses, email is your best bet for most types of complaints.

Use Keywords

We use a mail client specifically developed for abuse desks, and it is configured with a host of rules used for filtering and prioritization. Descriptive subject lines with keywords indicating the issue type are very useful. Including the words “Spam,” “Phishing” or “Copyright” in your subject line helps make sure your email is sent to the correct queue and, if applicable, receives expedited processing. Including the domain name and IP address in the body of the email is also helpful.

Follow Up

We work hard to investigate and resolve all complaints received however, due to volume, we typically do not respond to complaining parties. That said, we often rely on user complaints to determine if an issue has resumed or is ongoing so feel free to send a new complaint if activity persists.

Be Respectful

The only portion of your complaint we are likely to relay to our customer is the evidence itself along with any useful notes, which means that paragraph of profanity is read only by hardworking SoftLayer employees. We understand the frustration of being on the receiving end of spam or a DDOS, but please be professional and try to understand our position. We are on your side!

Hopefully you’ve found some of this information useful. When in doubt, submit your complaint to abuse@softlayer.com and we can offer further guidance. Stay tuned for Part 2, where I’ll offer suggestions for SoftLayer customers about how to facilitate better communication with our Abuse department to avoid service interruption if an abuse complaint is filed against you.

-Jennifer

January 18, 2012

Keep Fighting: SOPA on the Ropes. PIPA Lurking.

By in Business, News, SoftLayer, Technology

The Internet is unnervingly quiet today. In response to the Stop Online Piracy Act (SOPA) in the House of Representatives and the Protect IP Act (PIPA) in the Senate, some of the most popular sites on the web have gone dark today – demonstrating the danger (and the potential unchecked power) of these two bills.

Late Friday afternoon, Judiciary Committee Chairman Lamar Smith announced that the DNS-blocking provisions would be removed from SOPA, and on Saturday, The White House responded to in opposition to the the bills as they stand today. Shortly thereafter, SOPA was “shelved.”

The Internet was abuzz … but the Champagne wasn’t getting popped yet. After digging into the details, it was revealed that SOPA being “shelved” just meant that it is being temporarily put to sleep. Judiciary Committee Chairman Lamar Smith stood explained:

“To enact legislation that protects consumers, businesses and jobs from foreign thieves who steal America’s intellectual property, we will continue to bring together industry representatives and Members to find ways to combat online piracy.

Due to the Republican and Democratic retreats taking place over the next two weeks, markup of the Stop Online Piracy Act is expected to resume in February.”

I only mention this because it’s important not to forget that SOPA isn’t dead, and it’s still very dangerous. If you visit sites like reddit, Wikipedia, Mozilla and Boing Boing today (January 18, 2012), you experience the potential impact of the legislation.

The Internet’s outrage against SOPA has brought about real change in our nation’s capital: The House is reconsidering the bill, and they’ll hopefully dismiss it. With our collective momentum, we need to look at the PROTECT IP Act (PIPA, or Senate Bill 968) – a similar bill with similarly harmful implications that’s been sneaking around in SOPA’s shadow.

As it is defined today, PIPA has a stated goal of providing the US Government and copyright holders an additional arsenal of tools to aide in taking down ‘rogue websites dedicated to infringing or counterfeit goods.’ The Senate bill details that an “information location tool shall take technically feasible and reasonable measures, as expeditiously as possible, to remove or disable access to the Internet site associated with the domain name set forth in the order.” In addition, it must delete all hyperlinks to the offending “Internet site.”

Our opposition to PIPA is nearly identical to our opposition to SOPA. Both require a form of essentially breaking a core aspect of how the Internet functions – whether that breakage happens in DNS (as detailed in my last blog post) or in the required rearchitecture of the way any site that accepts user-generated content has to respond to PIPA-related complaints.

PIPA is scheduled for Senate vote on January 24, 2012. It is important that you voice your opinion with your government representatives and let them know about your opposition to both SOPA and PIPA. We want to help you get started down that path. Find your local representatives’ contact information:

[SOPA Concerns]: Contact your congressperson in the U.S. House of Representatives
[PIPA Concerns]: Contact your Senator in the U.S. Senate

Keep spreading the word, and make sure your voice is heard.

-@toddmitchell

January 12, 2012

How the Internet Works (And How SOPA Would Break It)

By in Business, Executive Blog, SoftLayer, Technology

Last week, I explained SoftLayer’s stance against SOPA and mentioned that SOPA would essentially require service providers like SoftLayer to “break the Internet” in response to reports of “infringing sites.” The technical readers in our audience probably acknowledged the point and moved on, but our non-technical readers (and some representatives in Congress) might have gotten a little confused by the references to DNS, domains and IP addresses.

Given how pervasive the Internet is in our daily lives, you shouldn’t need to be “a techie” to understand the basics of what makes the Internet work … And given the significance of the SOPA legislation, you should understand where the bill would “break” the process. Let’s take a high level look at how the Internet works, and from there, we can contrast how it would work if SOPA were to pass.

The Internet: How Sites Are Delivered

  1. You access a device connected in some way to the Internet. This device can be a cell phone, a computer or even a refrigerator. You are connected to the Internet through an Internet Service Provider (ISP) which recognizes that you will be accessing various sites and services hosted remotely. Your ISP manages a network connected to the other networks around the globe (“inter” “network” … “Internet”).
  2. You enter a domain name or click a URL (for this example, we’ll use http://www.softlayer.com since we’re biased to that site).

Internet Basics

  1. Your ISP will see that you want to access “www.softlayer.com” and will immediately try to find someone/something that knows what “www.softlayer.com” means … This search is known as an NS (name server) lookup. In this case, it will find that “www.softlayer.com” is associated with several name servers.

Internet Basics

  1. The first of these four name servers to respond with additional information about “softlayer.com” will be used. Domains are typically required to be associated with two or three name servers to ensure if one is unreachable, requests for that domain name can be processed by another.
  2. The name server has Domain Name System (DNS) information that maps “www.softlayer.com” to an Internet Protocol (IP) address. When a domain name is purchased and provisioned, the owner will associate that domain name with an authoritative DNS name server, and a DNS record will be created with that name server linking the domain to a specific IP address. Think of DNS as a phone book that translates a name into a phone number for you.

Internet Basics

  1. When the IP address you reach sees that you requested “www.softlayer.com,” it will find the files/content associated with that request. Multiple domains can be hosted on the same IP address, just as multiple people can live at the same street address and answer the phone. Each IP address only exists in a single place at a given time. (There are some complex network tricks that can negate that statement, but in the interest of simplicity, we’ll ignore them.)
  2. When the requested content is located (and generated by other servers if necessary), it is returned to your browser. Depending on what content you are accessing, the response from the server can be very simple or very complex. In some cases, the request will return a single HTML document. In other cases, the content you access may require additional information from other servers (database servers, storage servers, etc.) before the request can be completely fulfilled. In this case, we get HTML code in return.

Internet Basics

  1. Your browser takes that code and translates the formatting and content to be displayed on your screen. Often, formatting and styling of pages will be generated from a Cascading Style Sheet (CSS) referenced in the HTML code. The purpose of the style sheet is to streamline a given page’s code and consolidate the formatting to be used and referenced by multiple pages of a given website.

Internet Basics

  1. The HTML code will reference sources for media that may be hosted on other servers, so the browser will perform the necessary additional requests to get all of the media the website is trying to show. In this case, the most noticeable image that will get pulled is the SoftLayer logo from this location: http://static2.softlayer.com/images/layout/logo.jpg

Internet Basics

  1. When the HTML is rendered and the media is loaded, your browser will probably note that it is “Done,” and you will have successfully navigated to SoftLayer’s homepage.

If SOPA were to pass, the process would look like this:

The Internet: Post-SOPA

  1. You access a device connected in some way to the Internet.
  2. You enter a domain name or click a URL (for this example, we’ll use http://www.softlayer.com since we’re biased to that site).

*The Change*

  1. Before your ISP runs an NS lookup, it would have to determine whether the site you’re trying to access has been reported as an “infringing site.” If http://www.softlayer.com was reported (either legitimately or illegitimately) as an infringing site, your ISP would not process your request, and you’d proceed to an error page. If your ISP can’t find any reference to the domain an infringing site, it would start looking for the name server to deliver the IP address.
  2. SOPA would also enforce filtering from all authoritative DNS provider. If an ISP sends a request for an infringing site to the name server for that site, the provider of that name server would be forced to prevent the IP address from being returned.
  3. One additional method of screening domains would happen at the level of the operator of the domain’s gTLD. gTLDs (generic top-level domains) are the “.____” at the end of the domain (.com, .net, .biz, etc.). Each gTLD is managed by a large registry organization, and a gTLD’s operator would be required to prevent an infringing site’s domain from functioning properly.
  4. If the gTLD registry operator, your ISP and the domain’s authoritative name server provider agree that the site you’re accessing has not been reported as an infringing site, the process would resume the pre-SOPA process.

*Back to the Pre-SOPA Process*

  1. The domain’s name server responds.
  2. The domain’s IP address is returned.
  3. The IP address is reached to get the content for http://www.softlayer.com.
  4. HTML is returned.
  5. Your browser translates the HTML into a visual format.
  6. External file references from the HTML are returned.
  7. The site is loaded.

The proponents of SOPA are basically saying, “It’s difficult for us to keep up with and shut down all of the instances of counterfeiting and copyright infringement online, but it would be much easier to target the larger sites/providers ‘enabling’ users to access that (possible) infringement.” Right now, the DMCA process requires a formal copyright complaint to be filed for every instance of infringement, and the providers who are hosting the content on their network are responsible for having that content removed. That’s what our abuse team does full-time. It’s a relatively complex process, but it’s a process that guarantees us the ability to investigate claims for legitimacy and to hear from our customers (who hear from their customers) in response to the claims.

SOPA does not allow for due process to investigate concerns. If a site is reported to be an infringing site, service providers have to do everything in their power to prevent users from getting there.

-@toddmitchell

January 6, 2012

SOPA: Bad for Hosting

By in Business, Executive Blog, SoftLayer, Technology

SoftLayer manages more than 100,000 servers in thirteen data centers around the world. We have more than 23,000 customers, and those customers are responsible for millions of websites (which get billions of pageviews every month). We’re one of the largest hosting providers in the world, and we want to talk a little about the Stop Online Piracy Act (H.R. 3261 or “SOPA”).

Many in our industry have already commented (and in some cases, “changed their minds”) on SOPA and its equally evil twin, the PROTECT IP Act (“PIPA”) in the Senate, but we wanted to share our perspective on the legislation. Even with these Dudley-Do-Right, Goody-Two-Shoes titles and their ambitious goals, SoftLayer opposes these bills in their current forms because they expose innocent and law-abiding hosting companies to uncertain liabilities.

Because this legislation has gotten quite a bit of attention in the past few months, you’re probably already familiar with it, but if you haven’t paid much attention, we can give you a quick summary: As you can read in the name of the bill, SOPA is being proposed to “Stop Online Piracy.” SOPA is under consideration by the House Judiciary Committee, and its intent is to provide additional enforcement tools to combat foreign ‘rogue’ websites that are dedicated to copyright infringement or counterfeiting. That’s a great goal, and SoftLayer does not oppose the intent of the Act … As you saw from Kevin Hazard’s blog post a few weeks ago, we have a team of people working all the time to track down and immediately address any violations of our terms of service (including copyright infringement), so we wholeheartedly agree that copyright infringement and counterfeiting are bad.

The way SOPA tries to address the problem is where we disagree with the bill, so let’s talk about the most pertinent part of the bill for a service provider like SoftLayer. If SOPA were to pass, when a case of infringement is reported, we would have to “take such measures as [we determine] to be the least burdensome, technically feasible, and reasonable means designed to prevent access by [our] subscribers located within the United States to the foreign infringing site that is subject to the order.”

What that means: We would be forced to turn off our customers’ access to a small piece of the Internet.

How are we to do that? Well the “least burdensome, technically feasible, and reasonable means designed to prevent access” are not made clear, but most of the discussions about the bill have focused on changing the way the Doman Name System (DNS) resolves to an “infringing site.” We’d be more or less ordered to break DNS … DNS was designed to simply, accurately and quickly match a domain name with the IP address that domain’s owner provides, and if SOPA were to pass, we’d have to tell DNS to behave correctly for every site EXCEPT the reported infringing sites. Again, that’s not spelled out in the legislation, so it’s like being given a job by someone who has no idea how to do the job nor whether the job is even possible to successfully complete.

And that’s all assuming that the order to suspend access to an “infringing site” is legitimate. Many of the organizations that oppose SOPA have explained possible scenarios where orders could be filed under the guise of preventing copyright infringement. A competing site/business could claim:

“the operator of the site operates the site with the object of promoting, or has promoted, its use to carry out acts that constitute a violation of section 501 or 1201 of title 17, United States Code, as shown by clear expression or other affirmative steps taken to foster such violation.”

In another scenario, a copyright holder could pull the trigger on an order simply at the thought that a user could infringe on a copyright on/via the “infringing site.”

When the United States House of Representatives reconvenes after its winter recess, we will be watching intently with hopes that the Internet’s response to the bill has effectively derailed it in its current form. As SoftLayer General Council Suzy Fulton mentioned in her post about Texas House Bill 1841, we’ve been working with an industry group called TechAmerica which submitted a letter to Congress about SOPA and many of the issues that could negatively affect our industry. Additionally, we’ve gotten involved with SaveHosting.org to speak out against laws that can hurt our customers.

As discussions continue about SOPA, we’ll look for opportunities to share more of our insight with you here on our blog. Please let us know your thoughts about the legislation below.

-@toddmitchell

December 15, 2011

Fighting SPAM and Abuse on a Global Network

By in Infrastructure, Social Media, SoftLayer, Technology

For better or worse, one of the most engaging posts on the SoftLayer Blog is “We are a No-Spam Network,” written by Jacob Linscott in June 2007. When it was posted, it celebrated a completely clear Spamhaus listing page – quite an accomplishment for a large hosting provider (for reasons I’ll illustrate below). Since the post was published, it has become a hotbed of conversation about any and all abuse-related issues. Google “SoftLayer SPAM,” and you’ll see the post show up as the second result, so a lot of Internet passers-by will come across the post and use the comment section as a platform to share abuse-related concerns they have for us.

That engagement is a double-edge sword: It’s good because we hear the concerns people have. It’s bad because the post was meant to be a celebration of the continuous work that the abuse department does, and uninitiated visitors seem to consider it a unilateral claim that we’ve beaten spam once and for all. In the course of responding to comments on that post, I shared an analogy to convey what it’s like to run abuse for a large hosting provider:

Scenario

Let’s say you’re the security manager for a huge mall. This mall has 100,000 stores with people walking in and out 24x7x365. In this scenario, there are “good guys” and “bad guys” who walk into and out of the mall, and every person looks exactly the same. Some of those people are store owners while others are customers of those stores. As the security manager for the mall, you want to maintain the safest, most well-maintained mall in the world, so when you find bad guys walking in and out of your mall, you do everything you can to kick them out and keep them out. Sometimes those bad guys are store owners who attract and send the wrong crowd; sometimes they are bad guy customers of a good guy store owner.

How would you manage your mall? It’s not possible to differentiate whether a store owner will be a good guy or a bad guy when they’re applying to lease space in your mall, so you can’t “keep the bad guys out” in that regard. You can’t have a security team of 100,000 people monitoring what’s happening in those 100,000 stores, much less have someone individually check the millions of visitors streaming in and out of the stores. What’s a security manager to do?

If you look at how Las Vegas casinos address that concern, it’s clear that your best bet is to install security cameras and have a team monitoring them all the time. You might not be able to watch everything at the same time, but you can document what’s happening around your mall and respond if you notice something unusual (or if someone calls in to report that they’ve seen bad guys coming from a store in your mall).

That’s the position we’re in.

SoftLayer Abuse Team

SoftLayer’s network is the mall, the stores are servers, the store owners are our customers (who are often responsible for several “stores”), and the good guys and bad guys are traffic into and out of the network. We try to differentiate good guys and bad guys, but even if we know that all good guys have purple eyes and all bad guys have neon green eyes, it’s still difficult to look 26,000+ store owners in the eye every day as they’re walking into and out of the mall.

We staff a team of people intent on clearing the bad guys from our mall, and we know that even though good guy store owners may inadvertently host their own bad guy customers, they want to remove those customers from their store as well, so they appreciate us helping them pinpoint those customers so they can be removed.

We keep an eye on our security cameras and get our security guards to the stores where bad guys are reported as quickly as possible. If no one reports that the people coming out of store #73,403 are all bad guys, it’s hard for us to know that they aren’t good guys … Which is why we encourage anyone and everyone to report abuse-related concerns to abuse@softlayer.com so we can mobilize our security force.

As Edmund Burke once said, “When bad men combine, the good must associate; else they will fall one by one, an unpitied sacrifice in a contemptible struggle.” Or more colloquially, “All that is necessary for the triumph of evil is that good men do nothing.”

Given that illustration, the abuse team deserves a LOT of credit for the work they do behind the scenes. They are constantly investigating reports and working with customers to get remove any and all content that violate SoftLayer’s MSA, and too often, that can be a thankless job. Fighting abuse is an ongoing process, and while the nature of the beast might suggest the overall war will never be won, we’re always getting faster and stronger, so the individual battles are easier and easier to win.

-@khazard