Posts Tagged 'Abuse'

December 5, 2013

How to Report Abuse to SoftLayer

When you find hosted content that doesn't meet our acceptable use policy or another kind of inappropriate Internet activity originating from a SoftLayer service, your natural reaction might be to assume, "SoftLayer must know about it, and the fact that it's going on suggests that they're allowing that behavior." I know this because every now and then, I come across a "@SoftLayer is phishing my email. #spamming #fail" Tweet or a "How about u stop hacking my computer???" Facebook post. It's easy to see where these users are coming from, so my goal for this post is to provide the background you need to understand how behavior we don't condone — what we consider "abuse" of our services — might occur on our platform and what we do when we learn about it.

The most common types of abuse reported from the SoftLayer network are spam, copyright/trademark infringement, phishing and abusive traffic (DDoS attacks). All four are handled by the same abuse team, but they're all handled a bit differently, so it's important to break them down to understand the most efficient way to report them to our team. When you're on the receiving end of abuse, all you want is to make it stop. In the hurry to report the abusive behavior, it's easy to leave out some of the key information we need to address your concern, so let's take a look at each type of abuse and the best ways to report it to the SoftLayer team:

If You Get Spam

Spam is the most common type of abuse that gets reported to SoftLayer. Spam email is unsolicited, indiscriminate bulk messaging that is sent to you without your explicit consent. If you open your email client right now, your junk mail folder probably has a few examples of spam ... Someone is trying to sell you discount drugs or arrange a multi-million dollar inheritance transfer. In many ways, it's great that email is so easy to use and pervasive to our daily lives, but that ease of use also makes it an easy medium for spammers to abuse. Whether the spammer is a direct SoftLayer customer or a customer of one of our customers or somewhere further down the line of customers of customers, spam messages sent from a SoftLayer server will point back to us, and our abuse team is the group that will help stop it.

When you receive spam sent through SoftLayer, you should forward it directly to our abuse team (abuse@softlayer.com). Our team needs a full copy of the email with its headers intact. If you're not sure what that means, check out these instructions on how to retrieve your email headers. The email headers help tell the story about where exactly the messages are coming from and which customer we need to contact to stop the abuse.

If You See Phishing

Phishing abuse might be encountered via spam or you might encounter it on a website. Phishing is best described as someone masquerading as someone else to get your sensitive information, and it's one of the most serious issues our abuse team faces. Every second that a phishing/scam site is online, another user might be fooled into giving up his or her credit card or login information, and we don't want that to happen. Often, the fact that a site is not legitimate is clear relatively quickly, but as defenses against phishing have gotten better, so have the phishing sites. Take a minute to go through this phishing IQ test to get an idea of how difficult phishing can be to trace.

When it comes to reporting phishing, you should send the site's URL to the abuse team (also using abuse@softlayer.com). If you came across the phishing site via a spam email, be sure to include the email headers with your message. To help us filter the phishing complaint, please make sure to include the word "phishing" in your email's subject line. Our team will immediately investigate and follow up with the infringing customer internally.

If You Find Copyright or Trademark Infringement

If infringement of your copyright or trademark is happening on our platform, we want to know about it so we can have it taken down immediately. Copyright complaints and trademark complaints are handled slightly differently, so let's look at each type to better understand how they work.

Complaints of copyright infringement are processed by our abuse team based on the strict DMCA complaint laws. When I say "strict" in that sentence, I'm not saying it lightly ... Because DMCA complaints are legal issues, every requirement in the DMCA must be met in order for our team to act on the complaint. That might seem arbitrary, but we're not given much leeway when it comes to the DMCA process, and we have to be sticklers.

On our DMCA legal page, we outline the process of reporting a DMCA complaint of copyright infringement (primarily citing the statute 17 U.S.C. Section 512(c)(3)). If you don't completely understand what needs to be included in the claim, we recommend that you seek independent legal advice. It sounds harsh, but failure to submit copyright infringement notification as described above will result in no legal notice or action on behalf of SoftLayer. When you've made sure all required evidence has been included in your DMCA complaint, make sure "copyright" or "DMCA" are included in your subject line and submit the complaint to copyright@softlayer.com.

Trademark complaints do not have the same requirements as copyright complaints, but the more information you can provide in your complaint, the easier it will be for our customer to locate and remove the offending material. If you encounter unauthorized use of your registered trademark on our network, please email copyright@softlayer.com with details — the exact location of the infringing content, your trademark registration information, etc. — along with an explanation that this trademark usage is unauthorized and should be removed. In your email, please add the word "trademark" to the subject line to help us filter and prioritize your complaint.

If You See Abusive Traffic

Spam, phishing and copyright infringement are relatively straightforward when it comes to finding and reporting abuse, but sometimes the abuse isn't as visible and tangible (though the effect usually is). If a SoftLayer server is sending abusive traffic to your site, we want to know about it as quickly as possible. Whether that behavior is part of a Denial of Service (DoS) attack or is just scanning ports to possibly attack later, it's important that you give us details so we can prevent any further activity.

To report this type of abuse, send a snippet from your log file including at least 10 lines of logs that show attempts to break into or overload your server. Here's a quick reference to where you can find the relevant logs to send:

  • Email Spam - Send Mail Logs:
    • /var/log/maillog
    • /usr/local/psa/var/log/maillog
  • Brute Force Attacks - Send SSH Logs:
    • /var/log/messages
    • /var/log/secure

Like spam and phishing reports, abusive traffic complaints should be sent to abuse@softlayer.com with a quick explanation of what is happening and any other details you can provide. When you submit a complaint about abusive traffic, make sure your message's subject line reflects the type of issue ("DDoS attack," "brute force attempts," etc.) so our team can investigate your report even quicker.

As I mentioned at the start of this post, these are just four types of abusive behavior that our abuse department addresses on a daily basis. Our Acceptable Use Policy (AUP) outlines what can and cannot be hosted using SoftLayer services, and the process of reporting other types of abuse is generally the same as what you see in the four examples I mentioned above ... Send a clear, concise report to abuse@softlayer.com with key words about the type of violation in the message's subject line. When our team is able to look into your complaint and find the evidence they need to take action, they do so quickly.

I can't wrap up this blog of tips without mentioning the "Tips from the Abuse Department" blog Jennifer Groves wrote about reporting abuse ... It touches on some of the same ideas as this post, and it also provides a little more perspective from behind the lines of the abuse department. As the social media gal, I don't handle abuse on a day-to-day basis, but I do help people dealing with abuse issues, and I know a simple guide like this will be of value.

If an abuse-related issue persists and you don't feel like anything has been fixed, double-check that you've included all the necessary information and evidence in your correspondence to the abuse team. In most cases, you will not receive a response from the abuse team, but that doesn't mean they aren't taking action. The abuse@ and copyright@ email aliases function as notification systems for our abuse teams, and they correspond with the infringing customers internally when a complaint is submitted. Given the fact that hundreds of users may report the same abusive behavior at the same time, responding directly to each message would slow down the process of actually resolving the issue (which is the priority).

If everything was included in your initial correspondence with the abuse team but you still don't notice a change in the abusive behavior, you can always follow up with our social media team at twitter@softlayer.com, and we'll do everything we can to help.

-Rachel

May 29, 2013

Tips from the Abuse Department: To Catch a Predator

We've all seen the emails exclaiming, "THE KING HAS SENT YOU 1,000,000$ US DOLLARS," or "I NEED A PERSONAL ASSISTANT PAYING 500$ A WEEK." Do people actually fall for these? The answer is YES, many do. They think, "What risk is there replying to this email and possibly getting $1,000,000 or even a fraction of that?" As it turns out, there's a lot of risk.

As the senior manager of SoftLayer's abuse department, I know all about these kinds of scams, and I thought I'd reply to one of those emails to show what the interaction usually looks like and explain how the scam works.

---------------------------------------------
From: "Freddy Scammer" <scammer@address>
To: "Freddy Scammer" <scammer@address>
Subject: PA URGENTLY NEEDED

Hi, I am looking for a Personal Assistant, Kindly let me know if you are interested, and i can send you more details. Thank you

Freddy Scammer
---------------------------------------------

First, notice that my address email address isn't listed in the TO field or even the CC field. I must be BCC'd along with many others. I've changed the scammer's fake name to a more fitting name, and I'll use masculine pronouns when I talk about "him." According to our friends over at 419scam.org, this guy has been flagged as a scammer using the same name and email address. The name he provided actually belongs to a company that produces lamps as well as an American historian who focuses on colonization, decolonization and African history.

In the initial message, you'll see that there's no "all or nothing" proposition. Just like any scam, the scammer requests and provides information slowly to reel in a victim.

I replied back:

---------------------------------------------
From: <MY-EMAIL-ADDRESS>
To: <scammer@address>
Subject: RE: PA URGENTLY NEEDED

Doing What?
---------------------------------------------

I wanted to keep it short to see if I could get him to tell me more. He didn't disappoint:

---------------------------------------------
From: Freddy Scammer <scammer@address>
To: <MY-EMAIL-ADDRESS>
Subject: Re: PA URGENTLY NEEDED

Hello

Thanks for your reply, I got your email through the Chamber of Commerce directory. I am looking for someone who can handle my business errands during his or her spare time. I need your service because I am constantly traveling abroad on a missionary trip to build homes for orphaned children and doing other business as we are franchise company into alot of things.

Responsibilities:
1. Receive my mail and drop them off. {Your location doesn't matter as long as you have a post office nearby}
2. Pay my bills.
3. pay our workers on a regular basis

I would have love to meet with you to discuss this job in more detail, but I am currently away on a missionary trip. If you decide to accept the position, please read the employment requirements listed below.
REQUIREMENTS:
A. You are an honest and trustworthy citizen.
B. You need to be able to check your email regular and answer calls.

The pay is $500 weekly and you are entitled to other additional incentives after 1 month if you are hardworking. First, If I were to mail you a payment to
pay people that are needed to and your payment for your service, where would you want it mailed to?

Secondly, how would you like your name to appear on the payment? Note, payment would come in form of Check.

Provide me with the following details below to get started.

Full Name:
Complete Address(No PO Box allowed):
City:
State:
Country:
Zip Code:
Home Phone:
Cell Phone:
Age:
Occupation(If any):
Alternative Email if available:

Awaiting your prompt reply.
---------------------------------------------

Sounds easy enough right? Well it is easy. Who couldn't use an extra $500 a week! But there are a few problems here. If this sounds a lot like a "money mule" (or money laundering) type of situation, that's because it is! A money mule is a person who transfers money acquired illegally (e.g., stolen) in person, through a courier service, or electronically, on behalf of others. The mule is paid for their services, typically a small part of the money transferred.

Money mules are often dupes recruited on-line for what they think is legitimate employment, not aware that the money they are transferring is the product of crime. The money is transferred from the mule's account to the scam operator, typically in another country. Similar techniques are used to transfer illegal merchandise.

After a quick Google search for a few of the sentences in his message, I found out that this guy is low-balling me! He's offering $600 a week in other listings ... I'm hurt! I replied to see if I could get him off script:

---------------------------------------------
From: <MY-EMAIL-ADDRESS>
To: <scammer@address>
Subject: RE: PA URGENTLY NEEDED

So all I have to do is receive packages and re-ship them to where you tell me to,, also receive payments and cash it out and re-pay workers? How will I be paying them, what method? How often will I have to mail packages out and how big are they, who will pay for shipping?
---------------------------------------------

He was quick to respond:

---------------------------------------------
From: Freddy Scammer <scammer@address>
To: <MY-EMAIL-ADDRESS>
Subject: Re: PA URGENTLY NEEDED

Your going to be receiving payment mostly and it has already been paid for, for the shipping . All you have to do is receive and go ahead and cash it ....... Then i will tell you what to do with the money or whoever to pay with it. got me?
---------------------------------------------

Color me amazed. All I have to do is receive a check and cash it?! What luck!

---------------------------------------------
From: <MY-EMAIL-ADDRESS>
To: <scammer@address>
Subject: RE: PA URGENTLY NEEDED
Ok seems easy enough. But I only have a PO BOX, why would this be a problem? I currently don't have a permanent address as I'm staying with a friend trying to get back on my feet and I'm not on the house lease so I can't receive mail here. Is that going to be a problem?
---------------------------------------------

Now none of this is true, but I knew that this would throw Freddy off of his game. Most scammers don't allow a post office box because they don't want to be scammed ... What's to prevent the "victim" from renting a P.O. Box for a month, getting the check, cashing it and cancelling that P.O. Box? That possibility is a risk that scammers don't like to take. There have even been reports that in some instances, the scammers will send goons to your house if you don't hold up your end of the deal.

This whole underground world that you can get quickly and easily sucked into is exciting isn't it?

---------------------------------------------
From: Freddy Scammer <scammer@address>
To: <MY-EMAIL-ADDRESS>
Subject: Re: PA URGENTLY NEEDED

I'm afraid a PO BOX will not suffice, you can perhaps use a family members address and we can start the payments as soon as you send me the info. Please reply with the most urgent intent as I only have a few positions left as my assistant.
---------------------------------------------

At this point, I didn't bother emailing back. It's pretty obvious how easy it could be for someone down on their luck financially (or just bored) to get sucked into this type of scam. What's actually happening here is that the scammer wants to send money from a compromised account to the victim's legit account and then have the victim withdraw 90%-95% of the money and send it to another account that the bad guy has legitimate access to (probably over-seas). The victim would get to keep 5% for their troubles. Often the checks that are sent won't clear, so a victim thinks the funds are in his/her account ... Money is forwarded to the scammer from the victim's legitimate account and it clears before the funds from the scammer's deposited check disappear.

In some instances, scammers will buy high-priced items online with stolen credit card numbers and have those items shipped to the victim's house. The victim will then ship them to a different address. The bad guy has nothing to lose, and the victim takes all the risk.

The challenge with pursuing these scammers from a legal perspective is that they are often based in regions and areas out of the jurisdiction of our law enforcement authorities. As a result, they usually aren't caught, and they just move along to their next unsuspecting victim.

If you receive a "too good to be true" email from someone you don't know, let me spoil the surprise for you: It's not true.

-Dody

Categories: 
May 7, 2013

Tips from the Abuse Department: DMCA Takedown Notices

If you are in the web hosting business or you provide users with access to store content on your servers, chances are that you're familiar with the Digital Millennium Copyright Act (DMCA). If you aren't familiar with it, you certainly should be. All it takes is one client plagiarizing an article or using a filesharing program unscrupulously, and you could find yourself the recipient of a scary DMCA notice from a copyright holder. We've talked before about how to file a DMCA complaint with SoftLayer, but we haven't talked in detail about SoftLayer's role in processing DMCA complaints or what you should do if you find yourself on the receiving end of a copyright infringement notification.

The most important thing to understand when it comes to the way the abuse team handles DMCA complaints is that our procedures aren't just SoftLayer policy — they are the law. Our role in processing copyright complaints is essentially that of a middleman. In order to protect our Safe Harbor status under the Online Copyright Infringement Liability Limitation Act (OCILLA), we must enforce any complaint that meets the legal requirements of a takedown notice. That DMCA complaint must contain specific elements and be properly formatted in order to be considered valid.

Responding to a DMCA Complaint

When we receive a complaint that meets the legal requirements of a DMCA takedown notice, we must relay the complaint to our direct customer and enforce a deadline for removal of the violating material. We are obligated to remove access to infringing content when we are notified about it, and we aren't able to make a determination about the validity of a claim beyond confirming that all DMCA requirements are met.

The law states that SoftLayer must act expeditiously, so if you receive notification of a DMCA complaint, it's important that you acknowledge the ticket that the abuse department opened on your account and let us know your intended course of action. Sometimes that action is as simple as removing an infringing URL. Sometimes you may need to contact your client and instruct them to take the material down. Whatever the case may be, it's important to be responsive and to expressly confirm when you have complied and removed the material. Failure to acknowledge an abuse ticket can result in disconnection of service, and in the case of copyright infringement, SoftLayer has a legal obligation to remove access to the material or we face serious liability.

DMCA Counter Notifications

Most DMCA complaints are resolved without issue, but what happens if you disagree with the complaint? What if you own the material and a disgruntled former business partner is trying to get revenge? What if you wrote the content and the complaining party is copying your website? Thankfully there are penalties for filing a false DMCA complaint, but you also have recourse in the form of a counter notification. Keep in mind that while it may be tempting to plead your case to the abuse department, our role is not to play judge or jury but to allow the process to work as it was designed.

In some cases, you may be able to work out a resolution with the complaining party directly (misunderstandings happen, licenses lapse, etc.) and have them send a retraction, but most of the time your best course of action is to submit a counter notification.

Just as a takedown notice must be crafted in a specific way, counter notifications have their own set of requirements. Once you have disabled the material identified in the original complaint, we can provide your valid, properly formatted counter notification to the complaining party. Unless we receive a court order from the complaining party within the legally mandated time frame the material can be re-enabled and the case is closed for the time being.

While it might sound complicated, it's actually pretty straightforward, but we urge you to do your research and make sure you know what to do in the event a client of yours is hit with a DMCA takedown notice. Just as we are unable to make judgment calls when it comes to takedown notices or counter notifications, we are also unable to offer any legal advice for you if you need help. Hopefully this post cleared up a few questions and misconceptions about how the abuse department handles copyright complaints. In short:

Do take DMCA notifications seriously. You are at risk for service interruption and possible legal liability.
Do respond to the abuse department letting them know the material has been disabled and, if applicable, if you plan to file a counter notification.
Don't refuse to disable the material. Even if you believe the claim is false and you wish to file a counter notification, the material must be disabled within the time period allotted by the abuse department or we have to block access to it.
Don't expect the abuse department to take sides.

As with any abuse issue, communication and responsiveness is important. Disconnecting your server is a last resort, but we have ethical and legal obligations to uphold. The DMCA process certainly has its weaknesses and it leaves a bit to be desired, but at the end of the day, it's the law, and we have to operate inside of our legal obligation to it.

-Jennifer

November 2, 2012

The Trouble with Open DNS Resolvers

In the last couple of days, there's been a bit of buzz about "open DNS resolvers" and DNS amplification DDoS attacks, and SoftLayer's name has been brought up a few times. In a blog post on October 30, CloudFlare explained DNS Amplification DDoS attacks and reported the geographic and network sources of open DNS resolvers that were contributing to a 20Gbps attack on their network. SoftLayer's AS numbers (SOFTLAYER and the legacy THEPLANET-AS number) show up on the top ten "worst offenders" list, and Dan Goodin contacted us to get a comment for a follow-up piece on Ars Technica — Meet the network operators helping to fuel the spike in big DDoS attacks.

While the content of that article is less sensationalized than the title, there are still a few gaps to fill about when it comes to how SoftLayer is actually involved in the big picture (*SPOILER ALERT* We aren't "helping to fuel the spike in big DDoS attacks"). The CloudFlare blog and the Ars Technica post presuppose that the presence of open recursive DNS resolvers is a sign of negligence on the part of the network provider at best and maliciousness at worst, and that's not the case.

The majority of SoftLayer's infrastructure is made up of self-managed dedicated and cloud servers. Customers who rent those servers on a monthly basis have unrestricted access to operate their servers in any way they'd like as long as that activity meets our acceptable use policy. Some of our largest customers are hosting resellers who provide that control to their customers who can then provide that control to their own customers. And if 23 million hostnames reside on the SoftLayer network, you can bet that we've got a lot of users hosting their DNS on SoftLayer infrastructure. Unfortunately, it's easier for those customers and customers-of-customers and customers-of-customers-of-customers to use "defaults" instead of looking for, learning and implementing "best practices."

It's all too common to find those DNS resolvers open and ultimately vulnerable to DNS amplification attacks, and whenever our team is alerted to that vulnerability on our network, we make our customers aware of it. In turn, they may pass the word down the customer-of-customer chain to get to the DNS owner. It's usually not a philosophical question about whether DNS resolvers should be open for the greater good of the Internet ... It's a question of whether the DNS owner has any idea that their "configuration" is vulnerable to be abused in this way.

SoftLayer's network operations, abuse and support teams have tools that flag irregular and potentially abusive traffic coming from any server on our network, and we take immediate action when we find a problem or are alerted to one by someone who sends details to abuse@softlayer.com. The challenge we run into is that flagging obvious abusive behavior from an active DNS server is a bit of a cat-and-mouse game ... Attackers cloak their activity in normal traffic. Instead of sending a huge amount of traffic from a single domain, they send a marginal amount of traffic from a large number of machines, and the "abusive" traffic is nearly impossible for even the DNS owner to differentiate from "regular" traffic.

CloudFlare effectively became a honeypot, and they caught a distributed DNS amplification DoS attack. The results they gathered are extremely valuable to teams like mine at SoftLayer, so if they go the next step to actively contact the abuse channel for each of the network providers in their list, I hope that each of the other providers will jump on that information as I know my team will.

If you have a DNS server on the SoftLayer network, and you're not sure whether it's configured to prevent it from being used for these types of attacks, our support team is happy to help you out. For those of you interested in doing a little DNS homework to learn more, Google's Developer Network has an awesome overview of DNS security threats and mitigations which gives an overview of potential attacks and preventative measures you can take. If you're just looking for an easy way to close an open recursor, scroll to the bottom of CloudFlare's post, and follow their quick guide.

If, on the other hand, you have your own DNS server and you don't want to worry about all of this configuration or administration, SoftLayer operates private DNS resolvers that are limited to our announced IP space. Feel free to use ours instead!

-Ryan

October 25, 2012

Tips from the Abuse Department: Save Your Sinking Ship

I often find that the easiest way to present a complex process is with a relatable analogy. By replacing esoteric technical details with a less intimidating real-world illustration, smart people don't have to be technically savvy to understand what's going on. When it comes to explaining abuse-related topics, I find analogies especially helpful. One that I'm particularly keen on in explaining Abuse tickets in the context of a sinking ship.

How many times have you received an Abuse ticket and responded to the issue by suspending what appears to be the culprit account? You provide an update in the ticket, letting our team know that you've "taken care of the problem," and you consider it resolved. A few moments later, the ticket is updated on our end, and an abuse administrator is asking follow-up questions: "How did the issue occur?" "What did you do to resolve the issue?" "What steps are being taken to secure the server in order to prevent further abuse?"

Who cares how the issue happened if it's resolved now, right? Didn't I respond quickly and address the problem in the ticket? What gives? Well, dear readers, it's analogy time:

You're sailing along in a boat filled with important goods, and the craft suddenly begins to take on water. It's not readily apparent where the water is coming from, but you have a trusty bucket that you fill with the water in the boat and toss over the side. When you toss out all the water onboard, is the problem fixed? Perhaps. Perhaps not.

You don't see evidence of the problem anymore, but as you continue along your way, your vessel might start riding lower and lower in the water — jeopardizing yourself and your shipment. If you were to search for the cause of the water intake and take steps to patch it, the boat would be in a much better condition to deliver you and your cargo safely to your destination.

In the same way that a hull breach can sink a ship, so too can a security hole on your server cause problems for your (and your clients') data. In the last installment of "Tips from the Abuse Department," Andrew explained some of the extremely common (and often overlooked) ways servers are compromised and used maliciously. As he mentioned in his post, Abuse tickets are, in many cases, the first notification for many of our customers that "something's wrong."

At a crucial point like this, it's important to get the water out of the boat AND prevent the vessel from taking on any more water. You won't be sailing smoothly unless both are done as quickly as possible.

Let's look at an example of what thorough response to an Abuse ticket might look like:

A long-time client of yours hosts their small business site on one of your servers. You are notified by Abuse that malware is being distributed from a random folder on their domain. You could suspend the domain and be "done" with the issue, but that long-time client (who's not in the business of malware distribution) would suffer. You decide to dig deeper.

After temporarily suspending the account to stop any further malware distribution, you log into the server and track down the file and what permissions it has. You look through access logs and discover that the file was uploaded via FTP just yesterday from an IP in another country. With this IP information, you search your logs and find several other instances where suspicious files were uploaded around the same time, and you see that several FTP brute force attempts were made against the server.

You know what happened: Someone (or something) scanned the server and attempted to break into the domain. When the server was breached, malware was uploaded to an obscure directory on the domain where the domain owners might not notice it.

With this information in hand, you can take steps to protect your clients and the server itself. The first step might be to implement a password policy that would make guessing passwords very difficult. Next, you might add a rule within your FTP configuration to block continued access after a certain number of failed logins. Finally, you would clean the malicious content from the server, reset the compromised passwords, and unsuspend the now-clean site.

While it's quite a bit more work than simply identifying the domain and account responsible for the abuse and suspending it, the extra time you spent investigating the cause of the issue will prevent the same issue from happening after your client "fixes" the problem by deleting the files/directories. Invariably, they'd get compromised again in the same way when the domain is restored, and you'd hear from the Abuse department again.

Server security goes hand in hand with systems administration, and even though it's not a very fun part of the job, it is a 24/7 responsibility that requires diligence and vigilance. By investing time and effort into securing your servers and fixing your hull breach rather than just bailing water overboard, your customers will see less downtime, you'll be using your server resources more efficiently, and (best of all) you won't have the Abuse team hounding you about more issues!

-Garrett

P.S. I came up with a brilliant analogy about DNS and the postal service, so that might be a topic for my next post ...

October 23, 2012

Tips from the Abuse Department: Know Spam. Stop Spam.

As an abuse administrator, I'm surrounded by spam on a daily basis. When someone sends an abuse-related complaint to our abuse@softlayer.com contact address, it gets added to our ticket queue, and our Abuse SLayers take time to investigate and follow up with the customers whose servers violate our acceptable use policy. The majority of those abuse-related submissions are reporting spam coming from our network, and in my interaction with customers, I've noticed that spam (and the source of spam) is widely misunderstood.

Most spam tickets we create on customer accounts pinpoint spam sent from a compromised or exploited server. Our direct customer didn't send the phishing email, malware distribution, pharmacy advertisement or pornographic spam, but that activity came from their account. While they're accountable for the abusive behavior coming from their server, in many cases, they don't know that there's a problem until we post an abuse ticket on their account. These servers are targeted and compromised by common techniques and exploits that could have been easily avoided, but they aren't very well known outside the world of abuse.

To protect yourself from a spammer, you need to think like a spammer. You need to understand how someone might try to exploit your environment so that you can prevent them from doing so. As you're looking at ways to secure your server proactively, make sure you target these five exploits in particular:

1. User Auth Login

This is by far the most common exploit to used to send spam. This method involves a person or script using the credentials of a user to send spam through a domain's mail server. The majority of these incidences are caused by malware on a client PC that obtains the login and password for a domain user and uses that information to log on and send mail from the client PC through the server. Often, these spam messages are sent through a botnet command structure.

When an account is compromised, simply changing the password for the compromised user on the server usually won't stop the abuse. We see quite a few accounts that continue to send spam after an initial abuse ticket results in a password change. Most servers that are sending spam with this method are found to only be sending a small amount of spam at any given time to avoid detection. The low volume of spam that is being sent per server is made up for by the fact that there are thousands of servers being used for the same spamming campaigns.

In order to stop the User Auth Login exploit, a customer needs to clean all of the malicious software (malware) from their environments. To prevent future User Auth Login compromises, users should be made aware of the potential dangers of untrusted software, and if they believe their machines are infected, they need to know what to do.

2. Tell-a-friend Exploitation

The User Auth Login technique is the most common method employed by spammers, but the "tell-a-friend" script exploitation isn't far behind when it comes to volume of affected servers. This spamming method find websites that use scripts to invite users to refer friends to a page or product. Spammers will use the 'Your Message' field in one of these scripts to input their own content and links, and they'll push the actual page referral link to the bottom of the message. When these site scripts aren't secure, the spammer will use them to send hundreds or thousands of messages.

To avoid having your website fall victim to this type of spam, be very wary of any widget or script you add. If you need to add Facebook, Twitter and email "share" functionality to your site, make sure you incorporate a tell-a-friend script that does not allow for customizable messages or does not accept input of more than one email address. Also, users won't need the "cc" or "bcc" fields, so you can be sure those are axed as well. If you can't find a good "share" script that you're comfortable with from a security perspective, it might be a good idea to remove that functionality to avoid exploitation.

3. Uploaded Mailers

Spam sent via an uploaded third party mailer can sometimes prove difficult for admins to locate. An uploaded third party mailer could be capable of creating it's own outbound SMTP connection, and that would allow a program to bypass the existing MTA on the server and render any legitimate mail logs useless for investigation. Another challenge is that a php mailer can be uploaded to a location within a user's web content, and that mailer is run by the user 'nobody' (the default Apache user).

We strongly suggest configuring your server to have the mail headers show the script's user (that's not the Apache default user) and the location the script is running from on the server. Many times, these kinds of mailers are maliciously uploaded after a user's FTP password is been compromised, so be sure your FTP login information is secure.

4. Software Exploits

The "software exploits" category casts a huge shadow. Every piece of software on a server — from mail servers, content management systems and control panels to the operating system itself — can be targeted by hackers. They probe servers to find security vulnerabilities and weak coding, and when they find a vulnerability, they take control.

The hacker who found the software vulnerability might not actually take advantage of the exploit immediately. That user may sell access to other entities for their use, and that use often ends up being spam. In addition to having strong firewall rules and access restrictions, you should update and maintain the current stable versions of all software on your servers.

5. WordPress Exploits

WordPress exploits would technically fall under the "Software Exploits" category, but I'm breaking it out into its own category simply due to the volume of spam issues that are the result of exploiting this particular piece of software. The first step to protecting against spam being sent through this source is to make sure you have the latest version of WordPress installed. With that done, be sure to research the latest security plugins for that version and install any that are applicable to your environment.

These five techniques are not the only ones used by spammers to take advantage of your environment, but they are some of the most common. To protect yourself from becoming a source of spam, make your servers a more difficult target to exploit. To stop spam, you need to know spam. Now that you know spam, it's time to stop it. Ask questions, test your environment regularly and watch your logs for any unexplained usage.

-Andrew

June 25, 2012

Tips from the Abuse Department: Part 2 - Responding to Abuse Reports

If you're a SoftLayer customer, you don't want to hear from the Abuse department. We know that. The unfortunate reality when it comes to hosting a server is that compromises can happen, mistakes can be made, and even the most scrupulous reseller can fall victim to a fraudulent sign-up or sly spammer. If someone reports abusive behavior originating from one of your servers on our network, it's important to be able to communicate effectively with the Abuse department and build a healthy working relationship.

Beyond our responsibility to enforce the law and our Acceptable Use Policy, the Abuse department is designed to be a valuable asset for our customers. We'll notify you of all valid complaints (and possibly highlight security vulnerabilities in the process), we'll assist you with blacklist removal, we can serve as a liaison between you and other providers if there are any problems, and if you operate an email-heavy platform or service, we can help you understand the steps you need to take to avoid activity that may be considered abuse.

At the end of the day, if the Abuse department can maintain a good rapport with our customers, both our jobs can be easier, so I thought this installment in the "Tips from the Abuse Department" series could focus on some best practices for corresponding with Abuse from a customer perspective.

Check Your Tickets

This is the easiest, most obvious recommendation I can give. You'd be surprised at how many service interruptions could be avoided if our customers were more proactive about keeping up with their open tickets. Our portal is a vital tool for your business, so make sure you are familiar with how to access and use it.

Keep Your Contact Information Current

Our ticket system will send notifications to the email address you have on file, so making sure this information is correct and current is absolutely crucial, especially if you aren't in the habit of checking the ticket system on a regular basis. You can even set a specific address for abuse notifications to be sent to, so make use of this option. The quicker you can respond to an abuse report, the quicker the complaint can be resolved, and by getting the complaint resolved quickly, you avoid any potential service interruption.

If we are unable to reach you by ticket, we may need to call you, so keep your current phone numbers on file as well.

Provide Frequent Updates

Stay in constant communication in the midst of responding to an abuse report, and adhere to the allotted timeline in the ticket. If we don't see updates that the abusive behavior is being addressed in the grace period we are able to offer, your server is at risk of disconnection. By keeping us posted about the action you're taking and the time you need to resolve the matter, we're able to be more flexible.

If a customer on your servers created a spamming script or a phishing account, taking immediate steps to mitigate the issue by suspending that customer is another great way to respond to the process while you're performing an investigation of how that activity was started. We'll still want a detailed resolution, but if the abuse is not actively ongoing we can work with you on deadlines.

Be Concise ... But Not Too Concise

One-word responses: bad. Page long responses: also not ideal. If given the option we would opt for the latter, but your goal should be to outline the cause and resolution of any reported abusive activity as clearly and succinctly as possible in order to ease communication and expedite closing of the ticket.

Responding to a ticket with, "Fixed," is not sufficient to for the Abuse department to consider the matter resolved, but we also don't need a dump of your entire log file. Before the Abuse team can close a ticket, we have to see details of how the complaint was resolved, so if you don't provide those details in your first response, you can bet we'll keep following up with you to get them. What details do we need?

Take a Comprehensive Approach

In addition to stopping the abusive activity we want to know:

  1. How/why the issue occurred
  2. What steps are being taken to prevent further issues of that nature

We understand that dealing with abuse issues can often feel like a game of Whack-A-Mole, but if you can show that you're digging a bit deeper and taking steps to avoid recurrence, that additional work is very much appreciated. Having the Abuse department consider you a proactive, ethical and responsible customer is a worthy goal.

Be Courteous

I'm ending on a similar note to my last blog post because it's just that important! We understand getting an abuse ticket is a hassle, but please remember that we're doing our best to protect our network, the Internet community and you.

Unplugging your server is a last resort for us, and we want to make sure everyone is on the same page to prevent us from getting to that last resort. In the unfortunate event that you do experience an abuse issue, please refer back to this blog — it just might save you some headaches and perhaps some unnecessary downtime.

-Jennifer

June 18, 2012

Tips from the Abuse Department: Part 1 - Reporting Abuse

SoftLayer has a dedicated team working around the clock to address complaints of abuse on our network. We receive these complaints via feedback loops from other providers, spam blacklisting services such as Spamcop and Spamhaus, various industry contacts and mailing lists. Some of the most valuable complaints we receive are from our users, though. We appreciate people taking the time to let us know about problems on our network, and we find these complaints particularly valuable as they are non-automated and direct from the source.

It stands to reason that the more efficient people are at reporting abuse, the more efficient we can be at shutting down the activity, so I've compiled some tips and resources to make this process easier. Enjoy!

Review our Legal Page

Not only does this page contain our contact details, there's a wealth of information on our policies including what we consider abuse and how we handle reported issues. For starters, you may want to review our AUP (Acceptable Use Policy) to get a feel for our stance on abuse and how we mitigate it.

Follow Proper Guidelines

In addition to our own policies, there are legal aspects we must consider. For example, a claim of copyright infringement must be submitted in the form of a properly formatted DMCA, pursuant to the Digital Millennium Copyright Act. Our legal page contains crucial information on what is required to make a copyright claim, as well as information on how to submit a subpoena or court order. We take abuse very seriously, but we must adhere to the law as well as our privacy policy in order to protect our customers' businesses and our company from liability.

Include Evidence

Evidence can take the form of any number of things. A few common examples:

  • A copy of the alleged spam message with full headers intact.
  • A snippet from your log file showing malicious activity.
  • The full URL of a phishing page.

Without evidence that clearly ties abusive activity to a server on our network, we are unable to relay a complaint to our customer. Keep in mind that the complaint must be in a format that allows us to verify it and pass it along, which typically means an email or hard copy. While our website does have contact numbers and addresses, email is your best bet for most types of complaints.

Use Keywords

We use a mail client specifically developed for abuse desks, and it is configured with a host of rules used for filtering and prioritization. Descriptive subject lines with keywords indicating the issue type are very useful. Including the words "Spam," "Phishing" or "Copyright" in your subject line helps make sure your email is sent to the correct queue and, if applicable, receives expedited processing. Including the domain name and IP address in the body of the email is also helpful.

Follow Up

We work hard to investigate and resolve all complaints received however, due to volume, we typically do not respond to complaining parties. That said, we often rely on user complaints to determine if an issue has resumed or is ongoing so feel free to send a new complaint if activity persists.

Be Respectful

The only portion of your complaint we are likely to relay to our customer is the evidence itself along with any useful notes, which means that paragraph of profanity is read only by hardworking SoftLayer employees. We understand the frustration of being on the receiving end of spam or a DDOS, but please be professional and try to understand our position. We are on your side!

Hopefully you've found some of this information useful. When in doubt, submit your complaint to abuse@softlayer.com and we can offer further guidance. Stay tuned for Part 2, where I'll offer suggestions for SoftLayer customers about how to facilitate better communication with our Abuse department to avoid service interruption if an abuse complaint is filed against you.

-Jennifer

January 18, 2012

Keep Fighting: SOPA on the Ropes. PIPA Lurking.

The Internet is unnervingly quiet today. In response to the Stop Online Piracy Act (SOPA) in the House of Representatives and the Protect IP Act (PIPA) in the Senate, some of the most popular sites on the web have gone dark today – demonstrating the danger (and the potential unchecked power) of these two bills.

Late Friday afternoon, Judiciary Committee Chairman Lamar Smith announced that the DNS-blocking provisions would be removed from SOPA, and on Saturday, The White House responded to in opposition to the the bills as they stand today. Shortly thereafter, SOPA was "shelved."

The Internet was abuzz ... but the Champagne wasn't getting popped yet. After digging into the details, it was revealed that SOPA being "shelved" just meant that it is being temporarily put to sleep. Judiciary Committee Chairman Lamar Smith stood explained:

"To enact legislation that protects consumers, businesses and jobs from foreign thieves who steal America's intellectual property, we will continue to bring together industry representatives and Members to find ways to combat online piracy.

Due to the Republican and Democratic retreats taking place over the next two weeks, markup of the Stop Online Piracy Act is expected to resume in February."

I only mention this because it's important not to forget that SOPA isn't dead, and it's still very dangerous. If you visit sites like reddit, Wikipedia, Mozilla and Boing Boing today (January 18, 2012), you experience the potential impact of the legislation.

The Internet's outrage against SOPA has brought about real change in our nation's capital: The House is reconsidering the bill, and they'll hopefully dismiss it. With our collective momentum, we need to look at the PROTECT IP Act (PIPA, or Senate Bill 968) – a similar bill with similarly harmful implications that's been sneaking around in SOPA's shadow.

As it is defined today, PIPA has a stated goal of providing the US Government and copyright holders an additional arsenal of tools to aide in taking down 'rogue websites dedicated to infringing or counterfeit goods.' The Senate bill details that an "information location tool shall take technically feasible and reasonable measures, as expeditiously as possible, to remove or disable access to the Internet site associated with the domain name set forth in the order." In addition, it must delete all hyperlinks to the offending "Internet site."

Our opposition to PIPA is nearly identical to our opposition to SOPA. Both require a form of essentially breaking a core aspect of how the Internet functions – whether that breakage happens in DNS (as detailed in my last blog post) or in the required rearchitecture of the way any site that accepts user-generated content has to respond to PIPA-related complaints.

PIPA is scheduled for Senate vote on January 24, 2012. It is important that you voice your opinion with your government representatives and let them know about your opposition to both SOPA and PIPA. We want to help you get started down that path. Find your local representatives' contact information:

[SOPA Concerns]: Contact your congressperson in the U.S. House of Representatives
[PIPA Concerns]: Contact your Senator in the U.S. Senate

Keep spreading the word, and make sure your voice is heard.

-@toddmitchell

January 12, 2012

How the Internet Works (And How SOPA Would Break It)

Last week, I explained SoftLayer's stance against SOPA and mentioned that SOPA would essentially require service providers like SoftLayer to "break the Internet" in response to reports of "infringing sites." The technical readers in our audience probably acknowledged the point and moved on, but our non-technical readers (and some representatives in Congress) might have gotten a little confused by the references to DNS, domains and IP addresses.

Given how pervasive the Internet is in our daily lives, you shouldn't need to be "a techie" to understand the basics of what makes the Internet work ... And given the significance of the SOPA legislation, you should understand where the bill would "break" the process. Let's take a high level look at how the Internet works, and from there, we can contrast how it would work if SOPA were to pass.

The Internet: How Sites Are Delivered

  1. You access a device connected in some way to the Internet. This device can be a cell phone, a computer or even a refrigerator. You are connected to the Internet through an Internet Service Provider (ISP) which recognizes that you will be accessing various sites and services hosted remotely. Your ISP manages a network connected to the other networks around the globe ("inter" "network" ... "Internet").
  2. You enter a domain name or click a URL (for this example, we'll use http://www.softlayer.com since we're biased to that site).

Internet Basics

  1. Your ISP will see that you want to access "www.softlayer.com" and will immediately try to find someone/something that knows what "www.softlayer.com" means ... This search is known as an NS (name server) lookup. In this case, it will find that "www.softlayer.com" is associated with several name servers.

Internet Basics

  1. The first of these four name servers to respond with additional information about "softlayer.com" will be used. Domains are typically required to be associated with two or three name servers to ensure if one is unreachable, requests for that domain name can be processed by another.
  2. The name server has Domain Name System (DNS) information that maps "www.softlayer.com" to an Internet Protocol (IP) address. When a domain name is purchased and provisioned, the owner will associate that domain name with an authoritative DNS name server, and a DNS record will be created with that name server linking the domain to a specific IP address. Think of DNS as a phone book that translates a name into a phone number for you.

Internet Basics

  1. When the IP address you reach sees that you requested "www.softlayer.com," it will find the files/content associated with that request. Multiple domains can be hosted on the same IP address, just as multiple people can live at the same street address and answer the phone. Each IP address only exists in a single place at a given time. (There are some complex network tricks that can negate that statement, but in the interest of simplicity, we'll ignore them.)
  2. When the requested content is located (and generated by other servers if necessary), it is returned to your browser. Depending on what content you are accessing, the response from the server can be very simple or very complex. In some cases, the request will return a single HTML document. In other cases, the content you access may require additional information from other servers (database servers, storage servers, etc.) before the request can be completely fulfilled. In this case, we get HTML code in return.

Internet Basics

  1. Your browser takes that code and translates the formatting and content to be displayed on your screen. Often, formatting and styling of pages will be generated from a Cascading Style Sheet (CSS) referenced in the HTML code. The purpose of the style sheet is to streamline a given page's code and consolidate the formatting to be used and referenced by multiple pages of a given website.

Internet Basics

  1. The HTML code will reference sources for media that may be hosted on other servers, so the browser will perform the necessary additional requests to get all of the media the website is trying to show. In this case, the most noticeable image that will get pulled is the SoftLayer logo from this location: http://static2.softlayer.com/images/layout/logo.jpg

Internet Basics

  1. When the HTML is rendered and the media is loaded, your browser will probably note that it is "Done," and you will have successfully navigated to SoftLayer's homepage.

If SOPA were to pass, the process would look like this:

The Internet: Post-SOPA

  1. You access a device connected in some way to the Internet.
  2. You enter a domain name or click a URL (for this example, we'll use http://www.softlayer.com since we're biased to that site).

*The Change*

  1. Before your ISP runs an NS lookup, it would have to determine whether the site you're trying to access has been reported as an "infringing site." If http://www.softlayer.com was reported (either legitimately or illegitimately) as an infringing site, your ISP would not process your request, and you'd proceed to an error page. If your ISP can't find any reference to the domain an infringing site, it would start looking for the name server to deliver the IP address.
  2. SOPA would also enforce filtering from all authoritative DNS provider. If an ISP sends a request for an infringing site to the name server for that site, the provider of that name server would be forced to prevent the IP address from being returned.
  3. One additional method of screening domains would happen at the level of the operator of the domain's gTLD. gTLDs (generic top-level domains) are the ".____" at the end of the domain (.com, .net, .biz, etc.). Each gTLD is managed by a large registry organization, and a gTLD's operator would be required to prevent an infringing site's domain from functioning properly.
  4. If the gTLD registry operator, your ISP and the domain's authoritative name server provider agree that the site you're accessing has not been reported as an infringing site, the process would resume the pre-SOPA process.

*Back to the Pre-SOPA Process*

  1. The domain's name server responds.
  2. The domain's IP address is returned.
  3. The IP address is reached to get the content for http://www.softlayer.com.
  4. HTML is returned.
  5. Your browser translates the HTML into a visual format.
  6. External file references from the HTML are returned.
  7. The site is loaded.

The proponents of SOPA are basically saying, "It's difficult for us to keep up with and shut down all of the instances of counterfeiting and copyright infringement online, but it would be much easier to target the larger sites/providers 'enabling' users to access that (possible) infringement." Right now, the DMCA process requires a formal copyright complaint to be filed for every instance of infringement, and the providers who are hosting the content on their network are responsible for having that content removed. That's what our abuse team does full-time. It's a relatively complex process, but it's a process that guarantees us the ability to investigate claims for legitimacy and to hear from our customers (who hear from their customers) in response to the claims.

SOPA does not allow for due process to investigate concerns. If a site is reported to be an infringing site, service providers have to do everything in their power to prevent users from getting there.

-@toddmitchell

Subscribe to abuse