Tips from the Abuse Department: Know Spam. Stop Spam.
Posted by Andrew Smith - Martinez in Infrastructure, Technology, Tips and TricksAs an abuse administrator, I’m surrounded by spam on a daily basis. When someone sends an abuse-related complaint to our abuse@softlayer.com contact address, it gets added to our ticket queue, and our Abuse SLayers take time to investigate and follow up with the customers whose servers violate our acceptable use policy. The majority of those abuse-related submissions are reporting spam coming from our network, and in my interaction with customers, I’ve noticed that spam (and the source of spam) is widely misunderstood.
Most spam tickets we create on customer accounts pinpoint spam sent from a compromised or exploited server. Our direct customer didn’t send the phishing email, malware distribution, pharmacy advertisement or pornographic spam, but that activity came from their account. While they’re accountable for the abusive behavior coming from their server, in many cases, they don’t know that there’s a problem until we post an abuse ticket on their account. These servers are targeted and compromised by common techniques and exploits that could have been easily avoided, but they aren’t very well known outside the world of abuse.
To protect yourself from a spammer, you need to think like a spammer. You need to understand how someone might try to exploit your environment so that you can prevent them from doing so. As you’re looking at ways to secure your server proactively, make sure you target these five exploits in particular:
1. User Auth Login
This is by far the most common exploit to used to send spam. This method involves a person or script using the credentials of a user to send spam through a domain’s mail server. The majority of these incidences are caused by malware on a client PC that obtains the login and password for a domain user and uses that information to log on and send mail from the client PC through the server. Often, these spam messages are sent through a botnet command structure.
When an account is compromised, simply changing the password for the compromised user on the server usually won’t stop the abuse. We see quite a few accounts that continue to send spam after an initial abuse ticket results in a password change. Most servers that are sending spam with this method are found to only be sending a small amount of spam at any given time to avoid detection. The low volume of spam that is being sent per server is made up for by the fact that there are thousands of servers being used for the same spamming campaigns.
In order to stop the User Auth Login exploit, a customer needs to clean all of the malicious software (malware) from their environments. To prevent future User Auth Login compromises, users should be made aware of the potential dangers of untrusted software, and if they believe their machines are infected, they need to know what to do.
2. Tell-a-friend Exploitation
The User Auth Login technique is the most common method employed by spammers, but the “tell-a-friend” script exploitation isn’t far behind when it comes to volume of affected servers. This spamming method find websites that use scripts to invite users to refer friends to a page or product. Spammers will use the ‘Your Message’ field in one of these scripts to input their own content and links, and they’ll push the actual page referral link to the bottom of the message. When these site scripts aren’t secure, the spammer will use them to send hundreds or thousands of messages.
To avoid having your website fall victim to this type of spam, be very wary of any widget or script you add. If you need to add Facebook, Twitter and email “share” functionality to your site, make sure you incorporate a tell-a-friend script that does not allow for customizable messages or does not accept input of more than one email address. Also, users won’t need the “cc” or “bcc” fields, so you can be sure those are axed as well. If you can’t find a good “share” script that you’re comfortable with from a security perspective, it might be a good idea to remove that functionality to avoid exploitation.
3. Uploaded Mailers
Spam sent via an uploaded third party mailer can sometimes prove difficult for admins to locate. An uploaded third party mailer could be capable of creating it’s own outbound SMTP connection, and that would allow a program to bypass the existing MTA on the server and render any legitimate mail logs useless for investigation. Another challenge is that a php mailer can be uploaded to a location within a user’s web content, and that mailer is run by the user ‘nobody’ (the default Apache user).
We strongly suggest configuring your server to have the mail headers show the script’s user (that’s not the Apache default user) and the location the script is running from on the server. Many times, these kinds of mailers are maliciously uploaded after a user’s FTP password is been compromised, so be sure your FTP login information is secure.
4. Software Exploits
The “software exploits” category casts a huge shadow. Every piece of software on a server — from mail servers, content management systems and control panels to the operating system itself — can be targeted by hackers. They probe servers to find security vulnerabilities and weak coding, and when they find a vulnerability, they take control.
The hacker who found the software vulnerability might not actually take advantage of the exploit immediately. That user may sell access to other entities for their use, and that use often ends up being spam. In addition to having strong firewall rules and access restrictions, you should update and maintain the current stable versions of all software on your servers.
5. WordPress Exploits
WordPress exploits would technically fall under the “Software Exploits” category, but I’m breaking it out into its own category simply due to the volume of spam issues that are the result of exploiting this particular piece of software. The first step to protecting against spam being sent through this source is to make sure you have the latest version of WordPress installed. With that done, be sure to research the latest security plugins for that version and install any that are applicable to your environment.
These five techniques are not the only ones used by spammers to take advantage of your environment, but they are some of the most common. To protect yourself from becoming a source of spam, make your servers a more difficult target to exploit. To stop spam, you need to know spam. Now that you know spam, it’s time to stop it. Ask questions, test your environment regularly and watch your logs for any unexplained usage.
-Andrew
October 25th, 2012 at 1:10 pm
[...] the last installment of “Tips from the Abuse Department,” Andrew explained some of the extremely common (and often overlooked) ways servers are compromised and used maliciously. As he mentioned in his post, Abuse tickets are, in many cases, the first [...]
November 1st, 2012 at 2:24 am
This is useful article for stop spamming.
December 19th, 2012 at 10:35 am
So, how do you explain softlayer’s terrible reputation:
http://www.spamhaus.org/sbl/listings/softlayer.com
has you holding 13 spam/phish sites (like msnbcnews3.com) and:
http://www.siteadvisor.com/sites/softlayer.com/msgpage
has 2 pages of user comments slamming softlayer and spammers and abuse@ as consistenly ignoring abuse reports.
Just curious.
December 19th, 2012 at 10:58 am
Hi Andy, Thanks for expressing your curiosity! This blog about Fighting Spam and Abuse on a Global Network provides some context about how the abuse department works. Additionally, an abuse department manager posted a blog about reporting abuse that explains how our team handles complaints that are submitted.
It’s important to note from Jennifer’s blog that complaints are not generally “responded to,” but that shouldn’t suggest that they haven’t been addressed. If a spammer sends 10,000 spam emails to 10,000 different users from one server, the abuse team may receive thousands of messages reporting the activity. Responding to each report — even with a canned response — would be a terribly inefficient use of resources because that time is better spent following up internally with our customers (who might be following up with their customers who might be following up with their customers, etc.) to confirm that action is being taken to stop the abusive activity.
If any complaints are submitted but you don’t believe they have been resolved, send them to me (khazard@softlayer.com), and I’ll follow up with the abuse managers to see where they are in the process.
January 26th, 2013 at 4:14 am
So, how long does the process take? I ask, as I have been reporting SPAM from softlayer since 18 January, and it hasn’t stopped yet?
January 28th, 2013 at 9:31 am
The timeline can vary a bit … In some cases, we’re contacting a customer who has to contact its own customer who has to contact its own customer, and we want to express urgency while being fair about the grace period to respond before we take action. Can you forward an example of a message you’ve sent to the abuse team to our social media team (twitter@softlayer.com)? We’ll follow up directly with an abuse manager and let you know what we find out.
March 5th, 2013 at 3:53 pm
[how long can it take?]
It may take years. Considering that softlayer hosts spam factories like “websitewelcome.com” as a regular part of their business, you have to consider their position. If they stop hosting spam factories, then softlayer stops getting paid. If they stop getting paid, then their employees have to go out and find other jobs. This is a crummy time to be looking for a job.
On the other hand, if softlayer/theplanet continue to host spam factories and spamvertised web pages, they continue to get paid. If they get paid, then they can pay their employees to pretend to care about spam problems, and their employees in turn can buy groceries.
The decision is easy: continue to host spam factories and continue in business, or not. I think softlayer has shown us that their thinking on the subject is clear.
March 5th, 2013 at 6:07 pm
Thanks for the contribution, andrews. You call websitewelcome a “spam factory,” but it might be useful to investigate what that business is and why it might appear to host a significant amount of spam. A quick search for that name would reveal that it’s a private nameserver brand for a hosting reseller. A hosting reseller can order a SoftLayer server and split it up to provide hosting to customers for as low as a few dollars per month.
Unfortunately, that kind of deal is like a honeypot to spammers because they don’t have to spend a lot of money to get an account, and they can be online quickly. The hosting reseller can implement protections and safeguards in the ordering process, but there’s no way to conclusively prevent spammers from ordering those accounts. As a result, shared hosting accounts often wind up being home to spammers until we hear about them and shut them down … In the process, the hosting reseller can update their system to prevent that specific user from becoming a customer again.
When a hosting reseller has hundreds of thousands (or millions) of customers, and thousands of servers, their name might be mentioned a lot, but that’s a product of their size rather than the hosting reseller’s approval or support of spammers and abusive users.
March 6th, 2013 at 7:05 pm
Of course I call it a “spam factory”, and I think you see why. They make it easy to get online and send spam. Since softlayer is collecting revenue for this, websitewelcome is indeed a welcome spam factory.
If softlayer were not so spam-friendly, their contracts would contain actual monetary penalties for sending spam. That might shift their target market from spam factories to legitimate servers, and I do not know how well softlayer would compete there.
However, we can be sure that websitewelcome and their ilk would make one of two choices when faced with real penalties for spam: either they would move to another hosting service, or they would find a way to collect the penalties with mark-ups from their customers.
All of this is pure hypothetical. Pigs will fly, and softlayer will cease to host spammers, and taxes will decline in Volusia County, probably all at about the same time.
March 7th, 2013 at 9:49 am
Interesting thoughts, andrews. How would you suggest those penalties be structured fairly? If it’s any easier, you can detail your proposal in terms of the mall analogy we shared in Fighting SPAM and Abuse on a Global Network.
March 8th, 2013 at 9:05 am
First, I should say that the term “penalties” was a poor choice of word. My error.
Structure the charges to track the amount of spam. First spam, $1.00; every spam within a week thereafter $10.00 or $100.00 depending on costs to deal with it. The charges, which we will term “liquidated damages” rather than raw charges because they are intended to compensate for hard-to-quantify costs, then pay to deal with the spammer.
The charges should be assessed against the softlayer spam client, e.g. websitewelcome, because you have a way to decide if it is collectable. I am assuming here that you have a way to collect charges from your customers generally; if not, then you are a large charity and I find that unlikely.
Possibly you cap the liquidated damages at some number of months’ regular charges on the theory that you are going to unplug that customer and wait for the stink to fade from their IP address. You may have some provision for rebated damages where customer was hijacked or where problem is fixed promptly.
It is clear that the present system, which consists of hosting spam factories and wringing hands over how difficult it is to stop them from spamming, does little to stop them from spamming.
And this does not address hosting of spamvertised websites, which do present more interesting issues.
March 8th, 2013 at 12:48 pm
The assumption here seems to be that the customer you reference actively supports spammers. That customer offers hosting accounts that provide individuals, entrepreneurs and business owners the Internet real estate they need to create a site, build a business or share their lives with the rest of the world. As one of the largest shared hosting providers in the world, it’s important to recognize the positive role that company plays in helping get people online and helping the Internet grow (rather than defining their business based on the illegal behavior of users who do not use the business for its intended purpose). Because their shared hosting accounts are easily accessible and inexpensive, an unfortunate side effect is that abusive users will try to take advantage of that.
As you concede, the challenge when it comes to shared hosting resellers is that the legitimate users who sign up for a shared hosting account don’t have to be technically savvy to get started, so those users are often targets of the exploits Andrew writes about in this blog post … They aren’t actually sending spam or creating phishing sites, but their accounts are responsible for that activity because they didn’t secure their account appropriately. Even with rebated damages, we’re effectively encouraging our customer to refuse an account to a non-technical small business owner who wants to pay $5/mo to put “random-mom-and-pop-shop.biz” online. And if every hosting provider in the world adopted the same policy, the Internet’s growth would be immediately stifled.
Right now, the hosting industry operates in a “stop spam when we find it” fashion rather than “prevent spam” fashion because there hasn’t been a reliable (and fair) process to find and discipline abusive users without innocent users becoming collateral damage. You have some interesting ideas, and I encourage you to keep building on them with feedback from other users and hosting providers.
March 9th, 2013 at 7:09 am
Considering that your customer is websitewelcome, yeah, I’d say it’s a safe assumption that they actively support spammers, and that softlayer in assisting them probably are at least very pink-sympathetic.
I would certainly discourage mom and pop from putting up their own websitewelcome spam factories. Were there consequences to sending spam, other than continued revenue for the pink service providers, resellers would either stop or make sure that they could collect damages from their customers.
Remember that we know where to find mom and pop if they start spamming. We do not know where to find websitewelcome customers, because spammers’ information is unreliable. Websitewelcome has no motivation for it to be otherwise, because every $5 from a spammer is an extra $5 toward their softlayer bill. Thus they find a pink-friendly hosting service, collect $5 from every sleazy character that walks in, and everyone is happy except the recipients of the spam.
I would not so much use the description “stop spam when we find it” as “collect money from spam factories when we can recruit them” to describe the business model. Softlayer, websitewelcome, and the actual spammer being advertised all play their parts. Maybe the slogan should be “blame someone else when someone else finds spam and notifies us”, I doubt that softlayer is actually finding much.
March 14th, 2013 at 8:03 am
Of course I assume that the softlayer customer directly supports spam. At least in the case of the one I cited, the frequency of spam from them makes that a no-brainer.
Even if they were merely re-selling to others who sent the spam, however, the analysis is the same. Shifting the costs of dealing with spam from the ISP to the ISP’s spam-happy customer motivates them to either stop spamming or find a way to shift it to their spammy customers.
Ultimately spam thrives because spam-friendly ISPs like softlayer make the marginal cost of each spam message so near to zero as to be indistinguishable. If the spammer gets some money per spam, and pays essentially nothing per spam, then the economics work.
March 15th, 2013 at 10:56 am
The analysis is perhaps a little different in assessing damages where you have a softlayer customer or customer’s customer who is spamvertising a softlayer website (e.g. purchasorder.asia 173.193.106.10) but the spam is actually sent via someone else’s hijacked mail server. Do you assess them for spamming, or do you figure that since the spam itself is not coming from softlayer IP blocks, perhaps no one will react by blocking softlayer?
It’s a more complicated question that the websitewelcome question, where the customer is sending spam directly from softlayer addresses. I doubt that softlayer wants to stop hosting spamvertised websites, since they pay, and there really is not much clean-up to do where the spamvertising goes through someone else’s server.