Fighting SPAM and Abuse on a Global Network

December 15, 2011

For better or worse, one of the most engaging posts on the SoftLayer Blog is "We are a No-Spam Network," written by Jacob Linscott in June 2007. When it was posted, it celebrated a completely clear Spamhaus listing page – quite an accomplishment for a large hosting provider (for reasons I'll illustrate below). Since the post was published, it has become a hotbed of conversation about any and all abuse-related issues. Google "SoftLayer SPAM," and you'll see the post show up as the second result, so a lot of Internet passers-by will come across the post and use the comment section as a platform to share abuse-related concerns they have for us.

That engagement is a double-edge sword: It's good because we hear the concerns people have. It's bad because the post was meant to be a celebration of the continuous work that the abuse department does, and uninitiated visitors seem to consider it a unilateral claim that we've beaten spam once and for all. In the course of responding to comments on that post, I shared an analogy to convey what it's like to run abuse for a large hosting provider:

Scenario

Let's say you're the security manager for a huge mall. This mall has 100,000 stores with people walking in and out 24x7x365. In this scenario, there are "good guys" and "bad guys" who walk into and out of the mall, and every person looks exactly the same. Some of those people are store owners while others are customers of those stores. As the security manager for the mall, you want to maintain the safest, most well-maintained mall in the world, so when you find bad guys walking in and out of your mall, you do everything you can to kick them out and keep them out. Sometimes those bad guys are store owners who attract and send the wrong crowd; sometimes they are bad guy customers of a good guy store owner.

How would you manage your mall? It's not possible to differentiate whether a store owner will be a good guy or a bad guy when they're applying to lease space in your mall, so you can't "keep the bad guys out" in that regard. You can't have a security team of 100,000 people monitoring what's happening in those 100,000 stores, much less have someone individually check the millions of visitors streaming in and out of the stores. What's a security manager to do?

If you look at how Las Vegas casinos address that concern, it's clear that your best bet is to install security cameras and have a team monitoring them all the time. You might not be able to watch everything at the same time, but you can document what's happening around your mall and respond if you notice something unusual (or if someone calls in to report that they've seen bad guys coming from a store in your mall).

That's the position we're in.

SoftLayer Abuse Team

SoftLayer's network is the mall, the stores are servers, the store owners are our customers (who are often responsible for several "stores"), and the good guys and bad guys are traffic into and out of the network. We try to differentiate good guys and bad guys, but even if we know that all good guys have purple eyes and all bad guys have neon green eyes, it's still difficult to look 26,000+ store owners in the eye every day as they're walking into and out of the mall.

We staff a team of people intent on clearing the bad guys from our mall, and we know that even though good guy store owners may inadvertently host their own bad guy customers, they want to remove those customers from their store as well, so they appreciate us helping them pinpoint those customers so they can be removed.

We keep an eye on our security cameras and get our security guards to the stores where bad guys are reported as quickly as possible. If no one reports that the people coming out of store #73,403 are all bad guys, it's hard for us to know that they aren't good guys ... Which is why we encourage anyone and everyone to report abuse-related concerns to abuse@softlayer.com so we can mobilize our security force.

As Edmund Burke once said, "When bad men combine, the good must associate; else they will fall one by one, an unpitied sacrifice in a contemptible struggle." Or more colloquially, "All that is necessary for the triumph of evil is that good men do nothing."

Given that illustration, the abuse team deserves a LOT of credit for the work they do behind the scenes. They are constantly investigating reports and working with customers to get remove any and all content that violate SoftLayer's MSA, and too often, that can be a thankless job. Fighting abuse is an ongoing process, and while the nature of the beast might suggest the overall war will never be won, we're always getting faster and stronger, so the individual battles are easier and easier to win.

-@khazard

Comments

December 21st, 2011 at 7:56am

Well written article and analogy about the challenges of providing security and the insight required to be effective. Thank you.

 

December 23rd, 2011 at 1:18pm

Good quesiton, P.

As I explained at the beginning of the post, the reason Jacob wrote his initial celebration of having a clear SPAMHUAS report is because of the challenge of maintaining a completely clear listing, especially at our size. We have more than 100,000 servers under management with more than ten million domains hosted, so the nine listings you see there are all being addressed by the abuse team to be resolved. You might search again a week from now and see 10 listings, but those listings will likely be different from the ones you see today (because our team will have resolved the ones you see and the others will be posted).

This shouldn't suggest that we "permit" spam or abusive behavior in any way ... It's simply the product of having 26,000+ customers who themselves have hundreds of thousands or millions of their own customers, some of whom end up being "bad guys" (using the blog's analogy) despite best efforts to keep those bad guys out. SPAMHAUS, Spam Cop and notes from users sent to abuse@softlayer.com are all great ways to help us keep our network clean and clear.

June 2nd, 2013 at 8:26pm

You might search again a week from now and see 10 listings, but those listings will likely be different from the ones you see today

But then,. again, they may be the exact same ones you see today. The "websitewelcome" spammers have found a happy home at softlayer and are going nowhere. The guys advertising "[Domain Redacted]" [IP redacted] are well positioned to send their spam, though I notice they use someone else for hosting. Indeed, softlayer even provide european-based spam hosting. Is anyone holding their breath until softlayer stops sending 419 spam? I didn't think so.

Generally, I think our expectations can be expressed thus: softlayer hosts spammers while ignoring spam reports, and softlayer profits from hosting spammers, so softlayer has no intention of cutting back on their spam transmissions.

June 3rd, 2013 at 10:12am

The easiest way to test the theory that the listings you see on Spamhaus are the same listings you saw last week is to look at the dates on each of the Spamhaus listings. Right now, Spamhaus shows 7 listings on our network, and 6 of them were posted between last Friday and today (Monday). The fact that you don't see listings from a month ago or two months ago (much less six months ago or longer) clearly demonstrates that SoftLayer actively responds to reported abuse on our network.

I'm happy to reiterate the offer to personally follow up with any abuse requests that you feel have not been handled in a timely manner: khazard@softlayer.com

Leave a Reply

Filtered HTML

  • Web page addresses and e-mail addresses turn into links automatically.
  • You can enable syntax highlighting of source code with the following tags: <pre>, <blockcode>, <bash>, <c>, <cpp>, <drupal5>, <drupal6>, <java>, <javascript>, <php>, <python>, <ruby>. The supported tag styles are: <foo>, [foo].
  • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.

Comments

December 21st, 2011 at 7:56am

Well written article and analogy about the challenges of providing security and the insight required to be effective. Thank you.

 

December 23rd, 2011 at 1:18pm

Good quesiton, P.

As I explained at the beginning of the post, the reason Jacob wrote his initial celebration of having a clear SPAMHUAS report is because of the challenge of maintaining a completely clear listing, especially at our size. We have more than 100,000 servers under management with more than ten million domains hosted, so the nine listings you see there are all being addressed by the abuse team to be resolved. You might search again a week from now and see 10 listings, but those listings will likely be different from the ones you see today (because our team will have resolved the ones you see and the others will be posted).

This shouldn't suggest that we "permit" spam or abusive behavior in any way ... It's simply the product of having 26,000+ customers who themselves have hundreds of thousands or millions of their own customers, some of whom end up being "bad guys" (using the blog's analogy) despite best efforts to keep those bad guys out. SPAMHAUS, Spam Cop and notes from users sent to abuse@softlayer.com are all great ways to help us keep our network clean and clear.

June 2nd, 2013 at 8:26pm

You might search again a week from now and see 10 listings, but those listings will likely be different from the ones you see today

But then,. again, they may be the exact same ones you see today. The "websitewelcome" spammers have found a happy home at softlayer and are going nowhere. The guys advertising "[Domain Redacted]" [IP redacted] are well positioned to send their spam, though I notice they use someone else for hosting. Indeed, softlayer even provide european-based spam hosting. Is anyone holding their breath until softlayer stops sending 419 spam? I didn't think so.

Generally, I think our expectations can be expressed thus: softlayer hosts spammers while ignoring spam reports, and softlayer profits from hosting spammers, so softlayer has no intention of cutting back on their spam transmissions.

June 3rd, 2013 at 10:12am

The easiest way to test the theory that the listings you see on Spamhaus are the same listings you saw last week is to look at the dates on each of the Spamhaus listings. Right now, Spamhaus shows 7 listings on our network, and 6 of them were posted between last Friday and today (Monday). The fact that you don't see listings from a month ago or two months ago (much less six months ago or longer) clearly demonstrates that SoftLayer actively responds to reported abuse on our network.

I'm happy to reiterate the offer to personally follow up with any abuse requests that you feel have not been handled in a timely manner: khazard@softlayer.com

Leave a Reply

Filtered HTML

  • Web page addresses and e-mail addresses turn into links automatically.
  • You can enable syntax highlighting of source code with the following tags: <pre>, <blockcode>, <bash>, <c>, <cpp>, <drupal5>, <drupal6>, <java>, <javascript>, <php>, <python>, <ruby>. The supported tag styles are: <foo>, [foo].
  • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.