Posts Tagged 'Response'

December 15, 2011

Fighting SPAM and Abuse on a Global Network

For better or worse, one of the most engaging posts on the SoftLayer Blog is "We are a No-Spam Network," written by Jacob Linscott in June 2007. When it was posted, it celebrated a completely clear Spamhaus listing page – quite an accomplishment for a large hosting provider (for reasons I'll illustrate below). Since the post was published, it has become a hotbed of conversation about any and all abuse-related issues. Google "SoftLayer SPAM," and you'll see the post show up as the second result, so a lot of Internet passers-by will come across the post and use the comment section as a platform to share abuse-related concerns they have for us.

That engagement is a double-edge sword: It's good because we hear the concerns people have. It's bad because the post was meant to be a celebration of the continuous work that the abuse department does, and uninitiated visitors seem to consider it a unilateral claim that we've beaten spam once and for all. In the course of responding to comments on that post, I shared an analogy to convey what it's like to run abuse for a large hosting provider:

Scenario

Let's say you're the security manager for a huge mall. This mall has 100,000 stores with people walking in and out 24x7x365. In this scenario, there are "good guys" and "bad guys" who walk into and out of the mall, and every person looks exactly the same. Some of those people are store owners while others are customers of those stores. As the security manager for the mall, you want to maintain the safest, most well-maintained mall in the world, so when you find bad guys walking in and out of your mall, you do everything you can to kick them out and keep them out. Sometimes those bad guys are store owners who attract and send the wrong crowd; sometimes they are bad guy customers of a good guy store owner.

How would you manage your mall? It's not possible to differentiate whether a store owner will be a good guy or a bad guy when they're applying to lease space in your mall, so you can't "keep the bad guys out" in that regard. You can't have a security team of 100,000 people monitoring what's happening in those 100,000 stores, much less have someone individually check the millions of visitors streaming in and out of the stores. What's a security manager to do?

If you look at how Las Vegas casinos address that concern, it's clear that your best bet is to install security cameras and have a team monitoring them all the time. You might not be able to watch everything at the same time, but you can document what's happening around your mall and respond if you notice something unusual (or if someone calls in to report that they've seen bad guys coming from a store in your mall).

That's the position we're in.

SoftLayer Abuse Team

SoftLayer's network is the mall, the stores are servers, the store owners are our customers (who are often responsible for several "stores"), and the good guys and bad guys are traffic into and out of the network. We try to differentiate good guys and bad guys, but even if we know that all good guys have purple eyes and all bad guys have neon green eyes, it's still difficult to look 26,000+ store owners in the eye every day as they're walking into and out of the mall.

We staff a team of people intent on clearing the bad guys from our mall, and we know that even though good guy store owners may inadvertently host their own bad guy customers, they want to remove those customers from their store as well, so they appreciate us helping them pinpoint those customers so they can be removed.

We keep an eye on our security cameras and get our security guards to the stores where bad guys are reported as quickly as possible. If no one reports that the people coming out of store #73,403 are all bad guys, it's hard for us to know that they aren't good guys ... Which is why we encourage anyone and everyone to report abuse-related concerns to abuse@softlayer.com so we can mobilize our security force.

As Edmund Burke once said, "When bad men combine, the good must associate; else they will fall one by one, an unpitied sacrifice in a contemptible struggle." Or more colloquially, "All that is necessary for the triumph of evil is that good men do nothing."

Given that illustration, the abuse team deserves a LOT of credit for the work they do behind the scenes. They are constantly investigating reports and working with customers to get remove any and all content that violate SoftLayer's MSA, and too often, that can be a thankless job. Fighting abuse is an ongoing process, and while the nature of the beast might suggest the overall war will never be won, we're always getting faster and stronger, so the individual battles are easier and easier to win.

-@khazard

May 21, 2009

Anti-Spyware Workshop

I just got back from participating in a panel discussion at the most recent Anti-Spyware Coalition Public Workshop. The title of the panel session was “Who Owns the Problem”. You can see who all of the participants were, but it was a good session with representation from the FBI, Symantec, Paypal, the Center for Democracy and Technology, Stopbadware.org and KnujOn.

A lot of the session was focused on end user security regarding spyware, rogue anti-virus, malware and other general badware. But part of the discussion was in regards to the security efforts of the hosting industry in general and SoftLayer specifically. Some of the things we deal with in the hosting industry are second nature to those of us that have been here for a while. But when you start talking about it in front of a different crowd, you begin to appreciate the different perspectives that are out there.

For instance, one common perception (held by some, but obviously not by all) is that once we are made aware of a server that has malware on it, all we have to do is pull the plug on the server and the problem is resolved. However, sometimes the consequences of doing so are high enough to be worthy of a second look. For instance, consider the scenario where SoftLayer rents a server to a customer. That customer slices the server into virtuals using Parallel’s Virtuozzo product and rents a virtual to another customer. That customer puts Cpanel on it to sell shared hosting accounts. Now SoftLayer is 2 layers removed from the actual end user. If that end user’s website gets compromised and begins to distribute malware, how do we at SoftLayer deal with the problem. Ideally, we tell our customer and they tell their customer and they tell the end user about the problem. The end user reacts quickly and cleans up the site. That’s not anywhere close to “best case scenario”, but I would call that a reasonable real-world response.

The problem is, if any of the individuals in that chain of communication fails to react quickly, then the response time for that issue is drastically impacted and more people are potentially victimized by the malware. At what point do we pull the plug on the server? At what point do we decide that all of the other customers on the server have to suffer because of the one bad apple or because of a slow response time from one customers in the chain of communication? Websense did a study that showed in the second half of 2007, over half of all sites distributing malware were themselves compromised sites so the scenario described above is actually a very common problem. It also highlights that there is one more victim in the incident; the web site owner.

We tend to deal with each case as prudently and expeditiously as possible in every abuse report that we receive. In some cases, we pull the plug immediately. In others, we try very hard to work with the customer to resolve the issue. But in all cases, we are constantly working to act as quickly as possible on each individual case.

This is just one of the many scenarios that we have to deal with and it highlights why having a good relationship with your provider is such an important factor when choosing someone to help supply or service your IT needs.

-SamF

Subscribe to response