I just got back from participating in a panel discussion at the most recent Anti-Spyware Coalition Public Workshop. The title of the panel session was “Who Owns the Problem”. You can see who all of the participants were, but it was a good session with representation from the FBI, Symantec, Paypal, the Center for Democracy and Technology, Stopbadware.org and KnujOn.
A lot of the session was focused on end user security regarding spyware, rogue anti-virus, malware and other general badware. But part of the discussion was in regards to the security efforts of the hosting industry in general and SoftLayer specifically. Some of the things we deal with in the hosting industry are second nature to those of us that have been here for a while. But when you start talking about it in front of a different crowd, you begin to appreciate the different perspectives that are out there.
For instance, one common perception (held by some, but obviously not by all) is that once we are made aware of a server that has malware on it, all we have to do is pull the plug on the server and the problem is resolved. However, sometimes the consequences of doing so are high enough to be worthy of a second look. For instance, consider the scenario where SoftLayer rents a server to a customer. That customer slices the server into virtuals using Parallel’s Virtuozzo product and rents a virtual to another customer. That customer puts Cpanel on it to sell shared hosting accounts. Now SoftLayer is 2 layers removed from the actual end user. If that end user’s website gets compromised and begins to distribute malware, how do we at SoftLayer deal with the problem. Ideally, we tell our customer and they tell their customer and they tell the end user about the problem. The end user reacts quickly and cleans up the site. That’s not anywhere close to “best case scenario”, but I would call that a reasonable real-world response.
The problem is, if any of the individuals in that chain of communication fails to react quickly, then the response time for that issue is drastically impacted and more people are potentially victimized by the malware. At what point do we pull the plug on the server? At what point do we decide that all of the other customers on the server have to suffer because of the one bad apple or because of a slow response time from one customers in the chain of communication? Websense did a study that showed in the second half of 2007, over half of all sites distributing malware were themselves compromised sites so the scenario described above is actually a very common problem. It also highlights that there is one more victim in the incident; the web site owner.
We tend to deal with each case as prudently and expeditiously as possible in every abuse report that we receive. In some cases, we pull the plug immediately. In others, we try very hard to work with the customer to resolve the issue. But in all cases, we are constantly working to act as quickly as possible on each individual case.
This is just one of the many scenarios that we have to deal with and it highlights why having a good relationship with your provider is such an important factor when choosing someone to help supply or service your IT needs.