Customer Service Posts

October 1, 2014

Virtual Server Update

Good morning, afternoon, evening, or night, SoftLayer nation.

We want to give you an update and some more information on maintenance taking place right now with SoftLayer public and private node virtual servers.

As the world is becoming aware today, over the past week a security risk associated with Xen was identified by the Xen community and published as Xen Security Advisory 108 (XSA-108).

And as many are aware, Xen plays a role in our delivery of SoftLayer virtual servers.

Eliminating the vulnerability requires updating software on host nodes, and that requires downtime for the virtual servers running on those nodes.

Yeah, that’s not something anyone likes to hear. But customer security is of the utmost importance to us, so not doing it was not an option.

As soon as the risk was identified, our systems engineers and technology partners have been working nonstop to prepare the update.

On Sunday we notified every customer account that would be affected that we would have emergency maintenance in the middle of this week, and updated that notice each day.

And then yesterday we published that the maintenance would begin today at 3pm UTC, with a preliminary order of how the maintenance would roll out across all of our data centers.

We are updating host nodes data center by data center to complete the emergency maintenance as quickly as possible. This approach will minimize disruption for customers with failover infrastructure in multiple data centers.

The maintenance is under way and SoftLayer customers can follow it, live, on our forum at http://sftlyr.com/xs101.

-@SoftLayer

August 11, 2014

I PLEB Allegiance to My Data!

As a "techy turned marketing turned social media turned compliance turned security turned management" guy, I have had the pleasure of talking to many different customers over the years and have heard horror stories about data loss, data destruction, and data availability. I have also heard great stories about how to protect data and the differing ways to approach data protection.

On a daily basis, I deal with NIST 800-53 rev.4, PCI, HIPAA, CSA, FFIEC, and SOC controls among many others. I also deal with specific customer security worksheets that ask for information about how we (SoftLayer) protect their data in the cloud.

My first response is always, WE DON’T!

The looks I’ve seen on faces in reaction to that response over the years have been priceless. Not just from customers but from auditors’ faces as well.

  • They ask how we back up customer data. We don’t.
  • They ask how we make it redundant. We don’t.
  • They ask how we make it available 99.99 percent of the time. We don’t.

I have to explain to them that SoftLayer is simply infrastructure as a service (IaaS), and we stop there. All other data planning should be done by the customer. OK, you busted me, we do offer managed services as an additional option. We help the customer using that service to configure and protect their data.

We hear from people about Personal Health Information (PHI), credit card data, government data, banking data, insurance data, proprietary information related to code and data structure, and APIs that should be protected with their lives, etc. What is the one running theme? It’s data. And data is data folks, plain and simple!

Photographers want to protect their pictures, chefs want to protect their recipes, grandparents want to protect the pictures of their grandkids, and the Dallas Cowboys want to protect their playbook (not that it is exciting or anything). Data is data, and it should be protected.

So how do you go about doing that? That's where PLEB, the weird acronym in the title of this post, comes in!

PLEB stands for Physical, Logical, Encryption, Backups.

If you take those four topics into consideration when dealing with any type of data, you can limit the risk associated with data loss, destruction, and availability. Let’s look at the details of the four topics:

  • Physical Security—In a cloud model it is on the shoulders of the cloud service provider (CSP) to meet strict requirements of a regulated workload. Your CSP should have robust physical controls in place. They should be SOC2 audited, and you should request the SOC2 report showing little or no exceptions. Think cameras, guards, key card access, bio access, glass alarms, motion detectors, etc. Some, if not all, of these should make your list of must-haves.
  • Logical Access—This is likely a shared control family when dealing with cloud. If the CSP has a portal that can make changes to your systems and the portal has a permissions engine allowing you to add users, then that portion of logical access is a shared control. First, the CSP should protect its portal permission system, while the customer should protect admin access to the portal by creating new privileged users who can make changes to systems. Second, and just as important, when provisioning you must remove the initial credentials setup and add new, private credentials and restrict access accordingly. Note, that it’s strictly a customer control.
  • Encryption—There are many ways to achieve encryption, both at rest and in transit. For data at rest you can use full disk encryption, virtual disk encryption, file or folder encryption, and/or volume encryption. This is required for many regulated workloads and is a great idea for any type of data with personal value. For public data in transit, you should consider SSL or TLS, depending on your needs. For backend connectivity from your place of business, office, or home into your cloud infrastructure, you should consider a secure VPN tunnel for encryption.
  • Backups—I can’t stress enough that backups are not just the right thing to do, they are essential, especially when using IaaS. You want a copy at the CSP you can use if you need to restore quickly. But, you want another copy in a different location upon the chance of a disaster that WILL be out of your control.

So take the PLEB and mitigate risk related to data loss, data destruction, and data availability. Trust me—you will be glad you did.

-@skinman454

June 9, 2014

Visualizing a SoftLayer Billing Order

In my time spent as a data and object modeler, I’ve dealt with both good and bad examples of model visualization. As an IBMer through the Rational acquisition, I have been using modeling tools for a long time. I can appreciate a nice diagram shining a ray of light on an object structure, and abhor a behemoth spaghetti diagram.

When I started studying SoftLayer’s API documentation, I saw both the relational and hierarchical nature of SoftLayer’s concept model. The naming convention of API services and data types embodies their hierarchical structure. While reading about “relational properties” in data types, I thought it would be helpful to see diagrams showing relationships between services and data types versus clicking through reference pages. After all, diagramming data models is a valuable complement to verbal descriptions.

One way people can deal with complex data models is to digest them a little at a time. I can’t imagine a complete data model diagram of SoftLayer’s cloud offering, but I can try to visualize small portions of it. In this spirit, after reviewing article and blog entries on creating product orders using SoftLayer’s API, I drew an E-R diagram, using IBM Rational Software Architect, of basic order elements.

The diagram, Figure 1, should help people understand data entities involved in creating SoftLayer product orders and the relationships among the entities. In particular, IBM Business Partners implementing custom re-branded portals to support the ordering of SoftLayer resources will benefit from visualization of the data model. Picture this!

Figure 1. Diagram of the SoftLayer Billing Order

A user account can have many associated billing orders, which are composed of billing order items. Billing order items can contain multiple order containers that hold a product package. Each package can have several configurations including product item categories. They can be composed of product items with each item having several possible prices.

-Andrew

Andrew Hoppe, Ph.D., is a Worldwide Channel Solutions Architect for SoftLayer, an IBM Company.

May 8, 2014

SoftLayer Security: Questions and Answers

When I talk to IBM Business Partners about SoftLayer, one of the most important topics of discussion is security. We ask businesses to trust SoftLayer with their business-critical data, so it’s important that SoftLayer’s physical and network security is as transparent and understandable as possible.

After going through the notes I’ve taken in many of these client meetings, I pulled out the ten most frequently asked questions about security, and I’ve compiled answers.

Q1: How is SoftLayer secured? What security measures does SoftLayer have in place to ensure my workloads are safe?

A: This “big picture” question is the most common security-related question I’ve heard. SoftLayer’s approach to security involves several distinct layers, so it’s tough to generalize every aspect in a single response. Here are some of the highlights:

  • SoftLayer’s security management is aligned with U.S. government standards based on NIST 800-53 framework, a catalog of security and privacy controls defined for U.S. federal government information systems. SoftLayer maintains SOC 2 Type II reporting compliance for every data center. SOC 2 reports are audits against controls covering security, availability, and process integrity. SoftLayer’s data centers are also monitored 24x7 for both network and on-site security.
  • Security is maintained through automation (less likely for human error) and audit controls. Server room access is limited to authorized employees only, and every location is protected against physical intrusion.
  • Customers can create a multi-layer security architecture to suit their needs. SoftLayer offers several on-demand server and network security devices, such as firewalls and gateway appliances.
  • SoftLayer integrates three distinct network topologies for each physical or virtual server and offers security solutions for systems, applications, and data as well. Each customer has one or many VLANs in each data center facility, and only users and servers the customer authorizes can access servers in those VLANs.
  • SoftLayer offers single-tenant resources, so customers have complete control and transparency into their servers.

Q2: Does SoftLayer destroy my data when I’ve de-provisioned a compute resource?

A: Yes. When a customer cancels any physical or virtual server, all data is erased using Department of Defense (DoD) 5220.22-m standards.

Q3: How does SoftLayer protect my servers against distributed denial of service (DDoS) attacks?

A: A SoftLayer Network Operations Center (NOC) team monitors network performance and security 24x7. Automated DDoS mitigation controls are in place should a DDoS attack occur.

It’s important to clarify here that the primary objective of this DDoS mitigation is to maintain performance integrity of the overall cloud infrastructure. With that in mind, SoftLayer can’t stop a customer from being attacked, but it can shield the customer (and any other customers in the same network) from the effects of the attack. If necessary, SoftLayer will remove the target from the public network for periods of time and null-routes incoming connections. Because of SoftLayer’s three-tiered network architecture, a customer would still have access to the targeted system via the private network.

Q4: How is communication segmented from other tenants using SoftLayer?

A: SoftLayer utilizes industry standard VLANs and switch access control lists (ACLs) to segment customer environments. Customers have the ability to add and manage their own VLANs, providing additional security even inside their own accounts. ACLs are configured to permit or deny any specified network packet (data) to be directed along a switch.

Q5: How is my data kept private? How can I confirm that SoftLayer can’t read my confidential data?

A: This question is common customers who deal with sensitive workloads such as HIPAA-protected documentation, employee records, case files, and so on.

SoftLayer customers are encouraged to deploy a gateway device (e.g. Vyatta appliance) on which they can configure encryption protocols. Because the gateway device is the first hop into SoftLayer’s network, it provides an encrypted tunnel to traverse the VLANs that reside on SoftLayer. When securing compute and storage resources, customers can deploy single tenant dedicated storage devices to establish isolated workloads, and they can even encrypt their hard drives from the OS level to protect data at rest. Encrypting the hard drive helps safeguard data even if SoftLayer were to replace a drive or something similar.

Q6: Does SoftLayer track and log customer environments?

A: Yes. SoftLayer audits and tracks all user activity in our customer portal. Some examples of what is tracked include:

  • User access, both failed and authenticated attempts (destination IP is shown on a report)
  • Compute resources users deploy or cancel
  • APIs for each call (who called the API, the API call and function, etc.)
  • Intrusion Protection and Detection services that observe traffic to customer hosts
  • Additionally, customers have root access to operating systems on their servers, so they can implement additional logging of their own.

Q7: Can I disable access to some of my users through the customer portal?

A: Yes. SoftLayer has very granular ACLs. User entitlements are segmented into different categories, including Support, Security, and Hardware. SoftLayer also gives customers the ability to limit access to public and private networks. Customers can even limit user access to specific bare metal or virtual server.

Q8: Does SoftLayer patch my operating system?

A: For unmanaged cloud servers, no. Once the updated operating system is deployed on a customer’s server, SoftLayer doesn’t touch it.

If you want help with that hands-on server administration, SoftLayer offers managed hosting. In a managed hosting environment, Technical Account Managers (TAMs) are assigned as focal points for customer requests and issues. TAMs help with reports and trending data that provide recommendations to mitigate potential issues (including OS patching).

Q9: Is SoftLayer suited to run HIPAA workloads?

A: Yes. SoftLayer has a number of customers running HIPAA workloads on both bare metal and single-tenant virtual servers. A Business Associate Agreement (BAA), signed by SoftLayer and the customers, clearly define the shared responsibilities for data security: SoftLayer is solely responsible for the security of the physical data center, along with the SoftLayer-provided infrastructure.

Q10: Can SoftLayer run government workloads? Does SoftLayer use the FISMA standards?

A: The Federal Information Security Management Act (FISMA) defines a framework for managing information security that must be followed for all federal information systems. Some state institutions don’t require FISMA, but look to cloud hosting companies to be aligned to the FIMSA guidelines.

Today, two SoftLayer data centers are audited to the FISMA standards – Dallas (DAL05) and Washington, D.C. (WDC01). Customers looking for the FISMA standard can deploy their workloads in those data centers. Future plans include having data centers that comply with more stringent FedRAMP requests.

For additional information, I highly recommend the on-demand SoftLayer Fundamentals session, “Keep safe – securing your SoftLayer virtual instance.” Also, check out Allan Tate’s Thoughts on Cloud blog, “HIPAA and cloud computing: What you need to know” for more on how SoftLayer handles HIPPA-related workloads.

-Darrel Haswell

Darrel Haswell is a Worldwide Channel Solutions Architect for SoftLayer, an IBM Company.

May 2, 2014

Keyboard Shortcuts in the SoftLayer Customer Portal

I’m excited to introduce a new feature in the SoftLayer customer portal: Keyboard shortcuts!

Keyboard shortcuts give you quick access to the most commonly used features by simply typing a few characters. For those who prefer never having to reach for the mouse to navigate an application, you should find these handy additions quite helpful.

After you log into the Customer Portal, type “?” (shift + forward slash) on any page, and you'll see a full list of available keyboard shortcuts:

Keyboard Shortcuts

On the Keyboard Shortcuts help page, you have the option to enable or disable the functionality based on your preference. Keyboard shortcuts are enabled by default. Disabling this feature will turn off all keyboard shortcuts except the “?” shortcut so that you can access the enable/disable feature preference in the future if you change your mind. This preference is stored in a cookie in your browser, so changing computers or deleting your cookie will re-enable the feature.

The shortcuts are grouped into three sets: Global, Tabs, and Grids.

Keyboard Shortcuts

Global Navigation

You have the ability to navigate to any page in our application by typing in the respective position number in the menu combined with dashes (-). For example, typing 1-5-2 will open Support (1) > Help (5) > Portal Tour (2).

Use the “go to” key combinations to jump to a new location from anywhere in the portal. For example, type (g) and (d) to visit the Device List. Typing (g) and (u) allows you to access the list of portal users, and (g) and (t) takes you to view tickets. If you want to add a new ticket from anywhere in the portal, type (+) and (t). It’s that simple.

Keyboard Shortcuts

Tabs

Many of the pages within the portal have tabs that appear just above the main content of the page. These tabs often allow content to be filtered, or provide access to additional features related to the page topic. Each tab can be accessed by using a simple two-keystroke combination, such as (t) then (f) to reveal the Filter tab on the page.

Keyboard Shortcuts

Grids

Whenever a page contains a grid — a tabular listing — you can now perform common operations from the keyboard. Jump quickly from page to page (first/last or next/previous) or refresh the grid contents with a single keystroke.

Keyboard Shortcuts

Please give this new feature a try for yourself! We welcome your feedback. Please let us know if you would like to have us implement any other keyboard shortcuts in the future.

-Daniel

May 1, 2014

New App Release: SoftLayer Mobile for Windows 8.1

Today, the SoftLayer development team is launching a new platform accessibility tool for SoftLayer customers who want to easily manage their infrastructure from Windows. We've gotten a great response from the users of SoftLayer Mobile app for Windows Phone, so we turned our attention to creating an app for customers on Windows 8.1: SoftLayer Mobile for Windows 8.1.

With a growing number of users adopting and embracing Windows 8.1 on their PCs, and the Windows Store is becoming a vibrant community of useful apps for those customers. There are more than 145,000 apps on the Windows Store, and that number is expected to increase exponentially following Microsoft’s recent introduction of "Universal Apps for Windows Phone 8.1 and Windows 8.1.” With all that goodness and an expanding market, it was imperative for our mobile development team to build an app for customers using Windows 8.1 as their default OS or carrying Windows RT tablets.

Why Windows 8.1?

Our team wants to provide simple, efficient ways for customers to connect to SoftLayer infrastructure and perform any necessary management tasks while on-the-go. Our team is inspired by the power of connected devices in Windows ecosystem. By developing an app for Windows 8.1, we will slowly bring the phone, tablet and PC onto one streamlined platform — a concept many smart devices are adopting quickly.

What’s Fresh?

New Dashboard

The SoftLayer Mobile app for Windows 8.1 is a fresh new approach to its Windows Phone sibling. The app provides a dashboard view after authentication that provides a snapshot of some of the most commonly used information and controls in the portal.

Currently, the dashboard supports four different panels: Tickets, devices, accounting and bandwidth. All display an overview of relevant information for you and your environment. The dashboard also allows you to quickly add a ticket or make a one-time payment on your account.

SoftLayer Mobile for Windows 8.1

In-line Ticket Updates

In the new tickets module, you can update tickets without ever leaving the page. This functionality is similar to what you see on many social websites, and it's integrated to be seamless.

SoftLayer Mobile for Windows 8.1

Search Everywhere!

One of the coolest additions to the new app is the introduction of search functionality in each module. Now, you can search a ticket, a device, or an invoice by just typing into the search box! The search capability lets you spend less time scrolling and more time working.

SoftLayer Mobile for Windows 8.1

Bandwidth Display

Smart phones have apps that measure and report how much data you are using, and your infrastructure should be similarly transparent Bandwidth usage is an important aspect of server management, so we built the bandwidth module to show your infrastructure's public and private traffic for current and previous billing cycles. This view also helps you see when a server is about to reach its limits so that you can plan accordingly.

SoftLayer Mobile for Windows 8.1

The module provides two ways to look at the data:

  • In a tabular form by clicking the “Show/Hide Traffic Details” button.
  • In a graphic representation by clicking the “View Graph” button.

SoftLayer Mobile for Windows 8.1

Same Functionality. Better Experience.

Sometimes change is not always needed for a nicely crafted feature. The new app keeps the same feature richness of the Windows Phone app and arranges it in a user-friendly way. For example, in the devices module, you can navigate to between different tabs to get the information you need, from password lists and attached tickets to a specific device or monitoring alarms.

The “Remote Control” section on the module allows you to perform actions such as rebooting, power cycles, restarts and pinging servers. In addition, you can view hardware and software installed on the device along with the hardware and network components attached. In the current phone version, you can only see the root password for the device, but in the Windows 8.1 app, you see all passwords for the server.

SoftLayer Mobile for Windows 8.1

What's Next?

During the development of this app, the team's goal was to test to adopt a framework that would be ideal for scaling. More and more developers are adopting a Model-View-Model (MVVM) approach to mobile and web app development, so our goal was to use that approach for this project. The significant challenge we faced when adopting this approach was finding a well-supported framework that met our application's needs. We weren't able to find suitable frameworks that committed regular updates in SDKs or in APIs, so we ended up using the same MVVM principles without any underlying framework. In the end, the project allowed us to create our own framework for future projects!

There are many exciting features that are lined up for the Windows 8.1 app. Download it now: SoftLayer Mobile for Windows 8.1

After you try it out, please submit your feedback ... We want to keep improving the app by providing the features and functionality that matter most to you.

-Imran

February 3, 2014

Risk Management: 5 Tips for Managing Risk in the Cloud

Security breaches have made front-page news in recent months. With stories about Target, Neiman Marcus, Yahoo! and GoDaddy in the headlines recently, the importance of good information security practices is becoming harder and harder to ignore — even for smaller businesses. Moving your business into the cloud offers a plethora of benefits; however, those benefits do not come without their challenges. Moving your business into the cloud involves risks such as multi-tenancy, so it's important to be able to properly manage and identify these risks.

1. Know the Security Your Provider Offers
While some SaaS providers may have security baked-in, most IaaS providers (including SoftLayer) leave much of the logical security responsibility of a customer's systems to the customer. For the security measures that an infrastructure provider handles, the provider should be able to deliver documentation attesting these controls. We perform an annual SOC2 audit, so we can attest to the status of our security and availability controls as a service organization. With this information, our customers use controls from our report as part of their own compliance requirements. Knowing a provider's security controls (and seeing proof of that security) allows business owners and Chief Information Security Officers (CISO) to have peace-of-mind that they can properly plan their control activities to better prevent or respond to a breach.

2. Use the Cloud to Distribute and Replicate Your Presence
The incredible scalability and geographical distribution of operating in the cloud can yield some surprising payoff. Experts in the security industry are leveraging the cloud to reduce their patch cycles to days, not weeks or months. Most cloud providers have multiple sites so that you can spread your presence nationally, or even globally. With this kind of infrastructure footprint, businesses can replicate failover systems and accommodate regional demand across multiple facilities with the minimal incremental investment (and with nearly identical security controls).

3. Go Back to the Basics
Configuration management. Asset management. Separation of duties. Strong passwords. Many organizations get so distracted by the big picture of their security measures that they fail to manage these basic rights. Take advantage of any of your provider's tools to assist in the ‘mundane’ tasks that are vitally important to your business's overall security posture. For example, you can use image templates or post-provisioning scripts to deploy a standard baseline configuration to your systems, then track them down to the specific server room. You’ll know what hardware is in your server at all times, and if you're using SoftLayer, you can even drill down to the serial numbers of your hard drives.

4. Have Sound Incident Response Plans
The industry is becoming increasingly cognizant of the fact that it’s not a matter of if, but when a security threat will present itself. Even with exceedingly high levels of baked-in security, most of the recent breaches resulted from a compromised employee. Be prepared to respond to security incidents with confidence. While you may be physically distanced from your systems, you should be able to meet defined Recovery Time Objectives (RTOs) for your services.

5. Maintain Constant Contact with Your Cloud Provider
Things happen. No amount of planning can completely halt every incident, whether it be a natural disaster or a determined attacker. Know that your hosting provider has your back when things take an unexpected turn.

With proper planning and good practice, the cloud isn't as risky and frightening as most think. If you're interested in learning a little more about the best practices around security in the cloud, check out the Cloud Security Alliance (CSA). The CSA provides a wealth of knowledge to assist business owners and security professionals alike. Build on the strengths, compensate for the weaknesses, and you and your CISO will be able to sleep at night (and maybe even sneak in a beer after work).

-Matt

December 5, 2013

How to Report Abuse to SoftLayer

When you find hosted content that doesn't meet our acceptable use policy or another kind of inappropriate Internet activity originating from a SoftLayer service, your natural reaction might be to assume, "SoftLayer must know about it, and the fact that it's going on suggests that they're allowing that behavior." I know this because every now and then, I come across a "@SoftLayer is phishing my email. #spamming #fail" Tweet or a "How about u stop hacking my computer???" Facebook post. It's easy to see where these users are coming from, so my goal for this post is to provide the background you need to understand how behavior we don't condone — what we consider "abuse" of our services — might occur on our platform and what we do when we learn about it.

The most common types of abuse reported from the SoftLayer network are spam, copyright/trademark infringement, phishing and abusive traffic (DDoS attacks). All four are handled by the same abuse team, but they're all handled a bit differently, so it's important to break them down to understand the most efficient way to report them to our team. When you're on the receiving end of abuse, all you want is to make it stop. In the hurry to report the abusive behavior, it's easy to leave out some of the key information we need to address your concern, so let's take a look at each type of abuse and the best ways to report it to the SoftLayer team:

If You Get Spam

Spam is the most common type of abuse that gets reported to SoftLayer. Spam email is unsolicited, indiscriminate bulk messaging that is sent to you without your explicit consent. If you open your email client right now, your junk mail folder probably has a few examples of spam ... Someone is trying to sell you discount drugs or arrange a multi-million dollar inheritance transfer. In many ways, it's great that email is so easy to use and pervasive to our daily lives, but that ease of use also makes it an easy medium for spammers to abuse. Whether the spammer is a direct SoftLayer customer or a customer of one of our customers or somewhere further down the line of customers of customers, spam messages sent from a SoftLayer server will point back to us, and our abuse team is the group that will help stop it.

When you receive spam sent through SoftLayer, you should forward it directly to our abuse team (abuse@softlayer.com). Our team needs a full copy of the email with its headers intact. If you're not sure what that means, check out these instructions on how to retrieve your email headers. The email headers help tell the story about where exactly the messages are coming from and which customer we need to contact to stop the abuse.

If You See Phishing

Phishing abuse might be encountered via spam or you might encounter it on a website. Phishing is best described as someone masquerading as someone else to get your sensitive information, and it's one of the most serious issues our abuse team faces. Every second that a phishing/scam site is online, another user might be fooled into giving up his or her credit card or login information, and we don't want that to happen. Often, the fact that a site is not legitimate is clear relatively quickly, but as defenses against phishing have gotten better, so have the phishing sites. Take a minute to go through this phishing IQ test to get an idea of how difficult phishing can be to trace.

When it comes to reporting phishing, you should send the site's URL to the abuse team (also using abuse@softlayer.com). If you came across the phishing site via a spam email, be sure to include the email headers with your message. To help us filter the phishing complaint, please make sure to include the word "phishing" in your email's subject line. Our team will immediately investigate and follow up with the infringing customer internally.

If You Find Copyright or Trademark Infringement

If infringement of your copyright or trademark is happening on our platform, we want to know about it so we can have it taken down immediately. Copyright complaints and trademark complaints are handled slightly differently, so let's look at each type to better understand how they work.

Complaints of copyright infringement are processed by our abuse team based on the strict DMCA complaint laws. When I say "strict" in that sentence, I'm not saying it lightly ... Because DMCA complaints are legal issues, every requirement in the DMCA must be met in order for our team to act on the complaint. That might seem arbitrary, but we're not given much leeway when it comes to the DMCA process, and we have to be sticklers.

On our DMCA legal page, we outline the process of reporting a DMCA complaint of copyright infringement (primarily citing the statute 17 U.S.C. Section 512(c)(3)). If you don't completely understand what needs to be included in the claim, we recommend that you seek independent legal advice. It sounds harsh, but failure to submit copyright infringement notification as described above will result in no legal notice or action on behalf of SoftLayer. When you've made sure all required evidence has been included in your DMCA complaint, make sure "copyright" or "DMCA" are included in your subject line and submit the complaint to copyright@softlayer.com.

Trademark complaints do not have the same requirements as copyright complaints, but the more information you can provide in your complaint, the easier it will be for our customer to locate and remove the offending material. If you encounter unauthorized use of your registered trademark on our network, please email copyright@softlayer.com with details — the exact location of the infringing content, your trademark registration information, etc. — along with an explanation that this trademark usage is unauthorized and should be removed. In your email, please add the word "trademark" to the subject line to help us filter and prioritize your complaint.

If You See Abusive Traffic

Spam, phishing and copyright infringement are relatively straightforward when it comes to finding and reporting abuse, but sometimes the abuse isn't as visible and tangible (though the effect usually is). If a SoftLayer server is sending abusive traffic to your site, we want to know about it as quickly as possible. Whether that behavior is part of a Denial of Service (DoS) attack or is just scanning ports to possibly attack later, it's important that you give us details so we can prevent any further activity.

To report this type of abuse, send a snippet from your log file including at least 10 lines of logs that show attempts to break into or overload your server. Here's a quick reference to where you can find the relevant logs to send:

  • Email Spam - Send Mail Logs:
    • /var/log/maillog
    • /usr/local/psa/var/log/maillog
  • Brute Force Attacks - Send SSH Logs:
    • /var/log/messages
    • /var/log/secure

Like spam and phishing reports, abusive traffic complaints should be sent to abuse@softlayer.com with a quick explanation of what is happening and any other details you can provide. When you submit a complaint about abusive traffic, make sure your message's subject line reflects the type of issue ("DDoS attack," "brute force attempts," etc.) so our team can investigate your report even quicker.

As I mentioned at the start of this post, these are just four types of abusive behavior that our abuse department addresses on a daily basis. Our Acceptable Use Policy (AUP) outlines what can and cannot be hosted using SoftLayer services, and the process of reporting other types of abuse is generally the same as what you see in the four examples I mentioned above ... Send a clear, concise report to abuse@softlayer.com with key words about the type of violation in the message's subject line. When our team is able to look into your complaint and find the evidence they need to take action, they do so quickly.

I can't wrap up this blog of tips without mentioning the "Tips from the Abuse Department" blog Jennifer Groves wrote about reporting abuse ... It touches on some of the same ideas as this post, and it also provides a little more perspective from behind the lines of the abuse department. As the social media gal, I don't handle abuse on a day-to-day basis, but I do help people dealing with abuse issues, and I know a simple guide like this will be of value.

If an abuse-related issue persists and you don't feel like anything has been fixed, double-check that you've included all the necessary information and evidence in your correspondence to the abuse team. In most cases, you will not receive a response from the abuse team, but that doesn't mean they aren't taking action. The abuse@ and copyright@ email aliases function as notification systems for our abuse teams, and they correspond with the infringing customers internally when a complaint is submitted. Given the fact that hundreds of users may report the same abusive behavior at the same time, responding directly to each message would slow down the process of actually resolving the issue (which is the priority).

If everything was included in your initial correspondence with the abuse team but you still don't notice a change in the abusive behavior, you can always follow up with our social media team at twitter@softlayer.com, and we'll do everything we can to help.

-Rachel

May 7, 2013

Tips from the Abuse Department: DMCA Takedown Notices

If you are in the web hosting business or you provide users with access to store content on your servers, chances are that you're familiar with the Digital Millennium Copyright Act (DMCA). If you aren't familiar with it, you certainly should be. All it takes is one client plagiarizing an article or using a filesharing program unscrupulously, and you could find yourself the recipient of a scary DMCA notice from a copyright holder. We've talked before about how to file a DMCA complaint with SoftLayer, but we haven't talked in detail about SoftLayer's role in processing DMCA complaints or what you should do if you find yourself on the receiving end of a copyright infringement notification.

The most important thing to understand when it comes to the way the abuse team handles DMCA complaints is that our procedures aren't just SoftLayer policy — they are the law. Our role in processing copyright complaints is essentially that of a middleman. In order to protect our Safe Harbor status under the Online Copyright Infringement Liability Limitation Act (OCILLA), we must enforce any complaint that meets the legal requirements of a takedown notice. That DMCA complaint must contain specific elements and be properly formatted in order to be considered valid.

Responding to a DMCA Complaint

When we receive a complaint that meets the legal requirements of a DMCA takedown notice, we must relay the complaint to our direct customer and enforce a deadline for removal of the violating material. We are obligated to remove access to infringing content when we are notified about it, and we aren't able to make a determination about the validity of a claim beyond confirming that all DMCA requirements are met.

The law states that SoftLayer must act expeditiously, so if you receive notification of a DMCA complaint, it's important that you acknowledge the ticket that the abuse department opened on your account and let us know your intended course of action. Sometimes that action is as simple as removing an infringing URL. Sometimes you may need to contact your client and instruct them to take the material down. Whatever the case may be, it's important to be responsive and to expressly confirm when you have complied and removed the material. Failure to acknowledge an abuse ticket can result in disconnection of service, and in the case of copyright infringement, SoftLayer has a legal obligation to remove access to the material or we face serious liability.

DMCA Counter Notifications

Most DMCA complaints are resolved without issue, but what happens if you disagree with the complaint? What if you own the material and a disgruntled former business partner is trying to get revenge? What if you wrote the content and the complaining party is copying your website? Thankfully there are penalties for filing a false DMCA complaint, but you also have recourse in the form of a counter notification. Keep in mind that while it may be tempting to plead your case to the abuse department, our role is not to play judge or jury but to allow the process to work as it was designed.

In some cases, you may be able to work out a resolution with the complaining party directly (misunderstandings happen, licenses lapse, etc.) and have them send a retraction, but most of the time your best course of action is to submit a counter notification.

Just as a takedown notice must be crafted in a specific way, counter notifications have their own set of requirements. Once you have disabled the material identified in the original complaint, we can provide your valid, properly formatted counter notification to the complaining party. Unless we receive a court order from the complaining party within the legally mandated time frame the material can be re-enabled and the case is closed for the time being.

While it might sound complicated, it's actually pretty straightforward, but we urge you to do your research and make sure you know what to do in the event a client of yours is hit with a DMCA takedown notice. Just as we are unable to make judgment calls when it comes to takedown notices or counter notifications, we are also unable to offer any legal advice for you if you need help. Hopefully this post cleared up a few questions and misconceptions about how the abuse department handles copyright complaints. In short:

Do take DMCA notifications seriously. You are at risk for service interruption and possible legal liability.
Do respond to the abuse department letting them know the material has been disabled and, if applicable, if you plan to file a counter notification.
Don't refuse to disable the material. Even if you believe the claim is false and you wish to file a counter notification, the material must be disabled within the time period allotted by the abuse department or we have to block access to it.
Don't expect the abuse department to take sides.

As with any abuse issue, communication and responsiveness is important. Disconnecting your server is a last resort, but we have ethical and legal obligations to uphold. The DMCA process certainly has its weaknesses and it leaves a bit to be desired, but at the end of the day, it's the law, and we have to operate inside of our legal obligation to it.

-Jennifer

October 25, 2012

Tips from the Abuse Department: Save Your Sinking Ship

I often find that the easiest way to present a complex process is with a relatable analogy. By replacing esoteric technical details with a less intimidating real-world illustration, smart people don't have to be technically savvy to understand what's going on. When it comes to explaining abuse-related topics, I find analogies especially helpful. One that I'm particularly keen on in explaining Abuse tickets in the context of a sinking ship.

How many times have you received an Abuse ticket and responded to the issue by suspending what appears to be the culprit account? You provide an update in the ticket, letting our team know that you've "taken care of the problem," and you consider it resolved. A few moments later, the ticket is updated on our end, and an abuse administrator is asking follow-up questions: "How did the issue occur?" "What did you do to resolve the issue?" "What steps are being taken to secure the server in order to prevent further abuse?"

Who cares how the issue happened if it's resolved now, right? Didn't I respond quickly and address the problem in the ticket? What gives? Well, dear readers, it's analogy time:

You're sailing along in a boat filled with important goods, and the craft suddenly begins to take on water. It's not readily apparent where the water is coming from, but you have a trusty bucket that you fill with the water in the boat and toss over the side. When you toss out all the water onboard, is the problem fixed? Perhaps. Perhaps not.

You don't see evidence of the problem anymore, but as you continue along your way, your vessel might start riding lower and lower in the water — jeopardizing yourself and your shipment. If you were to search for the cause of the water intake and take steps to patch it, the boat would be in a much better condition to deliver you and your cargo safely to your destination.

In the same way that a hull breach can sink a ship, so too can a security hole on your server cause problems for your (and your clients') data. In the last installment of "Tips from the Abuse Department," Andrew explained some of the extremely common (and often overlooked) ways servers are compromised and used maliciously. As he mentioned in his post, Abuse tickets are, in many cases, the first notification for many of our customers that "something's wrong."

At a crucial point like this, it's important to get the water out of the boat AND prevent the vessel from taking on any more water. You won't be sailing smoothly unless both are done as quickly as possible.

Let's look at an example of what thorough response to an Abuse ticket might look like:

A long-time client of yours hosts their small business site on one of your servers. You are notified by Abuse that malware is being distributed from a random folder on their domain. You could suspend the domain and be "done" with the issue, but that long-time client (who's not in the business of malware distribution) would suffer. You decide to dig deeper.

After temporarily suspending the account to stop any further malware distribution, you log into the server and track down the file and what permissions it has. You look through access logs and discover that the file was uploaded via FTP just yesterday from an IP in another country. With this IP information, you search your logs and find several other instances where suspicious files were uploaded around the same time, and you see that several FTP brute force attempts were made against the server.

You know what happened: Someone (or something) scanned the server and attempted to break into the domain. When the server was breached, malware was uploaded to an obscure directory on the domain where the domain owners might not notice it.

With this information in hand, you can take steps to protect your clients and the server itself. The first step might be to implement a password policy that would make guessing passwords very difficult. Next, you might add a rule within your FTP configuration to block continued access after a certain number of failed logins. Finally, you would clean the malicious content from the server, reset the compromised passwords, and unsuspend the now-clean site.

While it's quite a bit more work than simply identifying the domain and account responsible for the abuse and suspending it, the extra time you spent investigating the cause of the issue will prevent the same issue from happening after your client "fixes" the problem by deleting the files/directories. Invariably, they'd get compromised again in the same way when the domain is restored, and you'd hear from the Abuse department again.

Server security goes hand in hand with systems administration, and even though it's not a very fun part of the job, it is a 24/7 responsibility that requires diligence and vigilance. By investing time and effort into securing your servers and fixing your hull breach rather than just bailing water overboard, your customers will see less downtime, you'll be using your server resources more efficiently, and (best of all) you won't have the Abuse team hounding you about more issues!

-Garrett

P.S. I came up with a brilliant analogy about DNS and the postal service, so that might be a topic for my next post ...

Subscribe to customer-service