Customer Service Posts

June 9, 2014

Visualizing a SoftLayer Billing Order

In my time spent as a data and object modeler, I’ve dealt with both good and bad examples of model visualization. As an IBMer through the Rational acquisition, I have been using modeling tools for a long time. I can appreciate a nice diagram shining a ray of light on an object structure, and abhor a behemoth spaghetti diagram.

When I started studying SoftLayer’s API documentation, I saw both the relational and hierarchical nature of SoftLayer’s concept model. The naming convention of API services and data types embodies their hierarchical structure. While reading about “relational properties” in data types, I thought it would be helpful to see diagrams showing relationships between services and data types versus clicking through reference pages. After all, diagramming data models is a valuable complement to verbal descriptions.

One way people can deal with complex data models is to digest them a little at a time. I can’t imagine a complete data model diagram of SoftLayer’s cloud offering, but I can try to visualize small portions of it. In this spirit, after reviewing article and blog entries on creating product orders using SoftLayer’s API, I drew an E-R diagram, using IBM Rational Software Architect, of basic order elements.

The diagram, Figure 1, should help people understand data entities involved in creating SoftLayer product orders and the relationships among the entities. In particular, IBM Business Partners implementing custom re-branded portals to support the ordering of SoftLayer resources will benefit from visualization of the data model. Picture this!

Figure 1. Diagram of the SoftLayer Billing Order

A user account can have many associated billing orders, which are composed of billing order items. Billing order items can contain multiple order containers that hold a product package. Each package can have several configurations including product item categories. They can be composed of product items with each item having several possible prices.

-Andrew

Andrew Hoppe, Ph.D., is a Worldwide Channel Solutions Architect for SoftLayer, an IBM Company.

May 8, 2014

SoftLayer Security: Questions and Answers

When I talk to IBM Business Partners about SoftLayer, one of the most important topics of discussion is security. We ask businesses to trust SoftLayer with their business-critical data, so it’s important that SoftLayer’s physical and network security is as transparent and understandable as possible.

After going through the notes I’ve taken in many of these client meetings, I pulled out the ten most frequently asked questions about security, and I’ve compiled answers.

Q1: How is SoftLayer secured? What security measures does SoftLayer have in place to ensure my workloads are safe?

A: This “big picture” question is the most common security-related question I’ve heard. SoftLayer’s approach to security involves several distinct layers, so it’s tough to generalize every aspect in a single response. Here are some of the highlights:

  • SoftLayer’s security management is aligned with U.S. government standards based on NIST 800-53 framework, a catalog of security and privacy controls defined for U.S. federal government information systems. SoftLayer maintains SOC 2 Type II reporting compliance for every data center. SOC 2 reports are audits against controls covering security, availability, and process integrity. SoftLayer’s data centers are also monitored 24x7 for both network and on-site security.
  • Security is maintained through automation (less likely for human error) and audit controls. Server room access is limited to authorized employees only, and every location is protected against physical intrusion.
  • Customers can create a multi-layer security architecture to suit their needs. SoftLayer offers several on-demand server and network security devices, such as firewalls and gateway appliances.
  • SoftLayer integrates three distinct network topologies for each physical or virtual server and offers security solutions for systems, applications, and data as well. Each customer has one or many VLANs in each data center facility, and only users and servers the customer authorizes can access servers in those VLANs.
  • SoftLayer offers single-tenant resources, so customers have complete control and transparency into their servers.

Q2: Does SoftLayer destroy my data when I’ve de-provisioned a compute resource?

A: Yes. When a customer cancels any physical or virtual server, all data is erased using Department of Defense (DoD) 5220.22-m standards.

Q3: How does SoftLayer protect my servers against distributed denial of service (DDoS) attacks?

A: A SoftLayer Network Operations Center (NOC) team monitors network performance and security 24x7. Automated DDoS mitigation controls are in place should a DDoS attack occur.

It’s important to clarify here that the primary objective of this DDoS mitigation is to maintain performance integrity of the overall cloud infrastructure. With that in mind, SoftLayer can’t stop a customer from being attacked, but it can shield the customer (and any other customers in the same network) from the effects of the attack. If necessary, SoftLayer will remove the target from the public network for periods of time and null-routes incoming connections. Because of SoftLayer’s three-tiered network architecture, a customer would still have access to the targeted system via the private network.

Q4: How is communication segmented from other tenants using SoftLayer?

A: SoftLayer utilizes industry standard VLANs and switch access control lists (ACLs) to segment customer environments. Customers have the ability to add and manage their own VLANs, providing additional security even inside their own accounts. ACLs are configured to permit or deny any specified network packet (data) to be directed along a switch.

Q5: How is my data kept private? How can I confirm that SoftLayer can’t read my confidential data?

A: This question is common customers who deal with sensitive workloads such as HIPAA-protected documentation, employee records, case files, and so on.

SoftLayer customers are encouraged to deploy a gateway device (e.g. Vyatta appliance) on which they can configure encryption protocols. Because the gateway device is the first hop into SoftLayer’s network, it provides an encrypted tunnel to traverse the VLANs that reside on SoftLayer. When securing compute and storage resources, customers can deploy single tenant dedicated storage devices to establish isolated workloads, and they can even encrypt their hard drives from the OS level to protect data at rest. Encrypting the hard drive helps safeguard data even if SoftLayer were to replace a drive or something similar.

Q6: Does SoftLayer track and log customer environments?

A: Yes. SoftLayer audits and tracks all user activity in our customer portal. Some examples of what is tracked include:

  • User access, both failed and authenticated attempts (destination IP is shown on a report)
  • Compute resources users deploy or cancel
  • APIs for each call (who called the API, the API call and function, etc.)
  • Intrusion Protection and Detection services that observe traffic to customer hosts
  • Additionally, customers have root access to operating systems on their servers, so they can implement additional logging of their own.

Q7: Can I disable access to some of my users through the customer portal?

A: Yes. SoftLayer has very granular ACLs. User entitlements are segmented into different categories, including Support, Security, and Hardware. SoftLayer also gives customers the ability to limit access to public and private networks. Customers can even limit user access to specific bare metal or virtual server.

Q8: Does SoftLayer patch my operating system?

A: For unmanaged cloud servers, no. Once the updated operating system is deployed on a customer’s server, SoftLayer doesn’t touch it.

If you want help with that hands-on server administration, SoftLayer offers managed hosting. In a managed hosting environment, Technical Account Managers (TAMs) are assigned as focal points for customer requests and issues. TAMs help with reports and trending data that provide recommendations to mitigate potential issues (including OS patching).

Q9: Is SoftLayer suited to run HIPAA workloads?

A: Yes. SoftLayer has a number of customers running HIPAA workloads on both bare metal and single-tenant virtual servers. A Business Associate Agreement (BAA), signed by SoftLayer and the customers, clearly define the shared responsibilities for data security: SoftLayer is solely responsible for the security of the physical data center, along with the SoftLayer-provided infrastructure.

Q10: Can SoftLayer run government workloads? Does SoftLayer use the FISMA standards?

A: The Federal Information Security Management Act (FISMA) defines a framework for managing information security that must be followed for all federal information systems. Some state institutions don’t require FISMA, but look to cloud hosting companies to be aligned to the FIMSA guidelines.

Today, two SoftLayer data centers are audited to the FISMA standards – Dallas (DAL05) and Washington, D.C. (WDC01). Customers looking for the FISMA standard can deploy their workloads in those data centers. Future plans include having data centers that comply with more stringent FedRAMP requests.

For additional information, I highly recommend the on-demand SoftLayer Fundamentals session, “Keep safe – securing your SoftLayer virtual instance.” Also, check out Allan Tate’s Thoughts on Cloud blog, “HIPAA and cloud computing: What you need to know” for more on how SoftLayer handles HIPPA-related workloads.

-Darrel Haswell

Darrel Haswell is a Worldwide Channel Solutions Architect for SoftLayer, an IBM Company.

May 2, 2014

Keyboard Shortcuts in the SoftLayer Customer Portal

I’m excited to introduce a new feature in the SoftLayer customer portal: Keyboard shortcuts!

Keyboard shortcuts give you quick access to the most commonly used features by simply typing a few characters. For those who prefer never having to reach for the mouse to navigate an application, you should find these handy additions quite helpful.

After you log into the Customer Portal, type “?” (shift + forward slash) on any page, and you'll see a full list of available keyboard shortcuts:

Keyboard Shortcuts

On the Keyboard Shortcuts help page, you have the option to enable or disable the functionality based on your preference. Keyboard shortcuts are enabled by default. Disabling this feature will turn off all keyboard shortcuts except the “?” shortcut so that you can access the enable/disable feature preference in the future if you change your mind. This preference is stored in a cookie in your browser, so changing computers or deleting your cookie will re-enable the feature.

The shortcuts are grouped into three sets: Global, Tabs, and Grids.

Keyboard Shortcuts

Global Navigation

You have the ability to navigate to any page in our application by typing in the respective position number in the menu combined with dashes (-). For example, typing 1-5-2 will open Support (1) > Help (5) > Portal Tour (2).

Use the “go to” key combinations to jump to a new location from anywhere in the portal. For example, type (g) and (d) to visit the Device List. Typing (g) and (u) allows you to access the list of portal users, and (g) and (t) takes you to view tickets. If you want to add a new ticket from anywhere in the portal, type (+) and (t). It’s that simple.

Keyboard Shortcuts

Tabs

Many of the pages within the portal have tabs that appear just above the main content of the page. These tabs often allow content to be filtered, or provide access to additional features related to the page topic. Each tab can be accessed by using a simple two-keystroke combination, such as (t) then (f) to reveal the Filter tab on the page.

Keyboard Shortcuts

Grids

Whenever a page contains a grid — a tabular listing — you can now perform common operations from the keyboard. Jump quickly from page to page (first/last or next/previous) or refresh the grid contents with a single keystroke.

Keyboard Shortcuts

Please give this new feature a try for yourself! We welcome your feedback. Please let us know if you would like to have us implement any other keyboard shortcuts in the future.

-Daniel

May 1, 2014

New App Release: SoftLayer Mobile for Windows 8.1

Today, the SoftLayer development team is launching a new platform accessibility tool for SoftLayer customers who want to easily manage their infrastructure from Windows. We've gotten a great response from the users of SoftLayer Mobile app for Windows Phone, so we turned our attention to creating an app for customers on Windows 8.1: SoftLayer Mobile for Windows 8.1.

With a growing number of users adopting and embracing Windows 8.1 on their PCs, and the Windows Store is becoming a vibrant community of useful apps for those customers. There are more than 145,000 apps on the Windows Store, and that number is expected to increase exponentially following Microsoft’s recent introduction of "Universal Apps for Windows Phone 8.1 and Windows 8.1.” With all that goodness and an expanding market, it was imperative for our mobile development team to build an app for customers using Windows 8.1 as their default OS or carrying Windows RT tablets.

Why Windows 8.1?

Our team wants to provide simple, efficient ways for customers to connect to SoftLayer infrastructure and perform any necessary management tasks while on-the-go. Our team is inspired by the power of connected devices in Windows ecosystem. By developing an app for Windows 8.1, we will slowly bring the phone, tablet and PC onto one streamlined platform — a concept many smart devices are adopting quickly.

What’s Fresh?

New Dashboard

The SoftLayer Mobile app for Windows 8.1 is a fresh new approach to its Windows Phone sibling. The app provides a dashboard view after authentication that provides a snapshot of some of the most commonly used information and controls in the portal.

Currently, the dashboard supports four different panels: Tickets, devices, accounting and bandwidth. All display an overview of relevant information for you and your environment. The dashboard also allows you to quickly add a ticket or make a one-time payment on your account.

SoftLayer Mobile for Windows 8.1

In-line Ticket Updates

In the new tickets module, you can update tickets without ever leaving the page. This functionality is similar to what you see on many social websites, and it's integrated to be seamless.

SoftLayer Mobile for Windows 8.1

Search Everywhere!

One of the coolest additions to the new app is the introduction of search functionality in each module. Now, you can search a ticket, a device, or an invoice by just typing into the search box! The search capability lets you spend less time scrolling and more time working.

SoftLayer Mobile for Windows 8.1

Bandwidth Display

Smart phones have apps that measure and report how much data you are using, and your infrastructure should be similarly transparent Bandwidth usage is an important aspect of server management, so we built the bandwidth module to show your infrastructure's public and private traffic for current and previous billing cycles. This view also helps you see when a server is about to reach its limits so that you can plan accordingly.

SoftLayer Mobile for Windows 8.1

The module provides two ways to look at the data:

  • In a tabular form by clicking the “Show/Hide Traffic Details” button.
  • In a graphic representation by clicking the “View Graph” button.

SoftLayer Mobile for Windows 8.1

Same Functionality. Better Experience.

Sometimes change is not always needed for a nicely crafted feature. The new app keeps the same feature richness of the Windows Phone app and arranges it in a user-friendly way. For example, in the devices module, you can navigate to between different tabs to get the information you need, from password lists and attached tickets to a specific device or monitoring alarms.

The “Remote Control” section on the module allows you to perform actions such as rebooting, power cycles, restarts and pinging servers. In addition, you can view hardware and software installed on the device along with the hardware and network components attached. In the current phone version, you can only see the root password for the device, but in the Windows 8.1 app, you see all passwords for the server.

SoftLayer Mobile for Windows 8.1

What's Next?

During the development of this app, the team's goal was to test to adopt a framework that would be ideal for scaling. More and more developers are adopting a Model-View-Model (MVVM) approach to mobile and web app development, so our goal was to use that approach for this project. The significant challenge we faced when adopting this approach was finding a well-supported framework that met our application's needs. We weren't able to find suitable frameworks that committed regular updates in SDKs or in APIs, so we ended up using the same MVVM principles without any underlying framework. In the end, the project allowed us to create our own framework for future projects!

There are many exciting features that are lined up for the Windows 8.1 app. Download it now: SoftLayer Mobile for Windows 8.1

After you try it out, please submit your feedback ... We want to keep improving the app by providing the features and functionality that matter most to you.

-Imran

February 3, 2014

Risk Management: 5 Tips for Managing Risk in the Cloud

Security breaches have made front-page news in recent months. With stories about Target, Neiman Marcus, Yahoo! and GoDaddy in the headlines recently, the importance of good information security practices is becoming harder and harder to ignore — even for smaller businesses. Moving your business into the cloud offers a plethora of benefits; however, those benefits do not come without their challenges. Moving your business into the cloud involves risks such as multi-tenancy, so it's important to be able to properly manage and identify these risks.

1. Know the Security Your Provider Offers
While some SaaS providers may have security baked-in, most IaaS providers (including SoftLayer) leave much of the logical security responsibility of a customer's systems to the customer. For the security measures that an infrastructure provider handles, the provider should be able to deliver documentation attesting these controls. We perform an annual SOC2 audit, so we can attest to the status of our security and availability controls as a service organization. With this information, our customers use controls from our report as part of their own compliance requirements. Knowing a provider's security controls (and seeing proof of that security) allows business owners and Chief Information Security Officers (CISO) to have peace-of-mind that they can properly plan their control activities to better prevent or respond to a breach.

2. Use the Cloud to Distribute and Replicate Your Presence
The incredible scalability and geographical distribution of operating in the cloud can yield some surprising payoff. Experts in the security industry are leveraging the cloud to reduce their patch cycles to days, not weeks or months. Most cloud providers have multiple sites so that you can spread your presence nationally, or even globally. With this kind of infrastructure footprint, businesses can replicate failover systems and accommodate regional demand across multiple facilities with the minimal incremental investment (and with nearly identical security controls).

3. Go Back to the Basics
Configuration management. Asset management. Separation of duties. Strong passwords. Many organizations get so distracted by the big picture of their security measures that they fail to manage these basic rights. Take advantage of any of your provider's tools to assist in the ‘mundane’ tasks that are vitally important to your business's overall security posture. For example, you can use image templates or post-provisioning scripts to deploy a standard baseline configuration to your systems, then track them down to the specific server room. You’ll know what hardware is in your server at all times, and if you're using SoftLayer, you can even drill down to the serial numbers of your hard drives.

4. Have Sound Incident Response Plans
The industry is becoming increasingly cognizant of the fact that it’s not a matter of if, but when a security threat will present itself. Even with exceedingly high levels of baked-in security, most of the recent breaches resulted from a compromised employee. Be prepared to respond to security incidents with confidence. While you may be physically distanced from your systems, you should be able to meet defined Recovery Time Objectives (RTOs) for your services.

5. Maintain Constant Contact with Your Cloud Provider
Things happen. No amount of planning can completely halt every incident, whether it be a natural disaster or a determined attacker. Know that your hosting provider has your back when things take an unexpected turn.

With proper planning and good practice, the cloud isn't as risky and frightening as most think. If you're interested in learning a little more about the best practices around security in the cloud, check out the Cloud Security Alliance (CSA). The CSA provides a wealth of knowledge to assist business owners and security professionals alike. Build on the strengths, compensate for the weaknesses, and you and your CISO will be able to sleep at night (and maybe even sneak in a beer after work).

-Matt

December 5, 2013

How to Report Abuse to SoftLayer

When you find hosted content that doesn't meet our acceptable use policy or another kind of inappropriate Internet activity originating from a SoftLayer service, your natural reaction might be to assume, "SoftLayer must know about it, and the fact that it's going on suggests that they're allowing that behavior." I know this because every now and then, I come across a "@SoftLayer is phishing my email. #spamming #fail" Tweet or a "How about u stop hacking my computer???" Facebook post. It's easy to see where these users are coming from, so my goal for this post is to provide the background you need to understand how behavior we don't condone — what we consider "abuse" of our services — might occur on our platform and what we do when we learn about it.

The most common types of abuse reported from the SoftLayer network are spam, copyright/trademark infringement, phishing and abusive traffic (DDoS attacks). All four are handled by the same abuse team, but they're all handled a bit differently, so it's important to break them down to understand the most efficient way to report them to our team. When you're on the receiving end of abuse, all you want is to make it stop. In the hurry to report the abusive behavior, it's easy to leave out some of the key information we need to address your concern, so let's take a look at each type of abuse and the best ways to report it to the SoftLayer team:

If You Get Spam

Spam is the most common type of abuse that gets reported to SoftLayer. Spam email is unsolicited, indiscriminate bulk messaging that is sent to you without your explicit consent. If you open your email client right now, your junk mail folder probably has a few examples of spam ... Someone is trying to sell you discount drugs or arrange a multi-million dollar inheritance transfer. In many ways, it's great that email is so easy to use and pervasive to our daily lives, but that ease of use also makes it an easy medium for spammers to abuse. Whether the spammer is a direct SoftLayer customer or a customer of one of our customers or somewhere further down the line of customers of customers, spam messages sent from a SoftLayer server will point back to us, and our abuse team is the group that will help stop it.

When you receive spam sent through SoftLayer, you should forward it directly to our abuse team (abuse@softlayer.com). Our team needs a full copy of the email with its headers intact. If you're not sure what that means, check out these instructions on how to retrieve your email headers. The email headers help tell the story about where exactly the messages are coming from and which customer we need to contact to stop the abuse.

If You See Phishing

Phishing abuse might be encountered via spam or you might encounter it on a website. Phishing is best described as someone masquerading as someone else to get your sensitive information, and it's one of the most serious issues our abuse team faces. Every second that a phishing/scam site is online, another user might be fooled into giving up his or her credit card or login information, and we don't want that to happen. Often, the fact that a site is not legitimate is clear relatively quickly, but as defenses against phishing have gotten better, so have the phishing sites. Take a minute to go through this phishing IQ test to get an idea of how difficult phishing can be to trace.

When it comes to reporting phishing, you should send the site's URL to the abuse team (also using abuse@softlayer.com). If you came across the phishing site via a spam email, be sure to include the email headers with your message. To help us filter the phishing complaint, please make sure to include the word "phishing" in your email's subject line. Our team will immediately investigate and follow up with the infringing customer internally.

If You Find Copyright or Trademark Infringement

If infringement of your copyright or trademark is happening on our platform, we want to know about it so we can have it taken down immediately. Copyright complaints and trademark complaints are handled slightly differently, so let's look at each type to better understand how they work.

Complaints of copyright infringement are processed by our abuse team based on the strict DMCA complaint laws. When I say "strict" in that sentence, I'm not saying it lightly ... Because DMCA complaints are legal issues, every requirement in the DMCA must be met in order for our team to act on the complaint. That might seem arbitrary, but we're not given much leeway when it comes to the DMCA process, and we have to be sticklers.

On our DMCA legal page, we outline the process of reporting a DMCA complaint of copyright infringement (primarily citing the statute 17 U.S.C. Section 512(c)(3)). If you don't completely understand what needs to be included in the claim, we recommend that you seek independent legal advice. It sounds harsh, but failure to submit copyright infringement notification as described above will result in no legal notice or action on behalf of SoftLayer. When you've made sure all required evidence has been included in your DMCA complaint, make sure "copyright" or "DMCA" are included in your subject line and submit the complaint to copyright@softlayer.com.

Trademark complaints do not have the same requirements as copyright complaints, but the more information you can provide in your complaint, the easier it will be for our customer to locate and remove the offending material. If you encounter unauthorized use of your registered trademark on our network, please email copyright@softlayer.com with details — the exact location of the infringing content, your trademark registration information, etc. — along with an explanation that this trademark usage is unauthorized and should be removed. In your email, please add the word "trademark" to the subject line to help us filter and prioritize your complaint.

If You See Abusive Traffic

Spam, phishing and copyright infringement are relatively straightforward when it comes to finding and reporting abuse, but sometimes the abuse isn't as visible and tangible (though the effect usually is). If a SoftLayer server is sending abusive traffic to your site, we want to know about it as quickly as possible. Whether that behavior is part of a Denial of Service (DoS) attack or is just scanning ports to possibly attack later, it's important that you give us details so we can prevent any further activity.

To report this type of abuse, send a snippet from your log file including at least 10 lines of logs that show attempts to break into or overload your server. Here's a quick reference to where you can find the relevant logs to send:

  • Email Spam - Send Mail Logs:
    • /var/log/maillog
    • /usr/local/psa/var/log/maillog
  • Brute Force Attacks - Send SSH Logs:
    • /var/log/messages
    • /var/log/secure

Like spam and phishing reports, abusive traffic complaints should be sent to abuse@softlayer.com with a quick explanation of what is happening and any other details you can provide. When you submit a complaint about abusive traffic, make sure your message's subject line reflects the type of issue ("DDoS attack," "brute force attempts," etc.) so our team can investigate your report even quicker.

As I mentioned at the start of this post, these are just four types of abusive behavior that our abuse department addresses on a daily basis. Our Acceptable Use Policy (AUP) outlines what can and cannot be hosted using SoftLayer services, and the process of reporting other types of abuse is generally the same as what you see in the four examples I mentioned above ... Send a clear, concise report to abuse@softlayer.com with key words about the type of violation in the message's subject line. When our team is able to look into your complaint and find the evidence they need to take action, they do so quickly.

I can't wrap up this blog of tips without mentioning the "Tips from the Abuse Department" blog Jennifer Groves wrote about reporting abuse ... It touches on some of the same ideas as this post, and it also provides a little more perspective from behind the lines of the abuse department. As the social media gal, I don't handle abuse on a day-to-day basis, but I do help people dealing with abuse issues, and I know a simple guide like this will be of value.

If an abuse-related issue persists and you don't feel like anything has been fixed, double-check that you've included all the necessary information and evidence in your correspondence to the abuse team. In most cases, you will not receive a response from the abuse team, but that doesn't mean they aren't taking action. The abuse@ and copyright@ email aliases function as notification systems for our abuse teams, and they correspond with the infringing customers internally when a complaint is submitted. Given the fact that hundreds of users may report the same abusive behavior at the same time, responding directly to each message would slow down the process of actually resolving the issue (which is the priority).

If everything was included in your initial correspondence with the abuse team but you still don't notice a change in the abusive behavior, you can always follow up with our social media team at twitter@softlayer.com, and we'll do everything we can to help.

-Rachel

May 7, 2013

Tips from the Abuse Department: DMCA Takedown Notices

If you are in the web hosting business or you provide users with access to store content on your servers, chances are that you're familiar with the Digital Millennium Copyright Act (DMCA). If you aren't familiar with it, you certainly should be. All it takes is one client plagiarizing an article or using a filesharing program unscrupulously, and you could find yourself the recipient of a scary DMCA notice from a copyright holder. We've talked before about how to file a DMCA complaint with SoftLayer, but we haven't talked in detail about SoftLayer's role in processing DMCA complaints or what you should do if you find yourself on the receiving end of a copyright infringement notification.

The most important thing to understand when it comes to the way the abuse team handles DMCA complaints is that our procedures aren't just SoftLayer policy — they are the law. Our role in processing copyright complaints is essentially that of a middleman. In order to protect our Safe Harbor status under the Online Copyright Infringement Liability Limitation Act (OCILLA), we must enforce any complaint that meets the legal requirements of a takedown notice. That DMCA complaint must contain specific elements and be properly formatted in order to be considered valid.

Responding to a DMCA Complaint

When we receive a complaint that meets the legal requirements of a DMCA takedown notice, we must relay the complaint to our direct customer and enforce a deadline for removal of the violating material. We are obligated to remove access to infringing content when we are notified about it, and we aren't able to make a determination about the validity of a claim beyond confirming that all DMCA requirements are met.

The law states that SoftLayer must act expeditiously, so if you receive notification of a DMCA complaint, it's important that you acknowledge the ticket that the abuse department opened on your account and let us know your intended course of action. Sometimes that action is as simple as removing an infringing URL. Sometimes you may need to contact your client and instruct them to take the material down. Whatever the case may be, it's important to be responsive and to expressly confirm when you have complied and removed the material. Failure to acknowledge an abuse ticket can result in disconnection of service, and in the case of copyright infringement, SoftLayer has a legal obligation to remove access to the material or we face serious liability.

DMCA Counter Notifications

Most DMCA complaints are resolved without issue, but what happens if you disagree with the complaint? What if you own the material and a disgruntled former business partner is trying to get revenge? What if you wrote the content and the complaining party is copying your website? Thankfully there are penalties for filing a false DMCA complaint, but you also have recourse in the form of a counter notification. Keep in mind that while it may be tempting to plead your case to the abuse department, our role is not to play judge or jury but to allow the process to work as it was designed.

In some cases, you may be able to work out a resolution with the complaining party directly (misunderstandings happen, licenses lapse, etc.) and have them send a retraction, but most of the time your best course of action is to submit a counter notification.

Just as a takedown notice must be crafted in a specific way, counter notifications have their own set of requirements. Once you have disabled the material identified in the original complaint, we can provide your valid, properly formatted counter notification to the complaining party. Unless we receive a court order from the complaining party within the legally mandated time frame the material can be re-enabled and the case is closed for the time being.

While it might sound complicated, it's actually pretty straightforward, but we urge you to do your research and make sure you know what to do in the event a client of yours is hit with a DMCA takedown notice. Just as we are unable to make judgment calls when it comes to takedown notices or counter notifications, we are also unable to offer any legal advice for you if you need help. Hopefully this post cleared up a few questions and misconceptions about how the abuse department handles copyright complaints. In short:

Do take DMCA notifications seriously. You are at risk for service interruption and possible legal liability.
Do respond to the abuse department letting them know the material has been disabled and, if applicable, if you plan to file a counter notification.
Don't refuse to disable the material. Even if you believe the claim is false and you wish to file a counter notification, the material must be disabled within the time period allotted by the abuse department or we have to block access to it.
Don't expect the abuse department to take sides.

As with any abuse issue, communication and responsiveness is important. Disconnecting your server is a last resort, but we have ethical and legal obligations to uphold. The DMCA process certainly has its weaknesses and it leaves a bit to be desired, but at the end of the day, it's the law, and we have to operate inside of our legal obligation to it.

-Jennifer

October 25, 2012

Tips from the Abuse Department: Save Your Sinking Ship

I often find that the easiest way to present a complex process is with a relatable analogy. By replacing esoteric technical details with a less intimidating real-world illustration, smart people don't have to be technically savvy to understand what's going on. When it comes to explaining abuse-related topics, I find analogies especially helpful. One that I'm particularly keen on in explaining Abuse tickets in the context of a sinking ship.

How many times have you received an Abuse ticket and responded to the issue by suspending what appears to be the culprit account? You provide an update in the ticket, letting our team know that you've "taken care of the problem," and you consider it resolved. A few moments later, the ticket is updated on our end, and an abuse administrator is asking follow-up questions: "How did the issue occur?" "What did you do to resolve the issue?" "What steps are being taken to secure the server in order to prevent further abuse?"

Who cares how the issue happened if it's resolved now, right? Didn't I respond quickly and address the problem in the ticket? What gives? Well, dear readers, it's analogy time:

You're sailing along in a boat filled with important goods, and the craft suddenly begins to take on water. It's not readily apparent where the water is coming from, but you have a trusty bucket that you fill with the water in the boat and toss over the side. When you toss out all the water onboard, is the problem fixed? Perhaps. Perhaps not.

You don't see evidence of the problem anymore, but as you continue along your way, your vessel might start riding lower and lower in the water — jeopardizing yourself and your shipment. If you were to search for the cause of the water intake and take steps to patch it, the boat would be in a much better condition to deliver you and your cargo safely to your destination.

In the same way that a hull breach can sink a ship, so too can a security hole on your server cause problems for your (and your clients') data. In the last installment of "Tips from the Abuse Department," Andrew explained some of the extremely common (and often overlooked) ways servers are compromised and used maliciously. As he mentioned in his post, Abuse tickets are, in many cases, the first notification for many of our customers that "something's wrong."

At a crucial point like this, it's important to get the water out of the boat AND prevent the vessel from taking on any more water. You won't be sailing smoothly unless both are done as quickly as possible.

Let's look at an example of what thorough response to an Abuse ticket might look like:

A long-time client of yours hosts their small business site on one of your servers. You are notified by Abuse that malware is being distributed from a random folder on their domain. You could suspend the domain and be "done" with the issue, but that long-time client (who's not in the business of malware distribution) would suffer. You decide to dig deeper.

After temporarily suspending the account to stop any further malware distribution, you log into the server and track down the file and what permissions it has. You look through access logs and discover that the file was uploaded via FTP just yesterday from an IP in another country. With this IP information, you search your logs and find several other instances where suspicious files were uploaded around the same time, and you see that several FTP brute force attempts were made against the server.

You know what happened: Someone (or something) scanned the server and attempted to break into the domain. When the server was breached, malware was uploaded to an obscure directory on the domain where the domain owners might not notice it.

With this information in hand, you can take steps to protect your clients and the server itself. The first step might be to implement a password policy that would make guessing passwords very difficult. Next, you might add a rule within your FTP configuration to block continued access after a certain number of failed logins. Finally, you would clean the malicious content from the server, reset the compromised passwords, and unsuspend the now-clean site.

While it's quite a bit more work than simply identifying the domain and account responsible for the abuse and suspending it, the extra time you spent investigating the cause of the issue will prevent the same issue from happening after your client "fixes" the problem by deleting the files/directories. Invariably, they'd get compromised again in the same way when the domain is restored, and you'd hear from the Abuse department again.

Server security goes hand in hand with systems administration, and even though it's not a very fun part of the job, it is a 24/7 responsibility that requires diligence and vigilance. By investing time and effort into securing your servers and fixing your hull breach rather than just bailing water overboard, your customers will see less downtime, you'll be using your server resources more efficiently, and (best of all) you won't have the Abuse team hounding you about more issues!

-Garrett

P.S. I came up with a brilliant analogy about DNS and the postal service, so that might be a topic for my next post ...

August 15, 2012

Managing Support Tickets: Email Subscriptions

This week, the development team rolled out some behind-the-scenes support functionality that I think a lot of our customers will want to take advantage of, so I put together this quick blog post to spread the word about it. With the new release, the support department is able to create "Ticket Email Subscriptions" for different ticket groups on every customer account. As a customer, you might not be jumping up and down with joy after reading that one-sentence description, but after you hear a little more about the functionality, if you're not clapping, I hope you'll at least give us a thumbs-up.

To understand the utility of the new ticket email subscription functionality, let's look at how normal tickets work in the SoftLayer portal without email subscriptions:

User Creates Ticket

  1. User A creates a ticket.
  2. User A becomes the owner of that ticket.
  3. When SoftLayer responds to the ticket, an email notification is sent to User A to let him/her know that the ticket has been updated.

SoftLayer Creates Ticket

  1. SoftLayer team creates a ticket on a customer's account.
  2. The primary customer contact on the account is notified of the new ticket.
  3. Customer logs into the portal and responds to ticket.
  4. Customer gets notifications of updates (as described above).

There's nothing wrong with the existing support notification process, but that doesn't mean there aren't ways to make the process better. What if User A creates an urgent ticket on his/her way out the door to go on vacation? User B and User C aren't notified when an update is posted on User A's ticket, so the other users aren't able to get to the ticket and respond as quickly as they would have if they received the notification. What if the primary customer contact on the account isn't the best person to receive a monitoring alert? The administrator who will investigate the monitoring alert has to see the new ticket on the account or hear about it from the primary contact (who got the notification).

Ticket email subscriptions allow for customers to set contact addresses to be notified when a ticket is created, edited or moved in a particular ticket group. Here are the ticket groups differentiated in our initial release:

  • Billing - Any ticket in our Billing department
  • Maintenance - Scheduled maintenance notifications for specific servers
  • Network Protection - DDoS mitigation and Null Routes
  • Monitoring - Host Down Alerts
  • CST, SysAdmin and Hardware - Any ticket in our support and data center departments
  • Managed Services - Tickets that relate to any managed services
  • Network Maintenance - Scheduled network maintenance

You'll notice that Abuse isn't included in this list, and the only reason it's omitted is because you've always been able to designate a contact on your account for abuse-related tickets ... Ticket subscriptions extend that functionality to other ticket groups.

Because only one email address can be "subscribed" to notifications in each ticket group, we recommend that customers use their own distribution lists as the email contacts. With a DL as the contact, you can enable multiple users in your organization to receive notifications, and you can add and remove users from each distribution list on your end quickly and easily.

When User A creates a ticket with the data center and goes on vacation, as soon as SoftLayer responds to the ticket, User A will be notified (as usual), and the supportsubscription@yourdomain.com distribution will get notified as well. When a network maintenance is ticket is created by SoftLayer, the netmaintsubscription@yourdomain.com distribution will be notified.

Ticket email subscriptions are additive to the current update notification structure, and they are optional. If you want to set up ticket email subscriptions on your account, create a ticket for the support department and provide us with the email addresses you'd like to subscribe to each of the ticket groups.

We hope this tool helps provide an even better customer experience for you ... If you don't mind, I'm going to head back to the lab to work with the dev team to cook up more ways to add flexibility and improvements into the customer experience.

-Chris

June 25, 2012

Tips from the Abuse Department: Part 2 - Responding to Abuse Reports

If you're a SoftLayer customer, you don't want to hear from the Abuse department. We know that. The unfortunate reality when it comes to hosting a server is that compromises can happen, mistakes can be made, and even the most scrupulous reseller can fall victim to a fraudulent sign-up or sly spammer. If someone reports abusive behavior originating from one of your servers on our network, it's important to be able to communicate effectively with the Abuse department and build a healthy working relationship.

Beyond our responsibility to enforce the law and our Acceptable Use Policy, the Abuse department is designed to be a valuable asset for our customers. We'll notify you of all valid complaints (and possibly highlight security vulnerabilities in the process), we'll assist you with blacklist removal, we can serve as a liaison between you and other providers if there are any problems, and if you operate an email-heavy platform or service, we can help you understand the steps you need to take to avoid activity that may be considered abuse.

At the end of the day, if the Abuse department can maintain a good rapport with our customers, both our jobs can be easier, so I thought this installment in the "Tips from the Abuse Department" series could focus on some best practices for corresponding with Abuse from a customer perspective.

Check Your Tickets

This is the easiest, most obvious recommendation I can give. You'd be surprised at how many service interruptions could be avoided if our customers were more proactive about keeping up with their open tickets. Our portal is a vital tool for your business, so make sure you are familiar with how to access and use it.

Keep Your Contact Information Current

Our ticket system will send notifications to the email address you have on file, so making sure this information is correct and current is absolutely crucial, especially if you aren't in the habit of checking the ticket system on a regular basis. You can even set a specific address for abuse notifications to be sent to, so make use of this option. The quicker you can respond to an abuse report, the quicker the complaint can be resolved, and by getting the complaint resolved quickly, you avoid any potential service interruption.

If we are unable to reach you by ticket, we may need to call you, so keep your current phone numbers on file as well.

Provide Frequent Updates

Stay in constant communication in the midst of responding to an abuse report, and adhere to the allotted timeline in the ticket. If we don't see updates that the abusive behavior is being addressed in the grace period we are able to offer, your server is at risk of disconnection. By keeping us posted about the action you're taking and the time you need to resolve the matter, we're able to be more flexible.

If a customer on your servers created a spamming script or a phishing account, taking immediate steps to mitigate the issue by suspending that customer is another great way to respond to the process while you're performing an investigation of how that activity was started. We'll still want a detailed resolution, but if the abuse is not actively ongoing we can work with you on deadlines.

Be Concise ... But Not Too Concise

One-word responses: bad. Page long responses: also not ideal. If given the option we would opt for the latter, but your goal should be to outline the cause and resolution of any reported abusive activity as clearly and succinctly as possible in order to ease communication and expedite closing of the ticket.

Responding to a ticket with, "Fixed," is not sufficient to for the Abuse department to consider the matter resolved, but we also don't need a dump of your entire log file. Before the Abuse team can close a ticket, we have to see details of how the complaint was resolved, so if you don't provide those details in your first response, you can bet we'll keep following up with you to get them. What details do we need?

Take a Comprehensive Approach

In addition to stopping the abusive activity we want to know:

  1. How/why the issue occurred
  2. What steps are being taken to prevent further issues of that nature

We understand that dealing with abuse issues can often feel like a game of Whack-A-Mole, but if you can show that you're digging a bit deeper and taking steps to avoid recurrence, that additional work is very much appreciated. Having the Abuse department consider you a proactive, ethical and responsible customer is a worthy goal.

Be Courteous

I'm ending on a similar note to my last blog post because it's just that important! We understand getting an abuse ticket is a hassle, but please remember that we're doing our best to protect our network, the Internet community and you.

Unplugging your server is a last resort for us, and we want to make sure everyone is on the same page to prevent us from getting to that last resort. In the unfortunate event that you do experience an abuse issue, please refer back to this blog — it just might save you some headaches and perhaps some unnecessary downtime.

-Jennifer

Subscribe to customer-service