Author Archive: Dani Roisman

October 3, 2013

Improving Communications for Customer-Affecting Events

Service disruptions are never a good thing. Though SoftLayer invests extensively in design, equipment, and personnel training to reduce the risk of disruptions to our customers, in the technology world there are times where scheduled events or unplanned incidents are inevitable. During those times, we understand that restoring service is top priority, and almost as important is communicating to customers regarding the cause of the incident and the current status of our work to resolve it.

To date we've used a combination of tickets, emails, forum posts, portal "yellow" notifications, as well as RSS and Twitter feeds to provide status updates during service-affecting events. Many of these methods require customers to "come and get it," so we've been working on a more targeted, proactive approach to disseminating information.

I'm excited to report that our Development and Operations teams have collaborated on new functionality in the SoftLayer portal that will improve the way we share information with customers about unplanned infrastructure troubles or upcoming planned maintenances. With our new Event Communications toolset, we're able to pinpoint the accounts affected by an event and update users who opt-in to receive notifications about how these events may impact their services.

Notifications

As the development work is finalized, we plan to roll out a few phases of improvements. The first phase of implementation, which is ready today, enables email alerts for unplanned incidents, and any portal user account can opt-in to receive them. These emails provide details about the impact and current status of an unplanned incident in progress (UIP). In this phase, notifications can be sent for devices such as physical servers, CCIs and shared SLB VIPs, and we will be adding additional services over time.

In future phases of this project, we plan to include:

  • A new "Event" section of the Customer Portal which will allow customers to browse upcoming scheduled maintenances or current/recent unplanned incidents which may impact their services. In the past, we generated tickets for scheduled maintenances, so separating these event notifications will improve customer visibility.
  • Enhanced visibility for events in our mobile apps (phone/tablet).
  • Updates to affected services for a given event as customers add / change services.
  • Notification of newly added or newly updated events that have not been read by the user (similar email "inbox" functionality) in the portal.
  • Identification of any related current or recent events as a customer begins to open a ticket in the portal.
  • Reminders of upcoming scheduled maintenances along with progress updates to the event notification throughout the maintenance in some cases.
  • Improved ability to correlate specific incidents to customer service troubles.
  • Dissemination of RFO (reason-for-outage) statements to customers following a post-incident review of an unplanned service disruption.

Since we respect our customers' inboxes, these notifications will only be sent to user accounts that have opted in. If you'd like to receive them, simply log into the Customer Portal and navigate to "Notification Subscriptions" under the "Administration" menu (direct link). From that page, individual users can control event subscriptions, and portal logins that have administrative control over multiple users on the account can control the opt-in for themselves and their downstream users. For a more detailed walkthrough of the opt-in process, visit the KnowledgeLayer: "Update Subscription Settings for the Event Management System"

The Network Operations Center has already begun using this customer notification toolset for customer-affecting events, so we recommend that you opt-in as soon as possible to benefit from this new functionality.

-Dani

July 5, 2012

Bandwidth Utilization: Managing a Global Network

SoftLayer has over 1,750 Gbit/s of network capacity. In each of our data centers and points of presence, we have an extensive library of peering relationships and multiple 10 Gbit/s connections to independent Tier 1 carriers. We operate one of the fastest, most reliable networks on the planet, and our customers love it:

From a network operations standpoint, that means we have our work cut out for us to keep everything running smoothly while continuing to build the network to accommodate a steady increase in customer demand. It might be easier to rest on our laurels to simply maintain what we already have in place, but when you look at the trend of bandwidth usage over the past 18 months, you'll see why we need to be proactive about expanding our network:

Long Term Bandwidth Usage Trend

The purple line above plots the 95th percentile of weekly outbound bandwidth utilization on the SoftLayer network, and the red line shows the linear trend of that consumption over time. From week to week, the total usage appears relatively consistent, growing at a steady rate, but when you look a little deeper, you get a better picture of how dynamic our network actually is:

SoftLayer Weekly Bandwidth Usage

The animated gif above shows the 2-hour average of bandwidth usage on our entire network over a seven-week period (times in CDT). As you can see, on a day-to-day basis, consumption fluctuates pretty significantly. The NOC (Network Operations Center) needs to be able to accommodate every spike of usage at any time of day, and our network engineering and strategy teams have to stay ahead of the game when it comes to planning points of presence and increasing bandwidth capacity to accommodate our customers' ever-expanding needs.

But wait. There's more.

Let's go one level deeper and look a graph of the 95th percentile bandwidth usage on 5-minute intervals from one week in a single data center:

Long Term Bandwidth Usage Trend

The variations in usage are even more dramatic. Because we have thirteen data centers geographically dispersed around the world with an international customer base, the variations you see in total bandwidth utilization understate the complexity of our network's bandwidth usage. Customers targeting the Asian market might host content in SNG01, and the peaks in bandwidth consumption from Singapore will counterbalance the valleys of consumption at the same time in the United States and Europe.

With that in mind, here's a challenge for you: Looking at the graph above, if the times listed are in CDT, which data center do you think that data came from?

It would be interesting to look at weekly usage trends, how those trends are changing and what those trends tell us about our customer base, but that assessment would probably be "information overload" in this post, so I'll save that for another day.

-Dani

P.S. If you came to this post expecting to see "a big truck" or "a series of tubes," I'm sorry I let you down.

June 11, 2012

"World IPv6 Launch Day" and What it Means for You

June 6, 2012, marked a milestone in the further advancement of the Internet: World IPv6 Launch Day. It was by no means an Earth-shattering event or a "flag day" where everyone switched over to IPv6 completely ... What actually happened was that content providers enabled AAAA DNS records for their websites and other applications, and ISPs committed to providing IPv6 connectivity to at least 1% of their customers by this date.

What's all of this fuss about the IPv6 transition about? The simplest way to explain the situation is that the current Internet can stay working as it does, using IPv4 addresses, forever ... if we're okay with it not growing any more. If no more homes and businesses wanted to get on the Internet, and no more new phones or tablets were produced, and no more websites or applications were created. SoftLayer wouldn't be able to keep selling new servers either. To prevent or lose that kind of organic growth would be terrible, so an alternative had to be created to break free from the limitations of IPv4.

IPv4 to IPv6

The long-term goal is to migrate the entire Internet to the IPv6 standard in order to eliminate the stifling effect of impending and inevitable IP address shortages. It is estimated that there are roughly 2.5 billion current connections to the Internet today, so to say the transition has a lot of moving parts would be an understatement. That complexity doesn't lessen the urgency of the need to make the change, though ... In the very near future, end-users and servers will no longer be able to get IPv4 connections to the Internet, and will only connect via IPv6.

The primary transition plan is to "dual-stack" all current devices by adding IPv6 support to everything that currently has an IPv4 address. By adding native IPv6 functionality to devices using IPv4, all of that connectivity will be able to speak via IPv6 without transitional technologies like NAT (Network Address Translation). This work will take several years, and time is not a luxury we have with the dwindling IPv4 pool.

Like George mentioned in a previous post, I see World IPv6 Launch day as a call-to-action for a "game changer." The IPv6 transition has gotten a ton of visibility from some of the most recognizable names on the Internet, but the importance and urgency of the transition can't be overstated.

So, what does that mean for you?

To a certain extent, that depends on what your involvement is on the Internet. Here are a few steps everyone can take:

  • Learn all you can about IPv6 to prepare for the work ahead. A few good books about IPv6 have been published, and resources like ARIN's IPv6 Information Wiki are perfect places to get more information.
  • If you own servers or network equipment, check them for IPv6 functionality. Upgrade or replace any software or devices to ensure that you can deliver native IPv6 connectivity end-to-end without any adverse impact to IPv6 users. If any piece of gear isn't IPv6-capable, IPv6 traffic won't be able to pass through your network.
  • If you are a content provider, make your content available via IPv6. This starts with requesting IPv6 service from your ISP. At SoftLayer, that's done via a zero-cost sales request to add IPv6 addresses to your VLANs. You should target 100% coverage for your services or applications — providing the same content via IPv6 as you do via IPv4. Take an inventory of all your DNS records, and after you've tested extensively, publish AAAA records for all hostnames to start attracting IPv6 traffic.
  • If you are receiving Internet connectivity to your home or business desktops, demand IPv6 services from your upstream ISP. Also be sure to check your access routers, switches and desktops to ensure they are running the most recent code with stable IPv6 support.
  • If you are running equipment such as firewalls, load balancers, IDS, etc., contact your vendors to learn about their IPv6 support and how to properly configure those devices. You want to make sure you aren't limiting performance or exposing any vulnerabilities.

Starting now, there are no more excuses. It's time to get IPv6 up and running if you want to play a part in tomorrow's Internet.

-Dani

January 19, 2012

IPv6 Milestone: "World IPv6 Launch Day"

On Tuesday, the Internet Society announced "World IPv6 Launch Day", a huge step in the transition from IPv4 to IPv6. Scheduled for June 6, 2012, this "launch day" comes almost one year after the similarly noteworthy World IPv6 Day, during which many prominent Internet businesses enabled IPv6 AAAA record resolution for their primary websites for a 24-hour period.

With IPv6 Day serving as a "test run," we confirmed a lot of what we know about IPv6 compatibility and interoperability with deployed systems throughout the Internet, and we even learned about a few areas that needed a little additional attention. Access troubles for end-users was measured in fractions of a percentage, and while some sites left IPv6 running, many of them ended up disabling the AAAA IPv6 records at the end of the event, resuming their legacy IPv4-only configuration.

We're past the "testing" phase now. Many of the IPv6-related issues observed in desktop operating systems (think: your PCs, phones, and tablets) and consumer network equipment (think: your home router) have been resolved. In response – and in an effort to kick IPv6 deployment in the butt – the same businesses which ran the 24-hour field test last year have committed to turning on IPv6 for their content and keeping it on as of 6/6/2012.

But that's not all, folks!

In the past, IPv6 availability would have simply impacted customers connecting to the Internet from a few universities, international providers and smaller technology-forward ISPs. What's great about this event is that a significant number of major broadband ISPs (think: your home and business Internet connection) have committed to enabling IPv6 to their subscribers. June 6, 2012, marks a day where at least 1% of the participating ISPs' downstream customers will be receiving IPv6 addresses.

While 1% may not seem all that impressive at first, in order to survive the change, these ISPs must slowly roll out IPv6 availability to ensure that they can handle the potential volume of resulting customer support issues. There will be new training and technical challenges that I suspect all of these ISPs will face, and this type of approach is a good way to ensure success. Again, we must appreciate that the ISPs are turning it on for good now.

What does this mean for SoftLayer customers? Well the good news is that our network is already IPv6-enabled ... In fact, it has been so for a few years now. Those of you who have taken advantage of running a dual-stack of IPv4 and IPv6 addresses may have noticed surprisingly low IPv6 traffic volume. When 6/6/2012 comes around, you should see that volume rise (and continue to rise consistently from there). For those of you without IPv6 addresses, now's the time to get started and get your feet wet. You need to be prepared for the day when new "eyeballs" are coming online with IPv6-only addresses. If you don't know where to start, go back through this article and click on a few of the hyperlinks, and if you want more information, ARIN has a great informational IPv6 wiki that has been enjoying community input for a couple years now.

The long term benefit of this June 6th milestone is that with some of the "big guys" playing in this space, the visibility of IPv6 should improve. This will help motivate the "little guys" who otherwise couldn't get motivated – or more often couldn't justify the budgetary requirements – to start implementing IPv6 throughout their organizations. The Internet is growing rapidly, and as our collective attentions are focused on how current legislation (SOPA/PIPA) could impede that growth, we should be intentional about fortifying the Internet's underlying architecture.

-Dani

May 26, 2011

IPv4 Justification Changes in IMS

When IANA ran out of IPv4 blocks in February, the belts tightened yet another notch in the IPv4 world. Regional registries such as ARIN have changed how they are allocating additional IPv4 blocks to ISPs, including performing more stringent reviews to ensure efficient utilization of current allocations and reducing future allocation sizes to more fairly distribute the remaining addresses. In turn, ISPs such as SoftLayer have to more closely monitor how we assign our IPv4 space to ensure efficient utilization by our customers. In order to maximize this limited resource, we have to make changes to the way customers order and receive additional IPv4 IPs.

Being down the line from ARIN, SoftLayer's networking department has to ensure we abide by all updated ARIN policies when we issue IPv4 addresses from our available pool in order to remain eligible to receive additional resource allocations. In addition to policy compliance, we are also focused on IPv4 conservation methods such as those referenced in RFC2050. Accordingly, we've made improvements to the way that we handle IPv4 requests to better streamline the process of collecting and reviewing usage and justification details.

Every SoftLayer server comes with one public IPv4 address, and until recently was assigned 4 additional IPs (/30) statically routed to that server. Our first step of IP conservation took place earlier this year, when we ceased the practice of automatically assigning the 4 additional IPs, and only issued them by customer request.

As we move forward, were trying to be as transparent with our customers about the IPv4 justification process as we can, so we're letting you know that additional justification requirements have been imposed on all ISPs by ARIN, and the best way we can meet those requirements is to have our customers follow the same guidelines. Being SoftLayer, we're doing what we can to automate and streamline the IPv4 justification process where possible, and are therefore implementing changes in the ordering system. Beginning on June 1, you will now be presented with a brief questionnaire whenever requesting additional IPv4 addresses. We must collect the requester's contact details, number of IPs that are expected to be used immediately and within the next year, as well as a brief description of how the IPs will be used. What happens next depends on the information you have provided as well as the current IP usage on your account: either the request is automatically approved and fulfilled, or a ticket is spawned for additional manual review by one of the folks in our networking team.

When a ticket is spawned for our network folks, we will get back to you within one business day, and let you know whether we need additional details from you in order to consider the request. These manual reviews will take additional time, depending on the size and complexity of request, as well as the quality of information provided. Please be prepared that there will be some cases where we will not be able to approve a request, such as when name-based virtualization can be used for IP conservation, or when services such as SEO, email campaigns, or VPN termination are the intended use for the additional IPv4 addresses.

If you're familiar with SoftLayer's previous method of ordering additional IPv4 addresses, you might feel like we're making you jump through hoops. These "hoops" are not intended to make the process arbitrarily more difficult. Rather, they're being put in place specifically to make sure we're in the best position possible to meet customer demands with our current IPv4 allocations, and get additional allocations from ARIN before they are completely depleted.

The ISPs that have no problems giving away IPv4 addresses right now without more stringent review are also going to be the providers that have the hardest time getting additional IPv4 blocks from ARIN, and therefore will have a hard time fulfilling future customer needs. At SoftLayer we are going by the philosophy that the more cautiously we approach IPv4 depletion by justifying our customers' needs for each IPv4 address, the longer our IPv4 pool will last to meet those needs.

This is a good opportunity to mention IPv6 as the long-term solution for IPv4 address depletion. The sooner that the public Internet becomes fully IPv6-enabled, the better off the resource constraint will become. The SoftLayer network is enabled for IPv6, and you may place an order for 18.4 quintillion free IPv6 addresses for your server via our web portal today!

-Dani

Subscribe to Author Archive: %