Infrastructure Posts

October 28, 2015

Ongoing Actions to Eliminate Spam Hosting

We are announcing a new policy, effective today, as part of our regular efforts to reduce the ability for spam to be sent from the SoftLayer network.

Starting October 28, 2015 bare metal servers and virtual servers provisioned on new accounts will not have the ability to send email directly via outbound connections through TCP port 25 (SMTP). Port 25 can be used as a conduit for distributing unsolicited bulk email.

In a follow-up phase, we will roll out this network policy change to customers who established accounts before October 28. (A separate communications will be sent with timeline and implementation guidance to those customers.)

You can read the technical details on KnowledgeLayer.

SendGrid Services Available to Send and Track Emails

We have partnered with SendGrid™ since 2011 to provide email delivery services. We have arranged for SendGrid to provide SoftLayer customers with an account allowing sending of up to 25,000 emails per month at no charge, which can be activated via the SoftLayer customer portal.

SendGrid allows you to use a SmartHost to relay your outbound mail services while generating metrics, including tracking lists and bounce rates, open rates, and click-through rates. It also assists with newsletters and provides authentication. All of these services are designed to provide stronger email analytics for you to optimize your communications and eNurture programs. Full details on our SendGrid service, including free options, can be found here.

Use Your Email Service Through a Custom Email Port

You are welcome to use your own email service on a custom port following the API or SMTP guidelines provided by your mail provider to configure your servers to an email port other than TCP port 25. This is common practice for most mail providers and should not be an inhibitor to you sending and measuring your communications.

Need an Exception?

If you are a new client and need the ability to send outbound SMTP email via TCP port 25, please open a support ticket in the customer portal, and provide details about why you require an exception to this policy. Be sure to explain why the SendGrid email relaying solution does not fit your system or application needs. Our team is specialized to assist with most email relaying and blacklisting issues for recognized and reputable real-time blackhole lists (RBLs) and can evaluate your situation.

Dedicated to Your Success

We continuously work with established monitoring authorities and groups to eliminate fraudulent spammers and to block the usage of port 25 for email communications.

As we all know, spam is unsolicited bulk email. Our network architecture isolates devices so customers cannot see or share traffic across accounts. We follow ISO 27001. And for federal accounts, we are aligned to NIST 800-53 framework and maintain SOC 2 Type II reporting compliance for all data centers. We integrate three distinct network topologies for each physical or virtual server and offer security solutions for systems, applications, and data as well.

Thank you again to your commitment to SoftLayer as we continue to work hard to ensure a secure environment for you.


August 12, 2015

Network Performance 101: What is latency, and why does it matter?

We’ve all been there. Waiting for a web page to load can be so frustrating that we end up just closing out. You might ask yourself, “Hey, I have high-speed Internet. Why is this happening to me?” Well, there are a lot of factors outside your control that … control page loads. And whether you have an online store, run big data solutions, or have your employees set up on a network accessing files around the world, you never want to hear that your data, consumer products, information, or otherwise, is keeping you from a sale or slowing down employee productivity because of slow data transfer.

So why are some pages so much slower to load than others?
It could be that poorly written code or large images are slowing the load on the backend, but slow page loads can also be caused by network latency. This might sound elementary, but data is not just floating out there in some non-physical Internet space. In reality, data is stored on hard drives … somewhere. Network connectivity provides a path for that data to travel to end users around the world, and that connectivity can vary significantly—depending on how far it’s going, how many times the data has to hop between service providers, how much bandwidth is available along the way, the other data traveling across the same path, and a number of other variables.

The measurement of how quickly data travels between two connected points is called network latency. Network latency is an expression of the amount of time it takes a packet of data to get from one place to another.

Understanding Network Latency
Theoretically, data can travel at the speed of light across optical fiber network cables, but in practice, data typically travels slower than light due to the variables we referenced in the previous section. If a network connection doesn’t have any available bandwidth capacity, data might temporarily queue up to wait for its turn to travel across the line. If a service provider’s network doesn’t route a network path optimally, data could be sent hundreds or thousands of miles away from the destination in the process of routing to the destination. These kinds of delays and detours lead to higher network latency, which lead to slower page loads and download speeds.

We express network latency in milliseconds (that’s 1,000 milliseconds per second), and while a few thousandths of a second may not mean much to us as we’re living our daily lives, those milliseconds are often the deciding factors for whether we stay on a webpage or give up and try another site. As consumers of high-speed Internet, we like what we like, and we want what we want when we want it. In the financial sector, milliseconds can mean billions of dollars in gains or losses from trade transactions on a day-to-day basis.

Logical conclusion: Everyone wants the lowest network latency to the greatest number of users.

Common Approaches to Minimize Network Latency
If our shared goal is to minimize latency for our data, the most common approaches to addressing network latency involve limiting the number of potential variables that can impact the speed of data’s movement. While we don’t have complete control over how our data travels across the Internet, we can do a few things to keep our network latency in line:

  • Distribute data around the world: Users in different locations can pull data from a location that’s geographically close to them. Because the data is closer to the users, it is handed off fewer times, it has a shorter distance to travel, and inefficient routing is less likely to cause a significant performance impact.
  • Provision servers with high-capacity network ports: Huge volumes of data can travel to and from the server every second. If packets are delayed due to fully saturated ports, milliseconds of time pass, pages load slower, download speeds drop, and users get unhappy.
  • Understand how your providers route traffic: When you know how your data is transferred to users around the world, you can make better decisions about where you host your data.

How SoftLayer Minimizes Network Latency
To minimize latency, we took a unique approach to building our network. All of our data centers are connected to network points of presence. All of our network points of presence are connected to each other via our global backbone network. And by maintaining our own global backbone network, our network operations team is able to control network paths and data handoffs much more granularly than if we relied on other providers to move data between geographies.

SoftLayer Private Network

For example, if a user in Berlin wants to watch a cat video hosted on a SoftLayer server in Dallas, the packets of data that make up that cat video will travel across our backbone network (which is exclusively used by SoftLayer traffic) to Frankfurt, where the packets would be handed off to one of our peering or transit public network partners to get to the user in Berlin.

Without a global backbone network, the packets would be handed off to a peering or transit public network provider in Dallas, and that provider would route the packets across its network and/or hand the packets off to another provider at a network hop, and the packets would bounce their way to Germany. It’s entirely possible that the packets could get from Dallas to Berlin with the same network latency with or without the global backbone network, but without the global backbone network, there are a lot more variables.

In addition to building a global backbone network, we also segment public, private, and management traffic onto different network ports so that different types of traffic can be transferred without interfering with each other.

SoftLayer Private Network

But at the end of the day, all of that network planning and forethought doesn’t amount to a hill of beans if you can’t see the results for yourself. That’s why we put speed tests on our website so you can check out our network yourself (for more on speed tests, check out this blog post).

TL;DR: Network Latency
Your users want your data as quickly as you can get it to them. The time it takes for your data to get to them across the Internet is called network latency. The more control you (or your provider) have over your data’s network path, the more consistent (and lower) your network latency will be.

Stay tuned. Next month we will be discussing Network Performance 101: Security, where we’ll discuss all things cloud security—including answering your burning questions: Can other people see or access my data in a public cloud? Is my data more prone to hackers? And, what safeguards do SoftLayer have in place to protect data?


June 29, 2015

Opening Up the Cloud

This guest blog post is written by Alexia Emmanoulopoulou, marketing manager at Canonical.

With OpenStack, cloud computing becomes easily accessible to everyone. It tears down financial barriers to cloud deployments and tackles the fear of lock-in. One of the main benefits of OpenStack is the fact that it is open source and supported by a wide ecosystem, with contributions from more than 200 companies, including Canonical and IBM. Users can change service providers and hardware at any time, and compared to other clouds using virtualization technology, OpenStack can double server utilization to as much as 85 percent. This means that an OpenStack cloud is economical and delivers more flexibility, scalability, and agility to businesses. The challenge however lies in recruiting and retaining OpenStack experts, who are in high demand, making it hard for companies to deploy OpenStack on time and on budget. But BootStack, Canonical’s managed cloud product solved that problem by offering all the benefits of a private cloud without any of the pain of day-to-day infrastructure management.

Addressing the Challenge of Finding OpenStack Experts

Resourcing an OpenStack six-strong team to work 24x7 would cost between $900,000 and $1.5 million and can take months of headhunting. Thus the savings that OpenStack should bring companies are eroded so Canonical created BootStack, short for Build, Operate, and Optionally Transfer. It’s a new service for setting up and operating an OpenStack cloud, in both on-premises and hosted environments, and it gives users the option of taking over the management of your cloud in the future.

After working with each customer to define their requirements and specify the right cloud infrastructure for their business, Canonical’s experienced engineering and support team builds and manages the entire cloud infrastructure of the customer, including Ubuntu OpenStack, the underlying hypervisor, and deployment onto hosted or on-premises hardware. As a result, users get all the benefits of a private cloud without any of the pain of day-to-day infrastructure management. For added protection, BootStack is backed by a clear SLA that covers cloud availability at the user’s desired scale as well as uptime and responsiveness metrics.

Choosing Between On-premises and Hosted Cloud

Some companies prefer to host on-premises because they feel more secure knowing their cloud is running on their own site. However, when things go wrong, some companies find they don’t have the expertise on-hand to quickly recover. Furthermore, on-site hosting is at least three times as expensive as it is to outsource to a hosting specialist.

With the hosted option for BootStack, your OpenStack cloud will be hosted on Ubuntu-certified hardware in SoftLayer data centers. SoftLayer provides customizable bare metal and virtual servers run on the highest performing cloud infrastructure available. Users can seamlessly move data between servers at no cost and benefit from secure, fast, and low-latency communications between data centers. 24x7 expert staff in each data center can troubleshoot any rare issues that can’t be directly resolved through their self-service management portal. Canonical and SoftLayer also take care of patches and upgrades to both the operating system and OpenStack, hardware and software failure prevention and fix, proactive health monitoring of the cloud and hardware, and resolution of any other problems.

No Lock-In and Predictable Cost

The two features that set BootStack apart from other managed cloud products are the predictable cost structure and the lack of lock-in. With BootStack, users can access every tool and every machine, any time. A company can choose to take over the management of its cloud at any time, at which point it will receive training and support from Canonical to ensure a smooth transition. BootStack customers can then choose to either bring their cloud in-house or continue hosting with SoftLayer.

In terms of costs, BootStack cloud is priced at $15 per day per server, plus the cost of the hosting. SoftLayer offers a number of bare metal servers that exceed the OpenStack recommended configuration, starting at $699 per month. You pay as you go, and can scale as your business needs change.

All-in-all, it’s a flexible managed cloud at a predictable cost with expert staff to manage it until you’re ready to take over!

For more information about BootStack, SoftLayer, and OpenStack, download our free white paper: The Easiest Way to Build and Manage an OpenStack Cloud.


May 14, 2015

Update - VENOM Vulnerability

Yesterday, a security advisory designated CVE-2015-3456 / XSA-133 was publicly announced. The advisory identified a vulnerability, which has become commonly known as "VENOM", through which an attacker could exploit floppy driver support in QEMU to escalate their privileges.

SoftLayer engineers, in concert with our technology partners, completed a deep analysis of the vulnerability and determined that SoftLayer virtual servers are not affected by this issue.

We're always committed to ensuring our customers' operations and data are well protected. If customers have any questions or concerns, don't hesitate to reach out to SoftLayer support or your direct SoftLayer contacts.


March 30, 2015

The Importance of Data's Physical Location in the Cloud

If top-tier cloud providers use similar network hardware in their data centers and connect to the same transit and peering bandwidth providers, how can SoftLayer claim to provide the best network performance in the cloud computing industry?

Over the years, I've heard variations of that question asked dozens of times, and it's fairly easy to answer with impressive facts and figures. All SoftLayer data centers and network points of presence (PoPs) are connected to our unique global network backbone, which carries public, private, and management traffic to and from servers. Using our network connectivity table, some back-of-the-envelope calculations reveal that we have more than 2,500Gbps of bandwidth connectivity with some of the largest transit and peering bandwidth providers in the world (and that total doesn't even include the private peering relationships we have with other providers in various regional markets). Additionally, customers may order servers with up to 10Gbps network ports in our data centers.

For the most part, those stats explain our differentiation, but part of the bigger network performance story is still missing, and to a certain extent it has been untold—until today.

The 2,500+Gbps of bandwidth connectivity we break out in the network connectivity table only accounts for the on-ramps and off-ramps of our network. Our global network backbone is actually made up of an additional 2,600+Gbps of bandwidth connectivity ... and all of that backbone connectivity transports SoftLayer-related traffic.

This robust network architecture streamlines the access to and delivery of data on SoftLayer servers. When you access a SoftLayer server, the network is designed to bring you onto our global backbone as quickly as possible at one of our network PoPs, and when you're on our global backbone, you'll experience fewer hops (and a more direct route that we control). When one of your users requests data from your SoftLayer server, that data travels across the global backbone to the nearest network PoP, where it is handed off to another provider to carry the data the "last mile."

With this controlled environment, I decided to undertake an impromptu science experiment to demonstrate how location and physical distance affect network performance in the cloud.

Speed Testing on the SoftLayer Global Network Backbone

I work in the SoftLayer office in downtown Houston, Texas. In network-speak, this location is HOU04. You won't find that location on any data center or network tables because it's just an office, but it's connected to the same global backbone as our data centers and network points of presence. From my office, the "last mile" doesn't exist; when I access a SoftLayer server, my bits and bytes only travel across the SoftLayer network, so we're effectively cutting out a number of uncontrollable variables in the process of running network speed tests.

For better or worse, I didn't tell any network engineers that I planned to run speed tests to every available data center and share the results I found, so you're seeing exactly what I saw with no tomfoolery. I just fired up my browser, headed to our Data Centers page, and made my way down the list using the SpeedTest option for each facility. Customers often go through this process when trying to determine the latency, speeds, and network path that they can expect from servers in each data center, but if we look at the results collectively, we can learn a lot more about network performance in general.

With the results, we'll discuss how network speed tests work, what the results mean, and why some might be surprising. If you're feeling scientific and want to run the tests yourself, you're more than welcome to do so.

The Ookla SpeedTests we link to from the data centers table measured the latency (ping time), jitter (variation in latency), download speeds, and upload speeds between the user's computer and the data center's test server. To run this experiment, I connected my MacBook Pro via Ethernet to a 100Mbps wired connection. At the end of each speed test, I took a screenshot of the performance stats:

SoftLayer Network Speed Test

To save you the trouble of trying to read all of the stats on each data center as they cycle through that animated GIF, I also put them into a table (click the data center name to see its results screenshot in a new window):

Data Center Latency (ms) Download Speed (Mbps) Upload Speed (Mbps) Jitter (ms)
AMS01 121 77.69 82.18 1
DAL01 9 93.16 87.43 0
DAL05 7 93.16 83.77 0
DAL06 7 93.11 83.50 0
DAL07 8 93.08 83.60 0
DAL09 11 93.05 82.54 0
FRA02 128 78.11 85.08 0
HKG02 184 50.75 78.93 2
HOU02 2 93.12 83.45 1
LON02 114 77.41 83.74 2
MEL01 186 63.40 78.73 1
MEX01 27 92.32 83.29 1
MON01 52 89.65 85.94 3
PAR01 127 82.40 83.38 0
SJC01 44 90.43 83.60 1
SEA01 50 90.33 83.23 2
SNG01 195 40.35 72.35 1
SYD01 196 61.04 75.82 4
TOK02 135 75.63 82.20 2
TOR01 40 90.37 82.90 1
WDC01 43 89.68 84.35 0

By performing these speed tests on the SoftLayer network, we can actually learn a lot about how speed tests work and how physical location affects network performance. But before we get into that, let's take note of a few interesting results from the table above:

  • The lowest latency from my office is to the HOU02 (Houston, Texas) data center. That data center is about 14.2 miles away as the crow flies.
  • The highest latency results from my office are to the SYD01 (Sydney, Australia) and SNG01 (Singapore) data centers. Those data centers are at least 8,600 and 10,000 miles away, respectively.
  • The fastest download speed observed is 93.16Mbps, and that number was seen from two data centers: DAL01 and DAL05.
  • The slowest download speed observed is 40.35Mbps from SNG01.
  • The fastest upload speed observed is 87.43Mbps to DAL01.
  • The slowest upload speed observed is 72.35Mbps to SNG01.
  • The upload speeds observed are faster than the download speeds from every data center outside of North America.

Are you surprised that we didn't see any results closer to 100Mbps? Is our server in Singapore underperforming? Are servers outside of North America more selfish to receive data and stingy to give it back?

Those are great questions, and they actually jumpstart an explanation of how the network tests work and what they're telling us.

Maximum Download Speed on 100Mbps Connection

If my office is 2 milliseconds from the test server in HOU02, why is my download speed only 93.12Mbps? To answer this question, we need to understand that to perform these tests, a connection is made using Transmission Control Protocol (TCP) to move the data, and TCP does a lot of work in the background. The download is broken into a number of tiny chunks called packets and sent from the sender to the receiver. TCP wants to ensure that each packet that is sent is received, so the receiver sends an acknowledgement back to the sender to confirm that the packet arrived. If the sender is unable to verify that a given packet was successfully delivered to the receiver, the sender will resend the packet.

This system is pretty simple, but in actuality, it's very dynamic. TCP wants to be as efficient as possible ... to send the fewest number of packets to get the entire message across. To accomplish this, TCP is able to modify the size of each packet to optimize it for each communication. The receiver dictates how large the packet should be by providing a receive window to accommodate a small packet size, and it analyzes and adjusts the receive window to get the largest packets possible without becoming unstable. Some operating systems are better than others when it comes to tweaking and optimizing TCP transfer rates, but the processes TCP takes to ensure that the packets are sent and received without error takes overhead, and that overhead limits the maximum speed we can achieve.

Understanding the SNG01 Results

Why did my SNG01 speed test max out at a meager 40.35Mbps on my 100Mbps connection? Well, now that we understand how TCP is working behind the scenes, we can see why our download speeds from Singapore are lower than we'd expect. Latency between the sending and successful receipt of a packet plays into TCP’s considerations of a stable connection. Higher ping times will cause TCP to send smaller packet sizes than it would for lower ping times to ensure that no sizable packet is lost (which would have to be reproduced and resent).

With our global backbone optimizing the network path of the packets between Houston and Singapore, the more than 10,000-mile journey, the nature of TCP, and my computer's TCP receive window adjustments all factor into the download speeds recorded from SNG01. Looking at the results in the context of the distance the data has to travel, our results are actually well within the expected performance.

Because the default behavior of TCP is partially to blame for the results, we could actually tweak the test and tune our configurations to deliver faster speeds. To confirm that improvements can be made relatively easily, we can actually just look at the answer to our third question...

Upload > Download?

Why are the upload speeds faster than the download speeds after latency jumps from 50ms to 114ms? Every location in North America is within 2,000 miles of Houston, while the closest location outside of North America is about 5,000 miles away. With what we've learned about how TCP and physical distance play into download speeds, that jump in distance explains why the download speeds drop from 90.33Mbps to 77.41Mbps as soon as we cross an ocean, but how can the upload speeds to Europe (and even APAC) stay on par with their North American counterparts? The only difference between our download path and upload path is which side is sending and which side is receiving. And if the receiver determines the size of the TCP receive window, the most likely culprit in the discrepancy between download and upload speeds is TCP windowing.

A Linux server is built and optimized to be a server, whereas my MacOSX laptop has a lot of other responsibilities, so it shouldn't come as a surprise that the default TCP receive window handling is better on the server side. With changes to the way my laptop handles TCP, download speeds would likely be improved significantly. Additionally, if we wanted to push the envelope even further, we might consider using a different transfer protocol to take advantage of the consistent, controlled network environment.

The Importance of Physical Location in Cloud Computing

These real-world test results under controlled conditions demonstrate the significance of data's geographic proximity to its user on the user's perceived network performance. We know that the network latency in a 14-mile trip will be lower than the latency in a 10,000-mile trip, but we often don't think about the ripple effect latency has on other network performance indicators. And this experiment actually controls a lot of other variables that can exacerbate the performance impact of geographic distance. The tests were run on a 100Mbps connection because that's a pretty common maximum port speed, but if we ran the same tests on a GigE line, the difference would be even more dramatic. Proof: HOU02 @ 1Gbps v. SNG01 @ 1Gbps

Let's apply our experiment to a real-world example: Half of our site's user base is in Paris and the other half is in Singapore. If we chose to host our cloud infrastructure exclusively from Paris, our users would see dramatically different results. Users in Paris would have sub-10ms latency while users in Singapore have about 300ms of latency. Obviously, operating cloud servers in both markets would be the best way to ensure peak performance in both locations, but what if you can only afford to provision your cloud infrastructure in one location? Where would you choose to provision that infrastructure to provide a consistent user experience for your audience in both markets?

Given what we've learned, we should probably choose a location with roughly the same latency to both markets. We can use the SoftLayer Looking Glass to see that San Jose, California (SJC01) would be a logical midpoint ... At this second, the latency between SJC and PAR on the SoftLayer backbone is 149ms, and the latency between SJC and SNG is 162ms, so both would experience very similar performance (all else being equal). Our users in the two markets won't experience mind-blowing speeds, but neither will experience mind-numbing speeds either.

The network performance implications of physical distance apply to all cloud providers, but because of the SoftLayer global network backbone, we're able to control many of the variables that lead to higher (or inconsistent) latency to and from a given data center. The longer a single provider can route traffic, the more efficiently that traffic will move. You might see the same latency speeds to another provider's cloud infrastructure from a given location at a given time across the public Internet, but you certainly won't see the same consistency from all locations at all times. SoftLayer has spent millions of dollars to build, maintain, and grow our global network backbone to transport public and private network traffic, and as a result, we feel pretty good about claiming to provide the best network performance in cloud computing.


March 12, 2015

Sydney’s a Go

Transforming an empty room into a fully operational data center in just three months: Some said it couldn’t be done, but we did it. In less than three months, actually.

Placing a small team on-site and turning an empty room into a data center is what SoftLayer refers to as a Go Live. Now, of course there is more to bringing a data center online than the just the transformation of an empty room. In the months leading up to the Go Live deployment, there are details to work out, contracts to sign, and the electrical fit out (EFO) of the room itself. During my time with SoftLayer I have been involved in building several of our data centers, or SoftLayer pods as we call them. Pods are designed to facilitate infrastructure scalability, and although they have evolved over the years as newer, faster equipment has become available, the original principles behind the design are still intact—so much so that a data center technician could travel to any SoftLayer data center in the world and start working without missing a beat. And the same holds true to building a pod from the ground up. This uniformity is what allows us to fast track the build out of a new SoftLayer pod. This is one of the reasons why the Sydney data center launch was such a success.

Rewind Three Months

When we landed in Sydney on December 11, 2014, we had an empty server room and about 125 pallets of gear and equipment that had been carefully packed and shipped by our inventory and logistics team. First order of business: breaking down the pallets, inspecting the equipment for any signs of damage and checking that we received everything needed for the build. It’s really quite impressive to know that everything from screwdrivers to our 25U routers to even earplugs had been logged and accounted for. When you are more than 8,500 miles away from your base of operations, it’s imperative that the Go Live team has everything it needs on hand from the start. Something seemingly inconsequential as not having the proper screws can lead to costly delays during the build. Once everything’s been checked off, the real fun begins.

(From Left) Jackie Vong, Dennis Vollmer, Jon Bowden, Chris Stelly, Antonio Gomez, Harpal Singh, Kneeling - Zachary Schacht, Peter Panagopoulos, and Marcelo Alba

Next we set up the internal equipment that powers the pod: four rows of equipment that encompass everything from networking gear to storage to the servers that run various internal systems. Racking the internal equipment is done according to pre-planned layouts and involves far too many cage nuts, the bane of every server build technician’s existence.

Once the internal rows are completed, it’s time to start focusing on the customer rows that will contain bare metal and virtual servers. Each customer rack contains a minimum of five switches—two for the private network, two for the public network, and one out-of-band management switch. Each row has two power strips and in the case of the Sydney data center, two electrical transfer switches at the bottom of the rack that provide true power redundancy by facilitating the transfer of power from one independent feed to another in the case of an outage. Network cables from the customer racks route back to the aggregate switch rack located at the center of each row.

Right around the time we start to wrap up the internal and customer rows, a team of network engineers arrive on-site to run the interconnects between the networking gear and the rest of the internal systems and to light up the fiber lines connecting our new pod to our internal network (as well as the rest of the world). This is a big day because not only do we finally get Wi-Fi up in the pod, but no longer are we isolated on an island. We are connected, and teams thousands of miles away can begin the process of remotely logging in to configure, deploy, and test systems. The networking team will start work on configuring the switches, load balancers, and firewalls for their specific purposes. The storage team will begin the process of bringing massive storage arrays online, and information systems will start work on deploying the systems that manage the automation each pod provides.

(From Left) Zach Robbins, Grayson Schmidt, Igor Gorbatok and Alex Abin

During this time, we start the process of onboarding the newest members of the team, the local Sydney techs, who in a few short months will be responsible for managing the data center independently. But before they fully take over, customer racks are prepped and are waiting to house the final piece of the puzzle: the servers. They arrive via truck day [check out DAL05 Pod 2 truck day]; Sydney’s was around the beginning of February. Given the amount of hardware we typically receive, truck days are an event unto themselves—more than 1,500 of the newest and fastest SuperMicro servers of various shapes and sizes that will serve as the bare metal and virtual servers for our customers. Through a combination of manpower and automation, these servers get unboxed, racked, checked in, and tested before they are sold to our customers.

Now departments involved in bringing the Sydney data center online wrap up and sign off. Then we go live.

Bringing a SoftLayer pod online and on time is a beautifully choreographed process and is one of my greatest professional accomplishments. The level of coordination and cohesion required to pull it off, not once, not twice but ten times all over the world in the last year alone can’t be overstated enough.


December 17, 2014

Does physical location matter “in the cloud”?

By now everyone understands that the cloud is indeed a place on Earth, but there still seems to be confusion around why global expansion by way of adding data centers is such a big deal. After all, if data is stored “in the cloud,” why wouldn’t adding more servers in our existing data centers suffice? Well, there’s a much more significant reason for adding more data centers than just being able to host more data.

As we’ve explained in previous blog posts, Globalization and Hosting: The World Wide Web is Flat and Global Network: The Proof is in the Traceroute, our strategic objective is to get a network point of presence (PoP) within 40ms of all our users (and our users' users) in order to provide the best network stability and performance possible anywhere on the planet.

Data can travel across the Internet quickly, but just like anything, the farther something has to go, the longer it will take to get there. Seems pretty logical right? But we also need to take into account that not all routes are created equally. So to deliver the best network performance, we designed our global network to get data to the closest route possible to our network. Think of each SoftLayer PoP as an on-ramp to our global network backbone. The sooner a user is able to get onto our network, the quicker we can efficiently route them through our PoPs to a server in one of our data centers. Furthermore, once plugged into the network, we are able to control the flow of traffic.

Let’s take a look at this traceroute example from the abovementioned blog post. As you are probably aware, a traceroute shows the "hops" or routers along the network path from an origin IP to a destination IP. When we were building out the Singapore data center (before the network points of presence were turned up in Asia), the author ran a traceroute from Singapore to, and immediately after the launch of the data center, ran another one.

Pre-Launch Traceroute to from Singapore

traceroute to (, 64 hops max, 52 byte packets
 1 (  1.884 ms  1.089 ms  1.569 ms
 2 (  2.006 ms  1.669 ms  1.753 ms
 3 (  3.380 ms  3.388 ms  4.344 ms
 4 (  3.684 ms  3.348 ms  3.919 ms
 5 (  9.002 ms  3.516 ms  4.228 ms
 6 (  3.716 ms  3.965 ms  5.663 ms
 7 (  4.442 ms  4.117 ms  4.967 ms
 8 (  6.807 ms  55.288 ms  56.211 ms
 9 (  187.953 ms  188.447 ms  187.809 ms
10 (  184.143 ms (  189.510 ms (  289.039 ms
11 (  187.645 ms  188.700 ms  187.912 ms
12 (  186.482 ms  188.265 ms  187.021 ms
13 (  188.569 ms  191.100 ms  188.736 ms
14 (  381.645 ms  410.052 ms  420.311 ms
15 (  415.379 ms  415.902 ms  418.339 ms
16 (  417.426 ms  417.301 ms (  416.692 ms
17  * * *

Post-Launch Traceroute to from Singapore

traceroute to (, 64 hops max, 52 byte packets
 1 (  2.850 ms  1.409 ms  1.206 ms
 2 (  1.550 ms  1.680 ms  1.394 ms
 3 (  1.812 ms  1.341 ms  1.734 ms
 4 (  35.550 ms  1.999 ms  2.124 ms
 5 (  174.726 ms  175.484 ms  175.491 ms
 6 (  203.821 ms  203.749 ms  205.803 ms
 7 (  306.755 ms (  208.669 ms  203.127 ms
 8 (  203.518 ms (  305.534 ms (  204.150 ms
 9  * * *

After the Singapore data center launch, the number of hops was reduced by 50 percent, and the response time (in milliseconds) was reduced by 40 percent. Those are pretty impressive numbers from just lighting up a couple PoPs and a data center, and that was just the beginning of our global expansion in 2012.

That’s why we are so excited to announce the three new data centers launching this month: Mexico City, Tokyo, and Frankfurt.

Of course, this is great news for customers who require data residency in Mexico, Japan, and Germany. And yes, these new locations provide additional in-region redundancy within APAC, EMEA, and the Americas. But even customers without servers in these new facilities have reason to celebrate: Our global network backbone is expanding, so users in these markets will see even better network stability and speed to servers in every other SoftLayer data center around the world!


October 8, 2014

An Insider’s Look at Our Data Centers

I’ve been with Softlayer over four years now. It’s been a journey that has taken me around the world—from Dallas to Singapore to Washington D.C, and back again. Along the way, I’ve met amazingly brilliant people who have helped me sharpen the tools in my ‘data center toolbox’ thus allowing me to enhance the customer experience by aiding and assisting in a complex compute environment.

I like to think of our data centers as masterpieces of elegant design. We currently have 14 of these works of art, with many more on the way. Here’s an insider’s look at the design:

Keeping It Cool
Our POD layouts have a raised floor system. The air conditioning units chill from the front bottom of the servers on the ‘cold rows’ passing through the servers on the ‘warm rows.’ The warm rows have ceiling vents to rapidly clear the warm air from the backs of the servers.

Jackets are recommended for this arctic environment.

Pumping up the POWER
Nothing is as important to us as keeping the lights on. Every data center has a three-tiered approach to keeping your servers and services on. Our first tier being street power. Each rack has two power strips to distribute the load and offer true redundancy for redundant servers and switches with the remote ability to power down an individual port on either power strip.

The second tier is our batter backup for each POD. This offers emergency response for seamless failover when street power is no more.

This leads to the third step in our model, generators. We have generators in place for a sustainable continuity of power until street power has returned. Check out the 2-megawatt diesel generator installation at the DAL05 data center here.

The Ultimate Social Network
Neither power nor cooling matter if you can’t connect to your server, which is where our proprietary networking topography comes to play. Each bare metal server and each virtual server resides in a rack that connects to three switches. Each of those switches connects to an aggregate switch for a row. The aggregate switch connects to a router.

The first switch, our private backend network, allows for SSL and VPN connectivity to manage your server. It also gives you the ability to have server-to-server communication without the bounds of bandwidth overages.

The second switch, our public network, provides pubic Internet access to your device, which is perfect for shopping, gaming, coding, or whatever you want to use it for. With 20TB of bandwidth coming standard for this network, the possibilities are endless.

The third and final switch, management, allows you to connect to the Intelligent Platform Management Interface that provides tools such as KVM/hardware monitoring/and even virtual CDs to install an image of your choosing! The cables to your devices from the switches are color-coded, port-number-to-rack-unit labeled, and masterfully arranged to maximize identification and airflow.

A Soft Place for Hardware
The heart and soul of our business is the computing hardware. We use enterprise grade hardware from the ground up. We offer our smallest offering of 1 core, 1GB RAM, 25GB HDD virtual servers, to one of our largest quad 10-core, 512GB RAM, multi 4TB HDD bare metal servers. With excellent hardware comes excellent options. There is almost always a path to improvement. Meaning, unless you already have the top of the line, you can always add more. Whether it be additional drive, RAM, or even processor.

I hope you enjoyed the view from the inside. If you want to see the data centers up close and personal, I am sorry to say, those are closed to the public. But you can take a virtual tour of some of our data centers via YouTube: AMS01 and DAL05

-Joshua Fox

October 1, 2014

Virtual Server Update

Good morning, afternoon, evening, or night, SoftLayer nation.

We want to give you an update and some more information on maintenance taking place right now with SoftLayer public and private node virtual servers.

As the world is becoming aware today, over the past week a security risk associated with Xen was identified by the Xen community and published as Xen Security Advisory 108 (XSA-108).

And as many are aware, Xen plays a role in our delivery of SoftLayer virtual servers.

Eliminating the vulnerability requires updating software on host nodes, and that requires downtime for the virtual servers running on those nodes.

Yeah, that’s not something anyone likes to hear. But customer security is of the utmost importance to us, so not doing it was not an option.

As soon as the risk was identified, our systems engineers and technology partners have been working nonstop to prepare the update.

On Sunday we notified every customer account that would be affected that we would have emergency maintenance in the middle of this week, and updated that notice each day.

And then yesterday we published that the maintenance would begin today at 3pm UTC, with a preliminary order of how the maintenance would roll out across all of our data centers.

We are updating host nodes data center by data center to complete the emergency maintenance as quickly as possible. This approach will minimize disruption for customers with failover infrastructure in multiple data centers.

The maintenance is under way and SoftLayer customers can follow it, live, on our forum at


August 26, 2014

Bare Metal Power. By the Hour.

Think quickly. You hear that your new app will be featured on the front page of TechCrunch in less than two hours. Because it’s a resource-intensive application you know that a flood of new users will bog down its current cloud infrastructure and you’ll need to scale out.

What do you do? Choose virtual servers to guarantee quick deployment and more flexibility? Opt for bare metal servers to deliver the best user experience (while crossing your fingers that the servers are online in time for the flood of traffic)? In times like these, you shouldn’t have to choose between flexibility and power.

You need hourly bare metal servers.

We’ve streamlined the deployment of four of our most popular bare metal configurations, and with that speed, we’re able to offer them with hourly billing! With the hardware pre-configured, you tell us where you want the server to be provisioned—Dallas, San Jose, Washington D.C., London, Toronto, Amsterdam, Singapore, and Hong Kong—and which operating system you’d like us to install— CentOS, Red Hat, FreeBSD, or Ubuntu. And in less than 30 minutes, your server will be online, fully integrated with your other SoftLayer servers and services, and ready for you.

Use the server for as long as you need it. Spin it down when you’re done. Pay for the hours you had it on your account. It’s that easy. No virtualization. No noisy neighbors. Just your computing-intensive workload, the hardware configuration you need, and a phobia-proof commitment.

Why you need hourly bare metal servers in your cloud life?

  • Processing Power: You have short-term workloads that require significant amounts of processing power. To get the same performance from virtual servers, you might have to provision twice as many nodes or run them for twice as long.
    • Example: a business intelligence ELT (Extract/Load/Transform) application.
  • Schedule-based Workloads: You have a number of applications that require compute and storage resources on a set schedule (i.e., once every month), and you don’t want to deploy (and pay for) high-end machines that will sit idle at all other times.
    • Example: payroll processing or claims payment processing.
  • Performance Testing: Certify or validate how an application performs on a specific hardware configuration.
    • Example: Software or mobile application companies can validate performance on specific hardware platforms.

With bare metal performance available on demand and on hourly terms, you don’t have to compromise performance for flexibility. When TechCrunch comes calling, you have peace of mind that your app’s success and popularity won’t bring it down.


Subscribe to infrastructure