Posts Tagged 'Product'

December 4, 2012

Big Data at SoftLayer: MongoDB

In one day, Facebook's databases ingest more than 500 terabytes of data, Twitter processes 500 million Tweets and Tumblr users publish more than 75 million posts. With such an unprecedented volume of information, developers face significant challenges when it comes to building an application's architecture and choosing its infrastructure. As a result, demand has exploded for "big data" solutions — resources that make it possible to process, store, analyze, search and deliver data from large, complex data sets. In light of that demand, SoftLayer has been working in strategic partnership with 10gen — the creators of MongoDB — to develop a high-performance, on-demand, big data solution. Today, we're excited to announce the launch of specialized MongoDB servers at SoftLayer.

If you've configured an infrastructure to accommodate big data, you know how much of a pain it can be: You choose your hardware, you configure it to run NoSQL, you install an open source NoSQL project that you think will meet your needs, and you keep tweaking your environment to optimize its performance. Assuming you have the resources (and patience) to get everything running efficiently, you'll wind up with the horizontally scalable database infrastructure you need to handle the volume of content you and your users create and consume. SoftLayer and 10gen are making that process a whole lot easier.

Our new MongoDB solutions take the time and guesswork out of configuring a big data environment. We give you an easy-to-use system for designing and ordering everything you need. You can start with a single server or roll out multiple servers in a single replica set across multiple data centers, and in under two hours, an optimized MongoDB environment is provisioned and ready to be used. I stress that it's an "optimized" environment because that's been our key focus. We collaborated with 10gen engineers on hardware and software configurations that provide the most robust performance for MongoDB, and we incorporated many of their MongoDB best practices. The resulting "engineered servers" are big data powerhouses:

MongoDB Configs

From each engineered server base configuration, you can customize your MongoDB server to meet your application's needs, and as you choose your upgrades from the base configuration, you'll see the thresholds at which you should consider upgrading other components. As your data set's size and the number of indexes in your database increase, you'll need additional RAM, CPU, and storage resources, but you won't need them in the same proportions — certain components become bottlenecks before others. Sure, you could upgrade all of the components in a given database server at the same rate, but if, say, you update everything when you only need to upgrade RAM, you'd be adding (and paying for) unnecessary CPU and storage capacity.

Using our new Solution Designer, it's very easy to graphically design a complex multi-site replica set. Once you finalize your locations and server configurations, you'll click "Order," and our automated provisioning system will kick into high gear. It deploys your server hardware, installs CentOS (with OS optimizations to provide MongoDB performance enhancements), installs MongoDB, installs MMS (MongoDB Monitoring Service) and configures the network connection on each server to cluster it with the other servers in your environment. A process that may have taken days of work and months of tweaking is completed in less than four hours. And because everything is standardized and automated, you run much less risk of human error.

MongoDB Configs

One of the other massive benefits of working so closely with 10gen is that we've been able to integrate 10gen's MongoDB Cloud Subscriptions into our offering. Customers who opt for a MongoDB Cloud Subscription get additional MongoDB features (like SSL and SNMP support) and support direct from the MongoDB authority. As an added bonus, since the 10gen team has an intimate understanding of the SoftLayer environment, they'll be able to provide even better support to SoftLayer customers!

You shouldn't have to sacrifice agility for performance, and you shouldn't have to sacrifice performance for agility. Most of the "big data" offerings in the market today are built on virtual servers that can be provisioned quickly but offer meager performance levels relative to running the same database on bare metal infrastructure. To get the performance benefits of dedicated hardware, many users have chosen to build, roll out and tweak their own configurations. With our MongoDB offering, you get the on-demand availability and flexibility of a cloud infrastructure with the raw power and full control of dedicated hardware.

If you've been toying with the idea of rolling out your own big data infrastructure, life just got a lot better for you.

-Duke

July 26, 2012

Global IP Addresses - What Are They and How Do They Work?

SoftLayer recently released "Global IPs" to a good amount of internal fanfare, and I thought I'd share a little about it with the blog audience in case customers have questions about what Global IPs are and how they work. Simply put, Global IP addresses can be provisioned in any data center on the SoftLayer network and moved to another facility if necessary. You can point it to a server in Dallas, and if you need to perform maintenance on the server in Dallas, you can move the IP address to a server in Amsterdam to seamlessly (and almost immediately) transition your traffic. If you spin up and turn down workloads on cloud computing instances, you have the ability to maintain and a specific IP address when you completely turn down an environment, and you can quickly reprovision the IP on a new instance when you spin up the next workload.

How Do Global IPs Work?

The basics of how the Internet works are simple: Packets are sent between you and a server somewhere based on the location of the content you've requested. That location is pinpointed by an IP address that is assigned to a specific server or cloud. Often for various reasons, blocks of IP addresses are provisioned in one region or location, so Global IPs are a bit of a departure from the norm.

When you're sending/receiving packets, you might thing the packets "know" the exact physical destination as soon as they're directed to an IP address, but in practice, they don't have to ... The packets are forwarded along a path of devices with a general idea of where the exact location will be, but the primary concern of each device is to get the all packets to the next hop in the network path as quickly as possible by using default routes and routing tables. As an example, let's follow a packet as it comes from an external webserver and detail how it gets back to your machine:

  1. The external webserver sends the packet to a local switch.
  2. The switch passes it to a router.
  3. The packet traverses a number of network hops (other routers) and enters the Softlayer network at one of the backbone routers (BBR).
  4. The BBR looks at the IP destination and compares it to a table shared and updated with the other routers on SoftLayer's network, and it locates the subnet the IP belongs to.
  5. The BBR determines behind which distribution aggregate router (DAR) the IP is located, then it to the closest BBR to that DAR.
  6. The DAR gets the packet, looks at its own tables, and finds the front-end customer router (FCR) that the subnet lives on, and sends it there.
  7. The FCR routes the packet to the front-end customer switch (FCS) that has that IP mapped to the proper MAC address.
  8. The switch then delivers the packet through the proper switchport.
  9. Your server gets the packet from the FCS, and the kernel goes, "Oh yes, that IP on the public port, I'll accept this now."

All of those steps happen in an instant, and for you to be reading this blog, the packets carrying this content would have followed a similar pattern to the browser on your computer.

The process is slightly different when it comes to Global IP addresses. When a packet is destined for a Global IP, as soon as it gets onto the SoftLayer network (step 4 above), the routing process changes.

We allocate subnets of IP addresses specifically to the Global IP address pool, and we tell all the BBRs that these IPs are special. When you order a global IP, we peel off one of those IPs and add a static route to your chosen server's IP address, and then tell all the BBRs that route. Rather than the server's IP being an endpoint, the network is expecting your server to act as a router, and do something with the packet when it is received. I know that could sound a little confusing since we aren't really using the server as a router, so let's follow a packet to your Global IP (following the first three steps from above):

  1. The BBR notes that this IP belongs to one of the special Global IP address subnets, and matches the destination IP with the static route to the destination server you chose when you provisioned the Global IP.
  2. The BBR forwards the packet to the DAR, which then finds the FCR, then hands it off to the switch.
  3. The switch hands the packet to your server, and your server accepts it on the public interface like a regular secondary IP.
  4. Your server then essentially "routes" the packet to an IP address on itself.

Because the Global IP address can be moved to different servers in different locations, whenever you change the destination IP, the static route is updated in our routing table quickly. Because the change is happening exclusively on SoftLayer's infrastructure, you don't have to wait on other providers propagate the change. Think of updating your site's domain to a new IP address via DNS as an example: Even after you update your authoritative DNS servers, you have to wait for your users' DNS servers to recognize and update the new IP address. With Global IPs, the IP address would remain the same, and all users will follow the new path as soon as the routers update.

This initial release of Global IP addresses is just the tip of the iceberg when it comes to functionality. The product management and network engineering teams are getting customer feedback and creating roadmaps for the future of the product, so we'd love to hear your feedback and questions. If you want a little more in-depth information about installation and provisioning, check out the Global IP Addresses page on KnowledgeLayer.

-Jason

February 15, 2012

SoftLayer + OpenStack Swift = SoftLayer Object Storage

Since our inception in 2005, SoftLayer's goal has been to provide an array of on-demand data center and hosting services that combine exceptional access, control, scalability and security with unparalleled network robustness and ease of use ... That's why we're so excited to unveil SoftLayer Object Storage to our customers.

Based on OpenStack Object Storage (codenamed Swift) — open-source software that allows the creation of redundant, scalable object storage on clusters of standardized servers — SoftLayer Object Storage provides customers with new opportunities to leverage cost-effective cloud-based storage and to simultaneously realize significant capex-related cost savings.

OpenStack has been phenomenally successful thanks to a global software community comprised of developers and other technologists that has built and tweaked a standards-based, massively scalable open-source platform for public and private cloud computing. The simple goal of the OpenStack project is to deliver code that enables any organization to create and offer feature-rich cloud computing services from industry-standard hardware. The overarching OpenStack technology consists of several interrelated project components: One for compute, one for an image service, one for object storage, and a few more projects in development.

SoftLayer Object Storage
Like the OpenStack Swift system on which it is based, SoftLayer Object Storage is not a file system or real-time data-storage system, rather it's a long-term storage system for a more permanent type of static data that can be retrieved, leveraged and updated when necessary. Typical applications for this type of storage can involve virtual machine images, photo storage, email storage and backup archiving.

One of the primary benefits of Object Storage is the role that it can play in automating and streamlining data storage in cloud computing environments. SoftLayer Object Storage offers rich metadata features and search capability that can be leveraged to automate the way unstructured data gets accessed. In this way, SoftLayer Object Storage will provide organizations with new capabilities for improving overall data management and storage efficiency.

File Storage v. Object Storage
To better understand the difference between file storage and object storage, let's look at how file storage and object storage differ when it comes to metadata and search for a simple photo image. When a digital camera or camera-enabled phone snaps a photo, it embeds a series of metadata values in the image. If you save the image in a standard image file format, you can search for it by standard file properties like name, date and size. If you save the same image with additional metadata as an object, you can set object metadata values for the image (after reading them from the image file). This detail provides granular search capability based on the metadata keys and values, in addition to the standard object properties. Here is a sample comparison of an image's metadata value in both systems:

File Metadata Object Metadata
Name:img01.jpg Name:img01.jpg
Date: 2012-02-13 Date:2012-02-13
Size:1.2MB Size:1.2MB
Manufacturer:CASIO
Model:QV-4000
x-Resolution:72.00
y-Resolution:72.00
PixelXDimension:2240
PixelYDimension:1680
FNumber:f/4.0
Exposure Time:1/659 sec.

Using the rich metadata and search capability enabled by object storage, you would be able to search for all images with a dimension of 2240x1680 or a resolution of 72x72 in a quick/automated fashion. The object storage system "understands" more about what is being stored because it is able to differentiate files based on characteristics that you define.

What Makes SoftLayer Object Storage Different?
SoftLayer Object Storage features several unique features and ways for SoftLayer customers to upload, access and manage data:

  • Search — Quickly access information through user-defined metadata key-value pairs, file name or unique identifier
  • CDN — Serve your content globally over our high-performance content delivery network
  • Private Network — Free, secure private network traffic between all data centers and storage cluster nodes
  • API — Access to a full-feature OpenStack-compatible API with additional support for CDN and search integration
  • Portal — Web application integrated into the SoftLayer portal
  • Mobile — iPhone and Android mobile apps, with Windows Phone app coming soon
  • Language Bindings — Feature-complete bindings for Java, PHP, Python and Ruby*

*Language bindings, documentation, and guides are available on SLDN.

We think SoftLayer Object Storage will be attractive to a broad range of current and prospective customers, from web-centric businesses dependent on file sharing and content distribution to legal/medical/financial-services companies which possess large volumes of data that must be stored securely while remaining readily accessible.

SoftLayer Object Storage significantly extends our cloud-services portfolio while substantially enriching the storage capabilities that we bring to our customers. What are you waiting for? Go order yourself some object storage @ $0.12/GB!

-Marc

Subscribe to product