Posts Tagged 'Storage'

November 11, 2014

Which storage solution is best for your project?

Before building applications around our network storage, here’s a refresher on what network storage is, how it is used, the different types available, and the best uses for each.

What is network storage? Why would you use it?

Appropriately named, network storage is storage attached to a server over our network; not to be confused with directly attached storage (DAS), which is a hard drive located in the server (or connected with a device like a SCSI or USB cable). Although DAS transfers data to a server faster than network storage due to network latency and system caching, there is still a strong place for network storage.

Many different servers can access network storage, and with some network storage solutions, more than one server can get data from the same shared storage volume simultaneously. This comes in handy if one server dies, because another can pick up a storage device and start where the first left off.

With DAS, planned downtime for server upgrades, potential data loss, and provisioning larger or more servers can slow down productivity. The physical constraints of internal drives and costs associated with servers do not affect network storage.

Because SoftLayer manages the disk space of our network storage products, there’s no need to worry about rebuilding a redundant array of inexpensive disks (RAIDs) or failed disks. If a disk fails, SoftLayer automatically replaces it and rebuilds the RAID—in most cases you would be unaware that the changes occurred.

Select network storage solutions are available with tools for your important data. Schedule snapshots of your data, promote snapshots to full volumes, or reset your data to the snapshot point.

And with network storage, downtime is minimal. Disaster recovery tools available on select storage solutions let you send a command to quickly fail over to a different data center so you can access your data if our network is ever down in a data center.

Types of Network Storage And How They Are Different

Storage Area Network (SAN) or Block Storage

Block storage works like DAS, just remotely—only a single server can access a block storage volume at a time. Using an Internet small computer system interface (iSCSI) protocol over a secure transmission control protocol/Internet protocol (TCP/IP) connection, SoftLayer's block storage has excellent features for backup and disaster recovery, and adding snapshot schedules and failover redundancy make it a powerful enterprise solution.

Network Attached Storage (NAS) or File Storage

File storage acts like a remote file system. It has a slim operating system that allows servers to treat it like a remote directory structure. Multiple servers can share files on the same storage simultaneously. Our new consistent performance storage lets you share files quickly and easily using a network file system (NFS) with your choice of performance level and secure connections.

We also have a common Internet file system (CIFS) (Windows), which requires a credential that grants access to any server on our private network. File storage can only be accessed by SoftLayer servers.

Object Storage

Object storage is a standalone storage entity with its own representational state transfer (REST) API that grants applications (not operating systems) access to the files stored there. Located on a public network, servers in any of our data centers can directly access files stored there. Object storage is different in the way those files are stored as well. In object storage there is not a directory structure, but instead metadata tags are used to categorize and search for files. In conjunction with a content delivery network (CDN), you can quickly serve files to your users or to a mobile device in close proximity.

With pay-as-you-go pricing, you don’t have to worry about running out of space. We only charge based on the greatest usage in any given day. That means you can get started right now for free!

Which storage solution is best for your project?

If you are still confused about which network storage option you should build your applications around, take this eight-question quiz to find out if object, file or block storage will work best for you:

-Kevin

August 29, 2013

HTML5 Tips and Tricks - Local Storage

As I'm sure you've heard by now: HTML5 is all the rage. People are creating amazing games with canvases, media interactivity with embeds and mobile/response sites with viewports. We've come a long way since 1990s iFrames! In this blog, I wanted to introduce you to an HTML5 tool that you might find useful: Local Web Storage — quite possibly the holy grail of web development!

In the past (and still most of the present), web sites store information about a surfer's preferences/choices via cookies. With that information, a site can be customized for a specific user, and that customization makes for a much better user experience. For example, you might select your preferred language when you first visit a website, and when you return, you won't have to make that selection again. You see similar functionality at work when you select themes/colors on a site or when you enlist help from one of those "remember me" checkboxes next to where you log into an account. The functionality that cookies enable is extremely valuable, but it's often inefficient.

You might be aware of some of the drawbacks to using cookies (such as size limitation (4KB) and privacy issues with unencrypted cookies), but I believe the most significant problem with cookies is overhead. Even if you limit your site to just a few small cookies per user, as your userbase grows into the thousands and tens of thousands, you'll notice that you're transferring a LOT data of over HTTP (and those bandwidth bills might start adding up). Cookies are stored on the user's computer, so every time that user visits your domain, the browser is transferring cookies to your server with every HTTP request. The file size for each of these transactions is tiny, but at scale, it can feel like death by a thousand cuts.

Enter HTML5 and local storage.

Rather than having to transmit data (cookies) to a remote web server, HTML5 allows a site to store information within the client web browser. The information you need to customize your user's experience doesn't have to travel from the user's hard drive to your server because the customization is stored in (and applied by) the user's browser. And because data in local storage isn't sent with every HTTP request like it is with cookies, the capacity of local storage is a whopping 5MB per domain (though I wouldn't recommend pushing that limit).

Let's check out how easy it is to use HTML5's local storage with JavaScript:

<script type="text/javascript">
    localStorage.setItem('preferredLanguage', 'EN');
</script>

Boom! We just set our first variable. Once that variable has been set in local storage for a given user, that user can close his or her browser and return to see the correct variable still selected when we retrieve it on our site:

<script type="text/javascript">
    localStorage.getItem('preferredLanguage');
</script>

All of the lead-up in this post, you're probably surprised by the simplicity of the actual coding, but that's one of the biggest reasons HTML local storage is such an amazing tool to use. We set our user's preferred language in local storage and retrieved it from local storage with a few simple lines. If want to set an "expiration" for a given variable in local storage the way you would for a cookie, you can script in an expiration variable that removes an entry when you say the time's up:

<script type="text/javascript">
    localStorage.removeItem('preferredLanguage');
</script>

If we stopped here, you'd have a solid fundamental understanding of how HTML5 local storage works, but I want to take you beyond the standard functionality of local storage. You see, local storage is intended primarily to store only strings, so if we wanted to store an object, we'd be out of luck ... until we realized that developers can find workarounds for everything!

Using some handy JSON, we can stringify and parse any object we want to store as local storage:

<script type="text/javascript">
    var user = {};
    user.name = 'CWolff';
    user.job = 'Software Engineer II';
    user.rating = 'Awesome';
 
    //If we were to stop here, the entry would only read as [object Object] when we try to retrieve it, so we stringify with JSON!
    localStorage.setItem('user', JSON.stringify(user));
 
    //Retrieve the object and assign it to a variable
    var getVar = JSON.parse(localStorage.getItem('user'));
 
    //We now have our object in a variable that we can play with, let's try it out
    alert(getVar.name);
    alert(getVar.job);
    alert(getVar.rating);
</script>

If you guys have read any of my other blogs, you know that I tend to write several blogs in a series before I move on to the next big topic, and this won't be an exception. Local storage is just the tip of the iceberg of what HTML5 can do, so buckle up and get ready to learn more about the crazy features and functionality of this next-generation language.

Try local storage for yourself ... And save yourself from the major headache of trying to figure out where all of your bandwidth is going!

-Cassandra

August 1, 2013

The "Unified Field Theory" of Storage

This guest blog was contributed by William Rocca of OS NEXUS. OS NEXUS makes the Quantastor Software Defined Storage platform designed to tackle the storage challenges facing cloud computing, Big Data and high performance applications.

Over the last decade, the creation and popularization of SAN/NAS systems simplified the management of storage into a single appliance so businesses could efficiently share, secure and manage data centrally. Fast forward about 10 years in storage innovation, and we're now rapidly changing from a world of proprietary hardware sold by big-iron vendors to open-source, scale-out storage technologies from software-only vendors that make use of commodity off-the-shelf hardware. Some of the new technologies are derivatives of traditional SAN/NAS with better scalability while others are completely new. Object storage technologies such as OpenStack SWIFT have created a foundation for whole new types of applications, and big data technologies like MongoDB, Riak and Hadoop go even further to blur the lines between storage and compute. These innovations provide a means for developing next-generation applications that can collect and analyze mountains of data. This is the exciting frontier of open storage today.

This frontier looks a lot like the "Wild West." With ad-hoc solutions that have great utility but are complex to setup and maintain, many users are effectively solving one-off problems, but these solutions are often narrowly defined and specifically designed for a particular application. The question everyone starts asking is, "Can't we just evolve to having one protocol ... one technology that unites them all?"

If each of these data storing technologies have unique advantages for specific use cases or applications, the answer isn't to eliminate protocols. To borrow a well-known concept from Physics, the solution lies in a "Unified Field Theory of Storage" — weaving them together into a cohesive software platform that makes them simple to deploy, maintain and operate.

When you look at the latest generation of storage technologies, you'll notice a common thread: They're all highly-available, scale-out, open-source and serve as a platform for next-generation applications. While SAN/NAS storage is still the bread-and-butter enterprise storage platform today (and will be for some time to come) these older protocols often don't measure up to the needs of applications being developed today. They run into problems storing, processing and gleaning value out of the mountains of data we're all producing.

Thinking about these challenges, how do we make these next-generation open storage technologies easy to manage and turn-key to deploy? What kind of platform could bring them all together? In short, "What does the 'Unified Field Theory of Storage' look like?"

These are the questions we've been trying to answer for the last few years at OS NEXUS, and the result of our efforts is the QuantaStor Software Defined Storage platform. In its first versions, we focused on building a flexible foundation supporting the traditional SAN/NAS protocols but with the launch of QuantaStor v3 this year, we introduced the first scale-out version of QuantaStor and integrated the first next-gen open storage technology, Gluster, into the platform. In June, we launched support of ZFS on Linux (ZoL), and enhanced the platform with a number of advanced enterprise features, such as snapshots, compression, deduplication and end-to-end checksums.

This is just the start, though. In our quest to solve the "Unified Field Theory of Storage," we're turning our eyes to integrating platforms like OpenStack SWIFT and Hadoop in QuantaStor v4 later this year, and as these high-power technologies are streamlined under a single platform, end users will have the ability to select the type(s) of storage that best fit a given application without having to learn (or unlearn) specific technologies.

The "Unified Field Theory of Storage" is emerging, and we hope to make it downloadable. Visit OSNEXUS.com to keep an eye on our progress. If you want to incorporate QuantaStor into your environment, check out SoftLayer's preconfigured QuantaStor Mass Storage Server solution.

-William Rocca, OS NEXUS

December 27, 2012

Using SoftLayer Object Storage to Back Up Your Server

Before I came to my senses and moved my personal servers to SoftLayer, I was one of many victims of a SolusVM exploit that resulted in the wide-scale attack of many nodes in my previous host's Chicago data center. While I'm a firm believer in backing up my data, I could not have foreseen the situation I was faced with: Not only was my server in one data center compromised with all of its data deleted, but my backup server in one of the host's other data centers was also attacked ... This left me with old, stale backups on my local computer and not much else. I quickly relocated my data and decided that I should use SoftLayer Object Storage to supplement and improve upon my backup and disaster recovery plans.

With SoftLayer Object Storage Python Client set up and the SoftLayer Object Storage Backup script — slbackup.py — in hand, I had the tools I needed to build a solid backup infrastructure easily. On Linux.org, I contributed an article about how to perform MySQL backups with those resources, so the database piece is handled, but I also need to back up my web files, so I whipped up another quick bash script to run:

#!/bin/bash
 
# The path the backups will be dumped to
DUMP_DIR="/home/backups/"
 
# Path to the web files to be backed up
BACKUP_PATH="/var/www/sites /"
 
# Back up folder name (mmddyyyy)
BACKUP_DIR="`date +%m%d%Y`"
 
# Backup File Name
DUMP_FILE="`date +%m_%d_%Y_%H_%M_%S`_site_files"
 
# SL container name
CONTAINER="site_backups"
 
# Create backup dir if doesn't exist
if [ ! -d $DUMP_DIR$BACKUP_DIR ]; then
        mkdir -p $DUMP_DIR$BACKUP_DIR
fi
 
tar -zcvpf $DUMP_DIR$BACKUP_DIR/$DUMP_FILE.tar.gz $BACKUP_PATH
 
# Make sure the archive exists
if [ -f $DUMP_DIR$BACKUP_DIR/$DUMP_FILE.tar.gz ]; then
        /root/slbackup.py -s $DUMP_DIR$BACKUP_DIR/ -o "$CONTAINER" -r 30
 
        # Remove the backup stored locally
        rm -rf $DUMP_DIR$BACKUP_DIR
 
        # Success
        exit 0
else
        echo "$DUMP_DIR$BACKUP_DIR/$DUMP_FILE.tar.gz does not exist."
        exit 1
fi

It's not the prettiest bash script, but it gets the job done. By tweaking a few variables, you can easily generate backups for any important directory of files and push them to your SoftLayer Object Storage account. If you want to change the retention time of your backups to be longer or shorter, you can change the 30 after the –r in the line below to the number of days you want to keep each backup:

/root/slbackup.py -s $DUMP_DIR$BACKUP_DIR/ -o "$CONTAINER" -r 30

I created a script for each website on my server, and I set a CRON (crontab –e) entry to run each one on Sundays staggered by 5 minutes:

5 1 * * 0  /root/bin/cron/CRON-site1.com_web_files > /dev/null
10 1 * * 0  /root/bin/cron/CRON-site2.com_web_files > /dev/null
15 1 * * 0  /root/bin/cron/CRON-site3.com_web_files > /dev/null 

If you're looking for an easy way to automate and solidify your backups, this little bit of code could make life easier on you. Had I taken the few minutes to put this script together prior to the attack I experienced at my previous host, I wouldn't have lost any of my data. It's easy to get lulled into "backup apathy" when you don't need your backups, but just because nothing *has* happened to your data doesn't mean nothing *can* happen to your data.

Take it from me ... Be over-prepared and save yourself a lot of trouble.

-Ronald

February 15, 2012

SoftLayer + OpenStack Swift = SoftLayer Object Storage

Since our inception in 2005, SoftLayer's goal has been to provide an array of on-demand data center and hosting services that combine exceptional access, control, scalability and security with unparalleled network robustness and ease of use ... That's why we're so excited to unveil SoftLayer Object Storage to our customers.

Based on OpenStack Object Storage (codenamed Swift) — open-source software that allows the creation of redundant, scalable object storage on clusters of standardized servers — SoftLayer Object Storage provides customers with new opportunities to leverage cost-effective cloud-based storage and to simultaneously realize significant capex-related cost savings.

OpenStack has been phenomenally successful thanks to a global software community comprised of developers and other technologists that has built and tweaked a standards-based, massively scalable open-source platform for public and private cloud computing. The simple goal of the OpenStack project is to deliver code that enables any organization to create and offer feature-rich cloud computing services from industry-standard hardware. The overarching OpenStack technology consists of several interrelated project components: One for compute, one for an image service, one for object storage, and a few more projects in development.

SoftLayer Object Storage
Like the OpenStack Swift system on which it is based, SoftLayer Object Storage is not a file system or real-time data-storage system, rather it's a long-term storage system for a more permanent type of static data that can be retrieved, leveraged and updated when necessary. Typical applications for this type of storage can involve virtual machine images, photo storage, email storage and backup archiving.

One of the primary benefits of Object Storage is the role that it can play in automating and streamlining data storage in cloud computing environments. SoftLayer Object Storage offers rich metadata features and search capability that can be leveraged to automate the way unstructured data gets accessed. In this way, SoftLayer Object Storage will provide organizations with new capabilities for improving overall data management and storage efficiency.

File Storage v. Object Storage
To better understand the difference between file storage and object storage, let's look at how file storage and object storage differ when it comes to metadata and search for a simple photo image. When a digital camera or camera-enabled phone snaps a photo, it embeds a series of metadata values in the image. If you save the image in a standard image file format, you can search for it by standard file properties like name, date and size. If you save the same image with additional metadata as an object, you can set object metadata values for the image (after reading them from the image file). This detail provides granular search capability based on the metadata keys and values, in addition to the standard object properties. Here is a sample comparison of an image's metadata value in both systems:

File Metadata Object Metadata
Name:img01.jpg Name:img01.jpg
Date: 2012-02-13 Date:2012-02-13
Size:1.2MB Size:1.2MB
Manufacturer:CASIO
Model:QV-4000
x-Resolution:72.00
y-Resolution:72.00
PixelXDimension:2240
PixelYDimension:1680
FNumber:f/4.0
Exposure Time:1/659 sec.

Using the rich metadata and search capability enabled by object storage, you would be able to search for all images with a dimension of 2240x1680 or a resolution of 72x72 in a quick/automated fashion. The object storage system "understands" more about what is being stored because it is able to differentiate files based on characteristics that you define.

What Makes SoftLayer Object Storage Different?
SoftLayer Object Storage features several unique features and ways for SoftLayer customers to upload, access and manage data:

  • Search — Quickly access information through user-defined metadata key-value pairs, file name or unique identifier
  • CDN — Serve your content globally over our high-performance content delivery network
  • Private Network — Free, secure private network traffic between all data centers and storage cluster nodes
  • API — Access to a full-feature OpenStack-compatible API with additional support for CDN and search integration
  • Portal — Web application integrated into the SoftLayer portal
  • Mobile — iPhone and Android mobile apps, with Windows Phone app coming soon
  • Language Bindings — Feature-complete bindings for Java, PHP, Python and Ruby*

*Language bindings, documentation, and guides are available on SLDN.

We think SoftLayer Object Storage will be attractive to a broad range of current and prospective customers, from web-centric businesses dependent on file sharing and content distribution to legal/medical/financial-services companies which possess large volumes of data that must be stored securely while remaining readily accessible.

SoftLayer Object Storage significantly extends our cloud-services portfolio while substantially enriching the storage capabilities that we bring to our customers. What are you waiting for? Go order yourself some object storage @ $0.12/GB!

-Marc

February 13, 2012

Logic Challenge: SoftLayer Server Rack Riddle

After I spent a little time weaving together a story in response to SKinman's "Choose Your Own Adventure" puzzle (which you can read in the comments section), I was reminded of another famous logic puzzle that I came across a few years ago. Because it was begging to be SoftLayer-ized, I freshened it up to challenge our community.

In 1962, Life International magazine published a logic puzzle that was said to be so difficult that it could only be solved by two percent of the world's population. It's been attributed to Einstein, and apparently Lewis Carroll is given a claim to it as well, but regardless of the original author, it's a great brain workout.

If you haven't tried a puzzle like this before, don't get discouraged and go Googling for the answer. You're given every detail you need to answer the question at the end ... Take your time and think about how the components are interrelated. If you've solved this puzzle before, this iteration might only be light mental calisthenics, but with its new SoftLayer twist, it should still be fun:

Einstein's SoftLayer Riddle

The Scenario: You're in a SoftLayer data center. You walk up to a server rack and you see five servers in the top five slots on the rack. Each of the five servers has a distinct hard drive configuration, processor type, operating system, control panel (or absence thereof) and add-on storage. No two servers in this rack are the same in any of those aspects.

  • The CentOS6 operating system is being run on the Xeon 3230 server.
  • The Dual Xeon 5410 server is racked next to (immediately above or below) the server running the Red Hat 6 operating system.
  • The Dual Xeon 5610 server uses 50GB of CloudLayer Storage as its add-on storage.
  • The Quad Xeon 7550 server has no control panel.
  • The Cent OS 5 operating system is racked immediately below the server running the Red Hat 5 operating system.
  • The server using 80GB NAS add-on storage is racked next to (immediately above or below) the server with two 100GB SSD hard drives.
  • The server running the Red Hat 5 operating system uses Parallels Virtuozzo (3VPS) as a control panel.
  • The server running the Windows 2008 operating system has two 100GB SSD hard drives.
  • The server using Plesk 9 as a control panel is in the middle space in the five-server set in the rack.
  • The top server in the rack is the Dual Xeon 5410 server.
  • The Xeon 3450 server has two 147GB 10K RPM SA-SCSI hard drives.
  • The server using 20GB EVault as its add-on storage has one 250GB SATA II hard drive.
  • The server with four 600GB 15K RPM SA-SCSI hard drives is next to (immediately above or below) the server using 100GB iSCSI SAN add-on storage.
  • The server using cPanel as a control panel has two 2TB SATA II hard drives.
  • The server with four 600GB 15K RPM SA-SCSI hard drives is racked next to (immediately above or below) the server using Plesk 10 (Unlimited) as a control panel.
  • One server will use a brand new, soon-to-be-announced product offering as its add-on storage.

Question: What is the monthly cost of the server that will be using our super-secret new product offering for its add-on storage?

Use the SoftLayer Shopping Cart to come up with your answer. You can assume that the server has a base configuration (unless specifically noted in the clues above), that SoftLayer's promotions are not used, and that the least expensive version of the control panel is being used for any control panel with several price points. You won't be able to include the cost of the add-on storage (yet), so just provide the base configuration cost of that server in one of our US-based data centers with all of the specs you are given.

Bonus Question: If you ordered all five of those servers, how long would it take for them to be provisioned for you?

Submit your answers via comment, and we'll publish the comments in about a week so other people have a chance to answer it without the risk of scrolling down and seeing spoilers.

-@khazard

December 23, 2011

Back up Your Life: In the Clouds, On the Go

The value of our cloud options here at SoftLayer have never been more noticeable than during the holiday seasons. Such a hectic time of the year can cause a lot of stress ... Stress that can lead to human error on some of your most important projects, data and memories. Such a loss could result in weeks or even years of valuable time and memories gone.

In the past few months, I've gone through two major data-related incidents that I was prepared for, and I can't imagine what I would have done if I didn't have some kind of backups in place. In one instance, my backups were not very current, so I ended up losing two weeks worth of work and data, but every now and then, you hear horror stories of people losing (or having to pay a lot to restore) all of their data. The saddest part about the data loss is that it's so easily preventable these days with prevalent backup storage platforms. For example, SoftLayer's CloudLayer Storage is a reliable, inexpensive place to keep all of your valuable data so you're not up a creek if you corrupt/lose your local versions somehow (like dropping a camera, issuing an incorrect syntax command or simply putting a thumb-drive though the washer).

That last "theoretical" example was in fact was one of the "incidents" I dealt with recently. A very important USB thumb-drive that I keep with me at all times was lost to the evil water machine! Because the security of the data was very important to me, I made sure to keep the drive encrypted in case of loss or theft, but the frequency of my backup schedule was the crack in my otherwise well thought data security and redundancy plan. A thumb drive is probably one of the best examples of items that need an automatic system or ritual to ensure data concurrency. This is a device we carry on us at all times, so it sees many changes in data. If this data is not properly updated in a central (secure and redundant) location, then all of our other efforts to take care of that data are wasted.

My the problem with my "Angel" (the name of the now-washed USB drive) was related to concurrency rather than security, and looking back at my mistake, I see how "The Cloud" would have served as a platform to better improve the way I was protecting my data with both of those point in mind. And that's why my new backups-in-the-cloud practices let me sleep a little more soundly these days.

If you're venturing out to fight the crowds of last-minute holiday shoppers or if you're just enjoying the sights and sounds of the season, be sure your memories and keepsake digital property are part of a well designed SRCD (secure, redundant and concurrent data) structure. Here are a few best practices to keep in mind when setting up your system:

  • Create a frequent back-up schedule
  • Use at least two physically separate devices
  • Follow your back-up schedule strictly
  • Automate everything you can for when you forget to execute on the previous bullet*

*I've used a few different programs (both proprietary and non-proprietary) that allow an automatic back-up to be performed when you plug your "on the go" device into your computer.

I'll keep an eye out for iPhone, Android and Blackberry apps that will allow for automatic transfers to a central location, and I'll put together a fresh blog with some ideas when I find anything interesting and worth your attention.

Have a happy Holidays!

- Jonathan

October 4, 2011

An Introduction to Redis

I recently had the opportunity to get re-acquainted with Redis while evaluating solutions for a project on the Product Innovation team here at SoftLayer. I'd actually played with it a couple of times before, but this time it "clicked." Or my brain broke. Either way, I see a lot of potential for Redis now.

No one product is a perfect fit for all of your data storage needs, of course. There are such fundamental tradeoffs to be made in designing storage architectures that you should be immediately suspicious of any product that claims to fit every need.

The best solutions tend to be products that actually embrace these tradeoffs. Redis, for instance, has sacrificed a small amount of data durability in exchange for being awesome.

What is it?

Redis is a key/value store, but describing it that way is sort of like calling a helicopter a "vehicle." It's a technically correct description, but it leaves out some important stuff.

You can think of it like a sophisticated older brother of Memcached. It presents a flat keyspace, and you can set those keys to string values. Another feature of Memcached is the ability to perform remote atomic operations, like "incr" and "append." These are really handy, because you have the ability to modify remote data without fetching, and you have an assurance that you're the only one performing that operation at that instant.

Redis takes this concept of remote commands on data and goes completely nuts with it. The database is aware of data structures like hashes, lists and sets in addition to simple string values. You can sort, union, intersect, slice and dice to your heart's content without fetching any data. Redis is a data structure server. You can treat it like remote memory, and this has an awesome immediate benefit for a programmer: your code and brain are already optimized for these data types.

But it's not just about making storage simpler. It's fast, too. Crazy fast. If you make intelligent use of its data structures, it's possible to serve a lot of traffic from relatively modest hardware. Redis 2.4 can easily handle ~50k list appends a second on my notebook. With batching, it can append 2 million items to a list on a remote host in about 1.28 seconds.

It allows the remote, atomic and performant manipulation of data structures. It took me a little while to realize exactly how useful that is.

What's wrong with it?

Nothing. Move along.

OK, it's a little short on durability. Redis uses memory as its primary store and periodically flushes to disk. A common configuration is to do so every second.

That sounds pretty reasonable. If a server goes down, you could lose a second of data. Keep in mind, however, how many operations Redis can perform in a second. If you're in a high-volume environment, that could be a lot of data. It's not for your financial transactions.

It also supports relatively limited availability options. Currently, it only supports master/slave replication. Clustering support is planned for an upcoming release. It's looking pretty powerful, but it will take some real-world testing to know its performance impact.

These challenges should be taken into consideration, and it's probably clear if you're in a situation where the current tradeoffs aren't a good fit.

In my experience, a lot of developers seriously overestimate the consequences of their application losing small amounts of data. Also consider whether or not the chance of losing a second (or less) of data genuinely represents a bigger threat to your application than any other compromises you might have made.

More Information
You can check out the slightly aging docs or browse the impressively simple source. There are probably already bindings for your language of choice as well.

-Tim

July 25, 2011

Under the Hood of 'The Cloud'

When we designed the CloudLayer Computing platform, our goal was to create an offering where customers would be able to customize and build cloud computing instances that specifically meet their needs: If you go to our site, you're even presented with an opportunity to "Build Your Own Cloud." The idea was to let users choose where they wanted their instance to reside as well as their own perfect mix of processor power, RAM and storage. Today, we're taking the BYOC mantra one step farther by unveiling the local disk storage option for CloudLayer computing instances!

Local Disk

For those of you familiar with the CloudLayer platform, you might already understand the value of a local disk storage option, but for the uninitiated, this news presents a perfect opportunity to talk about the dynamics of the cloud and how we approach the cloud around here.

As the resident "tech guy" in my social circle, I often find myself helping friends and family understand everything from why their printer isn't working to what value they can get from the latest and greatest buzzed-about technology. As you'd probably guess, the majority of the questions I've been getting recently revolve around 'the cloud' (thanks especially to huge marketing campaigns out of Redmond and Cupertino). That abstract term effectively conveys the intentional sentiment that users shouldn't have to worry about the mechanics of how the cloud works ... just that it works. The problem is that as the world of technology has pursued that sentiment, the generalization of the cloud has abstracted it to the point where this is how large companies are depicting the cloud:

Cloud

As it turns out, that image doesn't exactly illicit the, "Aha! Now I get it!" epiphany of users actually understanding how clouds (in the technology sense) work. See how I pluralized "clouds" in that last sentence? 'The Cloud' at SoftLayer isn't the same as 'The Cloud' in Redmond or 'The Cloud' in Cupertino. They may all be similar in the sense that each cloud technology incorporates hardware abstraction, on-demand scalability and utility billing, but they're not created in the same way.

If only there were a cloud-specific Declaration of Independence ...

We hold these truths to be self-evident, that all clouds are not equal, that they are endowed by their creators with certain distinct characteristics, that among these are storage, processing power and the ability to serve content. That to secure these characteristics, information should be given to users, expressed clearly to meet the the cloud's users;

The Ability to Serve Content
Let's unpack that Jeffersonian statement a little by looking at the distinct characteristics of every cloud, starting with the third ("the ability to serve content") and working backwards. Every cloud lives on hardware. The extent to which a given cloud relies on that hardware can vary, but at the end of the day, you &nash; as a user – are not simply connecting to water droplets in the ether. I'll use SoftLayer's CloudLayer platform as a specific example of that a cloud actually looks like: We have racks of uniform servers – designated as part of our cloud infrastructure – installed in rows in our data centers. All of those servers are networked together, and we worked with our friends at Citrix to use the XenServer platform to tie all of those servers together and virtualize the resources (or more simply: to make each piece of hardware accessible independently of the rest of the physical server it might be built into). With that infrastructure as a foundation, ordering a cloud server on the CloudLayer platform simply involves reserving a small piece of that cloud where you can install your own operating system and manage it like an independent server or instance to serve your content.

Processing Power
Understanding the hardware architecture upon which a cloud is built, the second distinct characteristic of every cloud ("processing power") is fairly logical: The more powerful the hardware used for a given cloud, the better processing performance you'll get in an instance using a piece of that hardware.

You can argue about what software uses the least resources in the process of virtualizing, but apples-to-apples, processing power is going to be determined by the power of the underlying hardware. Some providers try to obfuscate the types of servers/processors available to their cloud users (sometimes because they are using legacy hardware that they wouldn't be able to sell/rent otherwise), but because we know how important consistent power is to users, we guarantee that CloudLayer instances are based on 2.0GHz (or faster) processors.

Storage
We walked backward through the distinct characteristics included in my cloud-specific Declaration of Independence because of today's CloudLayer Computing storage announcement, but before I get into the details of that new option, let's talk about storage in general.

If the primary goal of a cloud platform is to give users the ability to scale instantly from 1 CPU of power to 16 CPUs of power, the underlying architecture has to be as flexible as possible. Let's say your cloud computing instance resides on a server with only 10 CPUs available, so when you upgrade to a 16-CPU instance, your instance will be moved to a server with enough available resources to meet your need. To make that kind of quick change possible, most cloud platforms are connected to a SAN (storage area network) or other storage device via a back-end network to the cloud servers. The biggest pro of having this setup is that upgrading and downgrading CPU and RAM for a given cloud instance is relatively easy, but it introduces a challenge: The data lives on another device that is connected via switches and cables and is being used by other customers as well. Because your data has to be moved to your server to be processed when you call it, it's a little slower than if a hard disk was sitting in the same server as the instance's processor and RAM. For that reason, many users don't feel comfortable moving to the cloud.

In response to the call for better-performing storage, there has been a push toward incorporating local disk storage for cloud computing instances. Because local disk storage is physically available to the CPU and RAM, the transfer of data is almost immediate and I/O (input/output) rates are generally much higher. The obvious benefit of this setup is that the storage will perform much better for I/O-intensive applications, while the tradeoff is that the setup loses the inherent redundancy of having the data replicated across multiple drives in a SAN (which, is almost like its own cloud ... but I won't confuse you with that right now).

The CloudLayer Computing platform has always been built to take advantage of the immediate scalability enabled by storing files in a network storage device. We heard from users who want to use the cloud for other applications that they wanted us to incorporate another option, so today we're happy to announce the availability of local disk storage for CloudLayer Computing! We're looking forward to seeing how our customers are going to incorporate cloud computing instances with local disk storage into their existing environments with dedicated servers and cloud computing instances using SAN storage.

If you have questions about whether the SAN or local disk storage option would fit your application best, click the Live Chat icon on SoftLayer.com and consult with one of our sales reps about the benefits and trade-offs of each.

We want you to know exactly what you're getting from SoftLayer, so we try to be as transparent as we can when rolling out new products. If you have any questions about CloudLayer or any of our other offerings, please let us know!

-@nday91

August 3, 2010

How Clouds Killed The PC

Most days, it seems that technology progresses all too slowly. It is a different feeling when you work with cutting edge technology on a daily basis: deploying the first dual network datacenter infrastructure, being entrenched in solutions for everything from CDN to ISCI to DTS and more, testing the latest enterprise solutions from leading industry vendors long before money could buy them… it never really meant a whole lot to me; it was very much just, “How we roll”, as the gang would say.

But every so often, there is a day when a new technology catches my attention and reminds me why I got involved in the IT industry. Something that reminds me of the days spent tapping out QuickBasic 2.0 applications on my 18MHz 386 and 16 color EGA monitor. Surprisingly, the rise of cloud computing did just that. There was a day some still remember, when the cost of localized hardware was significant enough that terminals ruled the world. Occasionally, you may still see one at a grocery checkout stand or being used in a retail stockroom to check inventory across locations. Early terminals were commonly thin clients lacking a processor, non-volatile user storage, and only possessing enough memory to display what was on the screen at any given time. As the cost of memory declined, fat clients gained some popularity offering locally programmable memory. However, the concept was still the same: one host machine, usually a mainframe, serving applications over a distance to multiple (less capable) client machines.

Terminals were not destined to last though. In a twist of irony one of the innovations that they helped to inspire, the microprocessor, combined with the falling price and increased capacity of memory eventually led the decline of terminals. Left behind, in a cloud of dust, by hardware manufacturer’s race for speed capacity combined with advances in networking technology, the terminal PC became a historical relic looked upon as a necessary stop-gap solution used in the days when hardware was just too-darn-expensive. It was at that time the truly personal computer that we know and love was born and has forever-since reigned supreme. Then came the ARPANET, which gave way to the Information Super Highway, gave way to the World Wide Web, gave way to the internet we know today.

Mainframes gave way to servers. And today, I walk into a datacenter surrounded by servers boasting quad octo-core processors and Cloud Computing Instances, talking to customers who use their smart-phones to remotely access their web hosts, and quietly thinking to myself, “Have things really changed?” How far off is the day, when the benefits of remotely hosted applications outweigh the benefits of localized hardware? When we sit at the start of a new era where CCI’s can be created in minutes, regularly imaged for data security, migrated and restored quickly in the event of hardware failure, accessed from anywhere and from a variety of client hardware and software implementations, how much more would it take for us to return to the days of terminal PC’s. As bandwidth continues to improve, purchase and operational costs per processing core continues to fall, people demand more and more ‘anywhere access’, open source gains popularity and the idea of renting freely upgraded applications becomes accepted outside of the IT community, who knows what the future might hold. In a future where the concept of parallel uplinks may be no more foreign than that of parallel data transfer over CAT6 is to the layman, I wonder if personal computers will be thought of as the necessary stop-gap solution used while we waited for bandwidth to catch up to useable processing power; nothing more than a dinosaur that gave way to the green-movement and our need to be connected everywhere.

While I work on bringing my head out of the clouds, I remember why I am here. I am not here because technology’s past was all that fantastic, or because the present is all that glamorous, but because the future is still wide open. Whether-or-not clouds ever really kill the PC is anyone’s guess and only time will tell. However, one thing is currently known, as companies continue to see the benefit of having their staff conduct business through a web-portal interface, consumers continue trying to figure out what they are going to do with the extra two or three of the four cores they have, and the cost-to-performance ratio associated with remote resources continues to fall, we are steadily moving that way.

Categories: 
Subscribe to storage