With SoftLayer Object Storage Python Client set up and the SoftLayer Object Storage Backup script — slbackup.py — in hand, I had the tools I needed to build a solid backup infrastructure easily. On Linux.org, I contributed an article about how to perform MySQL backups with those resources, so the database piece is handled, but I also need to back up my web files, so I whipped up another quick bash script to run:

#!/bin/bash
 
# The path the backups will be dumped to
DUMP_DIR="/home/backups/"
 
# Path to the web files to be backed up
BACKUP_PATH="/var/www/sites /"
 
# Back up folder name (mmddyyyy)
BACKUP_DIR="`date +%m%d%Y`"
 
# Backup File Name
DUMP_FILE="`date +%m_%d_%Y_%H_%M_%S`_site_files"
 
# SL container name
CONTAINER="site_backups"
 
# Create backup dir if doesn't exist
if [ ! -d $DUMP_DIR$BACKUP_DIR ]; then
        mkdir -p $DUMP_DIR$BACKUP_DIR
fi
 
tar -zcvpf $DUMP_DIR$BACKUP_DIR/$DUMP_FILE.tar.gz $BACKUP_PATH
 
# Make sure the archive exists
if [ -f $DUMP_DIR$BACKUP_DIR/$DUMP_FILE.tar.gz ]; then
        /root/slbackup.py -s $DUMP_DIR$BACKUP_DIR/ -o "$CONTAINER" -r 30
 
        # Remove the backup stored locally
        rm -rf $DUMP_DIR$BACKUP_DIR
 
        # Success
        exit 0
else
        echo "$DUMP_DIR$BACKUP_DIR/$DUMP_FILE.tar.gz does not exist."
        exit 1
fi

It’s not the prettiest bash script, but it gets the job done. By tweaking a few variables, you can easily generate backups for any important directory of files and push them to your SoftLayer Object Storage account. If you want to change the retention time of your backups to be longer or shorter, you can change the 30 after the –r in the line below to the number of days you want to keep each backup:

/root/slbackup.py -s $DUMP_DIR$BACKUP_DIR/ -o "$CONTAINER" -r 30

I created a script for each website on my server, and I set a CRON (crontab –e) entry to run each one on Sundays staggered by 5 minutes:

5 1 * * 0  /root/bin/cron/CRON-site1.com_web_files > /dev/null
10 1 * * 0  /root/bin/cron/CRON-site2.com_web_files > /dev/null
15 1 * * 0  /root/bin/cron/CRON-site3.com_web_files > /dev/null

If you’re looking for an easy way to automate and solidify your backups, this little bit of code could make life easier on you. Had I taken the few minutes to put this script together prior to the attack I experienced at my previous host, I wouldn’t have lost any of my data. It’s easy to get lulled into “backup apathy” when you don’t need your backups, but just because nothing *has* happened to your data doesn’t mean nothing *can* happen to your data.

Take it from me … Be over-prepared and save yourself a lot of trouble.

-Ronald

Based on OpenStack Object Storage (codenamed Swift) — open-source software that allows the creation of redundant, scalable object storage on clusters of standardized servers — SoftLayer Object Storage provides customers with new opportunities to leverage cost-effective cloud-based storage and to simultaneously realize significant capex-related cost savings.

OpenStack has been phenomenally successful thanks to a global software community comprised of developers and other technologists that has built and tweaked a standards-based, massively scalable open-source platform for public and private cloud computing. The simple goal of the OpenStack project is to deliver code that enables any organization to create and offer feature-rich cloud computing services from industry-standard hardware. The overarching OpenStack technology consists of several interrelated project components: One for compute, one for an image service, one for object storage, and a few more projects in development.

SoftLayer Object Storage
Like the OpenStack Swift system on which it is based, SoftLayer Object Storage is not a file system or real-time data-storage system, rather it’s a long-term storage system for a more permanent type of static data that can be retrieved, leveraged and updated when necessary. Typical applications for this type of storage can involve virtual machine images, photo storage, email storage and backup archiving.

One of the primary benefits of Object Storage is the role that it can play in automating and streamlining data storage in cloud computing environments. SoftLayer Object Storage offers rich metadata features and search capability that can be leveraged to automate the way unstructured data gets accessed. In this way, SoftLayer Object Storage will provide organizations with new capabilities for improving overall data management and storage efficiency.

File Storage v. Object Storage
To better understand the difference between file storage and object storage, let’s look at how file storage and object storage differ when it comes to metadata and search for a simple photo image. When a digital camera or camera-enabled phone snaps a photo, it embeds a series of metadata values in the image. If you save the image in a standard image file format, you can search for it by standard file properties like name, date and size. If you save the same image with additional metadata as an object, you can set object metadata values for the image (after reading them from the image file). This detail provides granular search capability based on the metadata keys and values, in addition to the standard object properties. Here is a sample comparison of an image’s metadata value in both systems:

Using the rich metadata and search capability enabled by object storage, you would be able to search for all images with a dimension of 2240×1680 or a resolution of 72×72 in a quick/automated fashion. The object storage system “understands” more about what is being stored because it is able to differentiate files based on characteristics that you define.

What Makes SoftLayer Object Storage Different?
SoftLayer Object Storage features several unique features and ways for SoftLayer customers to upload, access and manage data:

• Search — Quickly access information through user-defined metadata key-value pairs, file name or unique identifier
• CDN — Serve your content globally over our high-performance content delivery network
• Private Network — Free, secure private network traffic between all data centers and storage cluster nodes
• API — Access to a full-feature OpenStack-compatible API with additional support for CDN and search integration
• Portal — Web application integrated into the SoftLayer portal
• Mobile — iPhone and Android mobile apps, with Windows Phone app coming soon
• Language Bindings — Feature-complete bindings for Java, PHP, Python and Ruby*

*Language bindings, documentation, and guides are available on SLDN.

We think SoftLayer Object Storage will be attractive to a broad range of current and prospective customers, from web-centric businesses dependent on file sharing and content distribution to legal/medical/financial-services companies which possess large volumes of data that must be stored securely while remaining readily accessible.

SoftLayer Object Storage significantly extends our cloud-services portfolio while substantially enriching the storage capabilities that we bring to our customers. What are you waiting for? Go order yourself some object storage @ $0.12/GB!

-Marc

In 1962, Life International magazine published a logic puzzle that was said to be so difficult that it could only be solved by two percent of the world’s population. It’s been attributed to Einstein, and apparently Lewis Carroll is given a claim to it as well, but regardless of the original author, it’s a great brain workout.

If you haven’t tried a puzzle like this before, don’t get discouraged and go Googling for the answer. You’re given every detail you need to answer the question at the end … Take your time and think about how the components are interrelated. If you’ve solved this puzzle before, this iteration might only be light mental calisthenics, but with its new SoftLayer twist, it should still be fun:

Einstein’s SoftLayer Riddle

The Scenario: You’re in a SoftLayer data center. You walk up to a server rack and you see five servers in the top five slots on the rack. Each of the five servers has a distinct hard drive configuration, processor type, operating system, control panel (or absence thereof) and add-on storage. No two servers in this rack are the same in any of those aspects.

• The CentOS6 operating system is being run on the Xeon 3230 server.
• The Dual Xeon 5410 server is racked next to (immediately above or below) the server running the Red Hat 6 operating system.
• The Dual Xeon 5610 server uses 50GB of CloudLayer Storage as its add-on storage.
• The Quad Xeon 7550 server has no control panel.
• The Cent OS 5 operating system is racked immediately below the server running the Red Hat 5 operating system.
• The server using 80GB NAS add-on storage is racked next to (immediately above or below) the server with two 100GB SSD hard drives.
• The server running the Red Hat 5 operating system uses Parallels Virtuozzo (3VPS) as a control panel.
• The server running the Windows 2008 operating system has two 100GB SSD hard drives.
• The server using Plesk 9 as a control panel is in the middle space in the five-server set in the rack.
• The top server in the rack is the Dual Xeon 5410 server.
• The Xeon 3450 server has two 147GB 10K RPM SA-SCSI hard drives.
• The server using 20GB EVault as its add-on storage has one 250GB SATA II hard drive.
• The server with four 600GB 15K RPM SA-SCSI hard drives is next to (immediately above or below) the server using 100GB iSCSI SAN add-on storage.
• The server using cPanel as a control panel has two 2TB SATA II hard drives.
• The server with four 600GB 15K RPM SA-SCSI hard drives is racked next to (immediately above or below) the server using Plesk 10 (Unlimited) as a control panel.
• One server will use a brand new, soon-to-be-announced product offering as its add-on storage.

Question: What is the monthly cost of the server that will be using our super-secret new product offering for its add-on storage?

Use the SoftLayer Shopping Cart to come up with your answer. You can assume that the server has a base configuration (unless specifically noted in the clues above), that SoftLayer’s promotions are not used, and that the least expensive version of the control panel is being used for any control panel with several price points. You won’t be able to include the cost of the add-on storage (yet), so just provide the base configuration cost of that server in one of our US-based data centers with all of the specs you are given.

Bonus Question: If you ordered all five of those servers, how long would it take for them to be provisioned for you?

Submit your answers via comment, and we’ll publish the comments in about a week so other people have a chance to answer it without the risk of scrolling down and seeing spoilers.

-@khazard

In the past few months, I’ve gone through two major data-related incidents that I was prepared for, and I can’t imagine what I would have done if I didn’t have some kind of backups in place. In one instance, my backups were not very current, so I ended up losing two weeks worth of work and data, but every now and then, you hear horror stories of people losing (or having to pay a lot to restore) all of their data. The saddest part about the data loss is that it’s so easily preventable these days with prevalent backup storage platforms. For example, SoftLayer’s CloudLayer Storage is a reliable, inexpensive place to keep all of your valuable data so you’re not up a creek if you corrupt/lose your local versions somehow (like dropping a camera, issuing an incorrect syntax command or simply putting a thumb-drive though the washer).

That last “theoretical” example was in fact was one of the “incidents” I dealt with recently. A very important USB thumb-drive that I keep with me at all times was lost to the evil water machine! Because the security of the data was very important to me, I made sure to keep the drive encrypted in case of loss or theft, but the frequency of my backup schedule was the crack in my otherwise well thought data security and redundancy plan. A thumb drive is probably one of the best examples of items that need an automatic system or ritual to ensure data concurrency. This is a device we carry on us at all times, so it sees many changes in data. If this data is not properly updated in a central (secure and redundant) location, then all of our other efforts to take care of that data are wasted.

My the problem with my “Angel” (the name of the now-washed USB drive) was related to concurrency rather than security, and looking back at my mistake, I see how “The Cloud” would have served as a platform to better improve the way I was protecting my data with both of those point in mind. And that’s why my new backups-in-the-cloud practices let me sleep a little more soundly these days.

If you’re venturing out to fight the crowds of last-minute holiday shoppers or if you’re just enjoying the sights and sounds of the season, be sure your memories and keepsake digital property are part of a well designed SRCD (secure, redundant and concurrent data) structure. Here are a few best practices to keep in mind when setting up your system:

• Create a frequent back-up schedule
• Use at least two physically separate devices
• Follow your back-up schedule strictly
• Automate everything you can for when you forget to execute on the previous bullet*

*I’ve used a few different programs (both proprietary and non-proprietary) that allow an automatic back-up to be performed when you plug your “on the go” device into your computer.

I’ll keep an eye out for iPhone, Android and Blackberry apps that will allow for automatic transfers to a central location, and I’ll put together a fresh blog with some ideas when I find anything interesting and worth your attention.

Have a happy Holidays!

- Jonathan

No one product is a perfect fit for all of your data storage needs, of course. There are such fundamental tradeoffs to be made in designing storage architectures that you should be immediately suspicious of any product that claims to fit every need.

The best solutions tend to be products that actually embrace these tradeoffs. Redis, for instance, has sacrificed a small amount of data durability in exchange for being awesome.

What is it?

Redis is a key/value store, but describing it that way is sort of like calling a helicopter a “vehicle.” It’s a technically correct description, but it leaves out some important stuff.

You can think of it like a sophisticated older brother of Memcached. It presents a flat keyspace, and you can set those keys to string values. Another feature of Memcached is the ability to perform remote atomic operations, like “incr” and “append.” These are really handy, because you have the ability to modify remote data without fetching, and you have an assurance that you’re the only one performing that operation at that instant.

Redis takes this concept of remote commands on data and goes completely nuts with it. The database is aware of data structures like hashes, lists and sets in addition to simple string values. You can sort, union, intersect, slice and dice to your heart’s content without fetching any data. Redis is a data structure server. You can treat it like remote memory, and this has an awesome immediate benefit for a programmer: your code and brain are already optimized for these data types.

But it’s not just about making storage simpler. It’s fast, too. Crazy fast. If you make intelligent use of its data structures, it’s possible to serve a lot of traffic from relatively modest hardware. Redis 2.4 can easily handle ~50k list appends a second on my notebook. With batching, it can append 2 million items to a list on a remote host in about 1.28 seconds.

It allows the remote, atomic and performant manipulation of data structures. It took me a little while to realize exactly how useful that is.

What’s wrong with it?

Nothing. Move along.

OK, it’s a little short on durability. Redis uses memory as its primary store and periodically flushes to disk. A common configuration is to do so every second.

That sounds pretty reasonable. If a server goes down, you could lose a second of data. Keep in mind, however, how many operations Redis can perform in a second. If you’re in a high-volume environment, that could be a lot of data. It’s not for your financial transactions.

It also supports relatively limited availability options. Currently, it only supports master/slave replication. Clustering support is planned for an upcoming release. It’s looking pretty powerful, but it will take some real-world testing to know its performance impact.

These challenges should be taken into consideration, and it’s probably clear if you’re in a situation where the current tradeoffs aren’t a good fit.

In my experience, a lot of developers seriously overestimate the consequences of their application losing small amounts of data. Also consider whether or not the chance of losing a second (or less) of data genuinely represents a bigger threat to your application than any other compromises you might have made.

More Information
You can check out the slightly aging docs or browse the impressively simple source. There are probably already bindings for your language of choice as well.

-Tim

For those of you familiar with the CloudLayer platform, you might already understand the value of a local disk storage option, but for the uninitiated, this news presents a perfect opportunity to talk about the dynamics of the cloud and how we approach the cloud around here.

As the resident “tech guy” in my social circle, I often find myself helping friends and family understand everything from why their printer isn’t working to what value they can get from the latest and greatest buzzed-about technology. As you’d probably guess, the majority of the questions I’ve been getting recently revolve around ‘the cloud’ (thanks especially to huge marketing campaigns out of Redmond and Cupertino). That abstract term effectively conveys the intentional sentiment that users shouldn’t have to worry about the mechanics of how the cloud works … just that it works. The problem is that as the world of technology has pursued that sentiment, the generalization of the cloud has abstracted it to the point where this is how large companies are depicting the cloud:

As it turns out, that image doesn’t exactly illicit the, “Aha! Now I get it!” epiphany of users actually understanding how clouds (in the technology sense) work. See how I pluralized “clouds” in that last sentence? ‘The Cloud’ at SoftLayer isn’t the same as ‘The Cloud’ in Redmond or ‘The Cloud’ in Cupertino. They may all be similar in the sense that each cloud technology incorporates hardware abstraction, on-demand scalability and utility billing, but they’re not created in the same way.

If only there were a cloud-specific Declaration of Independence …

We hold these truths to be self-evident, that all clouds are not equal, that they are endowed by their creators with certain distinct characteristics, that among these are storage, processing power and the ability to serve content. That to secure these characteristics, information should be given to users, expressed clearly to meet the the cloud’s users;

The Ability to Serve Content
Let’s unpack that Jeffersonian statement a little by looking at the distinct characteristics of every cloud, starting with the third (“the ability to serve content”) and working backwards. Every cloud lives on hardware. The extent to which a given cloud relies on that hardware can vary, but at the end of the day, you &nash; as a user – are not simply connecting to water droplets in the ether. I’ll use SoftLayer’s CloudLayer platform as a specific example of that a cloud actually looks like: We have racks of uniform servers – designated as part of our cloud infrastructure – installed in rows in our data centers. All of those servers are networked together, and we worked with our friends at Citrix to use the XenServer platform to tie all of those servers together and virtualize the resources (or more simply: to make each piece of hardware accessible independently of the rest of the physical server it might be built into). With that infrastructure as a foundation, ordering a cloud server on the CloudLayer platform simply involves reserving a small piece of that cloud where you can install your own operating system and manage it like an independent server or instance to serve your content.

Processing Power
Understanding the hardware architecture upon which a cloud is built, the second distinct characteristic of every cloud (“processing power”) is fairly logical: The more powerful the hardware used for a given cloud, the better processing performance you’ll get in an instance using a piece of that hardware.

You can argue about what software uses the least resources in the process of virtualizing, but apples-to-apples, processing power is going to be determined by the power of the underlying hardware. Some providers try to obfuscate the types of servers/processors available to their cloud users (sometimes because they are using legacy hardware that they wouldn’t be able to sell/rent otherwise), but because we know how important consistent power is to users, we guarantee that CloudLayer instances are based on 2.0GHz (or faster) processors.

Storage
We walked backward through the distinct characteristics included in my cloud-specific Declaration of Independence because of today’s CloudLayer Computing storage announcement, but before I get into the details of that new option, let’s talk about storage in general.

If the primary goal of a cloud platform is to give users the ability to scale instantly from 1 CPU of power to 16 CPUs of power, the underlying architecture has to be as flexible as possible. Let’s say your cloud computing instance resides on a server with only 10 CPUs available, so when you upgrade to a 16-CPU instance, your instance will be moved to a server with enough available resources to meet your need. To make that kind of quick change possible, most cloud platforms are connected to a SAN (storage area network) or other storage device via a back-end network to the cloud servers. The biggest pro of having this setup is that upgrading and downgrading CPU and RAM for a given cloud instance is relatively easy, but it introduces a challenge: The data lives on another device that is connected via switches and cables and is being used by other customers as well. Because your data has to be moved to your server to be processed when you call it, it’s a little slower than if a hard disk was sitting in the same server as the instance’s processor and RAM. For that reason, many users don’t feel comfortable moving to the cloud.

In response to the call for better-performing storage, there has been a push toward incorporating local disk storage for cloud computing instances. Because local disk storage is physically available to the CPU and RAM, the transfer of data is almost immediate and I/O (input/output) rates are generally much higher. The obvious benefit of this setup is that the storage will perform much better for I/O-intensive applications, while the tradeoff is that the setup loses the inherent redundancy of having the data replicated across multiple drives in a SAN (which, is almost like its own cloud … but I won’t confuse you with that right now).

The CloudLayer Computing platform has always been built to take advantage of the immediate scalability enabled by storing files in a network storage device. We heard from users who want to use the cloud for other applications that they wanted us to incorporate another option, so today we’re happy to announce the availability of local disk storage for CloudLayer Computing! We’re looking forward to seeing how our customers are going to incorporate cloud computing instances with local disk storage into their existing environments with dedicated servers and cloud computing instances using SAN storage.

If you have questions about whether the SAN or local disk storage option would fit your application best, click the Live Chat icon on SoftLayer.com and consult with one of our sales reps about the benefits and trade-offs of each.

We want you to know exactly what you’re getting from SoftLayer, so we try to be as transparent as we can when rolling out new products. If you have any questions about CloudLayer or any of our other offerings, please let us know!

-@nday91

But every so often, there is a day when a new technology catches my attention and reminds me why I got involved in the IT industry. Something that reminds me of the days spent tapping out QuickBasic 2.0 applications on my 18MHz 386 and 16 color EGA monitor. Surprisingly, the rise of cloud computing did just that. There was a day some still remember, when the cost of localized hardware was significant enough that terminals ruled the world. Occasionally, you may still see one at a grocery checkout stand or being used in a retail stockroom to check inventory across locations. Early terminals were commonly thin clients lacking a processor, non-volatile user storage, and only possessing enough memory to display what was on the screen at any given time. As the cost of memory declined, fat clients gained some popularity offering locally programmable memory. However, the concept was still the same: one host machine, usually a mainframe, serving applications over a distance to multiple (less capable) client machines.

Terminals were not destined to last though. In a twist of irony one of the innovations that they helped to inspire, the microprocessor, combined with the falling price and increased capacity of memory eventually led the decline of terminals. Left behind, in a cloud of dust, by hardware manufacturer’s race for speed capacity combined with advances in networking technology, the terminal PC became a historical relic looked upon as a necessary stop-gap solution used in the days when hardware was just too-darn-expensive. It was at that time the truly personal computer that we know and love was born and has forever-since reigned supreme. Then came the ARPANET, which gave way to the Information Super Highway, gave way to the World Wide Web, gave way to the internet we know today.

Mainframes gave way to servers. And today, I walk into a datacenter surrounded by servers boasting quad octo-core processors and Cloud Computing Instances, talking to customers who use their smart-phones to remotely access their web hosts, and quietly thinking to myself, “Have things really changed?” How far off is the day, when the benefits of remotely hosted applications outweigh the benefits of localized hardware? When we sit at the start of a new era where CCI’s can be created in minutes, regularly imaged for data security, migrated and restored quickly in the event of hardware failure, accessed from anywhere and from a variety of client hardware and software implementations, how much more would it take for us to return to the days of terminal PC’s. As bandwidth continues to improve, purchase and operational costs per processing core continues to fall, people demand more and more ‘anywhere access’, open source gains popularity and the idea of renting freely upgraded applications becomes accepted outside of the IT community, who knows what the future might hold. In a future where the concept of parallel uplinks may be no more foreign than that of parallel data transfer over CAT6 is to the layman, I wonder if personal computers will be thought of as the necessary stop-gap solution used while we waited for bandwidth to catch up to useable processing power; nothing more than a dinosaur that gave way to the green-movement and our need to be connected everywhere.

While I work on bringing my head out of the clouds, I remember why I am here. I am not here because technology’s past was all that fantastic, or because the present is all that glamorous, but because the future is still wide open. Whether-or-not clouds ever really kill the PC is anyone’s guess and only time will tell. However, one thing is currently known, as companies continue to see the benefit of having their staff conduct business through a web-portal interface, consumers continue trying to figure out what they are going to do with the extra two or three of the four cores they have, and the cost-to-performance ratio associated with remote resources continues to fall, we are steadily moving that way.

Something to ponder on when considering the layout of your IT infrastructure is the software is only as good as the under lying server and the server is only as good as the under lying network which is intern only as good as the power that feeds it and so for and so on. This is the core reason that here at SoftLayer we strive to provide the best we can from the ground up to help ensure our clients that they have the tools and foundation to compete in the world of online services.

There is some confusion out there on what’s a good way to back up your data. In this article we will go over several options for good ways to backup and sore your backups along with a few ways that are not recommended.

When it comes to backups storing them off site (off your server or on a secondary drive not running your system) is the best solution with storing them off site being the recommended course.

When raids come into consideration just because the drives are redundant (a lave mirror situation) there are several situations, which can cause a complete raid failure such as the raid controller failing, the array developing a bad stripe. Drive failure on more than one drive(this does happen though rarely) , out of date firmware on the drives and the raid card causing errors. Using a network storage device like our evault or a nas storege is also an excellent way to store backups off system. The last thing to consider is keeping your backups up to date. I suggest making a new back every week at minimum (if you have very active sites or data bases I would recommend a every other day backup or daily backup). It is up to you or your server administrator to keep up with your backups and make sure they are kept up to date. If you have a hardware failure and your backups are well out of date it’s almost like not having them at all.

In closing consider the service you provide and how your data is safe, secure, and recoverable. These things I key to running a successful server and website.

Working in the SoftLayer data center, we take incredible measures to protect our customer’s servers. In a sense, we try to keep Mr. Murphy away. From the biggies (like redundant power and MASSIVE cooling units) to the routine (such as the data center walkthroughs, and proactive RAID alerts), we do our best to keep the servers in the data center running smoothly, and free of surprises.

Beyond the punches our friend Mr. Murphy can throw at us now and again, it’s nice to know there are a few good surprises in store for you, too. You might be surprised at the great deals our SLales team can provide. You might also be surprised at not only the amazing features such as the new Cloudlayer™ Storage, but the incredible rate we keep bringing new features to the table. I’ve also seen customer’s surprise when we rescue their server from the brink of disaster, or when we are able to provide a few tweaks to give THEIR business the edge it needs.

Furthermore, our people keep the data center interesting. SoftLayer sees no shortage of antics. There’s John’s fully automatic Nerf gun. There’s also plenty of jokes played at the expense of someone unfortunate enough to leave their workstation unlocked (call it “security training” – favorite backgrounds include the Care Bears and My Little Pony). We also have that one hardware tech who likes to hide around corners or sneak up behind you, and scare the life out of you while you’re focused on the task at hand.

With so many surprises, SoftLayer continues to be a very interesting place to work, and most certainly a place where one would never get bored!