Posts Tagged 'Storage'

October 4, 2011

An Introduction to Redis

I recently had the opportunity to get re-acquainted with Redis while evaluating solutions for a project on the Product Innovation team here at SoftLayer. I'd actually played with it a couple of times before, but this time it "clicked." Or my brain broke. Either way, I see a lot of potential for Redis now.

No one product is a perfect fit for all of your data storage needs, of course. There are such fundamental tradeoffs to be made in designing storage architectures that you should be immediately suspicious of any product that claims to fit every need.

The best solutions tend to be products that actually embrace these tradeoffs. Redis, for instance, has sacrificed a small amount of data durability in exchange for being awesome.

What is it?

Redis is a key/value store, but describing it that way is sort of like calling a helicopter a "vehicle." It's a technically correct description, but it leaves out some important stuff.

You can think of it like a sophisticated older brother of Memcached. It presents a flat keyspace, and you can set those keys to string values. Another feature of Memcached is the ability to perform remote atomic operations, like "incr" and "append." These are really handy, because you have the ability to modify remote data without fetching, and you have an assurance that you're the only one performing that operation at that instant.

Redis takes this concept of remote commands on data and goes completely nuts with it. The database is aware of data structures like hashes, lists and sets in addition to simple string values. You can sort, union, intersect, slice and dice to your heart's content without fetching any data. Redis is a data structure server. You can treat it like remote memory, and this has an awesome immediate benefit for a programmer: your code and brain are already optimized for these data types.

But it's not just about making storage simpler. It's fast, too. Crazy fast. If you make intelligent use of its data structures, it's possible to serve a lot of traffic from relatively modest hardware. Redis 2.4 can easily handle ~50k list appends a second on my notebook. With batching, it can append 2 million items to a list on a remote host in about 1.28 seconds.

It allows the remote, atomic and performant manipulation of data structures. It took me a little while to realize exactly how useful that is.

What's wrong with it?

Nothing. Move along.

OK, it's a little short on durability. Redis uses memory as its primary store and periodically flushes to disk. A common configuration is to do so every second.

That sounds pretty reasonable. If a server goes down, you could lose a second of data. Keep in mind, however, how many operations Redis can perform in a second. If you're in a high-volume environment, that could be a lot of data. It's not for your financial transactions.

It also supports relatively limited availability options. Currently, it only supports master/slave replication. Clustering support is planned for an upcoming release. It's looking pretty powerful, but it will take some real-world testing to know its performance impact.

These challenges should be taken into consideration, and it's probably clear if you're in a situation where the current tradeoffs aren't a good fit.

In my experience, a lot of developers seriously overestimate the consequences of their application losing small amounts of data. Also consider whether or not the chance of losing a second (or less) of data genuinely represents a bigger threat to your application than any other compromises you might have made.

More Information
You can check out the slightly aging docs or browse the impressively simple source. There are probably already bindings for your language of choice as well.

-Tim

July 25, 2011

Under the Hood of 'The Cloud'

When we designed the CloudLayer Computing platform, our goal was to create an offering where customers would be able to customize and build cloud computing instances that specifically meet their needs: If you go to our site, you're even presented with an opportunity to "Build Your Own Cloud." The idea was to let users choose where they wanted their instance to reside as well as their own perfect mix of processor power, RAM and storage. Today, we're taking the BYOC mantra one step farther by unveiling the local disk storage option for CloudLayer computing instances!

Local Disk

For those of you familiar with the CloudLayer platform, you might already understand the value of a local disk storage option, but for the uninitiated, this news presents a perfect opportunity to talk about the dynamics of the cloud and how we approach the cloud around here.

As the resident "tech guy" in my social circle, I often find myself helping friends and family understand everything from why their printer isn't working to what value they can get from the latest and greatest buzzed-about technology. As you'd probably guess, the majority of the questions I've been getting recently revolve around 'the cloud' (thanks especially to huge marketing campaigns out of Redmond and Cupertino). That abstract term effectively conveys the intentional sentiment that users shouldn't have to worry about the mechanics of how the cloud works ... just that it works. The problem is that as the world of technology has pursued that sentiment, the generalization of the cloud has abstracted it to the point where this is how large companies are depicting the cloud:

Cloud

As it turns out, that image doesn't exactly illicit the, "Aha! Now I get it!" epiphany of users actually understanding how clouds (in the technology sense) work. See how I pluralized "clouds" in that last sentence? 'The Cloud' at SoftLayer isn't the same as 'The Cloud' in Redmond or 'The Cloud' in Cupertino. They may all be similar in the sense that each cloud technology incorporates hardware abstraction, on-demand scalability and utility billing, but they're not created in the same way.

If only there were a cloud-specific Declaration of Independence ...

We hold these truths to be self-evident, that all clouds are not equal, that they are endowed by their creators with certain distinct characteristics, that among these are storage, processing power and the ability to serve content. That to secure these characteristics, information should be given to users, expressed clearly to meet the the cloud's users;

The Ability to Serve Content
Let's unpack that Jeffersonian statement a little by looking at the distinct characteristics of every cloud, starting with the third ("the ability to serve content") and working backwards. Every cloud lives on hardware. The extent to which a given cloud relies on that hardware can vary, but at the end of the day, you &nash; as a user – are not simply connecting to water droplets in the ether. I'll use SoftLayer's CloudLayer platform as a specific example of that a cloud actually looks like: We have racks of uniform servers – designated as part of our cloud infrastructure – installed in rows in our data centers. All of those servers are networked together, and we worked with our friends at Citrix to use the XenServer platform to tie all of those servers together and virtualize the resources (or more simply: to make each piece of hardware accessible independently of the rest of the physical server it might be built into). With that infrastructure as a foundation, ordering a cloud server on the CloudLayer platform simply involves reserving a small piece of that cloud where you can install your own operating system and manage it like an independent server or instance to serve your content.

Processing Power
Understanding the hardware architecture upon which a cloud is built, the second distinct characteristic of every cloud ("processing power") is fairly logical: The more powerful the hardware used for a given cloud, the better processing performance you'll get in an instance using a piece of that hardware.

You can argue about what software uses the least resources in the process of virtualizing, but apples-to-apples, processing power is going to be determined by the power of the underlying hardware. Some providers try to obfuscate the types of servers/processors available to their cloud users (sometimes because they are using legacy hardware that they wouldn't be able to sell/rent otherwise), but because we know how important consistent power is to users, we guarantee that CloudLayer instances are based on 2.0GHz (or faster) processors.

Storage
We walked backward through the distinct characteristics included in my cloud-specific Declaration of Independence because of today's CloudLayer Computing storage announcement, but before I get into the details of that new option, let's talk about storage in general.

If the primary goal of a cloud platform is to give users the ability to scale instantly from 1 CPU of power to 16 CPUs of power, the underlying architecture has to be as flexible as possible. Let's say your cloud computing instance resides on a server with only 10 CPUs available, so when you upgrade to a 16-CPU instance, your instance will be moved to a server with enough available resources to meet your need. To make that kind of quick change possible, most cloud platforms are connected to a SAN (storage area network) or other storage device via a back-end network to the cloud servers. The biggest pro of having this setup is that upgrading and downgrading CPU and RAM for a given cloud instance is relatively easy, but it introduces a challenge: The data lives on another device that is connected via switches and cables and is being used by other customers as well. Because your data has to be moved to your server to be processed when you call it, it's a little slower than if a hard disk was sitting in the same server as the instance's processor and RAM. For that reason, many users don't feel comfortable moving to the cloud.

In response to the call for better-performing storage, there has been a push toward incorporating local disk storage for cloud computing instances. Because local disk storage is physically available to the CPU and RAM, the transfer of data is almost immediate and I/O (input/output) rates are generally much higher. The obvious benefit of this setup is that the storage will perform much better for I/O-intensive applications, while the tradeoff is that the setup loses the inherent redundancy of having the data replicated across multiple drives in a SAN (which, is almost like its own cloud ... but I won't confuse you with that right now).

The CloudLayer Computing platform has always been built to take advantage of the immediate scalability enabled by storing files in a network storage device. We heard from users who want to use the cloud for other applications that they wanted us to incorporate another option, so today we're happy to announce the availability of local disk storage for CloudLayer Computing! We're looking forward to seeing how our customers are going to incorporate cloud computing instances with local disk storage into their existing environments with dedicated servers and cloud computing instances using SAN storage.

If you have questions about whether the SAN or local disk storage option would fit your application best, click the Live Chat icon on SoftLayer.com and consult with one of our sales reps about the benefits and trade-offs of each.

We want you to know exactly what you're getting from SoftLayer, so we try to be as transparent as we can when rolling out new products. If you have any questions about CloudLayer or any of our other offerings, please let us know!

-@nday91

August 3, 2010

How Clouds Killed The PC

Most days, it seems that technology progresses all too slowly. It is a different feeling when you work with cutting edge technology on a daily basis: deploying the first dual network datacenter infrastructure, being entrenched in solutions for everything from CDN to ISCI to DTS and more, testing the latest enterprise solutions from leading industry vendors long before money could buy them… it never really meant a whole lot to me; it was very much just, “How we roll”, as the gang would say.

But every so often, there is a day when a new technology catches my attention and reminds me why I got involved in the IT industry. Something that reminds me of the days spent tapping out QuickBasic 2.0 applications on my 18MHz 386 and 16 color EGA monitor. Surprisingly, the rise of cloud computing did just that. There was a day some still remember, when the cost of localized hardware was significant enough that terminals ruled the world. Occasionally, you may still see one at a grocery checkout stand or being used in a retail stockroom to check inventory across locations. Early terminals were commonly thin clients lacking a processor, non-volatile user storage, and only possessing enough memory to display what was on the screen at any given time. As the cost of memory declined, fat clients gained some popularity offering locally programmable memory. However, the concept was still the same: one host machine, usually a mainframe, serving applications over a distance to multiple (less capable) client machines.

Terminals were not destined to last though. In a twist of irony one of the innovations that they helped to inspire, the microprocessor, combined with the falling price and increased capacity of memory eventually led the decline of terminals. Left behind, in a cloud of dust, by hardware manufacturer’s race for speed capacity combined with advances in networking technology, the terminal PC became a historical relic looked upon as a necessary stop-gap solution used in the days when hardware was just too-darn-expensive. It was at that time the truly personal computer that we know and love was born and has forever-since reigned supreme. Then came the ARPANET, which gave way to the Information Super Highway, gave way to the World Wide Web, gave way to the internet we know today.

Mainframes gave way to servers. And today, I walk into a datacenter surrounded by servers boasting quad octo-core processors and Cloud Computing Instances, talking to customers who use their smart-phones to remotely access their web hosts, and quietly thinking to myself, “Have things really changed?” How far off is the day, when the benefits of remotely hosted applications outweigh the benefits of localized hardware? When we sit at the start of a new era where CCI’s can be created in minutes, regularly imaged for data security, migrated and restored quickly in the event of hardware failure, accessed from anywhere and from a variety of client hardware and software implementations, how much more would it take for us to return to the days of terminal PC’s. As bandwidth continues to improve, purchase and operational costs per processing core continues to fall, people demand more and more ‘anywhere access’, open source gains popularity and the idea of renting freely upgraded applications becomes accepted outside of the IT community, who knows what the future might hold. In a future where the concept of parallel uplinks may be no more foreign than that of parallel data transfer over CAT6 is to the layman, I wonder if personal computers will be thought of as the necessary stop-gap solution used while we waited for bandwidth to catch up to useable processing power; nothing more than a dinosaur that gave way to the green-movement and our need to be connected everywhere.

While I work on bringing my head out of the clouds, I remember why I am here. I am not here because technology’s past was all that fantastic, or because the present is all that glamorous, but because the future is still wide open. Whether-or-not clouds ever really kill the PC is anyone’s guess and only time will tell. However, one thing is currently known, as companies continue to see the benefit of having their staff conduct business through a web-portal interface, consumers continue trying to figure out what they are going to do with the extra two or three of the four cores they have, and the cost-to-performance ratio associated with remote resources continues to fall, we are steadily moving that way.

Categories: 
June 8, 2010

Core Construction

While constructing a storage building with a friend of mine I came to several realizations the first and less important was I really didn’t know what I was doing and was glad my friend was there. The next and most important thing was that the further into the construction we got the more I noticed from looking at the structure of the building from top to bottom there was a definite change in the need for structural support. I know what you are thinking at this point well duh, but for some reason when you actually step back and look at the framed structure the way that each layer depends on the previous is even more pronounced. A perfect example of this concept can be seen in every human and most animals of the world for as you peel back layers you eventually end up at a core that is the skeletal structure.

Something to ponder on when considering the layout of your IT infrastructure is the software is only as good as the under lying server and the server is only as good as the under lying network which is intern only as good as the power that feeds it and so for and so on. This is the core reason that here at SoftLayer we strive to provide the best we can from the ground up to help ensure our clients that they have the tools and foundation to compete in the world of online services.

October 19, 2009

I have backups…Don’t I?

There is some confusion out there on what’s a good way to back up your data. In this article we will go over several options for good ways to backup and sore your backups along with a few ways that are not recommended.

There is some confusion out there on what’s a good way to back up your data. In this article we will go over several options for good ways to backup and sore your backups along with a few ways that are not recommended.

When it comes to backups storing them off site (off your server or on a secondary drive not running your system) is the best solution with storing them off site being the recommended course.

When raids come into consideration just because the drives are redundant (a lave mirror situation) there are several situations, which can cause a complete raid failure such as the raid controller failing, the array developing a bad stripe. Drive failure on more than one drive(this does happen though rarely) , out of date firmware on the drives and the raid card causing errors. Using a network storage device like our evault or a nas storege is also an excellent way to store backups off system. The last thing to consider is keeping your backups up to date. I suggest making a new back every week at minimum (if you have very active sites or data bases I would recommend a every other day backup or daily backup). It is up to you or your server administrator to keep up with your backups and make sure they are kept up to date. If you have a hardware failure and your backups are well out of date it’s almost like not having them at all.

In closing consider the service you provide and how your data is safe, secure, and recoverable. These things I key to running a successful server and website.

May 13, 2009

The Data Center is Full of Surprises

After having been in the IT industry in some form or fashion for the last decade or so, I’ve learned that no matter how well you prepare yourself for disaster, you never seem to be surprised by certain issues that present themselves. Yes, ladies and gentlemen, I’m talking about the many surprises our friend Mr. Murphy can throw at us. I’m sure many a tech will anecdotally speak of the time where their server borked on them, their backups failed despite numerous backup audits, and they were up the infamous creek (I’m only assuming at least a few readers are nodding right now). Sometimes painful lessons are the best times to learn, but it’s a bad day when it happens on a production server.

Working in the SoftLayer data center, we take incredible measures to protect our customer’s servers. In a sense, we try to keep Mr. Murphy away. From the biggies (like redundant power and MASSIVE cooling units) to the routine (such as the data center walkthroughs, and proactive RAID alerts), we do our best to keep the servers in the data center running smoothly, and free of surprises.

Beyond the punches our friend Mr. Murphy can throw at us now and again, it’s nice to know there are a few good surprises in store for you, too. You might be surprised at the great deals our SLales team can provide. You might also be surprised at not only the amazing features such as the new Cloudlayer™ Storage, but the incredible rate we keep bringing new features to the table. I’ve also seen customer’s surprise when we rescue their server from the brink of disaster, or when we are able to provide a few tweaks to give THEIR business the edge it needs.

Furthermore, our people keep the data center interesting. SoftLayer sees no shortage of antics. There’s John’s fully automatic Nerf gun. There’s also plenty of jokes played at the expense of someone unfortunate enough to leave their workstation unlocked (call it “security training” – favorite backgrounds include the Care Bears and My Little Pony). We also have that one hardware tech who likes to hide around corners or sneak up behind you, and scare the life out of you while you’re focused on the task at hand.

With so many surprises, SoftLayer continues to be a very interesting place to work, and most certainly a place where one would never get bored!

April 27, 2009

Into the Cloud

You'll see the word "cloud" bouncing around quite a bit in IT nowadays. If you have been following The Inner Layer you'll have seen it a few times here as well. A cloud service is just something that is hosted on the Internet. Typically in a cloud scenario you are not actually doing the hosting but rather using hosted resources someone else is providing. Usually you'll hear it in terms of computing and storage.

This is going to be a brief article on a cloud storage product we are doing here at SoftLayer called CloudLayer™ Storage.

CloudLayer™ Storage is a WebDAV based system which uses client software on the local machine in order to redirect filesystem operations to the storage repository here at SoftLayer. In Windows you end up with a drive letter; on a Unix system you end up with a mount point. In both cases when you create folders and save files on those locations the actions actually happen on our storage repository here. Because the files are located with us you are able to access them wherever you are. Through the CloudLayer™ web UI you're able to also set up files for sharing with others so even if where you are never changes there is still value to using the cloud storage.

Even using a cloud storage system you must maintain proper backups. Hardware, software and human error all happens. Tied in with that concept of "errors happen" ... if you have files on CloudLayer™ that you need for a presentation, download them ahead of time. You don't want to be caught without files simply because the Internet connection at your hotel decided to take a nap the morning of your event.

Now what of security? Well, the connection to the CloudLayer™ endpoint here at SoftLayer is done via an encrypted session so people cannot snoop on your transmissions. This does mean you need to allow 443/tcp outbound communications via your firewall but since that is the normal HTTPS port I'd imagine you already have it open. Within CloudLayer™ you can control with whom you share your files.

Since CloudLayer™ is a filesystem redirected over the Internet the performance you get will be dependent on your local connection speed. Its best to treat CloudLayer™ Storage as simply a file repository. If you find you need some kind of off-machine storage for running applications on your server you could look into our iSCSI product.

So, dear readers, go forth and have a cloudy day.

Subscribe to storage