IOPS — Input/Output Operations Per Second — measure computer storage in terms of the number of read and write operations it can perform in a second. IOPS are a primary concern for database environments where content is being written and queried constantly, and when we take those database environments to the extreme (big data), the importance of IOPS can’t be overstated: If you aren’t able perform database reads and writes quickly in a big data environment, it doesn’t matter how many gigabytes, terabytes or petabytes you have in your database … You won’t be able to efficiently access, add to or modify your data set.

As we worked with 10gen to create, test and tweak SoftLayer’s MongoDB engineered servers, our primary focus centered on performance. Since the performance of massively scalable databases is dictated by the read and write operations to that database’s data set, we invested significant resources into maximizing the IOPS for each engineered server … And that involved a lot more than just swapping hard drives out of servers until we found a configuration that worked best. Yes, “Disk I/O” — the amount of input/output operations a given disk can perform — plays a significant role in big data IOPS, but many other factors limit big data performance. How is performance impacted by network-attached storage? At what point will a given CPU become a bottleneck? How much RAM should included in a base configuration to accommodate the load we expect our users to put on each tier of server? Are there operating system changes that can optimize the performance of a platform like MongoDB?

The resulting engineered servers are a testament to the blood, sweat and tears that were shed in the name of creating a reliable, high-performance big data environment. And I can prove it.

Most shared virtual instances — the scalable infrastructure many users employ for big data — use network-attached storage for their platform’s storage. When data has to be queried over a network connection (rather than from a local disk), you introduce latency and more “moving parts” that have to work together. Disk I/O might be amazing on the enterprise SAN where your data lives, but because that data is not stored on-server with your processor or memory resources, performance can sporadically go from “Amazing” to “I Hate My Life” depending on network traffic. When I’ve tested the IOPS for network-attached storage from a large competitor’s virtual instances, I saw an average of around 400 IOPS per mount. It’s difficult to say whether that’s “not good enough” because every application will have different needs in terms of concurrent reads and writes, but it certainly could be better. We performed some internal testing of the IOPS for the hard drive configurations in our Medium and Large MongoDB engineered servers to give you an apples-to-apples comparison.

Before we get into the tests, here are the specs for the servers we’re using:

The numbers shown in the table below reflect the average number of IOPS we recorded with a 100% random read/write workload on each of these engineered servers. To measure these IOPS, we used a tool called fio with an 8k block size and iodepth at 128. Remembering that the virtual instance using network-attached storage was able to get 400 IOPS per mount, let’s look at how our “base” configurations perform:

Clearly, the 400 IOPS per mount results you’d see in SAN-based storage can’t hold a candle to the performance of a physical disk, regardless of whether it’s SAS or SSD. As you’d expect, the “Journal” reads and writes have roughly the same IOPS between all of the configurations because all four configurations use 2 x 64GB SSD drives in RAID1. In both configurations, SSD drives provide better Data mount read/write performance than the 15K SAS drives, and the results suggest that having more physical drives in a Data mount will provide higher average IOPS. To put that observation to the test, I maxed out the number of hard drives in both configurations (10 in the 2U MD server and 34 in the 4U LG server) and recorded the results:

It should come as no surprise that by adding more drives into the configuration, we get better IOPS, but you might be wondering why the results aren’t “betterer” when it comes to the IOPS in the SSD drive configurations. While the IOPS numbers improve going from four to ten drives in the medium engineered server and six to thirty-four drives in the large engineered server, they don’t increase as significantly as the IOPS differences in the SAS drives. This is what I meant when I explained that several factors contribute to and potentially limit IOPS performance. In this case, the limiting factor throttling the (ridiculously high) IOPS is the RAID card we are using in the servers. We’ve been working with our RAID card vendor to test a new card that will open a little more headroom for SSD IOPS, but that replacement card doesn’t provide the consistency and reliability we need for these servers (which is just as important as speed).

There are probably a dozen other observations I could point out about how each result compares with the others (and why), but I’ll stop here and open the floor for you. Do you notice anything interesting in the results? Does anything surprise you? What kind of IOPS performance have you seen from your server/cloud instance when running a tool like fio?

-Kelly

Citrusleaf and SoftLayer: Taking NoSQL to the Next Level

Citrusleaf is the NoSQL OLTP (transaction-oriented) database behind some of the world’s largest advertising platforms. Our record of reliability and performance is the reason our customers choose us over any other database. We specialize in low-latency transactions on terabyte sized, billion-object databases. We fit well with analytics systems such as Hadoop or SQL-based “ETL” analytics architectures. Since Citrusleaf is fully reliable like a traditional database and has the speed of a cache, complexity is greatly reduced which leads to higher reliability and substantial cost savings.

Customers store actionable data for their internet applications on our platform. A typical use case is a server-side user data store. The advertising industry has moved to server-based user information storage as end users have become concerned about “tracking cookies” and other browser-side storage. Sophisticated advertising platforms are capable of associating users even after cookies have been cleared – through logins at partner sites, IP addresses and browser fingerprints. In the case where the user has elected not to be “tracked,” session management techniques allow “frequency capping” to limit the repetition of ads.

Citrusleaf’s clear strengths in the high-volume advertising use case can also be applied to other scenarios like online social gaming and internet scale applications. Our interface is simple: primary key lookups, an extension of the memcache interface with sophisticated eviction, document orientation, and database scanning. The result of the interface’s simplicity is that powerful applications can be written and maintained quickly and easily.

One of the keys to our technology is the rapid adoption of fast transactional storage. With a database that is capable of millions of transactions per second – 300,000 per node, and linearly scalable – configurations with SSD storage are the most cost efficient. Since SSDs can’t be provisioned on a minute-by-minute basis like traditional cloud infrastructure (the disks can take over 20 minutes to clear), SSDs fit well with a dedicated or managed hosting approach. SoftLayer has aggressively rolled out SSD capabilities in their data centers, and we work closely with them to source the most appropriate SSD models and vendors.

SoftLayer’s mixture of managed hosting, dedicated and cloud infrastructure in the same data center allows the rapid provisioning of database machines alongside the flexibility of a cloud system. That kind of flexibility and power is what we absolutely need to be successful, and we’re happy to have found it at SoftLayer.

If you’ve ever been interested to learn a little more about NoSQL and how it might reinvent the way you look at databases, visit http://citrusleaf.com to learn a little and let us know if you have any questions.

-Brian Bulkowski, Citrusleaf

This new area of exploration for me started shortly after my first days at SoftLayer when I was introduced to RAID (Redundant Array of Inexpensive Disks) for our servers. In the past, I had heard mention of the term but never had any idea of what that entailed and was only familiar with our good ole bug killer brand Raid. You can imagine my excitement as I learned more about its intricacies and how the different types of RAID could benefit my computer’s performance.

Armed with this new knowledge, I was determined to reconfigure my gaming pc at home to reap the benefits. Upon looking at the different RAID setups, I decided to go with a RAID 0 because I did not want to sacrifice storage space and my data was not critical enough that I would need a mirror such as provided with RAID 1.

One thing led to another as I became occupied for a good amount of time with benchmarking drive performance in my old setup versus my new setup. In the end, I was happy to report a significant performance gain in what I now refer to as my “killer setup”. Applications would launch noticeably faster and even in games where videos were stored locally on hard drives, the cinematic scenes would come up faster than before.

To add to the hype, a coworker was also building a new computer in anticipation of a new game called Final Fantasy XIV. It felt like a competition to exceed each other with better scores. I’m already planning ahead for future upgrades since this time around I had only used SATA drives. For my next upgrade I would love to run a RAID 0 with two SSD drives to see what kind of boost I would get.

So for business or pleasure, have you ever considered the benefits of setting up a RAID system?

-Danny

But alas, Thomas Norris stole my thunder! Check out his article “SSD: A Peek into the Future” for the complete skinny on the SSD’s we use. I seem to be a bit to concise for a nice long blog anyways. But not to worry, I’ve got some nifty numbers that will blow the jam out of your toes!

Solid State Drives (SSD) represent a large jump in drive performance. Not to mention smaller physical size, lower power consumption, and lower heat emissions. The majority of drive activity is random read/write. SSD drives have drastically improved in this area compared to mechanical drives. This results in a drastic overall performance increase for SSD drives.

This is a comparison of the Intel 32GB X25-E Extreme drive vs. other drives we carry. Note the massive jump in the random read/write speed of the SSD drive.

No more waiting on physical R/W heads to move around. How archaic!

Please note that no performance utility should be used to definitively judge a component or system. In the end, only real time usage is the final judge. But performance tests can give you a good idea of how a component or system compares to others.

Single drive performance increases directly translate into big improvements for RAID configurations as well. I have compared two of our fastest SATA and SAS four drive RAID 10 setups to a four drive SSD RAID 10 using an Adaptec 5405 Controller.

The Adaptec 5405 RAID controller certainly plays a part in the performance increase, on top on the simple speed doubling due to 2 drives being read simultaneously. (See my future blog on the basics or RAID levels, or check Wikipedia) .

Propeller heads read on:

The numbers indicate a multiplied increase if you take the base drive speed (Cheetah – 11.7mbps / X25-E – 64.8mbps) and double it (the theoretical increase a RAID 10 would give): 23.4mbps and 129.6mbps respectively. Actually performance tests show 27.3mbps and 208.1mbps. That means the Cheetahs are getting a 15% performance boost on random read/write and the X25-E a whopping 37% due to the RAID card. Hooray for math!

Once again, this is all performance tests and a bit of math speculation. The only real measure of performance, IMO, is how it performs the job you need it to do.