Developers can be cynical. When “the next great thing in technology” is announced, I usually wait to see how it performs before I get too excited about it … Show me how that “next great thing” compares apples-to-apples with the competition, and you’ll get my attention. With the launch of MongoDB at SoftLayer, I’d guess a lot of developers outside of SoftLayer and 10gen have the same “wait and see” attitude about the new platform, so I put our new MongoDB engineered servers to the test.
When I shared MongoDB architectural best practices, I referenced a few of the significant optimizations our team worked with 10gen to incorporate into our engineered servers (cheat sheet). To illustrate the impact of these changes in MongoDB performance, we ran 10gen’s recommended benchmarking harness (freely available for download and testing of your own environment) on our three tiers of engineered servers alongside equivalent shared virtual environments commonly deployed by the MongoDB community. We’ve made a pretty big deal about the performance impact of running MongoDB on optimized bare metal infrastructure, so it’s time to put our money where our mouth is.
The Testing Environment
For each of the available SoftLayer MongoDB engineered servers, data sets of 512kb documents were preloaded onto single MongoDB instances. The data sets were created with varying size compared to available memory to allow for data sets that were both larger (2X) and smaller than available memory. Each test also ensured that the data set was altered during the test run frequently enough to prevent the queries from caching all of the data into memory.
Once the data sets were created, JMeter server instances with 4 cores and 16GB of RAM were used to drive ‘benchrun’ from the 10gen benchmarking harness. This diagram illustrates how we set up the testing environment (click for a better look):
These Jmeter servers function as the clients generating traffic on the MongoDB instances. Each client generated random query and update requests with a ratio of six queries per update (The update requests in the test were to ensure that data was not allowed to fully cache into memory and never exercise reads from disk). These tests were designed to create an extreme load on the servers from an exponentially increasing number of clients until the system resources became saturated, and we recorded the resulting performance of the MongoDB application.
At the Medium (MD) and Large (LG) engineered server tiers, performance metrics were run separately for servers using 15K SAS hard drive data mounts and servers using SSD hard drive data mounts. If you missed the post comparing the IOPS statistics between different engineered server hard drive configurations, be sure to check it out. For a better view of the results in a given graph, click the image included in the results below to see a larger version.
Test Case 1: Small MongoDB Engineered Servers vs Shared Virtual Instance
Single 4-core Intel 1270 CPU
2 x 500GB SATAII – RAID1
4 Virtual Compute Units
2 x 500GB Network Storage – RAID1
200 iterations of 6:1 query-to-update operations
Concurrent client connections exponentially increased from 1 to 32
Test duration spanned 48 hours