When I shared MongoDB architectural best practices, I referenced a few of the significant optimizations our team worked with 10gen to incorporate into our engineered servers (cheat sheet). To illustrate the impact of these changes in MongoDB performance, we ran 10gen’s recommended benchmarking harness (freely available for download and testing of your own environment) on our three tiers of engineered servers alongside equivalent shared virtual environments commonly deployed by the MongoDB community. We’ve made a pretty big deal about the performance impact of running MongoDB on optimized bare metal infrastructure, so it’s time to put our money where our mouth is.

The Testing Environment

For each of the available SoftLayer MongoDB engineered servers, data sets of 512kb documents were preloaded onto single MongoDB instances. The data sets were created with varying size compared to available memory to allow for data sets that were both larger (2X) and smaller than available memory. Each test also ensured that the data set was altered during the test run frequently enough to prevent the queries from caching all of the data into memory.

Once the data sets were created, JMeter server instances with 4 cores and 16GB of RAM were used to drive ‘benchrun’ from the 10gen benchmarking harness. This diagram illustrates how we set up the testing environment (click for a better look):

These Jmeter servers function as the clients generating traffic on the MongoDB instances. Each client generated random query and update requests with a ratio of six queries per update (The update requests in the test were to ensure that data was not allowed to fully cache into memory and never exercise reads from disk). These tests were designed to create an extreme load on the servers from an exponentially increasing number of clients until the system resources became saturated, and we recorded the resulting performance of the MongoDB application.

At the Medium (MD) and Large (LG) engineered server tiers, performance metrics were run separately for servers using 15K SAS hard drive data mounts and servers using SSD hard drive data mounts. If you missed the post comparing the IOPS statistics between different engineered server hard drive configurations, be sure to check it out. For a better view of the results in a given graph, click the image included in the results below to see a larger version.

Test Case 1: Small MongoDB Engineered Servers vs Shared Virtual Instance

Servers

Tests Performed

Average Read Operations per Second
by Concurrent Client

Peak Read Operations per Second
by Concurrent Client

Average Write Operations per Second
by Concurrent Client

Peak Write Operations per Second
by Concurrent Client

Test Case 2: Medium MongoDB Engineered Servers vs Shared Virtual Instance

Servers (15K SAS Data Mount Comparison)

Tests Performed

Average Read Operations per Second
by Concurrent Client

Peak Read Operations per Second
by Concurrent Client

Average Write Operations per Second
by Concurrent Client

Peak Write Operations per Second
by Concurrent Client

Servers (SSD Data Mount Comparison)

Tests Performed

Average Read Operations per Second
by Concurrent Client

Peak Read Operations per Second
by Concurrent Client

Average Write Operations per Second
by Concurrent Client

Peak Write Operations per Second
by Concurrent Client

Test Case 3: Large MongoDB Engineered Servers vs Shared Virtual Instance

Servers (15K SAS Data Mount Comparison)

Tests Performed

Average Read Operations per Second
by Concurrent Client

Peak Read Operations per Second
by Concurrent Client

Average Write Operations per Second
by Concurrent Client

Peak Write Operations per Second
by Concurrent Client

Servers (SSD Data Mount Comparison)

Tests Performed

Average Read Operations per Second
by Concurrent Client

Peak Read Operations per Second
by Concurrent Client

Average Write Operations per Second
by Concurrent Client

Peak Write Operations per Second
by Concurrent Client

Impressions from Performance Testing

The results speak for themselves. Running a Mongo DB big data solution on a shared virtual environment has significant drawbacks when compared to running MongoDB on a single-tenant bare metal offering. Disk I/O is by far the most limiting resource for MongoDB, and relying on shared network-attached storage (with much lower disk I/O) makes this limitation very apparent. Beyond the average and peak statistics above, performance varied much more significantly in the virtual instance environment, so it’s not as consistent and predictable as a bare metal.

Highlights:

• When a working data set is smaller than available memory, query performance increases.
• The number of clients performing queries has an impact on query performance because more data is being actively cached at a rapid rate.
• The addition of a separate Journal Mount volume significantly improves performance. Because the Small (SM) engineered server does not include a secondary mount for Journals, whenever MongoDB began to journal, the disk I/O associated with journalling was disruptive to the query and update operations performed on the Data Mount.
• The best deployments in terms of operations per second, stability and control were the configurations with a RAID10 SSD Data Mount and a RAID1 SSD Journal Mount. These configurations are available in both our Medium and Large offerings, and I’d highly recommend them.

-Harold

The MongoDB engineered servers that you order from SoftLayer already implement several of the recommendations you’ll see below, and I’ll note which have been incorporated as we go through them. Given the scope of the topic, it’s probably easiest to break down this guide into a few sections to make it a little more digestible. Let’s take a look at the architectural best practices of running MongoDB through the phases of the roll-out process: Selecting a deployment strategy to prepare for your MongoDB installation, the installation itself, and the operational considerations of running it in production.

Deployment Strategy

When planning your MongoDB deployment, you should follow Sun Tzu’s (modified) advice: “If you know the [friend] and know yourself, you need not fear the result of a hundred battles.” “Friend” was substituted for the “enemy” in this advice because the other party is MongoDB. If you aren’t familiar with MongoDB, the top of your to-do list should be to read MongoDB’s official documentation. That information will give you the background you’ll need as you build and use your database. When you feel comfortable with what MongoDB is all about, it’s time to “know yourself.”

Your most important consideration will be the current and anticipated sizes of your data set. Understanding the volume of data you’ll need to accommodate will be the primary driver for your choice of individual physical nodes as well as your sharding plans. Once you’ve established an expected size of your data set, you need to consider the importance of your data and how tolerant you are of the possibility of lost or lagging data (especially in replicated scenarios). With this information in hand, you can plan and start testing your deployment strategy.

It sounds a little strange to hear that you should test a deployment strategy, but when it comes to big data, you want to make sure your databases start with a strong foundation. You should perform load testing scenarios on a potential deployment strategy to confirm that a given architecture will meet your needs, and there are a few specific areas that you should consider:

Installation

When performing the installation of MongoDB, a few considerations can help create both a stable and performance-oriented solution. 10gen recommends the use CentOS (64-bit) as the base operating system if at all possible. If you try installing MongoDB on a 32-bit operating system, you might run into file size limits that cause issues, and if you feel the urge to install it on Windows, you’ll see performance issues if virtual memory begins to be utilized by the OS to make up for a lack of RAM in your deployment. As a result, 32-bit operating systems and Windows operating systems should be avoided on MongoDB servers. SoftLayer provisions CentOS 6.X 64-bit operating systems by default on all of our MongoDB engineered server deployments.

When you’ve got CentOS 64-bit installed, you should also make the following changes to maximize your performance (all of which are included by default on all SoftLayer engineered servers):

One last tip on installation: Make the Journal and Data volumes be distinct physical volumes. If the Journal and Data directories reside on a single physical volume, flushes to the Journal will interrupt the access of data and provide spikes of high latency within your MongoDB deployment.

Operations

Once a MongoDB deployment has been promoted to production, there are a few recommendations for monitoring and optimizing performance. You should always have the MMS agent running on all MongoDB instances to help monitor the health and performance of your deployment. Additionally, this tool is also very useful if you have 10gen MongoDB Cloud Subscriptions because it provides useful debugging data for the 10gen team during support interactions. In addition to MMS, you can use the mongostat command (mentioned in the deployment section) to see runtime information about the performance of a MongoDB node. If either of these tools flags performance issues, sharding or indexing are first-line options to resolve them:

I know it seems like we’ve covered a lot over the course of this blog post, but this list of best practices is far from exhaustive. If you want to learn more, the MongoDB forums are a great resource to connect with the rest of the MongoDB community and learn from their experiences, and the documentation on MongoDB’s site is another phenomenal resource. The best people to talk to when it comes to questions about MongoDB are the folks at 10gen, so I also highly recommend taking advantage of MongoDB Cloud Subscriptions to get their direct support for your one-off questions and issues.

-Harold