By far when running ‘top’, iostat, and mongostat during these tests you could observe that the limiting factor on both platforms ended up being storage I/O. If the tests performed were query-only centric, then you are 100% correct that these tests would have been heavily skewed and the data fairly worthless. .5gb and 2gb differences in memory size given that overall data set size and the use of updating records while querying did not have a significant impact on these tests. Even given the larger difference in the final offering (once again the virtual provider just didn’t have an offering that matched) doesn’t indicate that the RAM had as much of an impact. If it had, it would have shown as a large skewing to the query results only which doesn’t appear to be the case in the numbers. Once again, this is because care was taken to avoid allowing MongoDB to cache too much data into memory.

I hope that helps, it was a little challenging to line up the two platforms.

You can obviously also create seperate groups in Ceph of different types of devices, like SSD- and HDD-backed group.

Ceph basically has a similair setup as Lustre (possibly Ceph has a superior design) and Lustre is in use for a number of top500 systems, so at least in theory it should scale pretty much as far as the budget can take you. As Ceph is fairly new, I’m sure their are still some limits of course.

The largest part of the performance hit comes from having a network hop to the network attached storage. I have done some personal testing on our CCI’s with local storage and tests where the data was completely cached in memory on CCI’s with remote storage. In both cases MongoDB performed well. Now to be fair, I didn’t do a local storage test on the other vendor (it wasn’t available), but given our own internal testing the data indicates that the majority of the performance hit is around the I/O to the storage. In addition to being a network hop, most network attached storage solutions don’t have the ability to offer QOS per mount. This makes a shared environment with network storage not only less performant in most cases, it also makes the deviation in performance unpredictable.

The compute side of the equation really comes into play when dealing with data sets smaller than available memory where MongoDB can cache the entire working set. When this happens, the bus speed/clock speed of the compute resources becomes the limiting factor and is no longer the storage I/O. In these cases obviously if your provider can clearly tell you what kind of CPU architecture you are running on, you can make a sound decision on if the platform will meet your needs. Raw compute power only matters (we are talking cores and architecture now) when you are doing map-reduce aggregation operations. These seem to be the only thing thing that begins to utilize CPU over the other resources.

On the flash cache, I would love to find a provider that would allow such an option. You could always “roll your own” and just have a local mounted SSD available. We haven’t tested any flash cache setups ourselves yet, but in theory that would help alleviate some of the I/O issues if your working set was smaller than the cache. I think if I had that option though, I would just mount the local SSD as the data drive itself and be done with it. In certain cases though where your total data set might be large but your working data set is smaller than the cache SSD drive it might be advisable to try out a flash cache setup and see if it helped. It would be fun to test it regardless!

My first question would be: what about non-virtual with shared network-attached storage ?

How does that perform ? To compare what hit you take by using the virtualisation and if the performance variance comes from the virtualisation or the shared network-attached storage.

What if you add a SSD as cache at the host that does the virtualisation (think flashcache or bcache) ? Does that solve a large part of the problem ?