Overview:

We are comparing performance of create/read/write/rw over two different architectures: Single Database vs Multiple Databases (15k-25k).

We prefer to use the Multi-DB architecture because that makes it easier to separate customers (customer = 1 company). However, due to performance degradation we fear this may not be a good solution.

Server Specification:

Single instance MongoDB server; 64GB RAM; 16 core; SSD HD

Test results:

Both test scenarios have the same total number of documents (and documents are roughly the same size). The variables are number of databases, collections per database and documents per collection.

All tests are conducted in parallel using 50 client threads (separate machine), with the exception of Read/Write, which uses 100 (50R/50W). 'directoryPerDB' is enabled. (All times are in milliseconds per doc operation)

Test                      Creation  Read     Write    Read/Write   Notes
25000 DB 4 Coll 250 Doc   23ms      1-10ms   1-4ms    2-10ms       Max 1400% CPU, noticeable "pauses" (CPU drops to 100%)
15000 DB 4 Coll 420 Doc   23ms      0.7-4ms  0.9-4ms  2-9ms        Max 1400% CPU, noticeable "pauses" (CPU drops to 100%)
1 DB 4 Coll 125000 Doc    0.8ms     0.6ms    0.8ms    1.2-1.6ms    Max 600% CPU, no pauses

Conclusion:

There seems to be noticeable performance degradation on a regular interval when the DB count is very high. It may be due to the sheer number of files (25000 DBs * 4 Colls * 2 files = 200k files) or some other bottleneck.

In the Single-DB test, the CPU stays around 600% and maintains that until completion. In the multi DB tests, the CPU (at peak performance) is somewhere between 800-1400%, but every so often the CPU drops to 100% and all operations are paused. This can be verified by watching the mongo log, as well as the logs from the test clients that are issuing R/W commands.

If it wasn't for these pauses, the Multi-DB architecture would be ~2x faster than Single-DB, however it appears there is some global contention that cannot be avoided.

I'm hoping someone might know what this global contention is and (if possible) how to solve it.