I currently have a MongoDB (on AWS) backup strategy involving automating daily snapshots of the data/journal volume on one node. The process is:
When replacing a RS member, I launch a new instance from the snapshot and add it to the replica set. The sole issue I have is on mongo process launch there is a very long delay (~20-30 minutes for ~500GB of data) before the new node jumps into SECONDARY and replays from another member's oplog. All I see is something like this in the mongodb.log file:
[initandlisten] Starting WiredTigerRecordStoreThread local.oplog.rs
[initandlisten] The size storer reports that the oplog contains 659681 records totaling to 4196461664 bytes
[initandlisten] Sampling from the oplog between Dec 20 05:29:14:2 and Jan 18 06:00:00:8 to determine where to place markers for truncation
[initandlisten] Taking 335 samples and assuming that each section of oplog contains approximately 19672 records totaling to 125140475 bytes