I am using getSibling to achieve this and I am unsure if this is the only way to attempt to not use $nin with writing mongodb shell script. I start out with this
//This runs for a 8-10 mins for 4Million docs
db = db.getSiblingDB("a")
akeys = db.getCollection("colla").aggregate([
{
$group: {
_id: "$_id"
}
}
]).toArray()
akeysarray = akeys.map(item => item._id)
['abc','a','d','erdfs',.......'anvdf']
I switch databases and I do the same
db = db.getSiblingDB("b")
bkeys = db.getCollection("collb").aggregate([
{
$group: {
_id: "$_id"
}
}
]).toArray()
akeysarray = akeys.map(item => item._id)
['abc','a',.......'anvdf']
Then I loop through with vanilla JavaScript because - it is the only way I can accomplish this in mongosh, but this is my problem area. It takes 5000 elements a minute and it will run at 16 hours or so over night.
difference = [];
arrays = []
size = 250000
// I have to splice the list of comparing arrays by 250000
// because it is very slow
while (bkeysarray.length > 0)
arrays.push(bkeysarray.splice(0, size))
difference = []
for (var idxo in arrays) {
for (var idx in arrays[i]) {
print("checking array set" + idxo + " with index " + idx + " and value " + arrays[idxo][idx])
var value = arrays[idxo][idx];
if (value == akeysarray[idxo]) {
continue;
}
if (akeysarray.indexOf(value) == -1) {
difference.push(value);
}
}
}
This will return 28K elements in an array and I want to use this in an $in statement to use the indexing for finding.
Please let me know if there is another way that can achieve a cross database _id comparison that is more performant than this that can only run in mongosh due to environment limitations with the client. I am aware it can be done with a driver in less than few seconds - but I can only run this script in mongosh.