I am querying data from mongodb in chunks having 100M + records in total. And then want to process those records within a function.
My query
cursor = collection.find({'facial_task':False}).sort("_id", -1).skip(1000000).limit(10000)
When I iterate over the cursor, then it is taking too much time and gets stuck a lot even I limit the number of results up to 10. I even retrieved the results using batch size but when processing the loop over it; it takes too much time.
My loop is like this
for dd in cursor:
ab = threading.Thread(target=insert_func, args=(dd,))
ab.start()
main_threads.append(ab)
if len(main_threads) >= 5000:
print("****Joining Main Thread***")
for ii in main_threads:
ii.join()
main_threads = []
If the slowdown occurs from the very first iteration, then it must have to do with the cursor itself.
Mongodb cursor is not like a 'chached' list of the entire collection; it queries the db one chunk at a time as you iterate over it. You can try to cast that cursor into a list and see if the iteration shows a significant speed-up.