We have a server running on express js with a database MongoDB accessing with the framework mongoose. The application is in production and we are facing the issue of APIs taking too long to respond sometimes they even get timed out and it is completely random and it happens for just 10-15 requests in a day. We have an AWS loader balancer and the server is hosted on 2 instances and MongoDB is hosted on a separate instance. We have put logs on the required places in the server. Firstly I thought it was an issue of TCP/IP connection so I configured keepAliveTimeout and headersTimeout in the application(Reference) but that too didn't help.
I tried to analyze the logs and found something the API comes to the server but it doesn't start processing for 5-10 seconds and sometimes it processes everything in a second but doesn't respond back. I even tried to replicate the issue on DEV through apache JMeter but the issue isn't replicating. I am thinking this might be due to node js being single-threaded but there there are so many applications running on node js. How are they managing such scenarios? Also, we have millions of records in the database so could they create an issue? By the way we have properly indexed the fields. Please guys help me out with this issue.
I assume that in beginning you are not facing this issue, right? and after data increase, you are facing an issue and timeout happens because API does not give response
according to my above assumptions you may have an issue with MongoDB when MongoDB have a huge dataset, my suggestion is to check your MongoDB operations when your request is being timeout your MongoDB will in load, to check MongoDB process
db.currentOp()
in mongo shell or any editor where you can run it
and check which process running through a long time resolve it and watch