I am using serverless on aws with nodejs and mongodb atlas as database
At the moment I am using the trial version which allow maximum 500 connections.
Seems that my code is not disconnecting the database when process end
I am using express to manage it
First I had no connection close thinking that the connection will be closed automatically once the process end but no I had a lot of connections open.
Then I added a middleware to close my connections after the response has been sent, it was not working, I was thinking that serverless was stopping the process once the response was sent.
Not on each route I am closing mongo connection, for example
router.get('/website/:id/page', async (req, res, next) => {
try {
const pages = await pageDataProvider.findByWebsite(req.params.id);
await mongodbDataProvider.close();
res.json(pages);
} catch (error) {
next(error)
}
})
This is how I handle connections with mongo
const MongoClient = require('mongodb').MongoClient
const config = require('../config')
const MONGODB_URI = config.stage === 'test' ?
global.__MONGO_URI__ :
`mongodb+srv://${config.mongodb.username}:${config.mongodb.password}@${config.mongodb.host}/admin?retryWrites=true&w=majority`;
const client = new MongoClient(MONGODB_URI);
let cachedDb = null;
module.exports.connect = async () => {
if (cachedDb) return cachedDb;
await client.connect();
const dbName = config.stage === 'test' ? global.__MONGO_DB_NAME__ : config.stage;
const db = client.db(dbName)
cachedDb = db;
return db;
}
module.exports.close = async () => {
if (!cachedDb) return;
await client.close();
cachedDb = null;
}
I do not understand why I have so many connections open
Isolate the call to the MongoClient.connect() function into its own module so that the connections can be reused across functions. Let's create a file mongo-client.js for that:
mongo-client.js:
const { MongoClient } = require('mongodb');
// Export a module-scoped MongoClient promise. By doing this in a separate
// module, the client can be shared across functions.
const client = new MongoClient(process.env.MONGODB_URI);
module.exports = client.connect();
Import the new module and use it in function handlers to connect to database.
some-file.js:
const clientPromise = require('./mongodb-client');
// Handler
module.exports.handler = async function(event, context) {
// Get the MongoClient by calling await on the connection promise. Because
// this is a promise, it will only resolve once.
const client = await clientPromise;
// Use the connection to return the name of the connected database for example.
return client.db().databaseName;
}
I think its a programmatic error in your close method. Please have a closer look at
if (!cachedDb) return;
I think it should have been
if (cachedDb != null) return;
As stated in other response, I would strongly advice against closing the DB connections with each request. You should be looking for a pool mechanism, where a connection from the pool is handed to your application. The application can wait till it receives the connection
Closure of the DB connections should be handled at the time when the application is exiting (shutting/going down). This way application will at least try to close the connections gracefully.
Nonetheless, here is an adaptation your program
index.js
const express = require('express')
const app = express()
const port = 3000
const dbProvider = require('./dbProvider');
dbProvider.connect();
app.get('/testConnection',async (req, res, next) => {
console.log('Doing something for fetching the request & closing connection');
dbProvider.close();
console.log('After closing the connection');
})
app.listen(port, () => {
console.log(`Example app listening at http://localhost:${port}`)
})
dbProvider.js
let cachedDb = null;
let db = {};
module.exports.connect = async () => {
if (cachedDb) {
console.log('Returning Cachedb');
return cachedDb;
}
else{
console.log('Not a cachedDB');
}
db.setup = 1;
return db;
}
module.exports.close = async () => {
if (!cachedDb) {
console.log('Since its cached DB not closing the connection');
return;
}
db=null;
return;
}
And here is the console output:
-> node index.js
Not a cachedDB
Example app listening at http://localhost:3000
Doing something for fetching the request & closing connection
Since its cached DB not closing the connection
After closing the connection
According to this: https://docs.atlas.mongodb.com/best-practices-connecting-from-aws-lambda/
It's a good idea to add this line so you keep your connection pool between requests.
context.callbackWaitsForEmptyEventLoop = false;