0
399
ViewsAWS Beanstalk - Share Database layer between worker and webserver
I am building some web application using NodeJS. I deployed the application to AWS beanstalk web server and I decoupled some heavy duty tasks, And deployed the heavy tasks to different AWS beanstalk workers, The main web server is executing the worker with SQS (aws queue).
The main web server has a big DB layer to connect, get and save to the database. The workers do need some db actions, every worker type need different actions that already implemented in the DB layer of the main web server.
So I'm trying to find a way to use the DB layer of the web server without duplicate it in every worker type (that's what I'm doing now).
What I thought is to decouple the DB layer from the main application and to deploy it to a web server to it self, but that's looks like a waste because I don't need to listen to requests in DB layer all the time, I need some on demand actions.
I attached a picture of how it looks:
- number 1 the main server (with full db layer)
- number 2 the queue
- number 3 the workers. (with duplicated db actions)
Is there any good solution to it?
2 answers
Answer question0
Just use native RDS integration with your Beanstalk environment: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.managing.db.html
0
How many save/load operations are you doing from workers? How many workers are running at the same time? I think it's hard to suggest a single solution without understanding how many workers are there and how much data will be saved.
In general, I can think of multiple solutions actually:
You can open the API on the web server, so once the worker finishes, it will call the web server. If you have many workers, it won't work, so you may need intermediate workers (let's call it the "saved worker"), so the current workers will queue the "saved worker", then the "saved worker" will call the API of the web server, this will ensure that the web server is not overloaded with save operations.
You can create a duplicate instance of the web server (let's call it "worker web server") which is for saving the worker results, so the workers will send the result to a new SQS, on this "worker web server" it will be listening to save requests in the SQS. You can control which mode the web server will work in by setting the environment variable, for example.
Maybe you can extract the database layer into a package or library that is used in the web server plus workers as well, but you have to be careful not to open many parallel connections to the database or do a lot of db operations at the same time.