0
131
ViewsLarge Scale Webscraping
I've been developing webscraping software for awhile for specific sites and I'm looking to take it to the next level on a large scale. I've got the funds ready and I'm trying to figure out the best way to proceed.
Here's a Diagram below of what the layout of processes looks like
In my current status, What I have been doing is just loading the scraper javascript files on a dedicated server that I can remote into and running them from the terminal there (which is probably such an ineffecient method)
So, I'm looking for advice as to how I can best run lets say 100+ site scrapers so that they all have maximum performance.
Such as would this be done through something Like AWS and doing something specifically?
I'm honestly just really stuck and want to know how to proceed and optimise as I build for the future.
The scrapers are written in Javascript so it would need to be something JS friendly.
I'm not looking for a webscraping software, I'm looking for a way to host / deploy my code in the best manner.
1 answers
Answer question0
What looks like you are trying to migrate your local service architecture to some cloud based to leverage their scale.
If you plan to utilize AWS, would suggest using its CDK to create your infrastructure and utilize AWS Lambda for your scrapers. https://docs.aws.amazon.com/cdk/v2/guide/home.html
There are some blogs on AWS for the solution of web scraping architecture which you can use for reference. Example : https://aws.amazon.com/blogs/architecture/serverless-architecture-for-a-web-scraping-solution/
Much better would be to convert your code to docker images, which then could be easily used on multiple cloud services, making it easier to move around the vendors to satisfy your needs. In AWS, it will be deployed via ECS.