I'm trying to build a web crawler with NodeJS. I'm debating what is the most efficient way to implement the crawler for scale and high number of requests.

Let assume that we only send 1 request to crawl a webpage and all the URLs inside it, and repeat the process until X depth or Y URLs crawled.

I came up with 2 design options (worker-threads vs event loop):

1. A queue containing the URLs. For each URL in the queue I create a worker-thread (from a pool) to run the crawl task and each time I see a link it will be added to the queue.

2. Use the native event loop, so that for each URL in the HTML page I'll run the same asynchronous function in a recursive way

I'm also planning on reading the HTML pages as a stream. Which is better in terms of performance?

I'll also like to know if each worker-thread has its own event loop.