I have an application that reads from JSON files and adds the data to a Kafka producer. As we're dealing with 5MB/s I needed to multithread this process. I have 3 workers reading files and 7 workers writing to Kafka. However, no matter what I do I keep getting the same error.

FATAL: Error: Worker terminated due to reaching memory limit: JS heap out of memory
    at new NodeError (node:internal/errors:372:5)
    at Worker.[kOnExit] (node:internal/worker:276:26)
    at Worker.<computed>.onexit (node:internal/worker:198:20) {
  code: 'ERR_WORKER_OUT_OF_MEMORY'
}

I've changed how much heap space (up to 16GB!) Javascript runs with by setting

NODE_OPTIONS=--max-old-space-size=$((16 * 1024))

But no matter what I do I keep running out of memory after around 20 minutes or so. How can I make sure I don't run out of memory?