I have a file with 2 large arrays, file takes about 2 GB. I want to combine both arrays by id. Problem is that the arrays are too large. Usually I would just map throught arrayMain, take the ID and then filter arrayItems by that ID. I believe the problem is that I store all the results in memory and there is not enough memory for this operation.

This is the how i would have normally combined both arrays and the expected result:

const composed = arrayMain.map((d) => {
  return {
    ...d,
    data: arrayItems.filter(({ ID }) => d.ID === ID),
  };
});

const arrayMain = [
  {
    ID: 30574062,
    number: 28234702,
    place: London,
  },
  {
    ID: 30574063,
    number: 45232502,
    place: Paris,
  },
  ...
];
const arrayItems = [
  //Objects with the same ID are not necessarily next to one another.
  {
    "ID": 30574062,
    "anotherNumber": "52,3",
    "color": "red"
  },
  {
    "ID": 30574062,
    "anotherNumber": "13",
    "color": "yellow"
  },
  {
    "ID": 30574063,
    "anotherNumber": "60,6",
    "color": "blue"
  },
  ...
]

//expected result
[
  {
    ID: 30574062,
    number: 28234702,
    place: London,
    data: [
      {
       "anotherNumber": "52,3",
       "color": "red"
      },
      {
       "anotherNumber": "13",
       "color": "yellow"
      }
    ]
  },
  {
    ID: 30574063,
    number: 45232502,
    place: Paris,
    data: [
      {
        "anotherNumber": "60,6",
        "color": "blue"
      },
    ]
  },
  ...
]

If i understand streams correctly, i have to map throught arrayMain and take an ID. Then stream objects from arrayItems, and filter that stream based on ID. After that I have to create a write stream to write the results to another file.

const stream = fs.createReadStream(filepath)
  .pipe(composed()) //composed is a function in this case, idea is the same as above
  .pipe(filepath)  //write data to a new file

Only writing results produces Error: Cannot create a string longer than 0x3fffffe7 characters

const stream = fs.createWriteStream(filepath, { flags:'a' });
stream.write('[');
const CHUNK_LENGTH = 20;
for (let i = 0; i < composed.length; i += CHUNK_LENGTH) {
  const chunkStr = JSON.stringify(
    composed.slice(i, i + CHUNK_LENGTH)
  );
  stream.write(chunkStr);
}
stream.write(']');