Executing the code below throws Error: Cannot create a string longer than 0x3fffffe7 characters. My file contains only 2 arrays and composed, the whole file contains 1,3GB. I wanted to combine them by id by using something like the code below to map through the arrayMain and return each object combined with the object from arrayItems with the same id and then restructure that object to throw out the id. But I think I am bumping up against a system limit. I am fairly new to working with large data files so any help is appreciated.
const composed = arrayMain.map((d) => {
return {
...d,
data: arrayItems.filter(({ ID }) => d.ID === ID).map(({notNeeded, ...needed}) => needed),
};
});
If anyone is wondering how my data is structured
const arrayMain = [
{
ID: 30574062,
number: 28234702,
place: London,
},
{
ID: 30574063,
number: 45232502,
place: Paris,
},
...
];
const arrayItems = [
{
"ID": 30574062,
"anotherNumber": "52,3",
"color": "red"
},
{
"ID": 30574062,
"anotherNumber": "13",
"color": "yellow"
},
{
"ID": 30574063,
"anotherNumber": "60,6",
"color": "blue"
},
...
]
//expected result
[
{
ID: 30574062,
number: 28234702,
place: London,
data: [
{
"anotherNumber": "52,3",
"color": "red"
},
{
"anotherNumber": "13",
"color": "yellow"
}
]
},
{
ID: 30574063,
number: 45232502,
place: Paris,
data: [
{
"anotherNumber": "60,6",
"color": "blue"
},
]
},
...
]
There's probably a nicer-looking solution, but perhaps you could stringify the array to be written in chunks, and only generate strings and call appendFile in those chunks, so that at no point there's a single huge string. After creating composed, create a write stream (to allow for many sequential writes without erroring) and keep calling .write on it.
const stream = fs.createWriteStream(filePath, { flags:'a' });
stream.write('[');
const CHUNK_LENGTH = 500; // alter as needed
for (let i = 0; i < composed.length; i += CHUNK_LENGTH) {
const chunkStr = JSON.stringify(
composed.slice(i, i + CHUNK_LENGTH)
);
stream.write(chunkStr);
}
stream.write(']');
// to wait for all of this to complete, watch for the stream's finish event