I have JSON file which has json data of size 914MB. I loading the file with fs-extra and parsing it. But when i parse it get error as
cannot create a string longer than 0x1fffffe8 characters
Below is code
const fs = require('fs-extra');
const rawdata = fs.readFileSync('src/raw.json');
const data = JSON.parse(rawdata);
I am running the project with npm and to run i have below command in package.json.
"scripts": {
"start:dev": "cross-env NODE_OPTIONS='--max-old-space-size=4096' ts-node -r tsconfig-paths/register ./src --env=development",
}
0x1fffffe8 is exactly 512MB.The many commenters are correct: you are bumping up against a system limit. I agree with @Pointy that it is mostly likely a Node string length limit. fs-extra has nothing to do with the limit
In any case, you're going to have to process that JSON in chunks.
Almost certainly your massive JSON data is an Array at the root level. So if you use a stream/SAX parser, you can process each element in that root array individually, or in batches, whichever makes sense.
ℹ️ You probably do not want a streaming parser that ONLY supports the JSON Streaming protocol, as that protocol is designed for a stream of multiple JSON objects concatenated in a stream. But from what I can tell you have one humungous monolithic JSON object, most likely an array at root as I mentioned above.
You have many parser options. To get you started, here are the ones with top usage on NPM:
https://www.npmjs.com/package/JSONStream (top, but archived)
https://gitlab.com/philbooth/bfj (#2, but also archived)
If you know the source JSON is very regular in its formatting, e.g. every record in the root array is N lines long, the most efficient thing might be to read the raw lines via a buffered reader, grab every N lines (adjusting for the opening lines for the root array at the top), and JSON.parse those individually (after removing the comma separating root array entries. Just a raw idea. I've never done it!