Problem

Whilst developing a chrome extension I encountered a problem where I try to retrieve large JSON file (~1GB) using XMLHttpRequest GET request. The file is located on my local machine and I retrieve the download link using chrome.runtime.getURL('data.json') from a content script.

The error I am getting is

Uncaught SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at xhr.onreadystatechange (<anonymous>:11:23)
    at XMLHttpRequest.<anonymous>

I assumed it is because the JSON file is too large and therefore, it is not possible to download it in one go (please correct me if this wouldn't be the case). Is there any workaround for this size limitation?

Additional info

I generate the JSON file from a python script, thus, it is completely possible to choose a different format (some binary format maybe) if necessary.

Furthermore, I cannot use any API provided for chrome extensions since the script I am actually calling the XMLHttpRequest from is not a content script but an appended script to the body of the website.

Here is a minimal code for reproduction (don't forget to configure the content_security_policy in the manifest to enable inline script execution).

// Content script

function init() {

    let script = document.createElement("script");
    script.appendChild(document.createTextNode(__magic.toString() + "; __magic('" + chrome.runtime.getURL('data.json') + "');"));
    script.id = "__magic";
    document.body.appendChild(script);
}

function __magic(url) {
    var db;
    var xhr = new XMLHttpRequest();
    xhr.onreadystatechange = () => { 
        if (xhr.readyState === 4) {
            db = JSON.parse(xhr.response);
            window.setInterval(check, 10);
        }
    };
    xhr.open("GET", url, true);
    xhr.send();
}

init();

Edit

I forgot to say that I was successful in loading the big JSON file in python, thus, I think it is safe to assume that the JSON file is not corrupted.

Furthermore, the JSON file is storing a graph structure - I choose following layout for the json file

{
 "node1": {
   "children": ["node2", "node3"],
   "foo": 15
 },
 "node2": {
   "children": ["node15", "node1"],
   "foo": 90
 },
 ...
}

I bet there is a more efficient way of storing a graph structure rather than using JSON, however, so far I could not find any that would be compatible with javascript.