I have a backend that returns large files as chunks of 206 partial content. I had logic:
fetch(url, query)
.then((resp)=>resp.json())
.then(...)
but surpricingly the code fails because the full json object does not get returned by the server.
Is there some commonly used library to solve this (monkeypatch fetch) or should I write a service-worker or a proxy for this? Why does the default browser fetch not support fetching the full object?
It's not surprising that JSON parsing fails on a partial object. fetch is just going to fetch what you ask it to (well, it follows redirects), so if the query has options for requesting a partial response, that's what you'll get.
You can build up the JSON string until you have the full response to parse, something along these lines:
async function getAll() {
// Start with a blank string
let json = "";
do {
// Get this chunk
const resp = await fetch(url, /*...query for next chunk...*/);
if (!resp.ok && resp.status !== 206) {
throw new Error(`HTTP error ${resp.status}`);
}
// Read this chunk as text and append it to the JSON
json += await resp.text();
} while (resp.status === 206);
return JSON.parse(json);
}
Obviously, that's a rough sketch, you'll need to handle the Range header, etc.
That code also requests each chunk in series, waiting for the chunk to arrive before requesting the next. If you know in advance what the chunk size is, you might be able to parallelize it, though it'll be more complicated with chunks possibly arriving out of sequence, etc.
It also assumes the server doesn't throw you a curveball, like giving you a wider range than you asked for that overlaps what you've already received. If that's a genuine concern, you'll need to add logic for it.