I'm doing a project that updates many patients' information with axios call. I parsed the patient's csv file to JSON format. And, call post api to update each patient's information like below.
/**
* this.csvJson.data : List of JSON format, each json contain one patient's information
*/
.
.
await this.csvJson.data.map(async (data)=>{
await axios.post(`api/MetaData/${this.projects.projectId}`,data)
.then(response=>{
//response handling
})
.catch(error=>{
//error handling
})
})
.
.
In this case, when the patient number is below 200, the uploading process is comparatively stable. But when patients number is over 20000, Browser has no response. I think calling multiple api for each patient cause browser crash.
What is the right way for handling multiple api call like this?
With that code you are making 20000 requests in an instant no wonder the browser is crashing. since you're using already an external library you can take a look at fastq.
Heres my implementation in React, basically fastq can take the each params of each request then wrap each of it in an async worker function then it will only do "concurrency" number of request at a time. If you make a 20000 requests depending on how fast the requests is made it would take time to finish. you may want to add more concurrency to make it faster.
import fastq from "fastq";
export async function consecutiveRequest(data, requester, concurrency = 10) {
const workerFunction= async (args) => {
try {
const data = await requester(args.params);
console.log(args.index, "Requested:", args, "Result:", data);
return data;
} catch (error) {
console.error(error);
return null;
}
};
const consecutiveQueue = fastq.promise(
workerFunction, // fastq take a worker function
concurrency // number of requests only allowed at a time
);
const req = [];
for (let i = 0; i < data.length; i++) {
const params = data[i];
req.push(consecutiveQueue.push({ params, index: i }));
}
let data = await Promise.all(req);
return data;
}
import _ from "lodash";
import React from "react";
import { consecutiveRequest } from "./request";
import "./styles.css";
export default function App() {
const randomNumbers = _.range(0, 50).map(() => _.random(0, 4));
const [results, setResults] = React.useState([]);
const [loading, setLoading] = React.useState(false);
const handleSendQueueRequest = async () => {
setLoading(true);
const res = await consecutiveRequest(
randomNumbers,
async (random) => {
const url = `https://cat-fact.herokuapp.com/facts`;
const res = await fetch(url, { method: "GET" });
const json = await res.json();
const facts = json || [];
return facts[random]?.text;
},
10
);
setResults(res);
setLoading(false);
};
return (
<div className="App">
<button disabled={loading} onClick={handleSendQueueRequest}>
{loading ? "Loading" : "Get Random Quote"}
</button>
<pre>{JSON.stringify(results, null, 2)}</pre>
</div>
);
}
Heres the codesandbox take a look in the console on how its working.
For multiple apiCall using Promise.all is often a better otpion, but i dont know if it can handle 20 000requests
const promises = csvJsonData.map((patientData) => {
return axios.post(`api/MetaData/${this.projects.projectId}`, data);
});
Promise.all(promises)
.then((response) => {
//response handling
})
.catch((error) => {
//error handling
});