0
159
Viewsuploading xlsx or csv files to firebase
Hello there, im trying to upload files (.csv, .xlsx of 5000 row and 6 col) to firebase with a cloud functions that i call from the frontend, the cloud function start and after a few seconds of run its make an error "code":"4" "details":"Deadline exceeded"
(Blaze plan)
- the upload is very slow
- getting an Deadline exceeded error with code 4
- its crashing the server after i get the error
explication of the code below:
i do a for of loop that iterate over an array of array that looks like this
[['Dufour', 'Annie', 'annie.duf@gmail.com', 33683333005, 'f'], ['john','Doe', 'john@gmail.com', 33223424, 'm']] then i format each row in order to
structure it as a firestore doc with many fields and try to add it to firestore
then its crash
i cant figure out when its crashing its really random
and when im trying a smaller file (1000 row) its also crashing after some seconds of execution
export const createNewLeads = functions.region("europe-west3").https.onCall(async (data, context) => {
if (!context.auth) {
throw new functions.https.HttpsError('failed-precondition', 'The function must be called' + 'while authenticated.');
}
const { category, subcategory, allLeads, downloadUrl, file_id, fileDetails } = data;
for (const lead of allLeads) {
const leads_infos = format_lead_infos(lead , fileDetails);
const leads_meta = format_lead_metadata(downloadUrl, file_id, fileDetails);
const db_lead_ref = admin.firestore().collection('leads');
const new_lead_doc_id = db_lead_ref.doc().id;
db_lead_ref
.doc(new_lead_doc_id).set({
lead_id: new_lead_doc_id,
category: category,
subcategory: subcategory,
createdOn: Date(),
uploaded_by: context.auth.uid,
purchased_by: null,
metadata: leads_meta,
infos: leads_infos
}).then(() => {
functions.logger.info('createLeads SUCCESS', { docId: new_lead_doc_id });
return {
user_id: new_lead_doc_id,
created: true,
};
}).catch((error: any) => {
functions.logger.info('createLeads ERROR', { error });
return {
error: true,
err_mess: error,
};
});
}
});
2 answers
Answer question0
As far as I can see you are not handling the asynchronous nature of writing to Firestore correctly, as you're not awaiting the call to set(). You're also returning a result on the first successful document write in the loop, which seems suspicious.
This should be closer to what you need:
try {
for (const lead of allLeads) {
const leads_infos = format_lead_infos(lead , fileDetails);
const leads_meta = format_lead_metadata(downloadUrl, file_id, fileDetails);
const db_lead_ref = admin.firestore().collection('leads');
const new_lead_doc_id = db_lead_ref.doc().id;
await db_lead_ref.doc(new_lead_doc_id).set({
lead_id: new_lead_doc_id,
category: category,
subcategory: subcategory,
createdOn: Date(),
uploaded_by: context.auth.uid,
purchased_by: null,
metadata: leads_meta,
infos: leads_infos
});
}
}
catch (error: any) => {
functions.logger.info('createLeads ERROR', { error });
return {
error: true,
err_mess: error,
};
}
functions.logger.info('createLeads SUCCESS', { docId: new_lead_doc_id
return {
user_id: new_lead_doc_id,
created: true,
};
In this code we await each document write operation and then return an error if any of them fails, and success only once all of them succeed.
0
Hello Frank thanks for helping us i copy the code you give to us bu after a few seconds of running i got this error error of the code from the firebase console log
export const createNewLeads = functions.region("europe-west3").https.onCall(async (data, context) => {
if (!context.auth) {
throw new functions.https.HttpsError('failed-precondition', 'The function must be called' + 'while authentificated.');
}
const { category, subcategory, allLeads, downloadUrl, file_id, fileDetails } = data;
const db_lead_ref = admin.firestore().collection('leads');
const new_lead_doc_id = db_lead_ref.doc().id;
try {
for (const lead of allLeads) {
const leads_infos = format_lead_infos(lead , fileDetails);
const leads_meta = format_lead_metadata(downloadUrl, file_id, fileDetails);
await db_lead_ref.doc(new_lead_doc_id).set({
lead_id: new_lead_doc_id,
category: category,
subcategory: subcategory,
createdOn: Date(),
uploaded_by: context.auth.uid,
purchased_by: null,
metadata: leads_meta,
infos: leads_infos
});
}
} catch (error:any) {
functions.logger.info('createLeads ERROR', { error });
return {
error: true,
err_mess: error,
}
}
functions.logger.info('createLeads SUCCESS', { docId: new_lead_doc_id })
return {
user_id: new_lead_doc_id,
created: true,
}
});
