We are running an application on Firestore and got a simple trigger that when order's details are created or updated some of it's information should be rewritten in the parent order collection.
The function for this got following code
export const updateOrderDetails = functions
.region(FUNCTION_REGION)
.firestore.document("orders/{orderId}/details/pickupAndDropoff")
.onWrite(async (change, context) => {
return await admin
.firestore()
.collection("orders")
.doc(context.params.orderId)
.set({ pickupAndDropoff: change.after.data() }, { merge: true });
});
It was work fine before, but now at random about every third of its executions is delayed. Sometimes by few minutes. In Cloud Function logs we see normal execution times <200ms, so it seems the trigger runs after a huge pause.
What's worse from time to time our change.after.data() is undefined, but we never delete anything - it's just updates and creates.
It was working fine, we did not changed nothing since last week, but now it started to have this unexpected delays. We've also checked the firebase status, but there are no malfunctions in firebase functions service. What can be the cause of this?
The problem can be due to the monotonically increasing orderId as the parameter passed here:
...
.collection("orders")
.doc(context.params.orderId)
...
If you can check once if the orderId passed here is monotonically increasing with each request? It can lead to hotspots which impacts latency.
To explain, I think the write rate must be changing at different day's and time's - as the user traffic using the application or load testing requests changes - which is creating the unexpected kind of behaviour. At low write rate, the requests are working as expected most of the time. At high write rate, the requests are facing hotspot situation in the firestore as mentioned in the firestore documentation resulting in delays (latency issue).
Here is the relevant link to firestore best practices documentation.
Thanks to Frank van Puffelen suggestion we've sent this question directly to Firebase support and after their internal investigation we've got the reply from an engineering team that it was in fact an infrastructure malfunction.
The reply I got from them was:
I escalated the issue to recover more information. So far it appears that there was an issue with pub/sub delivering and creating the event. The Firestore team is also communicating with the pub/sub team to investigate the issue and prevent future incidents.
It seems that the best way to deal with such problems is to quickly write directly to Firebase support team, because as they mentioned in the automatic reply I got after sending a support ticket:
For Firebase outages not listed on the status dashboard, we'll respond within 4 hours.
which seems to be the best option.