I am wondering how Lambda handles a backlog of S3 events when Lambda is throttled and can't keep up with events. Will S3 events be queued and for how long?
The hypothetical scenario is that a lambda function is being triggered by POST events on a S3 bucket and its concurrency limit is 10. Suppose there are suddenly a few million POST events...will those events be queued and how much time is allowed for the set of 10 lambda instances to process this steam of events?
I assume that the invocation (triggered by S3 Events Rule) follows the same behavior as you would trigger your Lambda through AWS SDK, AWS CLI, etc.
Which means that a lot of the few million POST events you mentioned in your hypothetical scenario will not be processed and even the retries will not work for all of the events.
As long as you didn't setup a DLQ (Dead Letter Queue) these events were discard and you dont have a chance to recover them.
I linked some articles to find more informations about the behavior of AWS Lambda.
AWS Lambda - Understanding Retry Behavior
Asynchronous invocation – Asynchronous events are queued before being used to invoke the Lambda function. If AWS Lambda is unable to fully process the event, it will automatically retry the invocation twice, with delays between retries.
AWS Lambda - Dead Letter Queues
AWS Lambda directs events that cannot be processed to the specified Amazon SNS topic or Amazon SQS queue. Functions that don't specify a DLQ will discard events after they have exhausted their retries.