I’d like to run a batch operation where my script calls the Pagespeed Insights API for a list of URLs, returns the JSON and saves it to Dynamo. Each API response averages between 20-30 sec, so maybe 30 URLs max before my Lambda times out at 15 min max. I’ll eventually have the list of URLs at maybe 200+.
What’s the best approach/stack for this?
Have a Lambda function put the URLs into an SQS queue and another Lambda function that gets invoked for messages in the Queue. The second Lambda handles querying one URL and stores the result in DynamoDB. You also get retries in case something goes wrong by using SQS.
You can limit concurrent executions by setting a concurrency limit for the second Lambda function. You can also configure batching for the SQS integration and have each Lambda invocation handle up to 10 URLs.