I have a node lambda that parses an s3 folder for json files, reads the files, and then injects that json into a dynamo table. It maxes out at 50 files and I’m trying to understand why. I’ve changed the lambda timeout and memory to no effect. I’ve also commented out the dynamo put command so I know that’s not the culprit. It always seems to stop on the 51st file when trying to read its contents from s3.
Any ideas why?
Thanks!
I was able to solve this problem by having one lambda parse the s3 folder for files and then have a second lambda act on each file. The key difference being that a lambda invocation only reads a single s3 file and writes a single record to the database.
I still do not understand why the first design was limited at 50 file reads, but this new design is better in multiple ways.