I'm refactoring a job that uploads ~1.2mln small files to AWS; previously this upload was made file by file on a 64 CPUs machine with processes. I switched to an async + multiprocess approach following S3 rate limits and best practices and performance guidelines to make it faster. With sample data I can achieve execution times as low as 1/10th. With production loads S3 is returning "SlowDown" errors.

Actually the business logic makes the folder structure like this:

s3://bucket/this/will/not/change/<shard-key>/<items>

The objects will be equally splitted across ~30 shard-keys, making every prefix contain ~40k items.

We have every process writing on its own prefix and launching batches of 3k PUT requests in async until completion. There is a sleep after the batch write operation to ensure that we do not send another batch before 1.1sec has passed, so we will respect the 3500 PUT requests per second.

The problem is that we receive SlowDown errors for ~1 hour and then the job writes all the files in ~15 minutes. If we lower the limit to 1k/sec this gets even worse, running for hours and never finishing.

This is the distribution of the errors over time for the 3k/sec limit:

We are using Python 3.6 with aiobotocore to run async.

Doing some sort of trial and error to try to understand how to mitigate this takes forever on production data and testing with a lower quantity of data gives us different results (flawlessly works).

Did I miss any documentation regarding how to make the system scale up correctly?