On the page linked below, I found the following statement:
Make one concurrent request for each 85–90 MB/s of desired network throughput. To saturate a 10 Gb/s network interface card (NIC), you might use about 15 concurrent requests over separate connections. You can scale up the concurrent requests over more connections to saturate faster NICs, such as 25 Gb/s or 100 Gb/s NICs.
What is the origin of these numbers? I can't find any other documentation that justifies this. My guess is that this limitation is speaking more to the limitations of NIC on the EC2 instance rather than S3. Still, is there any other source that explains where these numbers came from?
To be clear, this is not a question about how to optimize S3 throughput -- I'm aware of the alternatives. This is a question about the AWS S3 documentation itself.
The only people who could answer this definitively are those who are working on S3 internals. And they're almost certainly covered by NDA. So what I'm about to write is complete speculation.
We know that S3 is distributed and redundant: each object is stored on multiple physical drives, across multiple availability zones.
We can infer, from the fact that S3 is available as a networked service, that there is some form of network interface between the S3 volume and the outside world. Obvious, yes, but if that network interface is limited to 1Gbit/sec, it would be able to achieve approximately 85-90 Mbyte/sec sustained throughput.
It's also important to remember that AWS uses a software-defined network: so while the S3 service may in fact have a network interface that supports 10 Gbit/sec, AWS may restrict the bandwidth that is available to any given connection.
Far more interesting to me is this quote, from the same link:
we suggest making concurrent requests for byte ranges of an object at the granularity of 8–16 MB
This implies that redundancy is managed at a sub-object level, so that a large object is split into multiple pieces of maybe 64 MB, and those pieces are individually distributed. Which is how HDFS manages large files, so not a giant leap.
As for your supposition that it's a limit of EC2 rather than S3, I think that the suggestion to use multiple connections rules that out. Although it's possible that a single connection is limited to 1Gbit/sec by EC2, I would expect the S3 designers to be more concerned about load on their system. You can always test that out by opening a single connection between two EC2 instances with high-bandwidth networking, and see if it's throttled.