I have an S3 bucket that contains a million objects, each object keys are quite different from each other and nothing standard at all.
I want to know if there's a way to search for specific key patterns and return those objects using the Amazon S3 SDK for Java?
For example, can I search for the keys using
What are the possible ways to search for keys with S3?
You can ListObjects() with a given Prefix. Amazon S3 does not support listing via suffix or regex.
The Prefix includes the full path of the object, so an object with a Key of 2020/06/10/foo.txt could be found with a prefix of 2020/06/10/, but not a prefix of foo.
The Java command is: ListObjects()
See also: Performing Operations on Amazon S3 Objects - AWS SDK for Java
With millions of objects, it could be quite slow to list your objects (even with a Prefix) since each API call will return a maximum of 1000 objects.
Alternatively you might want to use Amazon S3 Inventory, which can provide a daily or weekly CSV file containing a list of all objects.