I am using Amazon SES and storing emails to S3 buckets, reading from SNS Notifications. I can either create a single bucket and store all my emails in that single bucket or create individual bucket for each new email account. I expect a high volume of emails in each email account.
Which of the above two email:s3bucket design is recommended and why?
1) 1-email-account -> 1 s3-bucket
2) all-email-accounts -> 1 s3-bucket
According to the AWS documentation:
By default, you can create up to 100 buckets in each of your AWS accounts. If you need additional buckets, you can increase your bucket limit by submitting a service limit increase. For information about how to increase your bucket limit, go to AWS Service Limits in the AWS General Reference.
If you have less than 100 email accounts, you may use 1 bucket per account.
Here's my comparison:
Performance: 1 bucket per account wins.
There's no performance gain in using multiple buckets instead of a single one. But architecting your S3 object key is easier if you use 1 bucket per account. Choosing the right S3 object key pattern is crucial if you want to maximize the read/write throughput from/to S3.
For more info: http://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html
Security: Tie.
Using multiple buckets, you can create IAM policies per bucket instead of making them "per folder" (e.g. "arn:aws:s3:::email1-at-foobar.com/" vs "arn:aws:s3:::emails/email1/"). There's no difference.
Flexibility: 1 bucket for all email wins.
If you want to analyze/query the email stored into S3 using EMR or Athena, having one single bucket can make your life easier.