This is my scenario
I am a spark and aws enthusiast and I am itching to understand more about the technology.
Case 1: My spark application runs on an EMR cluster and the spark application read from a hive on s3 table and write into a hive table on s3. In this case , the S3 buckets belong to the same user usera so I added fs.s3.awsAccessKeyId and fs.s3.awsSecretAccessKey to a config file . In my case I added it to the hdfs-site.xml. usera had the right permissions to access the bucket so no problem .
Case 2: I am reading from 2 hive tables on s3. table1 and table2. table1 belongs to user1 and table2 belongs to user2.
given that i cannot specify multiple awsAccessKeyId in the config file for s3. [ I understand that s3a has a concept of bucket specific keys but I am not using s3a I am using s3.]
how are these scenarios supported in aws EMR ?
I understand that IAM , EC2 instance role and profile role can apply here
I think the solution to your problem is cross-account permissions. Thus, you can define permission for user1 to access user2's bucket. You can also take a look at this too.
Apache Hadoop 2.8 supports per-bucket configuration. AWS EMR doesn't, which is something you will have to take up with them.
As a workaround, you can put secrets in the URI, e.g. s3://user:secret@bucket, remembering to encode special characters in the secret. After doing that, the URL, logs and stack traces must be considered sensitive data and not shared.