Using pyspark2 (version 2.0.0.cloudera1) on Cloudera within AWS
I am trying to write out a dataframe from Spark to S3 storage, but failing because of authentication:
pyspark.sql.utils.IllegalArgumentException: u'AWS Access Key ID and Secret Access Key must be specified by setting the fs.s3n.awsAccessKeyId and fs.s3n.awsSecretAccessKey properties (respectively).'
My pyspark code is:
utp.coalesce(1).write.format('com.databricks.spark.csv').save('s3n://my_bucket/tmr_xfers/test_output')
We use roles for accessing S3, i.e. 'aws_iam_role=arn:aws:iam::123456789012:role/RoleName' -- not individual AccessKeyIDs
What do I need to change in my Spark code so that my csv gets written out to S3 using roles instead of individual AccessKeyId and SecretAccessKey?
I had this same problem and got around it by using s3a:// instead, which is apparently much more modern and performant in any case.
The problem is in the Hadoop driver code (I think the hadoop-aws.jar) that is responsible for accessing the S3 filesystem. Apparently the s3n 'native' protocol uses some old jets3t driver that is difficult to work with and everyone is afraid to mess with. The new s3a protocol implementation uses the AWS SDK directly, and supports instance profiles etc.
Have a look at these HADOOP-9680 and HADOOP-9384 to see why they WONTFIXed this problem.
Here is a solution in scala spark2, beware of security issue.
spark.sparkContext.hadoopConfiguration.set("fs.s3n.awsAccessKeyId", "xxxxx")
spark.sparkContext.hadoopConfiguration.set("fs.s3n.awsSecretAccessKey", "xxxxxxxx")
Df.write.
format("com.databricks.spark.csv").option("header", "true").
save("s3n://my_bucket/tmr_xfers/test_output")