When trying to access a file from a S3 bucket using Scala, the bucket name gets added to the front of endpoint and the endpoint becomes wrong and in-accessible.
Libraries:
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.0.0"
libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.0.0"
libraryDependencies += "org.apache.hadoop" % "hadoop-aws" % "2.8.0"
Code:
sc.hadoopConfiguration.set("fs.s3a.endpoint", "[endpoint]")
sc.textFile("s3a://[bucket_name]/testa.txt")
Enabling the path style URLs solves the issue as there are no virtual hosts configuration done on the server where S3 is installed. fs.s3a.path.style.access(docs) can be set to true to enable path style URLs.