So, I am new to AWS S3 and pyspark and linux. I am not sure where to start. Here is my question:
In linux I can issue the following command and can see files in the folder:
aws s3 ls 's3://datastore/L2/parquet'
Doing similar thing with python does not work
import os
os.listdir('s3://datastore/L2/parquet')
It gives error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OSError: [Errno 2] No such file or directory: 's3://datastore/L2/parquet'
However, pyspark and SQLContext.read.parquet understands it well:
from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)
df = sqlContext.read.parquet('s3://datastore/L2/parquet')
Any reason why it works in SQLContext and does not work in os.listdir? Where can I start to clear my confusion? Any reply besides 'get bachelor in cs' would be helpful.
So, AWS s3 is not the same as your operating system's file system. The AWS s3 ls command and the pyspark SQLContext.read commands are doing something different from the os.listdir command, which does not know how to read things from s3.
To read things from s3, I recommend looking at the boto3 library, or the s3fs library, which is a wrapper around boto3 that treats s3 more like a filesystem. There are a variety of options within boto3 for listing buckets and files within buckets.
From the s3 docs:
In terms of implementation, buckets and objects are resources, and Amazon S3 provides APIs for you to manage them.
If you don't know how the Linux file system works, I recommend reading about it, maybe something like this will be helpful.