I'm trying to write a csv file into an S3 bucket using AWS Lambda, and for this I used the following code:
data=[[1,2,3],[23,56,98]]
with open("s3://my_bucket/my_file.csv", "w") as f:
f.write(data)
And this raises the following error:
[Errno 2] No such file or directory: u's3://my_bucket/my_file.csv': IOError
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 51, in lambda_handler
with open("s3://my_bucket/my_file.csv", "w") as f:
IOError: [Errno 2] No such file or directory: u's3://my_bucket/my_file.csv'
Can I have some help with this please ?
PS: I'm using python 2.7
Thanking you in advance
Better to answer later than never. There are four steps to get your data in S3:
Something like this:
import csv
import requests
#all other apropriate libs already be loaded in lambda
#properly call your s3 bucket
s3 = boto3.resource('s3')
bucket = s3.Bucket('your-bucket-name')
key = 'yourfilename.txt'
#you would need to grab the file from somewhere. Use this incomplete line below to get started:
with requests.Session() as s:
getfile = s.get('yourfilelocation')
#Only then you can write the data into the '/tmp' folder.
with open('/tmp/yourfilename.txt', 'w', newline='') as f:
w = csv.writer(f)
w.writerows(filelist)
#upload the data into s3
bucket.upload_file('/tmp/yourfilename.txt', key)
Hope it helps.
I am not aware of using AWS Lambda, but I have been using Boto3 to do the same. It is a simple few line code.
#Your file path will be something like this:
#s3://<your_s3_bucket_name>/<Directory_name>/<File_name>.csv
import boto3
BUCKET_NAME = '<your_s3_bucket_name>'
PREFIX = '<Directory_name>/'
s3 = boto3.resource('s3')
obj = s3.Object(BUCKET_NAME, PREFIX + '<File_name>.csv')
obj.put(Body=content)
with open("s3://my_bucket/my_file.csv", "w+") as f:
instead of
with open("s3://my_bucket/my_file.csv", "w") as f:
notice the "w" has changed to "w+" this means that it will write to the file, and if it does not exist it will create it.