am trying to scroll over all buckets in s3 and see if there is a prefix that matches and get into those folders and read the json files.
I have tried to get the folders that contain a prefix, but failing to enter them.
Code:
import boto3
bucket = ['test-eob', 'test-eob-images']
client = boto3.client('s3')
for i in bucket:
result = client.list_objects(Bucket=i,Prefix = 'PROCESSED_BY/FILE_JSON', Delimiter='/')
print(result)
Using this am getting the ones with prefix and fails when bucket doesnt have that prefix.
structure of test-eob , test-eob/PROCESSED_BY/FILE_JSON/*.json
I have to read the json if only my prefix matches, else come out of the bucket.
Can anyone help me out here.
Try to catch the error(is it a KeyError?) when the bucket does not contain the prefix.
For example:
for i in bucket:
try:
result = client.list_objects(Bucket=i,Prefix = 'PROCESSED_BY/FILE_JSON', Delimiter='/')
print(result)
except KeyError:
pass
To read the json, there are several ways. For example with json.loads() from the json module.
So for each object in the bucket:
content_object = s3.Object(bucket_name, file_name)
file_content = content_object.get()['Body'].read().decode('utf-8')
json_content = json.loads(file_content)