I have connected to Amazon S3 and am trying to retrieve data from the JSON content from multiple buckets using the below code.
But I have to read only specific JSON files, but not all. How do I do it?
Code:
for i in bucket:
try:
result = client.list_objects(Bucket=i,Prefix = 'PROCESSED_BY/FILE_JSON', Delimiter='/')
content_object = s3.Object(i, "PROCESSED_BY/FILE_JSON/?Account.json")
file_content = content_object.get()['Body'].read().decode('utf-8')
json_content = json.loads(file_content)
except KeyError:
pass
Bucket structure example.
test-eob/PROCESSED_BY/FILE_JSON/222-Account.json
test-eob/PROCESSED_BY/FILE_JSON/1212121-Account.json
test-eob/PROCESSED_BY/FILE_JSON/122-multi.json
test-eob/PROCESSED_BY/FILE_JSON/qwqwq-Account.json
test-eob/PROCESSED_BY/FILE_JSON/wqwqw-multi.json
From the above list, I want to only read *-Account.json files.
How can I achieve this?
There are several ways to do this in Python. For example, checking if 'stringA' is in 'stringB':
list1=['test-eob/PROCESSED_BY/FILE_JSON/222-Account.json',
'test-eob/PROCESSED_BY/FILE_JSON/1212121-Account.json',
'test-eob/PROCESSED_BY/FILE_JSON/122-multi.json',
'test-eob/PROCESSED_BY/FILE_JSON/qwqwq-Account.json',
'test-eob/PROCESSED_BY/FILE_JSON/wqwqw-multi.json',]
for i in list1:
if 'Account' in i:
print (i)
else:
pass
You can make use of a regex that matches your pattern from the list of objects.
import re
MATCH = "FILE_JSON/.*?Account.json"
full_list = [
"test-eob/PROCESSED_BY/FILE_JSON/222-Account.json",
"test-eob/PROCESSED_BY/FILE_JSON/1212121-Account.json",
"test-eob/PROCESSED_BY/FILE_JSON/122-multi.json",
"test-eob/PROCESSED_BY/FILE_JSON/qwqwq-Account.json",
"test-eob/PROCESSED_BY/FILE_JSON/wqwqw-multi.json"
]
for item in full_list:
if re.search(MATCH, item):
print(item)