Say for example I have the following bucket set up:
bucketone
…/folderone
…/text1.txt
…/text2.txt
…/foldertwo
…/file1.json
…/folderthree
…/folderthreesub
…/file2.json
…/file3.json
But it only goes down one level.
What’s the proper way of retrieving information under a bucket?
Will be sure to accept/upvote answer.
Contrary to the way you'd think it will work, rsplit() actually returns the splits from left-right, even though it applies it right-to-left.
Therefore, you actually want to obtain the last element of the split:
filename = obj['Key'].rsplit('/', 1)[-1]
See: Python rsplit() documentation
Also, be careful of 'pretend directories' that might be created via the console. They are actually zero-length files the make the folder appear in the UI. Therefore, skip files with no name after the final slash.
Make those fixes and it works as desired:
import boto3
import os
s3client = boto3.client('s3')
for obj in s3client.list_objects_v2(Bucket='my-bucket')['Contents']:
filename = obj['Key'].rsplit('/', 1)[-1]
localfiledir = os.path.join('/tmp', filename)
if filename != '':
s3client.download_file('my-bucket', obj['Key'], localfiledir)
Whats wrong with just doing this from the CLI?
aws s3 cp s3://bucketing . --recursive