In Python/Boto 3, Found out that to download a file individually from S3 to local can do the following:
bucket = self._aws_connection.get_bucket(aws_bucketname)
for s3_file in bucket.list():
if filename == s3_file.name:
self._downloadFile(s3_file, local_download_directory)
break;
And to download all files under one chosen directory:
else:
bucket = self._aws_connection.get_bucket(aws_bucketname)
for s3_file in bucket.list():
self._downloadFile(s3_file, local_download_directory)
And helper function _downloadFile():
def _downloadFile(self, s3_file, local_download_destination):
full_local_path = os.path.expanduser(os.path.join(local_download_destination, s3_file.name))
try:
print "Downloaded: %s" % (full_local_path)
s3_file.get_contents_to_filename(full_local_path)
But both don’t seem to be working. Using Boto 3 and Python, would like to be able to download all files, as a zip preferably, under a defined directory on S3 to my local.
What could I be doing wrong, and what’s the correct implementation of the parameters?
Thank you in advance, and will be sure to accept/upvote answer
UPDATE CODE: Getting an error: “AttributeError: 'S3' object has no attribute
import sys
import json
import os
import subprocess
import boto3
from boto.s3.connection import S3Connection
s3 = boto3.resource('s3')
s3client = boto3.client('s3')
#This works
for bucket in s3.buckets.all():
print(bucket.name)
def main():
#Getting an error: “AttributeError: 'S3' object has no attribute 'download’”
s3client.download('testbucket', 'fileone.json', 'newfile')
if __name__ == "__main__": main()
To download files from S3 to Local FS, use the download_file() method
s3client = boto3.client('s3')
s3client.download_file(Bucket, Key, Filename)
If the S3 object is s3://mybucket/foo/bar/file.txt, then the arguments would be
Bucket --> mybucket
Key --> foo/bar/file.txt
Filename --> /local/path/file.txt
There aren't any methods to download the entire bucket. An alternative way would be to list all the objects in the bucket and download them individually as files.
for obj in s3client.list_objects(Bucket='mybucket')['Contents']:
try:
filename = obj['Key'].rsplit('/', 1)[1]
except IndexError:
filename = obj['Key']
localfilename = os.path.join('/home/username/Downloads/', filename) # The local directory must exist.
s3client.download_file('mybucket', obj['Key'], localfilename)
Note: The response of list_objects() is truncated to 1000 objects. Use the markers in the response to retrieve the remainder of objects in the bucket.