Business
Jobs
  • About Us
  • Solutions
    • Job Postings
      Post your job and receive qualified candidates in 48h.
    • Candidate Assessments
      500+ technical and psychological tests, plus anti-fraud.
    • Headhunting
      Tailor-made executive search from start to finish.
    • Payroll + EOR
      Payroll dispersal and EOR across 15+ LATAM countries.
  • Pricing
  • Jobs

0

430
Views
How to zip files in Amazon s3 Bucket and get its URL

I have a bunch of files inside Amazon s3 bucket, I want to zip those file and download get the contents via S3 URL using Java Spring.

over 4 years ago · Santiago Trujillo
3 answers
Answer question

0

S3 is not a file server, nor does it offer operating system file services, such as data manipulation.

If there is many "HUGE" files, your best bet is

  1. start a simple EC2 instance
  2. Download all those files to EC2 instance, compress them, reupload it back to S3 bucket with a new object name

Yes, you can use AWS lambda to do the same thing, but lambda is bounds to 900 seconds (15 mins) execution timeout (Thus it is recommended to allocate more RAM to boost lambda execution performance)

Traffics from S3 to local region EC2 instance and etc services is FREE.

If your main purpose is just to read those file within same AWS region using EC2/etc services, then you don't need this extra step. Just access the file directly.

(Update) : As mentioned by @Robert Reiz, now you can also use AWS Fargate to do the job.

Note :

It is recommended to access and share file using AWS API. If you intend to share the file publicly, you must look into security issue seriously and impose download restriction. AWS traffics out to internet is never cheap.

over 4 years ago · Santiago Trujillo Report

0

Hi I recently have to do that for my application -- serve a bundle of files in zip format through a url link that the users can download.

In a nutshell, first create an object using BytesIO method, then use the ZipFile method to write into this object by iterating all the s3 objects, then use put method on this zip object and create a presiged url for it.

The code I used looks like this:

First, call this function to get the zip object, ObjectKeys are the s3 objects that you need to put into the zip file.


def zipResults(bucketName, ObjectKeys):
    buffer = BytesIO()
    with zipfile.ZipFile(buffer, 'w', compression=zipfile.ZIP_DEFLATED) as zip_file:
        for ObjectKey in ObjectKeys:
            objectContent = S3Helper().readFromS3(bucketName, ObjectKey)
            fileName = os.path.basename(ObjectKey)
            zip_file.writestr(fileName, objectContent)

    buffer.seek(0)
    return buffer

Then call this function, key is the key you give to your zip object:

def uploadObject(bucketName, body, key):
    s3client = AwsHelper().getClient("s3")
    try:
        response = s3client.put_object(
            Bucket=bucketName,
            Body=body,
            Key=key
        )
    except ClientError as e:
        logging.error(e)
        return None

    return response

Of course, you would need io, zipfile and boto3 modules.

over 4 years ago · Santiago Trujillo Report

0

If you need individual files (objects) in S3 compressed, then it is possible to do so in a round-about way. You can define a CloudFront endpoint pointing to the S3 bucket, then let CloudFront compress the content on the way out: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/ServingCompressedFiles.html

over 4 years ago · Santiago Trujillo Report
Answer question
Find remote jobs

Discover the new way to find a job!

Top jobs
Top job categories
Business
Post vacancy Pricing Sales
Legal
Terms and conditions Privacy policy
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Show me some job opportunities
There's an error!