Question

0

158

Views

Boto3 download gzip and upload as stream

Using the python module boto3, let me say that again, using boto3, not boto. How can I download a file from S3, gzip and re-upload to S3 without the file ever being written to disk?

I am trying to write an AWS lambda function that Gzips all content uploaded to S3. The problem is a lambda function is limited to 512MB of disk space and my uploads could far exceed this.

My assumption is it may be possible to do this using streams, any help would be much appreciated! Thanks.

[UPDATE]

The below code works, kind of. It will upload the chunks to S3, and I can see the resulting *.gz file. However the gzip headers aren't getting added correctly. Opening the file results in on mac Error 32 - Broken Pipe.

Interesting fact, if the file size is less then the CHUNK_SIZE, i.e. only one iteration, the file is uploaded, and is not corrupt.

Any see something I am doing wrong?

CHUNK_SIZE = 10000000
gz_buffer = io.BytesIO()
gz_stream = gzip.GzipFile(fileobj=gz_buffer, mode='wb', compresslevel=9)
obj = resource.Object(bucket, key)
body = obj.get()['Body']
try:
    while True:
        data = body.read(CHUNK_SIZE)
        if data:
            compressed_bytes = gz_stream.write(data)
            if compressed_bytes < CHUNK_SIZE:
                gz_stream.close()
            cdata = gz_buffer.getvalue()[0:compressed_bytes]
            # Upload cdata as multipart upload
            # This is a little helper function that 
            # uses boto3 create_multipart_upload
            multipart.upload(cdata)
        else:
            # Signal to S3 complete multipart upload
            multipart.complete()
            break
except Exception as e:
    pass

over 4 years ago · Santiago Trujillo

1 answers

Answer question

0

Answer question

Find remote jobs

Answer 1 · 2022-06-13T16:40:16.118Z

I would do this:

import gzip,io

out_buffer = io.BytesIO()
f = gzip.open(out_buffer,"wb")

obj = resource.Object(bucket, key)
body = obj.get()['Body']
while True:
    read = body.read(500000)
    print('reading...')
    if read:
        # 1.) Stream chunks to gzip
        f.seek(0)
        nb_bytes = f.write(read)
        # 2.) Stream compressed chunks back to S3
        cdata = out_buffer.getvalue()[0:nb_bytes]
        # cdata now holds the compressed chunk of data
    else:
        break

use io.BytesIO to create a "fake" file in memory
map a gzip handle on it
loop to read (your code)
seek to start of the fake file handle before writing so it doesn't use too much memory (current write overwrites previous iteration write)
write the data read to the gzip handle, note down the number of bytes written (they vary depending on the data, and if it's shorter than the previous iteration, the out_buffer does not shrink so we must know the length
slice the contents of the buffer using this length to create your compressed chunk.

note that in python 2.x, you cannot pass a fileobject to gzip.open, you have to create a Gzip object instead, like this:

f = gzip.GzipFile("foo.gz","wb",fileobj=out_buffer)

0

158

Boto3 download gzip and upload as stream

1 answers

0

Find remote jobs

Andres GPT