Empresas
Empregos
  • Sobre nós
  • Soluções
    • Publicação de vagas
      Publique sua vaga e receba candidatos qualificados em 48h.
    • Avaliações de candidatos
      Mais de 500 testes técnicos e psicológicos, mais anti-fraude.
    • Headhunting
      Busca executiva personalizada do início ao fim.
    • Folha de Pagamento + EOR
      Dispersão de folha e EOR em mais de 15 países da LATAM.
  • Preços
  • Empregos

0

204
Visualizações
How to load a pickle file from S3 to use in AWS Lambda?

I am currently trying to load a pickled file from S3 into AWS lambda and store it to a list (the pickle is a list).

Here is my code:

import pickle
import boto3

s3 = boto3.resource('s3')
with open('oldscreenurls.pkl', 'rb') as data:
    old_list = s3.Bucket("pythonpickles").download_fileobj("oldscreenurls.pkl", data)

I get the following error even though the file exists:

FileNotFoundError: [Errno 2] No such file or directory: 'oldscreenurls.pkl'

Any ideas?

over 4 years ago · Santiago Trujillo
3 Respostas
Responde à pergunta

0

Super simple solution

import pickle
import boto3

s3 = boto3.resource('s3')
my_pickle = pickle.loads(s3.Bucket("bucket_name").Object("key_to_pickle.pickle").get()['Body'].read())
over 4 years ago · Santiago Trujillo Relatório

0

As shown in the documentation for download_fileobj, you need to open the file in binary write mode and save to the file first. Once the file is downloaded, you can open it for reading and unpickle.

import pickle
import boto3

s3 = boto3.resource('s3')
with open('oldscreenurls.pkl', 'wb') as data:
    s3.Bucket("pythonpickles").download_fileobj("oldscreenurls.pkl", data)

with open('oldscreenurls.pkl', 'rb') as data:
    old_list = pickle.load(data)

download_fileobj takes the name of an object in S3 plus a handle to a local file, and saves the contents of that object to the file. There is also a version of this function called download_file that takes a filename instead of an open file handle and handles opening it for you.

In this case it would probably be better to use S3Client.get_object though, to avoid having to write and then immediately read a file. You could also write to an in-memory BytesIO object, which acts like a file but doesn't actually touch a disk. That would look something like this:

import pickle
import boto3
from io import BytesIO

s3 = boto3.resource('s3')
with BytesIO() as data:
    s3.Bucket("pythonpickles").download_fileobj("oldscreenurls.pkl", data)
    data.seek(0)    # move back to the beginning after writing
    old_list = pickle.load(data)
over 4 years ago · Santiago Trujillo Relatório

0

This is the easiest solution. You can load the data without even downloading the file locally using S3FileSystem

from s3fs.core import S3FileSystem
s3_file = S3FileSystem()

data = pickle.load(s3_file.open('{}/{}'.format(bucket_name, file_path)))
over 4 years ago · Santiago Trujillo Relatório
Responde à pergunta
Encontrar trabalhos remotos

Descubra a nova forma de encontrar um emprego!

melhores empregos
Principais categorias de trabalho
Empresas
Postar vaga Preços Comercial
Jurídico
Termos e Condições Política de privacidade
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Recomende algumas ofertas para mim
Preciso de ajuda