0
336
ViewsHeroku web dyno running Django app not releasing memory
I'm struggling with a memory issue on Heroku when running a Django application (with gunicorn).
I have the following code that takes a user-uploaded image, removes all EXIF data, and returns the image ready for it to be uploaded to S3. This is used both as a form data cleaner and when reading base64 data into memory.
def sanitise_image(img): # img is InMemoryUploadedFile
try:
image = Image.open(img)
except IOError:
return None
# Move all pixel data into a new PIL image
data = list(image.getdata())
image_without_exif = Image.new(image.mode, image.size)
image_without_exif.putdata(data)
# Create new file with image_without_exif instead of input image.
thumb_io = StringIO.StringIO()
image_without_exif.save(thumb_io, format=image.format)
io_len = thumb_io.len
thumb_file = InMemoryUploadedFile(thumb_io, None, strip_tags(img.name), img.content_type,
io_len, None)
# DEL AND CLOSE EVERYTHING
thumb_file.seek(0)
img.close()
del img
thumb_io.close()
image_without_exif.close()
del image_without_exif
image.close()
del image
return thumb_file
I basically take an InMemoryUploadedFile and return a new one with just the pixel data.
del and closes may be redundant, but they represent my attempt to fix the situation where Heroku memory usage keeps growing and is not released every time this function terminates, even remaining overnight:
Running this on localhost with Guppy and following the tutorial, there are no remaining InMemoryUploadedFiles nor StringIOs nor PIL Image left in the heap, leaving me puzzled.
My suspicion is Python does not release the memory back to the OS, as I've read in multiple threads on SO. Has anyone played around with InMemoryUploadedFile and can give me an explanation as to why this memory is not being released?
When I do not perform this sanitisation, the issue does not occur.
Thanks a lot!
2 answers
Answer question0
I think the issue is creating the temporary list object:
data = list(image.getdata())
Try:
image_without_exif.putdata(image.getdata())
This is why I think that is the issue:
>>> images = [Image.new('RGBA', (100, 100)) for _ in range(100)]
Python memory usage increased ~4Mb.
>>> get_datas = [image.getdata() for image in images]
No memory increase.
>>> pixel_lists = [list(image.getdata()) for image in images]
Python memory usage increased by ~85Mb.
You probably don't want to make getdata() into a list unless you need the numbers explicitly. From the Pillow docs:
Note that the sequence object returned by this method is an internal PIL data type, which only supports certain sequence operations. To convert it to an ordinary sequence (e.g. for printing), use list(im.getdata()).
0
I found my own answer eventually. Huge thanks to Ryan Tran for pointing me in the right direction. list() does indeed cause the leak.
Using the equivalent split() and merge() method (docs) this is the updated code:
with Image.open(img) as image:
comp = image.split()
image_without_exif = Image.merge(image.mode, comp)
thumb_io = StringIO.StringIO()
image_without_exif.save(thumb_io, format=image.format)
io_len = thumb_io.len
clean_img = InMemoryUploadedFile(thumb_io, None, strip_tags(img.name), img.content_type,
io_len, None)
clean_img.seek(0)
return clean_img