I am using AWS Lambda and the functions I need to deploy require many different packages. Using serverless-python-requirements the zip file that is generated is 169.5MB, far greater than the 50MB limit. I have tried using Lambda Layers, but this doesn't solve the size issue.
I have also tried dumping the zip file in an s3 bucket, but it is still too large to load when invoking the function. I need all of these packages and I'm not sure how I can deploy them all.
My requirements.txt file looks like:
bs4==0.0.1
gensim==3.8.3
matplotlib==3.2.2
nltk==3.5
numpy==1.19.0
openpyxl==3.0.4
pandas==1.0.5
pyLDAvis==2.1.2
spacy==2.3.1
XlsxWriter==1.2.9
Very recently, AWS announced the support of EFS for Lambda.Read the announcement here. EFS or the Elastic File System is the NFS file system for compute nodes. Read more about them here.
With this now you can essentially attach a network storage to your lambda function. I have personally used it to load huge reference files which are over the limit of Lambda's file storage. For a walkthrough you can refer to this article by AWS. To pick the conclusion from the article:
EFS for Lambda allows you to share data across function invocations, read large reference data files, and write function output to a persistent and shared store. After configuring EFS, you provide the Lambda function with an access point ARN, allowing you to read and write to this file system. Lambda securely connects the function instances to the EFS mount targets in the same Availability Zone and subnet.