For a project I'm working on, there is a need to pull down a largish text file that's updated daily and made available at a specific customer URL, and store it in AWS S3 which then triggers downstream processing of the file (details unimportant).
I was thinking of having the download + store in S3 done by an AWS Lambda triggered every 24 hours by CloudWatch, which would work, but there's a catch: the file is 36MB in size and is served by a host that throttles downloads to 100kB/s (outside of my control). This means it takes at least 360s (i.e. 6 mins) to completely download the file. AWS Lambda functions however have an upper limit of 300s run time, which effectively makes it impossible to use for this task as the Lambda times-out and exits before the file is completely downloaded.
I'm looking for suggestions of ways of working around the 300s run time limit of AWS Lambda to achieve this goal.
As long as I'm sticking to AWS, the only alternative I see is to set up a cron job on an EC2 instance, but that seems expensive / overkill, especially if I end up not needing an always-on EC2 for anything else.
Thanks!
I'd have the Lambda spin up a small EC2 instance that runs the copy job. You could have either a custom AMI for the EC2 instance or a cloud-init script that sets everything up. Let the program on the EC2 run for a bit and remember that you get billed for an hour regardless of how much time you need. But if the entire process takes 15 mins (as there is no way to guarantee against some traffic congestion) and you're using a t2.nano, you got billed USD $0.006 (6 tenths of a cent) plus some I/O and, likely, EBS space. I'd be willing to bet you'd spend very little.
Once the job is done it terminates the EC2 instance it's running on.
I realize that this is a bit of a hassle - CloudWatch triggers Lambda which triggers the EC2. But CloudWatch alone isn't going to be able to do what you need for the EC2.