My spring-boot application running in AWS Elasic Beanstalk loads data from a location in Amazon S3. The data is refreshed every 10 minutes and I want all instances of beanstalk to reload the data from S3.
I had a hacky way to get this done. I did the following.
1. Implemented an end point in beanstalk application which, when called, triggers the reload.
2. I have a script that determines the IP address of each beanstalk instance given the beanstalk application's `environment name`.
3. I then iterate through all these IP addresses and call the end point mentioned in point 1 above.
This worked so far because the IP addresses of the instances were publicly addressable. This is no longer going to be the case as we have to use only private IPs.
What's the right way of triggering the data reload? I was thinking of the following:
1. Set up Amazon S3 bucket to generate an event and post to a SNS topic.
2. Have an SNS client in my beanstalk application which subscribes to that topic and listens for events.
3. Upon receiving an event from that topic, trigger the data reload.
I have done (1) but do not know how to do (2) and (3). How do I subscribe to the topic in Java code? SNS allows only 3 methods of notifying - using SQS (AWS simple queueing service), SMS and via HTTP/HTTPS URLs.
The URLs approach won't work because it will forward the request to only one instance behind load balancer. So not all instances will reload.
SQS also won't work because once one instance reads the message, the message will be de-queued and the other instances will not receive the trigger message.
Email also won't work (or I don't know how to make it work).
Any help/ideas? Pointers to Java code would be much appreciated.
I am in the same boat as you, I have an ElasticBeanstalk auto-scaling app and I want each EC2 instance to subscribe to an SNS topic. I therefore want to know how to build up a URL to use as my subscription listener endpoint. I'm not quite sure whether Java will pick up my private or public address with it's IP functions.
I have seen comments in forums online that criticise that by adding a HTTP Post listener in your code to receive the messages is an awful implementation (from a security perspective), this alone implies that it's possible. However in other contexts talking about VPC's it sounds like things start to break again.
Even if I did solve all of the above, the additional issue I would then have is running the same code on my development machine, most likely it would use my private IP address that's unreachable from the internet, causing my code to fail and might actually be the same issue on the EC2 instance if the private IP address is picked up.
From what I've read so far, you are better off using Amazon SQS, register some sort of pub/sub queue, then poll for updates from your EC2 instances with a timeout.
In my case I need to work out if my app is being shutdown (ContextListener will help here) then stop the SQS queue receiver loop, so after my timeout I'll likely check whether a shutdown flag is set or not, if not loop round again.
It certainly seems a whole lot more complicated than SNS pub/sub. I wanted to share what I have learned in the hope it may help you/others in the meantime. Also I've started a question here on stack overflow regarding pub/sub within the context of AWS Elastic Beanstalk EC2 instances.