I have an ECS Fargate cluster with a service that generates tasks based on how many messages are in a queue. Each task does long polling on the queue and processes one message at a time. If the queue gets over 5 messages a new task is spun up and it starts taking messages. When the queue then drops below 5 messages from the previous threshold, it shuts down a task.
My question is, when the service decides to scale down, how does it know what task to kill? All tasks could be processing a message. Each task continuously runs and does long pulling from SQS, so how would it know that a task is in valid shut down state (just completed a message) and a non-valid shut down state (currently processing a message).
There is an open issue for improving termination of tasks as people have same concerns as you:
From the issue, you can minimize the impact of termination of tasks on your processes by using stopTimeout. The parameter is:
Time duration (in seconds) to wait before the container is forcefully killed if it doesn't exit normally on its own.
But there is also a new feature for ESC:
With this you can setup:
dependencies for container startup and shutdown as well as a per-container start and stop timeout value.
So generally there is no full control over termination of tasks, like you have in termination of instances in AutoScaling Group. But the things are getting worked on.