0
128
ViewsHow to generate an alert when a SQS message is sent to the dead letter queue?
Goal
Aiming to have a CloudWatch Alert triggered when a message from an SQS queue to a lambda function exceeds the maximum retries.
Problem
I presumed that this would be easy and the NumberOfMessagesReceived metric would reflect this. Those familiar with this will know that this is not the case.
Solutions
The 'Limbo' Solution
My quick and easy solution for this problem was the introduce a "Limbo" which acts as the first DLQ and within seconds pushes the message to the final/actual DLQ. In the metrics this results in a spike in the "Limbo" queue's visible messages metric. So having an alert threshold of "> 0" means that every time that queue receives a message an alert can be issued.
However the powers above me are not happy with having a "Limbo" queue for every time we want this functionality.
As far as I have been able to figure out there are some alternative methods but these seem worse than the Limbo Solution.
New Lambda Function
The first is to have a new lambda function that uses a SQS DLQ as a source and generates the alert.
Lambda Runtime Interception
Second is to have the have logic inside the existing lambdas (that process SQS messages) read the amount of times a message has gone been retried and on the final time generate the alert. This kind of removes the advantage of using a queue and a re-drive policy in the first place, and is an over engineered solution.
Metric Maths
The last alternative I can think of is to is to use some Metric Maths to look at the DLQ and calculate if there was been an increase in the last X minutes.
These all seem like strange and overly complex solutions to what (I am convinced) must have a simple implementation. How do I create an alert every time a DLQ receives a message?
2 answers
Answer question0
I came across this same issue and had success implementing it using Metrics Math. Cloudwatch has a RATE() function which:
"Returns the rate of change of the metric per second. This is calculated as the difference between the latest data point value and the previous data point value, divided by the time difference in seconds between the two values."
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/using-metric-math.html
So I created an alarm which looks at the rate of change of the ApproximateNumberOfMessagesVisible metric on the Deadletter queue. It goes into alarm when the rate of change is greater than 0. Here is a Cloudformation template example for the alarm:
DeadletterAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: "DEADLETTER_ALARM"
ComparisonOperator: GreaterThanThreshold
EvaluationPeriods: 1
TreatMissingData: missing
Threshold: '0'
Metrics:
- Id: r1
Expression: RATE(FILL(m1, 0))
ReturnData: true
- Id: m1
Label: VisibleAverage
ReturnData: false
MetricStat:
Stat: Average
Period: '300'
Metric:
MetricName: ApproximateNumberOfMessagesVisible
Namespace: AWS/SQS
Dimensions:
- Name: QueueName
Value: "Deadletter_queue_name"
0
One other way to accomplish this is to alarm on ApproximateNumberOfMessagesDelayed. Then you just need to set a delay on your DLQ. So it could look something like this:
MyDLQAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: MyDLQAlarm
AlarmDescription: "Alarm when we have 1 or more failed messages in 10 minutes for MyQueue."
Namespace: "AWS/SQS"
MetricName: "ApproximateNumberOfMessagesDelayed"
Dimensions:
- Name: "QueueName"
Value:
Fn::GetAtt:
- "MyQueue"
- "QueueName"
Statistic: "Sum"
Period: 300
DatapointsToAlarm: 1
EvaluationPeriods: 2
Threshold: 1
ComparisonOperator: "GreaterThanOrEqualToThreshold"
AlarmActions:
- Ref: "SNSTopic"
Then your DLQ can look like:
MyQueueDLQ:
Type: AWS::SQS::Queue
Properties:
QueueName: MyQueueDLQ
MessageRetentionPeriod: 1209600
DelaySeconds: 60