Empresas
Empregos
  • Sobre nós
  • Soluções
    • Publicação de vagas
      Publique sua vaga e receba candidatos qualificados em 48h.
    • Avaliações de candidatos
      Mais de 500 testes técnicos e psicológicos, mais anti-fraude.
    • Headhunting
      Busca executiva personalizada do início ao fim.
    • Folha de Pagamento + EOR
      Dispersão de folha e EOR em mais de 15 países da LATAM.
  • Preços
  • Empregos

0

339
Visualizações
SQS fifo queues not ensuring single time delivery when used as lambda trigger

I have a fifo queue for which it acts trigger to a lambda and there are no other consumers for this fifo queue.

I want my lambda to not receive any duplicates, for that I am making sure that my messages have uniqueid and content based deduplication is enabled such that there are no duplicates. But as the documentation says

Amazon SQS FIFO queues ensure that the order of processing follows the message order within a message group. However, it does not guarantee only once delivery when used as a Lambda trigger. If only once delivery is important in your serverless application, it’s recommended to make your function idempotent. You could achieve this by tracking a unique attribute of the message using a scalable, low-latency control database like Amazon DynamoDB.

Does this mean I will receive duplicates to my lambda where it is the only consumer of the fifo queue even though deduplication is enabled?

over 4 years ago · Santiago Trujillo
2 Respostas
Responde à pergunta

0

There are two things in play here that you seem to be mixing together.

  • On one hand, there's the delivery model of SQS. With SQS FIFO queues, you correctly noted that it enables exactly-once delivery.

  • On the other hand, there's the execution model of Lambda functions. The execution model is at-least-once. This is independent of any concurrency settings, by the way.

There are multiple reasons why Lambda may execute a function more than once. Most notably, and independently of SQS FIFO queues, Lambda has built-in retry. Depending on what kind of error may happen, and what kind of externally observable side effects your Lambda code has, you may see your code run more than once for a single actual invocation.

That said, there are still other reasons why a message sent to an SQS FIFO queue may end up being processed by Lambda more than once. For example, if your Lambda function takes longer than the VisibilityTimeout settings of the queue or message to finish processing the entire batch it received, then all those messages are going to become Visible again in the queue, and another invocation of your Lambda function will (certainly, not just "probably") receive those messages again.

So the bottom line is: you need to write idempotent code in Lambda - not because of SQS FIFO (it does correctly allow for prevention of duplicate deliveries), but because of Lambda retries and th potential for the messages to become visible again in the queue due to slow processing (it's always at-least-once execution model).

over 4 years ago · Santiago Trujillo Relatório

0

Even you have a single lambda function as consumer, depends on the concurrency setting on your lambda function, there can be multiple invocations. Which means at a given time, multiple invocations can pick messages in your SQS queue.

Simple workaround for this is to set the lambda function's concurrency to 1, so that at a given time, only one invocation is allowed. So the messages in the queue will be processed in order.

(However this will leads to a bottleneck if you have a large number of messages in the queue to be processed.)

over 4 years ago · Santiago Trujillo Relatório
Responde à pergunta
Encontrar trabalhos remotos

Descubra a nova forma de encontrar um emprego!

melhores empregos
Principais categorias de trabalho
Empresas
Postar vaga Preços Comercial
Jurídico
Termos e Condições Política de privacidade
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Recomende algumas ofertas para mim
Preciso de ajuda