I have a Typescript project using amqp to connect to an AmazonMQ broker. I have many workers connecting to it and processing jobs from queues without issue most of the time.
It seems that for some reason, there are jobs in the queue being retried by RabbitMQ (as told by headers being set from RMQ) but when the worker goes to ack it, there is the following error thrown
Error: write ECONNRESET
at WriteWrap.onWriteComplete [as oncomplete] (internal/stream_base_commons.js:94:16)
at WriteWrap.callbackTrampoline (internal/async_hooks.js:130:17)
at handleWriteReq (internal/stream_base_commons.js:53:26)
at writeGeneric (internal/stream_base_commons.js:145:15)
at TLSSocket.Socket._writeGeneric (net.js:798:11)
at TLSSocket.Socket._write (net.js:810:8)
at writeOrBuffer (internal/streams/writable.js:358:12)
at TLSSocket.Writable.write (internal/streams/writable.js:303:10)
at roundrobin (/app/node_modules/amqplib/lib/mux.js:61:25)
at Mux._readIncoming (/app/node_modules/amqplib/lib/mux.js:67:3)
at Immediate._onImmediate (/app/node_modules/amqplib/lib/mux.js:97:12)
at processImmediate (internal/timers.js:464:21)
at process.topLevelDomainCallback (domain.js:152:15)
at process.callbackTrampoline (internal/async_hooks.js:128:24)
This only occurs when a long lived worker (an hour or less processing the job from the queue and it has heartbeat in the connection URL) tries to ack the completion of its job for a job that has already been completed. This failure causes my worker to die, causing RMQ to requeue the job it seems.
I am working on getting a dummy repro built but it's being difficult.
I have a feeling it is because I am trying to ack a job that has already been acked. I would have imagined that RMQ would not have re-queued the job if that was the case. Can anyone validate/show me where I could validate if RMQ would requeue a job that fails to be acked if it failed to be acked because it was acked already?
Is there another path I can go down to find out what the issue is? I don't have error handling attached to the channel released into production yet and only have these logs attached to queue. I am trying to push an update out to get logs for channel to see if there's more information there. Where else can I look for issues?