I'm trying to figure out what happens to write requests when a channel is non-responsive. For example, this happens when the peer suddenly drops off the network, and we never get a RST.
From reading docs and Understanding netty channel buffers and watermarks, it seems that once the high WriterBufferWaterMark is reached, ChannelOutboundBuffer will be considered full, and:
Channel.isWritable() will start to return false.
From Channel.isWritable():
Returns true if and only if the I/O thread will perform the requested write operation immediately. Any write requests made when this method returns false are queued until the I/O thread is ready to process the queued write requests.
My initial question was: What happens when we keep writing anyway?
Where will the data been queued if Netty channel isWritable return false states that there is an internal buffer, and implies it is unbounded.
The question becomes, where exactly are the write requests queued, and more importantly can I observe the size of these queues?
Or: is there some limit when Netty/OS will detect that the connection is broken and needs to be closed?
It will still be queued in ChannelOutboundBuffer. You are responsible to stop writing once the buffer is full and start again one it empties. You can observe this via ChannelInboundHandler.channelWritabilityChanged(...).
Looking at the source for ChannelOutboundBuffer in 4.1.37, the method addMessage always adds incoming messages to an internal linked list, regardless of the state of isWritable. This list grows unbounded.
The total byte length of this buffer is reachable via channel.unsafe().outboundBuffer().totalPendingWriteBytes().
For an indirect value, if you don't want to use unsafe(), you can alternatively use channel.bytesBeforeWritable() which will return the queued bytes minus the low watermark.
When you call the ChannelHandlerContext#write method, netty will eventually write the data you want to send into a queue called channelOutboundBuffer.
When the network data receiving speed and processing speed are getting smaller and smaller due to network congestion or the high load of the Netty client, the sliding window of TCP keeps shrinking to reduce the sending of network data until it reaches zero, while the Netty server has a lot of frequent Write operation, continuously write to ChannelOutboundBuffer.
As a result, the data cannot be sent out, but the Netty server is constantly writing data, which will slowly burst the ChannelOutboundBuffer and cause OOM.
So Netty introduced LOW_WATER_MARK and HIGH_WATER_MARK to control the capacity limit of ChannelOutboundBuffer to avoid unlimited growth of ChannelOutboundBuffer
you can use ctx.channel().unsafe().outboundBuffer().totalPendingWriteBytes() to Check the total memory usage of the ChannelOutboundBuffer