Ingress gateway is retrying if the upstream returns 502. Most of the time it is working as expected. Sometimes gateway returns 507 "exceeded request buffer limit while retrying upstream" without retrying.
From the logs I can see this: app 502 -> istio-proxy sidecar 502 -> ingress gateway 507 -> client. Unable to find other errors in logs related to ingress-gateway.
Requests are 1-30Mb in size. Any ideas where to look for the issue?
VirtualService:
...
retries:
attempts: 4
retryOn: 502,retriable-status-codes,connect-failure
retryRemoteLocalities: true
...
EnvoyFilter:
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: buffer-limit
spec:
configPatches:
- applyTo: LISTENER
listenerMatch: 0.0.0.0_8080
patch:
operation: MERGE
value:
per_connection_buffer_limit_bytes: 100000000
Solution was to use correct Envoy filter:
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
namespace: istio-system
name: buffer-limit
spec:
workloadSelector:
labels:
istio: ingressgateway
configPatches:
- applyTo: LISTENER
patch:
operation: MERGE
value:
per_connection_buffer_limit_bytes: 100000000
The filter is working on v1.8.4
To check if the filter is applied:
istioctl proxy-config listeners <istio-ingressgateway-pod> -o json -n istio-system