We have a GKE cluster with Ingress (kubernetes.io/ingress.class: "gce") where one backend is serving our production site.

The cluster is regional one with 3 zones enabled (autoscaling enabled).

The backend serving production site is a Varnish server running as Deployment - single replica. Behind Varnish there are multiple Nginx/PHP pods running under HorizontalPodAutoscaler.

The performance of of the site is slow. We have noticed by using GCP console that all traffic is routed only to one Backend and there is only 1/1 healthy endpoint in one zone?

We are getting exclamation mark next to the serving Backend with message 'Usage is at capacity, max = 1' and 'Backend utilization: 0%'. The other backend in second zone has no endpoint configured? And there is no third backed in third zone?

Initially we were getting a lot of 5xx responses from the backend at around 80RPS rate so we have turned on CDN via BackendConfig.

This has reduced 5xx responses including RPS on the backend to around 9RPS and around 83% RPS is being served from CDN.

We are trying to figure it out if it is possible to improve our backend utilization as clearly serving 80RPS from one Varnish server which has many pods behind should be easily achievable. We can not find any underperforming POD (varnish itself or nginx/php) in this scenario.

Is GKE/GCP throttling the backend/endpoint to only support 1RPS?

Is there any way to increase RPS per endpoint and increase number of endpoints, at least one per zone?

Is there any documentation available that explain how to scale such architecture on GKE/GCP?