I notice the cpu utilization of pods in same hpa varies from 31m to 1483m. Is this expected and normal? See below for the cpu utilization of 8 pods which are of the same hpa.
NAME CPU(cores)
myapp-svc-pod1 31m
myapp-svc-pod2 87m
myapp-svc-pod3 1061m
myapp-svc-pod4 35m
myapp-svc-pod5 523m
myapp-svc-pod6 1483m
myapp-svc-pod7 122m
myapp-svc-pod8 562m
HPA main goal is to spawn more pods to keep average load for a group of pods on specified level.
HPA is not responsible for Load Balancing and equal connection distribution.
For equal connection distribution is responsible k8s service, which works by deafult in iptables mode and - according to k8s docs - it picks pods by random.
Your uneven cpu load distribution is most probably caused by the data it processes. To make sure it's not the issue with k8s service, I'd recomment you to export some metrics like number of connections and time it takes to process one request. After you gathered this data, have a look at it and see if a pattern emerges.
Now to answer your question:
Is this expected and normal?
It depends what you consider as normal, but if you were expecting more equal cpu load distribution then you may want to rethink your design. It's hard to say what you can do to make it more equal because I don't know what myapp-svc-pods do, but as I already mentioned, it may be best to have a look at the metrics.