I created simple kubernetes cluster with demo app. When creating the cluster I installed prometheus-stack and nginx-ingress-controller with helm (default values files).
After the cluster is set up I create Ingress object to expose prometheus, grafana and alertmanager with:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: monitoring-ingress
namespace: monitoring
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- http:
paths:
- path: /prometheus(/|$)(.*)
pathType: Prefix
backend:
service:
name: prometheus-operated
port:
number: 9090
- path: /alertmanager(/|$)(.*)
pathType: Prefix
backend:
service:
name: prometheus-stack-kube-prom-alertmanager
port:
number: 9093
- path: /grafana(/|$)(.*)
pathType: Prefix
backend:
service:
name: prometheus-stack-grafana
port:
number: 80
When I try to access prometheus via <ingress-controller's external IP>/prometheus it resolves to <ingress-controller's external IP>/graph and displays 404 - page not found.
If I use for example kubectl port-forward svc/prometheus-operated 9090:9090 -n monitoring I can reach prometheus without problem.
I can reach alertmanager through <ingress-controller's external IP>/alertmanager. The path is resolved to <ingress-controller's external IP>/alertmanager/#/alerts
I suspect there is something wrong with the path rewriting but can't figure out what.
Please help...
In the end I was able to find 2 solutions to this problem.
Option 1
I had DNS zone in azure (where my cluster lives as well) and there I added subdomains for grafana, prometheus and alertmanager pointing to the ingress-controller external IP.
When deploying kube-prometheus-stack with Helm Chart I provided default root path for prometheus in values.yaml file with folowing configuration:
prometheus:
prometheusSpec:
externalUrl: http://prometheus.mydomainname.something
The Ingress manifest for (for example Prometheus) then needs to include the host address and contain only root path "/".
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: prometheus-new
namespace: monitoring
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:
- host: prometheus.mydomainname.something
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: prometheus-stack-kube-prom-prometheus
port:
number: 9090
Option 2
You can also provide default paths for each of the applications from kube-prometheus-stack (prometheus, grafana, alertmanager) as:
prometheus:
prometheusSpec:
externalUrl: http://mydomainname.something/prometheus
and make Ingress manifest to redirect based on the path with:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: prometheus-new
namespace: monitoring
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:
- http:
paths:
- path: /prometheus
pathType: Prefix
backend:
service:
name: prometheus-stack-kube-prom-prometheus
port:
number: 9090