I have a new server running Prometheus in docker-compose. I want to be able to re-load the configuration file (prometheus.yml) without have to stop and start the container.
Of course since I persist the storage of promethues in a volume the stop and start isn't really a problem but it seems like overkill, especially since prometheus itself has such a handy api to reload configs.
I see other people with similar questions (e.g. here) but I have been unable to get those solutions to work for me. Maybe I'm overlooking something there.
docker-compose.yml
version: "3"
services:
grafana:
restart: always
container_name: grafana
image: grafana/grafana:6.2.1
ports:
- 3000:3000
volumes:
- grafanadata:/var/lib/grafana
prometheus:
restart: always
container_name: prometheus
image: prom/prometheus:v2.10.0
privileged: true
volumes:
- ./configuration/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
- prometheusdata:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--web.enable-admin-api'
- '--web.enable-lifecycle'
ports:
- 9090:9090
node:
restart: always
container_name: node
image: prom/node-exporter:v0.18.0
ports:
- 9100:9100
volumes:
grafanadata:
prometheusdata:
Alas, my results..
When I run curl -X POST http://localhost:9090/-/reload the docker-compose logs give:
prometheus | level=info ts=2019-06-17T15:33:02.690Z caller=main.go:730 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
prometheus | level=info ts=2019-06-17T15:33:02.691Z caller=main.go:758 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml
So the prometheus' end is working fine.. All good so far.
However, when I edit ./configuration/prometheus/prometheus.yml the changes don't propogate to the container.
Furthermore, when I try to edit /etc/promethus/prometheus.yml in container I see that it is read only (and as an aside, the container does not have a 'sudo' command).
Is there a docker native way to hot/live reload these config files to the container directory?
As stated, the down/start option works for now but I'm curious if there is a more elegant solution.
docker-compose kill -s SIGHUP prometheus does the trick, so Vishrant was certainly on to something there.
Your problem might be, that your editor is not saving the changed file under the same inode. So dockerd does not realize that the file changed, since its written to a new inode. Solutions might be to mount the complete directory or tell your editor to not use a temporary file:
I.e. for Sublime, setting "atomic_save": false to user preferences might help (results unclear) or for vim see https://github.com/moby/moby/issues/15793#issuecomment-571932545
This issue is very interesting for this topic in general: https://github.com/moby/moby/issues/15793
I will assume you have already updated the /etc/prometheus/prometheus.yml file and you are just asking about the reloading of config. (Though the config can be part of configmap, and you can avoid the pod restart when the configmap changes).
Run the following command in order to reload the config without stopping the prometheus process:
docker exec <prometheus_container_name> sudo killall -HUP prometheus
Explanation: sudo killall -HUP prometheus command sends a SIGHUP signal to the prometheus process to reload the configurations.
Validate the changes by calling:
curl -X GET http://localhost:9090/api/v1/status/config