We are running our stack on Heroku. We are using Django 2.2 with a Postgres 11 DB. Our build pipeline (Github Actions) pushes to Heroku (git push https://git.heroku.com...) and immediately afterwards runs the migrations (heroku run python manage.py migrate --app heroku-app-name). All of that was working with a Postgres 9.6 database and is still working in our staging environment (Postgres 11). Now with production being on Postgres 11, the django migrate command is just stuck and doesn't produce any output, even so there are no actual migrations to apply.
The only differences between our production setup and our staging setup are a follower/slave in production attached to the master DB and "production workload".
In order to fix that deployment I have to run a
heroku pg:killall -a heroku-app-name
heroku restart -a heroku-app-name
At this point the migrations task in the build pipeline fails.
and afterwards migrations can be applied manually without problems:
heroku run python manage.py migrate --app heroku-app-name
So for some reason the migrations command is "waiting" for something, some database lock or whatever, yet I cannot put my finger on it. Especially odd for me is the fact that its also stuck where no migrations are to applied. Why would it be stuck there?
We found the solution. There are actually three things coming together.
pg_dump) the backup job on Postgres 11 (and others?) has now a more exclusiv lock on some operations. I assume that that a CREATE EXTENSION ... IF NOT EXISTS (even so the extension already exists) cannot be executed while having a backup job running in parallel.(I am sure there are some Postgres internas missing to explain this more correctly)
As a result of these three things, the DB blocks the migrate operation, waiting for the backup job to finish. I have moved the daily backup job to a different time and reconfigured our pipeline to wait for the "pre-deploy"-backup to finish first.