Any idea why I can't run docker using Airflow?
The same docker command runs fine from terminal using the user that runs airflow. So there is no permission issue on the Linux side.
But when put it into Airflow dag, it complains
docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.39/containers/create: dial unix /var/run/docker.sock: connect: permission denied.
The command is simply docker run --rm -v /data:/data:ro docker_image mycommand. It is wrapped inside a python subprocess, so I can't use docker operator.
Note this is not the same issue with the question: How to fix "dial unix /var/run/docker.sock: connect: permission denied" when group permissions seem correct?
There it is a linux permission issue as it cannot run the docker run command. Here the problem is more with Airflow, I think.
Faced a similar issue . Got it fixed by changing the default_owner(default=airflow) in airflow.cfg to the user having permission to access docker(i.e username belonging to docker group).
Interestingly passing owner in 'default_args' in the DAG definition didn't seem to work
Looks like the reason is in access rights to /var/run/docker.sock. For dev environment, you can do:
docker exec -ti -u root <container> bash
sudo chmod 777 /var/run/docker.sock