Question

0

610

Views

How to run a python function or script somescript.py on the KubernetesPodOperator in airflow?

I am running a Celery Executor and I'm trying to run some python script in the KubernetesPodOperator. Below are examples of what I have tried that didn't work. What am I doing wrong?

Running sctipt

org_node = KubernetesPodOperator(
    namespace='default',
    image="python",
    cmds=["python", "somescript.py" "-c"],
    arguments=["print('HELLO')"],
    labels={"foo": "bar"},
    image_pull_policy="Always",
    name=task,
    task_id=task,
    is_delete_operator_pod=False,
    get_logs=True,
    dag=dag
)

Running function load_users_into_table()

def load_users_into_table(postgres_hook, schema, path):
  gdf = read_csv(path)
  gdf.to_sql('users', con=postgres_hook.get_sqlalchemy_engine(), schema=schema)

org_node = KubernetesPodOperator(
    namespace='default',
    image="python",
    cmds=["python", "somescript.py" "-c"],
    arguments=[load_users_into_table],
    labels={"foo": "bar"},
    image_pull_policy="Always",
    name=task,
    task_id=task,
    is_delete_operator_pod=False,
    get_logs=True,
    dag=dag
)

over 4 years ago · Santiago Trujillo

2 answers

Answer question

0

Answer question

Find remote jobs

Answer 1 · 2022-05-13T16:23:22.795Z

The script somescript.py must be in Docker image.

Step-1: let's create a image https://docs.docker.com/develop/develop-images/dockerfile_best-practices/.

FROM python:3.8

# copy requirement.txt from local to container
COPY requirements.txt requirements.txt
# install dependencies into container (geopandas, sqlalchemy)
RUN pip install -r requirements.txt
# copy the python script from local to container
COPY somescript.py somescript.py
ENTRYPOINT [ "python", "somescript.py"]

Step-2: Build and push the image into public Docker repository https://hub.docker.com.

NB: kubernetes_pod_operator looks for image from public docker repo

# build image
docker build -t my-python-img:latest .
# test if your image works perfectly
docker run my-python-img:latest

# push image.
docker tag my-python-img username/my-python-img
docker push username/my-python-img
docker pull username/my-python-img

step-3: Lest's create k8s task.

p = KubernetesPodOperator(
        namespace='default',
        image='username/my-python-img:latest',
        labels={'dag-id': dag.dag_id},
        name='airflow-my-image-pod',
        task_id='load-users',
        in_cluster=False,       #False: local, True: cluster               
        cluster_context='microk8s',  
        config_file='/usr/local/airflow/include/.kube/config',
        is_delete_operator_pod=True,
        get_logs=True,
        dag=dag
    )

If you don't understand where configuration file comes from, look here: https://www.astronomer.io/docs/cloud/stable/develop/kubepodoperator-local.

Finally: I want to mention something important when working with databases (credentials). Kubernetes offers the use secret to secure sensitive information. https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html

Answer 2 · 2022-05-13T16:23:22.803Z

KubernetesPodOperator launches a Kubernetes pod that runs a container as specified in the operator's arguments.

First Example

In the first example, the following happens:

KubernetesPodOperator instructs K8s to lunch a pod and prepare to run a container in it using the python image (the image parameter) from hub.docker.com (the default image registry)
ENTRYPOINT of the python image is replaced by ["python", "somescript.py" "-c"] (the cmd parameter)
CMD of the python image is replaced by ["print('HELLO')"] (the arguments parameter)
...
The container is run

So, the complete command that is run in the container is

python somescript.py -c print('HELLO')

Obviously, the official Python image from Docker Hub does not have somescript.py in its working directory. Even if did, it probably would have been not the one that you wrote. That is why the command fails with something like:

python: can't open file 'somescrit.py': [Errno 2] No such file or directory

Second Example

In the second example, pretty much the same happens as in the first example, but the command that is run in the container (again based on the cmd and arguments parameters) is

python somescript.py -c None

(None is the string representation of the load_users_into_table()'s return value)

This command fails, because of the same reasons as in the first example.

How It Could be Done (a Sketch)

You could build a Docker image with somescript.py and all its dependencies. Push the image to an image registry. Specify the image, ENTRYPOINT, and CMD in the corresponding parameters of KubernetesPodOperator.

0

610