Business
Jobs
  • About Us
  • Solutions
    • Job Postings
      Post your job and receive qualified candidates in 48h.
    • Candidate Assessments
      500+ technical and psychological tests, plus anti-fraud.
    • Headhunting
      Tailor-made executive search from start to finish.
    • Payroll + EOR
      Payroll dispersal and EOR across 15+ LATAM countries.
  • Pricing
  • Jobs

0

505
Views
Best practices for syncing DAGs in worker pods in Apache Airflow + Kubernetes

I have an Airflow deployment in Kubernetes configured with Kubernetes Executor. I setup a Git sync initContainer to fetch DAGs from my own DAGs repository, and I can see my Worker and Scheduler containers are running fine and have their DAGs folder updated with the latest commit. Thus, I wrote a simple DAG with two BashOperator but my worker pod, when scheduled, shows the following error:

[2021-01-14 14:01:13,808] {dagbag.py:440} INFO - Filling up the DagBag from /var/lib/airflow/dags/first_pod.py
Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
    args.func(args)
  File "/usr/local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/cli.py", line 89, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 216, in task_run
    dag = get_dag(args.subdir, args.dag_id)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/cli.py", line 187, in get_dag
    raise AirflowException(
airflow.exceptions.AirflowException: dag_id could not be found: test. Either the dag did not exist or it failed to parse.

I understand my worker pod doesn't have any DAG. I would like to avoid to start a Git sync task every time a pod is spawned, so I would like to know if there are some best practices around for worker pod sync since I can't find any. Thanks in advance.

over 4 years ago · Santiago Trujillo
Answer question
Find remote jobs

Discover the new way to find a job!

Top jobs
Top job categories
Business
Post vacancy Pricing Sales
Legal
Terms and conditions Privacy policy
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Show me some job opportunities
There's an error!