Empresas
Empregos
  • Sobre nós
  • Soluções
    • Publicação de vagas
      Publique sua vaga e receba candidatos qualificados em 48h.
    • Avaliações de candidatos
      Mais de 500 testes técnicos e psicológicos, mais anti-fraude.
    • Headhunting
      Busca executiva personalizada do início ao fim.
    • Folha de Pagamento + EOR
      Dispersão de folha e EOR em mais de 15 países da LATAM.
  • Preços
  • Empregos

0

507
Visualizações
Best practices for syncing DAGs in worker pods in Apache Airflow + Kubernetes

I have an Airflow deployment in Kubernetes configured with Kubernetes Executor. I setup a Git sync initContainer to fetch DAGs from my own DAGs repository, and I can see my Worker and Scheduler containers are running fine and have their DAGs folder updated with the latest commit. Thus, I wrote a simple DAG with two BashOperator but my worker pod, when scheduled, shows the following error:

[2021-01-14 14:01:13,808] {dagbag.py:440} INFO - Filling up the DagBag from /var/lib/airflow/dags/first_pod.py
Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
    args.func(args)
  File "/usr/local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/cli.py", line 89, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 216, in task_run
    dag = get_dag(args.subdir, args.dag_id)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/cli.py", line 187, in get_dag
    raise AirflowException(
airflow.exceptions.AirflowException: dag_id could not be found: test. Either the dag did not exist or it failed to parse.

I understand my worker pod doesn't have any DAG. I would like to avoid to start a Git sync task every time a pod is spawned, so I would like to know if there are some best practices around for worker pod sync since I can't find any. Thanks in advance.

over 4 years ago · Santiago Trujillo
Responde à pergunta
Encontrar trabalhos remotos

Descubra a nova forma de encontrar um emprego!

melhores empregos
Principais categorias de trabalho
Empresas
Postar vaga Preços Comercial
Jurídico
Termos e Condições Política de privacidade
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Recomende algumas ofertas para mim
Preciso de ajuda