I have a python callable process_csv_entries that processes csv file entries. I want my task to complete successfully only if all entries were processed successfully. Task should fail otherwise
def process_csv_entries(csv_file):
# Boolean
file_completely_parsed = <call_to_module_to_parse_csv>
return not file_completely_parsed
CSV_FILE=<Sets path to csv file>
t1 = PythonOperator(dag=dag,
task_id='parse_csv_completely',
python_operator=process_csv_entries,
op_args=[CSV_FILE])
t1 seems to complete successfully irrespective of returned value. How do I force PythonOperator task to fail?
raise exception when you meet the error condition ( in your case: when file is not sucesfully parsed)
raise ValueError('File not parsed completely/correctly')
raise relevant error type with suitable message
Yes, raise AirflowException, this will cause the task to move immediately to failure state.
from airflow import AirflowException
ValueError can be used for fail and retry.
AirflowFailException is now available to make the task fail without retries