Practice Lab 1 Airflow Building your first pipeline

Course Q&A Data Engineering Source Systems, Data Ingestion, and Pipelines
1

Hello!

I was able to code what was required in each exercise and I finally upload it to the airflow platform. I have two errors, but I can’t track them.

How can I validate the file before uploading it?
How can I validate each function before uploading it to airflow?

I would appreciate it if someone could help me.

Luis Millao.

2

Hello @lumillao

There are two types of errors you may encounter with your DAGs.
First, if there is a problem with the definition of the DAG or its tasks, Airflow won’t be able to import your DAG. On such occasions, after uploading your python files to the DAG bucket, you will see a error pop up on top of the Airflow homepage saying where the compiler had problem reading the python file. If this happens, your DAG won’t show up in the list of DAGs in Airflow.
Secondly, there might be exceptions that raise only after the tasks are run. For example, you might have defined your tasks correctly, but one of your tasks has a bug that prevents it from completing. In these cases, there won’t be a problem uploading the DAGs, and Airflow will import them successfully. However, after you trigger your DAG, you will see that somewhere one of your tasks has failed. You can click on your DAG and go to the DAG page. There, you will see a red light for one of the tasks. The UI looks like this, but with a red light:

image
image1850×757 200 KB

If you want to debug, you can click on the failed task’s square, then there will be a Logs tab on the right gadget. You can open it, and all the logs from your task are there.

Hope this helps.

3

Hello! @Amir_zare

Ok, I understand. Do you know how to validate it before uploading it to the platform?

Best regards!
Luis Millao

4

@lumillao I don’t think there is a way to validate code before uploading to Airflow. You can always upload the files and see if they are imported by Airflow successfully.

5

Hello! @Amir_Zare

Could you use the Debugging Airflow DAGs?

For example:

if __name__ == “__main__”:
dag.test()

Add these two lines to the bottom of your dag file.

I think it could be helpful for other learners.

Best Regards!
Luis Millao.

6

Hello again @lumillao
I didn’t know Airflow DAGs have such a functionality. Have you tested it?
Thank you for sharing it.