C2 W4 Lab 1 - No DAGs appearing in airflow

Course Q&A Data Engineering Source Systems, Data Ingestion, and Pipelines
1

Airflow is not importing DAGs.

Initially had some typos in first uploaded script, these were flagged in AirFlow as parsing errors. Corrected them and re-sync src folder with S3.

Confirmed new files (timestamp) were uploaded to the dags bucket in S3.

Nothing in Airflow - waited over 10 mins. Restarted Airflow. No parsing errors. Nothing ??

Even uploaded the solution file in case i was blind to a silent failure. Still no DAGs appearing in AirFlow. Nothing in AirFlow audit trail.

Nuked the S3 folder , re uploaded files, restarted AirFlow – no DAGs or errors.

Help Appreciated

1 Like
2

Hello @pmgriffin,
Yes since you uploaded the file there should be at least import errors in the upper left corner of airflow. However could you check that you’ve uploaded to the correct location to see the DAG, you might used:
aws s3 sync src s3://de-c2w4lab1-<ACCOUNT_ID>-us-east-1-mwaa-dags
instead of:
aws s3 sync src s3://de-c2w4lab1-<ACCOUNT_ID>-us-east-1-mwaa-dags/dags

1 Like
3

Hi Georgios,

thanks for quick response…

Yes i believe i have uploaded to the correct bucket. And this is the bucket Airflow was previously reading from to find originaly parsing errors in my script.

image
image1005×512 55.4 KB

Paul

4

No DAGs listed under DAGs view

image
image1884×375 31.4 KB

however if i goto Browse->Dag Dependencies and untick “Only show DAGs with dependencies”

It appears to show “book_of_the_day”

image
image1884×375 25.1 KB

5

Hi Paul,
There are instructions to restart airflow and use the solution file to re-sync the bucket file to the correct one. Just make sure the file will find changes and get an output from the aws s3 sync src s3://de-c2w4lab1-<ACCOUNT_ID>-us-east-1-mwaa-dags/dags command. I could try to reproduce the issue. Hope it helps

6

Hi Georgios,

thanks for taking the time again. I had already actually tried those steps, you’ll see in my initial post i had but deleted all S3 contents and sync’d up the solution script. I also restarted AirFlow but no luck.

The 2 hour time limit has no expired (so annoying) and i’ll need to restart the lab. Hopefully new resources my fix this issue as it seems i’m doing everything correctly based on the instructions and your responses.

Paul

7

Sure @pmgriffin take your time, well I just see the DAG on my airflow just fine. Keep the tips and the solution file and if you need to restart airflow use bash ./scripts/restart_airflow.sh in the beginning. Thank you

8

I have no idea what’s different this time Georgios but i copied my file contents back in and Airflow Picked Up the DAG first time. :man_shrugging:

A classic turn it off / turn it on! Maybe the lab time out was a blessing.

THanks for your help anyway

1 Like
9

Hi All,

I need some support as I am not able to follow {{ dag_name }} instructions. How to replace the {{ dag_name }} is something which is really killing me off. No clues whatsoever.

image
image821×491 23.4 KB

10

Hello @amiguel,

In part 5.1 of the exercise you create the file template.py. As the name suggests you are supposed to create three different files with three different names with the help of this file.
This is done in part 5.1.2 with Jinja template. If you search the file (control+f) for model_trip_duration_easy_destiny you will find three occurrences. You need to replace them with {{ dag_name }}

The first one is a function and could be flagged as a syntax error, this is normal. The last two at the bottom of the file should keep their syntax (one start with dag_ and the second is a function()).
Same process goes with part 5.1.3 {{ vendor_name }} (four occurrences)
Hope its helpful