I started the first airflow lab and the airflow url does not work so I restarted it with the script. The guide says it should take less than 3 mins, but the airflow service is yet to become healthy
I am afraid the same thing happens in the 2nd Airflow practice lab.
same thing with the final assignment - Graded Programming Assignment 4: Building an Advanced Data Pipeline With Data Quality Checks
edit: interestingly enough even though I dont have access to the airflow UI, i just followed the instructions in the markdown, submitted and got full marks.
Hello Ivan,
You are right; I couldn’t open the Airflow UI either. I’ve reported the issue, and we will work on it. Thank you!
Hello @Ivan_Ivanov,
I am informed that the issue is resolved, and the Airflow UI in these labs is working now. Thank you for reporting this.
I am facing the same issue today. Not sure if this has been resolved yet. Can you please look into this.
Can you please let me know when the issue is fixed. I have to do the assignment all over again
Same here, it doesn’t work. There are a lot of issues in the labs, making it a truly disappointing experience
Hello @vishal1238
I’m sorry that you are facing issues in the labs. I tried this lab right now, and the Airflow UI is working fine. In the past, there has been issues with the UI at times, but we have always noted them and fixed them ASAP.
Please, let us know if you are facing the exact same issue of Airflow UI not being available in this exact same lab. In case you can’t restart the service as it is demonstrated in the lab instructions, please, share the error you are getting.
Had same issue, the problem is with stack probably failing in some cases and doesn’t execute all steps.
In my case, it was missing docker, docker compose and most other files in /opt/airflow in ec2 instance.
I tried to copy from other lab I had still running but missed couple of files and then env expired.
It would probably be useful (as exercise as well), to make airflow scripts made available in github somewhere so would be possible to rerun manually if creation fails again (which seems to happen quite often based on these replies).
For reference my aws accId: 774102081459
Issue still seems to persist for all W4 labs. Airlfow UI doesn’t show up even after restarting from cloud shell.
Hello,
I have the same issue. impossible to open Airflow UI for Labs for C2W4. Even after trying to make the restart
Lost 3h of progress because of this issue, thanks for reporting it. I thought was just me and the work laptop…
@Amir_Zare still having this issue on my end, I will try to reboot the server and see if works after.
EDIT1:
rebooting did nothing, still same issue, UI not accessible:
Do I have to ask to have the lab restarted from scratch?
It seems the issue is a failed apt-get update when docket is being build, that causes airflow not to start and dbt are not created…
I tried to fix it using the command shell, but it gets stuck in a bunch of further issues due to deprecated libraries…
I was able to pass without the need to have a successful DAG run, just unpausing the DAGs from the AWS Sloud Shell, it took a bunch of hours to be able to find the proper way…
Here are the steps:
1. Find the Airflow EC2 Instance
From CloudShell:
aws cloudformation list-stack-resources --stack-name <YOUR_STACK_NAME>
Look for:
ApiAirflowInstance
AWS::EC2::Instance
i-xxxxxxxxxxxxxxxxx
2. Connect to the Instance
aws ssm start-session --target <INSTANCE_ID>
3. Go to the Airflow Directory
cd /opt/airflow
4. Start Airflow Services
Use these commands (important: use --no-build):
docker compose up -d --no-build postgres redis airflow-init
docker compose up -d --no-build airflow-webserver airflow-scheduler airflow-worker airflow-triggerer
Verify:
docker compose ps
5. Unpause the DAG
For the songs pipeline:
docker compose exec airflow-webserver airflow dags unpause deftunes_songs_pipeline_dag
Or for the API pipeline:
docker compose exec airflow-webserver airflow dags unpause deftunes_api_pipeline_dag
6. Verify the DAG is Unpaused
docker compose exec airflow-webserver airflow dags list
Example:
dag_id | is_paused
deftunes_api_pipeline_dag | False
deftunes_songs_pipeline_dag | False
``
False means the DAG is unpaused.
Monitoring DAG Runs
Show DAG Runs
Songs pipeline:
docker compose exec airflow-webserver airflow dags list-runs -d deftunes_songs_pipeline_dag
API pipeline:
docker compose exec airflow-webserver airflow dags list-runs -d deftunes_api_pipeline_dag
Example:
run_id state
scheduled__2020-02-01T00:00:00+00:00 success
scheduled__2020-03-01T00:00:00+00:00 failed
Watch Airflow Logs
Scheduler:
docker compose logs -f airflow-scheduler
Worker:
docker compose logs -f airflow-worker