C4W4 Capstone project 2 - Data Quality checks

konutech · January 26, 2025, 8:25pm

* Link to the classroom item you are referring to:

Data quality checks run as Tasks via Airflow are not working.
Had to turn these tasks off to run DAGs successfully

9433eeb05036
*** Found local files:
***   * /opt/airflow/logs/dag_id=deftunes_songs_pipeline_dag/run_id=scheduled__2020-02-01T00:00:00+00:00/task_id=dq_check_songs/attempt=2.log
*** Found logs served from host http://9433eeb05036:8793/log/dag_id=deftunes_songs_pipeline_dag/run_id=scheduled__2020-02-01T00:00:00+00:00/task_id=dq_check_songs/attempt=2.log
[2025-01-26, 19:50:54 UTC] {local_task_job_runner.py:123} ▶ Pre task execution logs
[2025-01-26, 19:50:54 UTC] {base_aws.py:606} WARNING - Unable to find AWS Connection ID 'aws_default', switching to empty.
[2025-01-26, 19:50:54 UTC] {base_aws.py:180} INFO - No connection ID provided. Fallback on boto3 credential strategy (region_name='us-east-1'). See: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
[2025-01-26, 19:50:56 UTC] {credentials.py:1075} INFO - Found credentials from IAM Role: de-c4w4a2-ec2-role
[2025-01-26, 19:50:56 UTC] {glue.py:440} INFO - Submitting AWS Glue data quality ruleset evaluation run for RulesetNames ['songs_dq_ruleset']
[2025-01-26, 19:50:56 UTC] {glue.py:471} INFO - Waiting for AWS Glue data quality ruleset evaluation run RunId: dqrun-ce615f2ead4ef7c01e28a74140e56fae30855a7b to complete.
[2025-01-26, 19:52:57 UTC] {taskinstance.py:3310} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 767, in _execute_task
    result = _execute_callable(context=context, **execute_callable_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 733, in _execute_callable
    return ExecutionCallableRunner(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/utils/operator_helpers.py", line 252, in run
    return self.func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/baseoperator.py", line 406, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/amazon/aws/operators/glue.py", line 473, in execute
    self.hook.get_waiter("data_quality_ruleset_evaluation_run_complete").wait(
  File "/home/airflow/.local/lib/python3.11/site-packages/botocore/waiter.py", line 55, in wait
    Waiter.wait(self, **kwargs)
  File "/home/airflow/.local/lib/python3.11/site-packages/botocore/waiter.py", line 374, in wait
    raise WaiterError(
botocore.exceptions.WaiterError: Waiter data_quality_ruleset_evaluation_run_complete failed: Waiter encountered a terminal failure state: For expression "Status" we matched expected path: "FAILED"
[2025-01-26, 19:52:57 UTC] {taskinstance.py:1225} INFO - Marking task as UP_FOR_RETRY. dag_id=deftunes_songs_pipeline_dag, task_id=dq_check_songs, run_id=scheduled__2020-02-01T00:00:00+00:00, execution_date=20200201T000000, start_date=20250126T195054, end_date=20250126T195257
[2025-01-26, 19:52:57 UTC] {taskinstance.py:340} ▶ Post task execution logs
[2025-01-26, 20:23:35 UTC] {local_task_job_runner.py:123} ▶ Pre task execution logs
[2025-01-26, 20:23:36 UTC] {base_aws.py:606} WARNING - Unable to find AWS Connection ID 'aws_default', switching to empty.
[2025-01-26, 20:23:36 UTC] {base_aws.py:180} INFO - No connection ID provided. Fallback on boto3 credential strategy (region_name='us-east-1'). See: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
[2025-01-26, 20:23:37 UTC] {credentials.py:1075} INFO - Found credentials from IAM Role: de-c4w4a2-ec2-role
[2025-01-26, 20:23:37 UTC] {glue.py:440} INFO - Submitting AWS Glue data quality ruleset evaluation run for RulesetNames ['songs_dq_ruleset']
[2025-01-26, 20:23:38 UTC] {glue.py:471} INFO - Waiting for AWS Glue data quality ruleset evaluation run RunId: dqrun-f594ba68f1dc7c69cf2ef9f4c6edc3f40535a098 to complete.

Georgios · January 27, 2025, 1:04pm

@konutech glad you got it to work. Hope it helps

zvika_sinkevich · January 28, 2025, 11:01am

@Georgios It’s not working, he turned it off… and I have the same issue.

Mine is at songs ( in the tf file songs have default values, I edit only sessions & users)

Georgios · January 28, 2025, 11:45am

@zvika_sinkevich sorry for the inconvenience, could you fill this form for a lab refresh. That might refresh existing data quality checks, thanks

zvika_sinkevich · January 28, 2025, 4:46pm

@Georgios form submited

Georgios · January 28, 2025, 5:00pm

Hello @zvika_sinkevich,
Are your jobs from steps 2.4 and 2.5 succesfull and afterwards could you go to the Databases menu from AWS Glue, delete the rulesets, and then run terraform apply -target=module.data_quality again. Thanks

zvika_sinkevich · January 29, 2025, 5:14pm

Now the lab is working fine - everything is working

pawanshirbhate · March 7, 2025, 10:02am

I’m facing the same issue. what’s the problem here? how to fix it? @Georgios

eb534735d22d
*** Found local files:
***   * /opt/airflow/logs/dag_id=deftunes_songs_pipeline_dag/run_id=scheduled__2020-02-01T00:00:00+00:00/task_id=dq_check_songs/attempt=2.log
[2025-03-07, 09:48:58 UTC] {local_task_job_runner.py:123} ▶ Pre task execution logs
[2025-03-07, 09:48:59 UTC] {base_aws.py:606} WARNING - Unable to find AWS Connection ID 'aws_default', switching to empty.
[2025-03-07, 09:48:59 UTC] {base_aws.py:180} INFO - No connection ID provided. Fallback on boto3 credential strategy (region_name='us-east-1'). See: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
[2025-03-07, 09:49:00 UTC] {credentials.py:1075} INFO - Found credentials from IAM Role: de-c4w4a2-ec2-role
[2025-03-07, 09:49:00 UTC] {taskinstance.py:3310} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 767, in _execute_task
    result = _execute_callable(context=context, **execute_callable_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 733, in _execute_callable
    return ExecutionCallableRunner(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/utils/operator_helpers.py", line 252, in run
    return self.func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/baseoperator.py", line 406, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/amazon/aws/operators/glue.py", line 438, in execute
    self.validate_inputs()
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/amazon/aws/operators/glue.py", line 435, in validate_inputs
    raise AirflowException(f"Following RulesetNames are not found {not_found_ruleset}")
airflow.exceptions.AirflowException: Following RulesetNames are not found ['songs_dq_ruleset']
[2025-03-07, 09:49:00 UTC] {taskinstance.py:1225} INFO - Marking task as FAILED. dag_id=deftunes_songs_pipeline_dag, task_id=dq_check_songs, run_id=scheduled__2020-02-01T00:00:00+00:00, execution_date=20200201T000000, start_date=20250307T094858, end_date=20250307T094900
[2025-03-07, 09:49:01 UTC] {taskinstance.py:340} ▶ Post task execution logs

Georgios · March 7, 2025, 5:41pm

Hello @pawanshirbhate,

Could you check that you succesfuly created the songs_dq_ruleset since it says is not found. You could find it in the AWS console :

if you go to AWS Glue>tables under Data Catalog> and choose view data quality in songs

See if it succeded then run DAG again. Hope it helps:

sapiensush · March 11, 2025, 5:23pm

Hi @Georgios, Mine is failing under user.

Georgios · March 11, 2025, 6:22pm

Hello @sapiensush,

I couldn’t reproduce your issue, the rules looks good but I can see it fails with that value. Could you check at the logs for that specific DAG for more info. Thank you

Topic		Replies	Views
C4 - W4 - capstone project 2 Data Modeling, Transformation, and Serving week-4	4	47	November 6, 2024
Week 4 Part 2 - Error running DAG RDS Data Modeling, Transformation, and Serving week-4	1	39	December 24, 2024
C2W4_final lab with grade Source Systems, Data Ingestion, and Pipelines week-4	3	23	March 24, 2025
Course 2 Week 4 Graded Programming Assignment 4: Building an Advanced Data Pipeline With Data Quality Checks no DAGs appear in the Airflow Source Systems, Data Ingestion, and Pipelines week-4	6	60	November 20, 2024
Capstone Project Part 2,** 4.2 - DAG for Songs Data in RDS Source Airflow task failed Data Modeling, Transformation, and Serving week-4	16	129	December 11, 2024

C4W4 Capstone project 2 - Data Quality checks

Related topics