Capstone Project Part 2 moduel.data_quality

Course Q&A Data Engineering Data Modeling, Transformation, and Serving
1

Hello,

I also have the same issue with Capstone project part 2 after running :

terraform apply -target=module.data_quality

Error: creating Glue Data Quality Ruleset (songs_dq_ruleset): operation error Glue: CreateDataQualityRuleset, https response error StatusCode: 400, RequestID: 2464f86e-6e24-48ed-a225-eb3232341f7c, InvalidInputException: Entity Not Found (Service: AmazonDataCatalog; Status Code: 400; Error Code: EntityNotFoundException; Request ID: f4e24140-cbce-4df3-88c9-d117a1c950d5; Proxy: null)

with module.data_quality.aws_glue_data_quality_ruleset.songs_dq_ruleset,
on modules/data_quality/glue.tf line 1, in resource “aws_glue_data_quality_ruleset” “songs_dq_ruleset”:
1: resource “aws_glue_data_quality_ruleset” “songs_dq_ruleset” {

Error: creating Glue Data Quality Ruleset (sessions_dq_ruleset): operation error Glue: CreateDataQualityRuleset, https response error StatusCode: 400, RequestID: 24a2f9ea-8749-4a5e-a78b-14e31a21a972, InvalidInputException: Entity Not Found (Service: AmazonDataCatalog; Status Code: 400; Error Code: EntityNotFoundException; Request ID: dad3805a-6fb8-4ce4-b6ac-e3b98e725c22; Proxy: null)

with module.data_quality.aws_glue_data_quality_ruleset.sessions_dq_ruleset,
on modules/data_quality/glue.tf line 10, in resource “aws_glue_data_quality_ruleset” “sessions_dq_ruleset”:
10: resource “aws_glue_data_quality_ruleset” “sessions_dq_ruleset” {

Error: creating Glue Data Quality Ruleset (users_dq_ruleset): operation error Glue: CreateDataQualityRuleset, https response error StatusCode: 400, RequestID: d6eac947-a755-4984-a067-b083146b805c, InvalidInputException: Entity Not Found (Service: AmazonDataCatalog; Status Code: 400; Error Code: EntityNotFoundException; Request ID: 7e3e89b2-2fa2-4fd7-9e62-8d04be05b39a; Proxy: null)

with module.data_quality.aws_glue_data_quality_ruleset.users_dq_ruleset,
on modules/data_quality/glue.tf line 19, in resource “aws_glue_data_quality_ruleset” “users_dq_ruleset”:
19: resource “aws_glue_data_quality_ruleset” “users_dq_ruleset” {

This is what I try :
1- Reboot the Lab
2-clear previous resource created

Since 02 days, I have been struggling with this error

2

Hello @euloge
I guess the problem is with the target_table in the rulesets. These ruleset depend on these tables, and apparently terraform is not able to find these entities. Please, make sure that you complete all the previous steps and that all the extract and transform glue jobs succeed before proceeding to this section.