Module-4 C4W4 Assignment Part 2 - Issue running terraform apply -target=module.data_quality

Course Q&A Data Engineering Data Modeling, Transformation, and Serving
1

In C4W4 Assignment Part 2, I was able to complete the earlier steps successfully in Module 4, but I am running into an issue when I try to run:

terraform apply -target=module.data_quality

Each time I run it, the terminal closes/kills itself. I also tried different variations of glue.tf under the data_quality module, but I am still facing the same problem. And even, these codes aren’t helpful as it also leads the terminal to kills itself: terraform apply -no-color 2> errors.txt or terraform apply -target=module.data_quality -no-color 2> errors.txt

Has anyone else encountered this behavior? Any guidance on how to debug why Terraform terminates like this or how to correctly configure glue.tf for the data_quality module would be very helpful.

2

Anyway, i solved the problem myself. You can delete the topic now.

3

I am facing the same issue. How did you solve it?

4

I solved it by manually deleting the existing Glue Data Quality rulesets in the AWS console and then letting the lab recreate them. Here’s how you can do it:

  1. Go to the AWS Glue Console → Data Quality → Rulesets.

  2. Locate the rulesets that were created earlier (for example: users_dq_ruleset, transactions_dq_ruleset, etc.).

  3. Select them and choose Delete.

  4. Once deleted, re-run the lab steps. The Terraform/Airflow tasks will then create fresh rulesets without conflict.

In my case, the issue happened because leftover rulesets from a previous run were conflicting with the new setup. Cleaning them up from the console fixed the problem.

6

@ahmetkarabas: Facing similar issue, even deleting AWS Glue Console → Data Quality → Rulesets is not helping. Getting error as Error: creating Glue Connection (de-c4w4a2-connection-rds): operation error Glue: CreateConnection, https response error StatusCode: 400, RequestID: 2119b4ec-172a-4dfe-8c18-9345d8d47668, AlreadyExistsException: Connection already exists.

with module.extract_job.aws_glue_connection.rds_connection,

on modules/extract_job/glue.tf line 2, in resource “aws_glue_connection” “rds_connection”:

2: resource “aws_glue_connection” “rds_connection” {

Error: creating IAM Role (de-c4w4a2-glue-role): operation error IAM: CreateRole, https response error StatusCode: 409, RequestID: 7739c3f4-1493-43e6-b8b1-44dd322586d7, EntityAlreadyExists: Role with name de-c4w4a2-glue-role already exists.

with module.extract_job.aws_iam_role.glue_role,

on modules/extract_job/iam.tf line 1, in resource “aws_iam_role” “glue_role”:

1: resource “aws_iam_role” “glue_role” {

Any pointer to resolve to this issue will be really appreicated

7

@Alejandro_Ceren Are you able to resolve the issue. If yes please let me know you resolved it, terraform apply is crashing again and again for me

8

I didn’t solved it. I just went ahead with the rest of the lab.

9

already fixed it after several hours, 1 important is apply this cli 2>&1 | tee -a output.log after each terraform apply to avoid crash the terminal, after that, if any resource is existed, just go to aws console and delete it corresponding, and rerun again, repeat through 3 apply