C4w4 capstone project part 2 assignment 5 Incorrect grading

Course Q&A Data Engineering Data Modeling, Transformation, and Serving
1

C4w4 capstone project part 2 assignment 5 Incorrect grading.

image
image679Ă—512 16.2 KB

these file exist in s3 as shown below.Let me know if anything missing from mypart
image
image1535Ă—719 81.6 KB

2

Hello @ami_il,

Could check in the s3 bucket you have a typo in the folder name, bi_viwes instead of bi_views that the grader needs. Could you rename your folder in models and try again, thank you

3

Hi thanks for pointing out the typo.

Tried doing again , but ended up getting below errors, looks like the account needs to be refreshed
.pls help

Error: creating Glue Connection (de-c4w4a2-connection-rds): operation error Glue: CreateConnection, https response error StatusCode: 400, RequestID: 8f15a63b-1185-495e-b662-d0b9e6bd8a72, AlreadyExistsException: Connection already exists.

with module.extract_job.aws_glue_connection.rds_connection,
on modules/extract_job/glue.tf line 2, in resource “aws_glue_connection” “rds_connection”:
2: resource “aws_glue_connection” “rds_connection” {

Error: creating IAM Role (de-c4w4a2-glue-role): operation error IAM: CreateRole, https response error StatusCode: 409, RequestID: 8eb566c3-8b95-48b8-99f2-7de7487346a9, EntityAlreadyExists: Role with name de-c4w4a2-glue-role already exists.

with module.extract_job.aws_iam_role.glue_role,
on modules/extract_job/iam.tf line 1, in resource “aws_iam_role” “glue_role”:
1: resource “aws_iam_role” “glue_role” {

4

Hello @ami_il,

Sorry for the inconvenience. I got that error as well before when ran terraform apply -target=module.transform_job -no-color 2> errors.txt and the terminal disconnected. You should be able to find those resources in AWS console and delete them:

  1. de-c4w4a2-connection-rds at AWS glue > connections:

C4W4A2_connections
C4W4A2_connections1170Ă—685 133 KB

  1. de-c4w4a2-glue-role at IAM > roles

C4W4A2_role
C4W4A2_role1252Ă—809 219 KB

Afterwards you should use source scripts/setup.sh again and navigate to cd terraform and continue with terraform commands (init and plan) and try again to create the transform jobs.

Finally in case you need to delete dq_rulesets you could find them in AWS glue>tables>view data quality. Hope it helps

C4W4A2_data_quality
C4W4A2_data_quality1907Ă—465 150 KB

C4W4_A2_songsdq_ruleset
C4W4_A2_songsdq_ruleset1867Ă—755 183 KB

5

Hi Thanks for the response and details steps
, Getting below error, its a mistake form my end , I deleted the view tables along instead of the rule set(I mean the rules got deleted too ).

So looks like I will have to execute the transform jobs again to get the tables created , and then proceed ahead, will do that , incase any ther issue will reach out

Error: creating Glue Data Quality Ruleset (songs_dq_ruleset): operation error Glue: CreateDataQualityRuleset, https response error StatusCode: 400, RequestID: 51022745-6be0-4f7b-bafd-e334e24ff757, InvalidInputException: Entity Not Found (Service: AmazonDataCatalog; Status Code: 400; Error Code: EntityNotFoundException; Request ID: 838e88f4-f05d-45f9-8cdb-d1b28ee746d7; Proxy: null)

with module.data_quality.aws_glue_data_quality_ruleset.songs_dq_ruleset,
on modules/data_quality/glue.tf line 1, in resource “aws_glue_data_quality_ruleset” “songs_dq_ruleset”:
1: resource “aws_glue_data_quality_ruleset” “songs_dq_ruleset” {

Error: creating Glue Data Quality Ruleset (sessions_dq_ruleset): operation error Glue: CreateDataQualityRuleset, https response error StatusCode: 400, RequestID: 395e337b-8e66-49c8-859a-de8982ae11c6, InvalidInputException: Entity Not Found (Service: AmazonDataCatalog; Status Code: 400; Error Code: EntityNotFoundException; Request ID: 5aed3c9b-02c5-491d-b0d1-3db8e279e487; Proxy: null)

with module.data_quality.aws_glue_data_quality_ruleset.sessions_dq_ruleset,
on modules/data_quality/glue.tf line 10, in resource “aws_glue_data_quality_ruleset” “sessions_dq_ruleset”:
10: resource “aws_glue_data_quality_ruleset” “sessions_dq_ruleset” {

Error: creating Glue Data Quality Ruleset (users_dq_ruleset): operation error Glue: CreateDataQualityRuleset, https response error StatusCode: 400, RequestID: 51b57e22-0d96-4ac5-baf3-dc21d0a795f0, InvalidInputException: Entity Not Found (Service: AmazonDataCatalog; Status Code: 400; Error Code: EntityNotFoundException; Request ID: 0a33d989-d742-4713-bd42-f6c53ece2c98; Proxy: null)

with module.data_quality.aws_glue_data_quality_ruleset.users_dq_ruleset,
on modules/data_quality/glue.tf line 19, in resource “aws_glue_data_quality_ruleset” “users_dq_ruleset”:
19: resource “aws_glue_data_quality_ruleset” “users_dq_ruleset” {

6

Hello @ami_il,

Could you make sure you are following the correct order of the commands in part 2.3:

terraform init
terraform plan
terraform apply -target=module.extract_job
terraform apply -target=module.transform_job
terraform apply -target=module.serving

Later in step 3.1.2 you create the rulesets with this command terraform apply -target=module.data_quality if you run it before serving for example you might get this error. Hope it helps

7

Hi , Thanks completed the assignment

1 Like