C1W4 - terraform apply does not work

Course Q&A Data Engineering Introduction to Data Engineering
1

Hi, when running terrafrom apply -no-color 2>errors.txt (after following instructions prior to that) I get

Error: creating Glue Catalog Database (de-c1w4-ml-db): operation error Glue: CreateDatabase, https response error StatusCode: 400, RequestID: 88945bfe-788e-4e4b-b6d4-eb5bfe6e89ec, AlreadyExistsException: Database already exists.

with module.etl.aws_glue_catalog_database.ml_database,
on modules/etl/glue.tf line 1, in resource “aws_glue_catalog_database” “ml_database”:
1: resource “aws_glue_catalog_database” “ml_database” {

Error: creating Glue Connection (de-c1w4-rds-connection): operation error Glue: CreateConnection, https response error StatusCode: 400, RequestID: 82ce880b-4cef-4ae5-9d1e-c3f6a6ee43c4, AlreadyExistsException: Connection already exists.

with module.etl.aws_glue_connection.rds_connection,
on modules/etl/glue.tf line 6, in resource “aws_glue_connection” “rds_connection”:
6: resource “aws_glue_connection” “rds_connection” {

Error: creating IAM Role (de-c1w4-glue-role): operation error IAM: CreateRole, https response error StatusCode: 409, RequestID: 09c45136-9f5d-4064-afe6-bd961dba8a67, EntityAlreadyExists: Role with name de-c1w4-glue-role already exists.

with module.etl.aws_iam_role.glue_role,
on modules/etl/iam_roles.tf line 1, in resource “aws_iam_role” “glue_role”:
1: resource “aws_iam_role” “glue_role” {

Couldn’t find these resources to delete them manually

1 Like
2

Hello @MilovicM,

Could you check in AWS console and delete them manually:

  1. de-c1w4-ml-db should be in AWS Glue> databases:

C1W4_rds_connection
C1W4_rds_connection1893×696 190 KB

  1. de-c1w4-rds-connection in AWS Glue > connections

C1W4_glue_catalog_ml_db
C1W4_glue_catalog_ml_db1894×462 111 KB

  1. For de-c1w4-glue-role go to IAM and then Roles:

C1W4_glue_role_iam
C1W4_glue_role_iam1781×717 212 KB

  1. If the issue continues you could wait for a new lab session cleanup (2 hours) or complete this form for a lab refresh. Thanks
3

Hi, I followed all the instructions but could not find database. After running terraform init, plan and apply afterwards I got the following error:

Error: creating Glue Catalog Database (de-c1w4-ml-db): operation error Glue: CreateDatabase, https response error StatusCode: 400, RequestID: 400a1bf2-6945-4757-a136-1d225ef5e1da, AlreadyExistsException: Database already exists.

with module.etl.aws_glue_catalog_database.ml_database,
on modules/etl/glue.tf line 1, in resource “aws_glue_catalog_database” “ml_database”:
1: resource “aws_glue_catalog_database” “ml_database” {

Screenshot from 2025-02-16 16-09-50
Screenshot from 2025-02-16 16-09-501855×549 68.9 KB

4

Hello @MilovicM,

Please complete this form for a lab refresh. Note it will take 2 business days and you should be notified after it’s done. Thank you

5

After I fixed the first 3 items that showed up the Error.txt log, I reran it and a 4th item showed up that needed to be deleted.
Error: creating Glue Catalog Database (de-c1w4-ml-db): operation error Glue: CreateDatabase, https response error StatusCode: 400, RequestID: 137e9b4c-8a6e-45d9-8684-a6077a6d71b2, AlreadyExistsException: Database already exists.

with module.etl.aws_glue_catalog_database.ml_database,
on modules/etl/glue.tf line 1, in resource “aws_glue_catalog_database” “ml_database”:
1: resource “aws_glue_catalog_database” “ml_database” {

Error: creating Glue Connection (de-c1w4-rds-connection): operation error Glue: CreateConnection, https response error StatusCode: 400, RequestID: b6428978-cc4e-4df5-b5c2-aa7c775a9684, AlreadyExistsException: Connection already exists.

with module.etl.aws_glue_connection.rds_connection,
on modules/etl/glue.tf line 6, in resource “aws_glue_connection” “rds_connection”:
6: resource “aws_glue_connection” “rds_connection” {

Error: creating IAM Role (de-c1w4-glue-role): operation error IAM: CreateRole, https response error StatusCode: 409, RequestID: 5a4d43af-0117-4442-9e78-0fcb33347559, EntityAlreadyExists: Role with name de-c1w4-glue-role already exists.

with module.etl.aws_iam_role.glue_role,
on modules/etl/iam_roles.tf line 1, in resource “aws_iam_role” “glue_role”:
1: resource “aws_iam_role” “glue_role” {

Error: creating Glue Crawler (de-c1w4-ml-db-crawler): operation error Glue: CreateCrawler, https response error StatusCode: 400, RequestID: 5d2dda68-f410-456b-94fd-c93c998d8b7a, AlreadyExistsException: 654654372386:de-c1w4-ml-db-crawler already exists

with module.etl.aws_glue_crawler.s3_crawler,
on modules/etl/glue.tf line 22, in resource “aws_glue_crawler” “s3_crawler”:
22: resource “aws_glue_crawler” “s3_crawler” {

Those are all four that I had to delete (a few times) to eventually get it to move on past that point.

1 Like
6

Additionally to the @Georgios’s answer. > You need to remove the crawler as well to make it work.

The issue for me is that after this step I got the following error in the step 1.13:

An error occurred while calling o111.pyWriteDynamicFrame. Insufficient Lake Formation permission(s) on ratings_ml_training (Service: AWSGlue; Status Code: 400; Error Code: AccessDeniedException; Request ID: e4157a6f-c798-44e7-9f17-6620a87465d0; Proxy: 169.254.76.0)

7

Hello @fpcoutinho,

If you get Insufficient Lake Formation Permissions when you run the glue job you could fill this form to fix it. It should take 1-2 business days and perhaps @hawraa.salami could refresh your lab. Thanks