Capstone project PARt 1 constantly failing

The jobs in Terraform are constantly failing.
I have to run them a couple of times - refreshing terraform again and again. Refreshing the lab a couple of times in order to work with the same script.
Super annoying and making the progress super slow.

The transformation job is currently failing. I have checked the date to be as it should be per other topics. I have re-run the previous jobs - which run fine with no issue.
What could be the problem?

Hello, I have the same problem, when trying to re-run terraform i get an error

Error: creating Glue Connection (de-c4w4a2-connection-rds): operation error Glue: CreateConnection, https response error StatusCode: 400, RequestID: 33c63310-9334-4e5b-b314-5e3b7573ab63, AlreadyExistsException: Connection already exists.

with module.extract_job.aws_glue_connection.rds_connection,
on modules/extract_job/glue.tf line 2, in resource “aws_glue_connection” “rds_connection”:
2: resource “aws_glue_connection” “rds_connection” {
…

Is there a way to remove all resources i created to be able to re-create them with terraform. I tried deleting them manually, but i do not have enough IAM access to do so.

Thank you for help

Hello @Kiril_Milev
Does both your transformation jobs fail? Are there any errors you see when you run terraform or glue commands?
Also, have you waited for the extract jobs to complete before running the transformation jobs?

Hello @abedelb
If you keep getting Connection already exists error in your next try, your labs need to be refreshed. You can request a lab refresh via this link.

Hello, after the first run of terraform (extractor-jobs),
i keep getting connection already exists.

Is there a way to delete all my resources ( i tried reboot but it does not remove resources )

LOGS
Error: creating Glue Connection (de-c4w4a1-connection-rds): operation error Glue: CreateConnection, https response error StatusCode: 400, RequestID: 976f8b28-58ff-4411-a31c-474a62b5c06f, AlreadyExistsException: Connection already exists.

with module.extract_job.aws_glue_connection.rds_connection,
on modules/extract_job/glue.tf line 2, in resource “aws_glue_connection” “rds_connection”:
2: resource “aws_glue_connection” “rds_connection” {

Error: creating IAM Role (de-c4w4a1-glue-role): operation error IAM: CreateRole, https response error StatusCode: 409, RequestID: 38231229-7cbb-45ab-adad-2701b9a81040, EntityAlreadyExists: Role with name de-c4w4a1-glue-role already exists.

with module.extract_job.aws_iam_role.glue_role,
on modules/extract_job/iam.tf line 1, in resource “aws_iam_role” “glue_role”:
1: resource “aws_iam_role” “glue_role” {

Error: pq: Schema “deftunes_serving” already exists

with module.serving.redshift_schema.serving_schema,
on modules/serving/redshift.tf line 2, in resource “redshift_schema” “serving_schema”:
2: resource “redshift_schema” “serving_schema” {

Error: pq: Schema “deftunes_transform” already exists

with module.serving.redshift_schema.transform_external_from_glue_data_catalog,
on modules/serving/redshift.tf line 12, in resource “redshift_schema” “transform_external_from_glue_data_catalog”:
12: resource “redshift_schema” “transform_external_from_glue_data_catalog” {

@abedelb please, fill out the lab refresh form.

Hello, i filled it twice :slight_smile: every day i try and run into same errors :sob: i don’t know if it is a miss configuration in terraform, or insufficient IAM roles.


Error: creating Glue Catalog Database (de_c4w4a1_silver_db): operation error Glue: CreateDatabase, https response error StatusCode: 400, RequestID: cc025dee-92ba-4efc-8b1e-f16e755a45c8, AlreadyExistsException: Database already exists.

  with module.transform_job.aws_glue_catalog_database.transform_db,
  on modules/transform_job/glue.tf line 3, in resource "aws_glue_catalog_database" "transform_db":
   3: resource "aws_glue_catalog_database" "transform_db" {

@abedelb Please, wait up to 3 business days after filling out this form. Lab refreshes are done manually and take a while to be handled.

1 Like

it worked for me. I deleted the de_c4w4a1_silver_db manually from the UI and then it worked

Thanks for the link to reset the lab