C4W4 - Capstone Project Part 1 - 4.2 Terraform Apply Error : Database Already Exists

[Christophe_Lancien]

Hello everyone,

I keep encountering the same error when trying to apply a Terraform configuration in part 4.2 of the notebook to create an AWS Glue Catalog Database.

Everything worked perfectly in part 4.1, where the extract job completed successfully.

However, the issue arises with the aws_glue_catalog_database resource. Terraform tries to create a database that already exists, and it fails with an AlreadyExistsException.

Here is the output of my Terraform apply command :

Plan: 5 to add, 0 to change, 0 to destroy.

Changes to Outputs:

• • glue_catalog_db = (sensitive value)*
• • glue_json_transformation_job = “de-c4w4a1-json-transform-job”*
• • glue_songs_transformation_job = “de-c4w4a1-songs-transform-job”*

Do you want to perform these actions?

• Terraform will perform the actions described above.*

• Only ‘yes’ will be accepted to approve.*

• Enter a value: yes*

module.transform_job.aws_glue_catalog_database.transform_db: Creating…
module.transform_job.aws_s3_object.glue_job_transform_json: Creating…
module.transform_job.aws_s3_object.glue_job_transform_db: Creating…
module.transform_job.aws_s3_object.glue_job_transform_json: Creation complete after 1s [id=de-c4w4a1-transform-json-job.py]
module.transform_job.aws_s3_object.glue_job_transform_db: Creation complete after 1s [id=de-c4w4a1-transform-songs-job.py]
╷
│ Error: creating Glue Catalog Database (de_c4w4a1_silver_db): operation error Glue: CreateDatabase, https response error StatusCode: 400, RequestID: 3e89b7d9-b1b6-40c9-b521-b91da264f282, AlreadyExistsException: Database already exists.
*│ *
│ with module.transform_job.aws_glue_catalog_database.transform_db,
│ on modules/transform_job/glue.tf line 2, in resource “aws_glue_catalog_database” “transform_db”:
│ 2: resource “aws_glue_catalog_database” “transform_db” {

Questions:

1. How can I configure Terraform to ignore this error and proceed without attempting to recreate an existing Glue Catalog Database?
2. Is there a way to check if the database exists and skip its creation programmatically within Terraform?
3. Has anyone else faced this issue and found a clean solution?

@Amir_Zare, I think my environment may be corrupt, could you please assist me with this ?

This is my final part for this specialization, Capstone Project Part 2 is already done.

Any advice or insights would be greatly appreciated. Thank you !

[Amir_Zare]

Hello @Christophe_Lancien
Unfortunately, I don’t know if it is possible to fix this programmatically from withing terraform config files. However, I too have faced the same issue in the past. I believe it is stemming from nonperfect lab refreshes. To fix this, go to AWS Glue, select Databases from the menu on the left, select the database, and press delete.

[Christophe_Lancien]

Hello @Amir_Zare,
What is strange is that I see nothing in AWS Glue

I made a request via the form you already shared because I think maybe something is wrong with the environment (an other user is facing the same issue with Capstone Project Part 2)

Any other idea to solve the problem ?

[Amir_Zare]

@Christophe_Lancien that is strange. I hope the lab refresh fixes this.

[Christophe_Lancien]

How can I know when the fix refresh is applied ?

[Amir_Zare]

@Christophe_Lancien The staff will notify you. Thank you for your patience.

[Christophe_Lancien]

The problem was resolved thanks to the refresh environment form shared by @Amir_Zare.

I successfully completed the lab, and what a joy it was to finally see those transformation jobs succeed in AWS Glue haha

I even took the luxury of tackling the optional section 5.4 - Data Modeling. It wasn’t straightforward due to the shallow explanations provided for creating all the required files, but I managed to get it done almost last minute !

This was my last lab to achieve, and I’m thrilled to share that I have officially completed the Data Engineering Specialization

What a journey it has been, so much learned along the way !

[Amir_Zare]

@Christophe_Lancien glad to hear the issue was resolved and that you successfully completed the specialization! I understand that these issues can be frustrating at times, but I believe overcoming them can even enhance the learning experience. Cheers, and wishing you all the best in your future endeavors.

[Christophe_Lancien]

Thank you @Amir_Zare ! I completely agree, these unrelated issues really pushed me to dig deeper into how everything works. It often involved trying to debug things that couldn’t be solved on my own, but in the end, I realized everything was set up correctly, and the cause was likely external.

To illustrate, I became so familiar with the Terraform process that I was able to complete it without having to follow the notebook. It was a great learning experience, and I’m grateful for the challenge !

Cheers, and thanks again for your support !

[BonAppetite]

I encounter this issue then I request for the lab reset. but still stuck with this issue. can anyone help?

[Amir_Zare]

Hello @hawraa.salami
@BonAppetite here is getting the database already exists error even after a lab refresh. Can you please help us?
Thank you in advance.

[corentin1]

Hello @Amir_Zare

I have the same issue than Christophe and I don’t know how to solve it.

Could you help me to refresh the lab ?

thanks in advance

[Amir_Zare]

Hello @corentin1
Please, fill out this form.

[corentin1]

Thanks a lot @Amir_Zare I will do this

[corentin1]

what a great moment to finish this great training !

thanks again for your help @Amir_Zare