Hello everyone,
I keep encountering the same error when trying to apply a Terraform configuration in part 4.2 of the notebook to create an AWS Glue Catalog Database.
Everything worked perfectly in part 4.1, where the extract job completed successfully.
However, the issue arises with the aws_glue_catalog_database resource. Terraform tries to create a database that already exists, and it fails with an AlreadyExistsException.
Here is the output of my Terraform apply command :
Plan: 5 to add, 0 to change, 0 to destroy.
Changes to Outputs:
-
- glue_catalog_db = (sensitive value)*
-
- glue_json_transformation_job = âde-c4w4a1-json-transform-jobâ*
-
- glue_songs_transformation_job = âde-c4w4a1-songs-transform-jobâ*
Do you want to perform these actions?
module.transform_job.aws_glue_catalog_database.transform_db: CreatingâŚ
module.transform_job.aws_s3_object.glue_job_transform_json: CreatingâŚ
module.transform_job.aws_s3_object.glue_job_transform_db: CreatingâŚ
module.transform_job.aws_s3_object.glue_job_transform_json: Creation complete after 1s [id=de-c4w4a1-transform-json-job.py]
module.transform_job.aws_s3_object.glue_job_transform_db: Creation complete after 1s [id=de-c4w4a1-transform-songs-job.py]
âˇ
â Error: creating Glue Catalog Database (de_c4w4a1_silver_db): operation error Glue: CreateDatabase, https response error StatusCode: 400, RequestID: 3e89b7d9-b1b6-40c9-b521-b91da264f282, AlreadyExistsException: Database already exists.
*â *
â with module.transform_job.aws_glue_catalog_database.transform_db,
â on modules/transform_job/glue.tf line 2, in resource âaws_glue_catalog_databaseâ âtransform_dbâ:
â 2: resource âaws_glue_catalog_databaseâ âtransform_dbâ {
Questions:
- How can I configure Terraform to ignore this error and proceed without attempting to recreate an existing Glue Catalog Database?
- Is there a way to check if the database exists and skip its creation programmatically within Terraform?
- Has anyone else faced this issue and found a clean solution?
@Amir_Zare, I think my environment may be corrupt, could you please assist me with this ?
This is my final part for this specialization, Capstone Project Part 2 is already done.
Any advice or insights would be greatly appreciated. Thank you !
Hello @Christophe_Lancien
Unfortunately, I donât know if it is possible to fix this programmatically from withing terraform config files. However, I too have faced the same issue in the past. I believe it is stemming from nonperfect lab refreshes. To fix this, go to AWS Glue, select Databases from the menu on the left, select the database, and press delete.
Hello @Amir_Zare,
What is strange is that I see nothing in AWS Glue
I made a request via the form you already shared because I think maybe something is wrong with the environment (an other user is facing the same issue with Capstone Project Part 2)
Any other idea to solve the problem ?
@Christophe_Lancien that is strange. I hope the lab refresh fixes this.
How can I know when the fix refresh is applied ?
@Christophe_Lancien The staff will notify you. Thank you for your patience.
1 Like
The problem was resolved thanks to the refresh environment form shared by @Amir_Zare.
I successfully completed the lab, and what a joy it was to finally see those transformation jobs succeed in AWS Glue haha
I even took the luxury of tackling the optional section 5.4 - Data Modeling. It wasnât straightforward due to the shallow explanations provided for creating all the required files, but I managed to get it done almost last minute !
This was my last lab to achieve, and Iâm thrilled to share that I have officially completed the Data Engineering Specialization 
What a journey it has been, so much learned along the way !
1 Like
@Christophe_Lancien glad to hear the issue was resolved and that you successfully completed the specialization! I understand that these issues can be frustrating at times, but I believe overcoming them can even enhance the learning experience. Cheers, and wishing you all the best in your future endeavors.
1 Like
Thank you @Amir_Zare ! I completely agree, these unrelated issues really pushed me to dig deeper into how everything works. It often involved trying to debug things that couldnât be solved on my own, but in the end, I realized everything was set up correctly, and the cause was likely external.
To illustrate, I became so familiar with the Terraform process that I was able to complete it without having to follow the notebook. It was a great learning experience, and Iâm grateful for the challenge !
Cheers, and thanks again for your support !
1 Like
I encounter this issue then I request for the lab reset. but still stuck with this issue. can anyone help?
1 Like
Hello @hawraa.salami
@BonAppetite here is getting the database already exists error even after a lab refresh. Can you please help us?
Thank you in advance.
Hello @Amir_Zare
I have the same issue than Christophe and I donât know how to solve it.
Could you help me to refresh the lab ?
thanks in advance
Hello @corentin1
Please, fill out this form.
1 Like
Thanks a lot @Amir_Zare I will do this
what a great moment to finish this great training !
thanks again for your help @Amir_Zare
Iâve also faced a consistency issue with Capstone 1, where the state of the lab was unable to proceed due to the existence of the Glue Database, but the console does not have a record of it.
Error: creating Glue Catalog Database (de_c4w4a1_silver_db): operation error Glue: CreateDatabase, https response error StatusCode: 400, RequestID: f0dc7067-368d-44a7-9225-23f3f61e1eb5, AlreadyExistsException: Database already exists.
with module.transform_job.aws_glue_catalog_database.transform_db,
on modules/transform_job/glue.tf line 2, in resource "aws_glue_catalog_database" "transform_db":
2: resource "aws_glue_catalog_database" "transform_db" {
Itâs evident the AWS account still thinks the database exists, when the console does not show it.
Yet, when attempting to manually create the named Glue Data Catalog, it shows that the Database does exist.
This is my 2nd full attempt at redoing the Capstone because the state of the first attempt was inconsistently grading the submission, but Iâve waited for the environment to be cleared twice but the symptoms persist.
Iâve raised a form request similar to others.
Hello @taepyongyang
Sorry for the inconvenience. The issue you are facing is fixed with a lab refresh. If you have already filled out the form. Please, wait for up to 3 business days and try again after that. Hopefully, it will be resolved.
May i get the refresh link? Thanks
