C4W4 Capstone Part 2 Issues - Insufficient Lake Formation permission(s) Glue Jobs

Hello,

Yes, since you submited the form the lab refresh should allow your permissions to complete the transform jobs. Note that it takes 1-2 business days (no weekend) since it is a manual process performed by the engineers. Hope it helps

Hi everyone,

I’m running into persistent issues while progressing through the lab that relies on Terraform. Every time I execute terraform apply, I encounter errors stating that certain resources already exist. This blockage is preventing me from moving forward in the lab exercises.

Below are the error messages I keep receiving:

Error: pq: Schema "deftunes_serving" already exists

  with module.serving.redshift_schema.serving_schema,
  on modules/serving/redshift.tf line 1, in resource "redshift_schema" "serving_schema":
   1: resource "redshift_schema" "serving_schema" {

Error: pq: Schema "deftunes_transform" already exists

  with module.serving.redshift_schema.transform_external_from_glue_data_catalog,
  on modules/serving/redshift.tf line 9, in resource "redshift_schema" "transform_external_from_glue_data_catalog":
   9: resource "redshift_schema" "transform_external_from_glue_data_catalog" {

Error: creating Glue Catalog Database (de_c4w4a2_silver_db): operation error Glue: CreateDatabase, https response error StatusCode: 400, RequestID: 9baaf9d1-b1f9-4179-a93d-93dc97b7760f, AlreadyExistsException: Database already exists.

  with module.transform_job.aws_glue_catalog_database.transform_db,
  on modules/transform_job/glue.tf line 1, in resource "aws_glue_catalog_database" "transform_db":
   1: resource "aws_glue_catalog_database" "transform_db" {

I have tried removing the local Terraform cache and checking for any potential conflicts, but the issues persist. Has anyone encountered these errors before or does anyone have recommendations on how to resolve them? Any help or pointers would be greatly appreciated!

Thank you in advance.

Hello @liberty2008,

The issue with de_c4w4a2_silver_db and other existing resources happened to @floojeda as well. The solution that @pmk209 explained that after terraform apply the lab rebooted and you need to delete them.

I could at least find silver_db at AWS glue databases and deleting the others as well should be straightforward. You could try in a new lab session and hopefully terraform apply succeds. Thank you

I’ve tried to delete manually, but faced another issue that I don’t have permission to delete those resources :grinning_face:. So how to get my specialization certificate?)

Hello @liberty2008,

Sorry for the inconvenience, if you can’t delete them manually you could wait for a new lab session. Hopefully that helps clear them, you could fill this form for a lab refresh but it takes 1-2 business days (no weekends). Thank you

If this is an ongoing issue, then why must we request a lab refresh? This should be fixed for all students.

Hello,

Sorry for the inconvenience, it seems the lab refresh is the best way to fix permissions issues. It also helps clear any resources that could be found in the AWS console. There are resources that might fix after a new lab session a few hours later.

It is very difficult to explain when a lab refresh will work. For example waiting during the weekend with a bug that could’ve been addressed differently. The staff is able to fix that for all students, thank you