C3-W2 Assignment Metadata problem

Course Q&A Data Engineering Data Storage and Queries
,
1

When creating a post, please add:

2

Hello @Noddy,

Sorry for the inconvenience you are facing. It seems you are having issues with terraform apply or successfully running extract/transform jobs. There are a few other learners who had those issues with terraform because of existing resources when running terraform apply (you could check the errors.txt file when if you run terraform apply -no-color 2> errors.txt).

If you use the forum search tool you could see people who submitted the lab refresh form, that takes 1-2 business days to complete. This form also helps with Insufficient Lake Formation issues that cause the jobs fail because of metadata. Hope it helps

3

Hi @Georgios, As Mentioned earlier i have done everything stated in the conversation and refreshing the lab with other ways, Need Your Support atleast the Lab should go back to Zero ground so that i can start again that also is not happening. Solving the Lab is not much of the task but the problem is with infrastructure from last 2 weeks

4

@Noddy When you try again the lab, the terraform files contain your saved work from a previous session.

The lab asks you first to uncomment first Landing zone in main.tf and outputs.tf and then run the jobs. If you try again the lab and let’s say you have the first two modules uncommented from the previous session, and your run terraform in the first section, all the resources of these two modules will be created. So when you get to the second section and run terraform again, you’ll get that all the resources have been already created.

So when you try the lab again, I’d suggest that you either comment back everything you uncommented from previous sessions (in main.tf and outputs.tf) Or you can retrieve the original lab files as follows:

  1. Create a new folder and move all current lab files to this folder (you can label the folder as old_files)
  2. Click on lab help which is the question mark on the top right
  3. Click on “Get the latest version”.
5

@Noddy, In order to pass the grader you only need to run the Landing zone jobs (two) and Curated zone jobs (three). In order to do that you need to follow the instructions until step 4.3. If you have any issues with those jobs you could check again at the edited scripts. That’s all you need to do, thank you:

  1. Uncomment main.tf until line 32 (transform_etl),Uncomment outputs.tf until line 25 (transform_etl)
  2. Run the terraform commands
  3. Run the 2 landing_etl_jobs
  4. Run the 3 transform_etl_jobs
  5. Submit the assignment
6

I have Done step 2 you have mentioned, but i when i run the ETL Jobs, the error is appearing, i have uncommented Main.tf & Outputs.tf

7

@Noddy,
The error like metadata means that after the you ran terraform applythe output is not showing as apply complete. That means there is an issue with existing resources in AWS console. Could you try terraform apply -no-color 2> errors.txt instead of terraform apply and check at the file created errors.txt. Thank you

8

@Noddy Have you copied your files into the new folder and used get latest version like I suggested so you can edit your files frorm the beginning. What is the error you are getting, in the first terrafrom deployment steps 3.2.2 or after you edited the glue.tf file in step 4.2.3. Please share a screenshot of the issue so I can see it, thank you