C3_W2_Assignment Assignment 2: Building a Data Lakehouse with AWS Lake Formation and Apache Iceberg, terraform

Course Q&A Data Engineering Data Storage and Queries
, ,
1

Working on Coursera , Data Storage and queries, week 2, assignment 2, trying to run terraform init, terraform plan || echo ”$?”, command asks for input for var.currated_db_name, or var.data_lake_name, …etc. The variables are not being used from the terraform/variables.tf file. The variables are all there but have no default values. The landing_etl will not work, nor the transform_etl. The varibles in the main.tf file.

2

Hi @vincentanggaines
Run the following command in the terminal to set up the environment, this should be before running the terraform commands.

source scripts/setup.sh

3

I’m currently working on the C3 W2 assignment and I’m stuck at section 4.3 Deployment while running Terraform.

Issue:

When I run:

terraform apply

Terraform keeps prompting me to manually enter a large number of variables (e.g., curated_db_name, data_lake_name, source_host, etc.).

What I expected:

Based on the lab instructions and mentor comments, these values should be automatically populated after running:

source scripts/setup.sh

What I’ve already tried:

  • I ran source scripts/setup.sh before running Terraform

  • I did NOT restart the terminal

  • I uncommented only the required sections in:

    • terraform/main.tf

    • terraform/outputs.tf

  • I replaced all None values in the Terraform files (as required in earlier steps)

Problem:

Despite doing all of the above, Terraform still asks me to input every variable manually.

Additional details:

  • This happens even without restarting the terminal

  • It occurs specifically after enabling the transform_etl module

  • I was able to run earlier sections without this issue

  • Restarting the Lab is not helping

Question:

  • Is this expected behavior for this lab?

  • Is there something missing in setup.sh or environment variable setup?

I’ve been stuck on this for a few days now, so any guidance would be really appreciated.

4

Hi

Minit Chitroda , it seems like you have a good approach to solving this problem. As you may know running the setup.sh script sets up the environmental variables for terraform. I imagine that you are repeating some of your previous work. If I remember correctly this lab has different terraform phases to be applied. Depending on which phase you are using the scripts use different variables. However one would think that the setup script would set all required environment variables for terraform. I found this environment very frustrating. It seems to get confused very easily. After much effort I found the only true way is to let your aws session expire and start again from scratch. You might try rebooting your corserra session first, but this does not help much. After i waited and started from scratch with a new lag start, following the instructions worked great.. Good luck. Let me know what the solution is.
5

I believe it takes a couple of hours of inactivity for your session to time-out and all the assets to be released.