Frustrating problems with part 2 of the capstone: terraform command crash problems

Course Q&A Data Engineering Data Modeling, Transformation, and Serving
1

I have experienced a number of problems with this lab. First I was being told I didn’t have permissions to run the glue jobs in section 2.4. Nothing I could do or find in the course discussions could remedy my issues, so I assumed that the issue was with the lab environment. I rebooted the lab but can no longer get to that part because of issues with the terraform apply commands in section 2.3:

cd terraform
terraform init
terraform plan
terraform apply -target=module.extract_job
terraform apply -target=module.transform_job
terraform apply -target=module.serving

The first ‘apply’ command fails constantly and I can’t get an error printed for it via the usual suggested means. I’m at the end of my rope for the troubleshooting on this and it seems there are a number of people trying to get this finished having somewhat related errors. Can you assist me so that I can finish with this course asap. Is my issue needing a ‘clean slate’ restart somehow ? If so, can you please facilitate.
Thanks and regards,
Peter

2

Note that when I try to redo the terraform section I also get this (I am asked for values) :

coder@e84f42d88f32:~/project/terraform$ terraform plan
var.catalog_database
Curated DB name

Enter a value:

var.data_lake_bucket
Data lake bucket name

Enter a value:

var.db_sg_id
Security group ID for RDS

Enter a value:

And etc.

3

Hi Peter,

Sorry to hear you’re running into so many issues—it definitely sounds frustrating.

This might be an environment-related problem. Before we request a full reset, could you please try the workaround shared in this post?

I’ve also tagged @hawraa.salami and @mubsi in case further support is needed. If the workaround doesn’t resolve it, we’ll help you get a clean restart.

Hang in there—we’ll get you through this!

4

Thanks @benjamintabaresjr !
@PST

  • Regarding the second issue, make sure to run: source scripts/setup.sh the very first thing. This step defines the environment variables needed for terraform. So if you ever get prompted by terraform to enter the values for these variables, it means they are not set up. Running source scripts/setup.sh defines these variables.
  • as I mentioned in the other post, please avoid rebooting the lab within the same session, this will make you lose the terraform state (which what keeps track of what’s been created)
  • make sure you’re doing terraform apply for the modules in the order stated in the instructions.
  • you can append || echo "$?" to all your commands to avoid the terminal from crashing, i.e.,
    • terraform apply -target=module.extract_job || echo "$?"
1 Like
5

@PST Please let me know how things will go with this part. Please feel free to tag me so I can get notified and see your reply.

6

@hawraa.salami @benjamintabaresjr
Those suggestions solved the problem. Thank you for the quick help.
Best,
Peter

2 Likes