C4W4 vscode disconnects. Terraform doesn't find variables

I got a message that vscode had disconnected. Jupyterlab was also disconnected. So I ran the setup shell script again and got a new url for jupyterlab. But now when I try to run terraform, it doesn’t seem to be finding the variables. Instead it’s asking me to type in the value of the variables. I might be able to do this but unfortunately it doesn’t show me what I’ve typed so I can’t detect the frequent typos my mac’s butterfly keyboard always makes.

Trying again, the terminal keeps quitting with this error: The terminal process “/bin/bash” terminated with exit code: 100.

Hello @drl
Terraform variables are defined in the setup.sh file as can be seen hereunder.


So, whenever you start a new terminal instance and see that it’s asking you to input terraform variables, you can simply run this file once again, and you will be good to go. The command to run this file from the default directory would be sh scripts/setup.sh.

Hi @Amir_Zare I am having a similar issue and this just doesn’t seem to work. It keeps asking me to input the variables.

Also, I am getting trouble here as well:

 # At the `physical_connection_requirements` configuration, set the `subnet_id` to `data.aws_subnet.private_a.id` and 
  # the `security_group_id_list` to a list containing the element `data.aws_security_group.db_sg.id`
  physical_connection_requirements {
    availability_zone      = data.aws_subnet.public_a.availability_zone
    security_group_id_list = [data.aws_security_group.db_sg.id]
    subnet_id              = data.aws_subnet.private_a.id
  }

There are no private_a variables but when I change it to public_a, I am still getting issues. I am so lost.

@Amir_Zare I have a similar mistake. There is no private_a subnet created on aws, so when running terraform apply it fails. You cant use data.aws_subnet.private_a if it doesnt exist. I created one on my own but then glue jobs fail because of miss configuration

Hi @SophiaBrowne and @floojeda
Seems like there is a mistake in the instructions, and it should be public_a instead of private_a. Thank you for reporting this, and I will ask the team to fix it.
@SophiaBrowne you should run sh scripts/setup.sh in the same terminal that you run your terraform commands. If you still get prompted for variables, please let us know what variable you are being prompted for.

1 Like

Thank you so much, I passed the Capstone 1! I found another mistake in the instructions transform_job/glue.tf file where we are asked to use the same object key in for both the transform-json-job and the transform-songs-job.

# Use the scripts_bucket variable and `de-c4w4a1-transform-songs-job.py` as the object key
    script_location = "s3://${var.scripts_bucket}/de-c4w4a1-transform-json-job.py"

Are they really mistakes though? Or are they meant to see if we are paying attention to what we are doing and/or able to identify potential errors?

Hello @SophiaBrowne
Thank you for pointing out the second mistake. These aren’t intentional errors and have been committed during a recent update, I guess. However, you are right. Although facing these issues can be frustrating at times, trying to resolve them can be an experience that enriches the learning process.

@Amir_Zare I cant put the terraform infrastructure up in the second capstone project in C4W4. It shows an error because of resources that already exist and then the glue job does not work. I already updated it and reboot it.

@Amir_Zare This is the error, and then I cant run the glue jobs

Error: creating Glue Connection (de-c4w4a2-connection-rds): operation error Glue: CreateConnection, https response error StatusCode: 400, RequestID: 9d8d89e3-8b24-4e58-b83d-5be9e0877641, AlreadyExistsException: Connection already exists.

  with module.extract_job.aws_glue_connection.rds_connection,
  on modules/extract_job/glue.tf line 2, in resource "aws_glue_connection" "rds_connection":
   2: resource "aws_glue_connection" "rds_connection" {


Error: creating IAM Role (de-c4w4a2-glue-role): operation error IAM: CreateRole, https response error StatusCode: 409, RequestID: 855b9fa7-fee3-4be4-ae31-b76435cec20b, EntityAlreadyExists: Role with name de-c4w4a2-glue-role already exists.

  with module.extract_job.aws_iam_role.glue_role,
  on modules/extract_job/iam.tf line 1, in resource "aws_iam_role" "glue_role":
   1: resource "aws_iam_role" "glue_role" {


Error: pq: Schema "deftunes_serving" already exists

  with module.serving.redshift_schema.serving_schema,
  on modules/serving/redshift.tf line 1, in resource "redshift_schema" "serving_schema":
   1: resource "redshift_schema" "serving_schema" {


Error: pq: Schema "deftunes_transform" already exists

  with module.serving.redshift_schema.transform_external_from_glue_data_catalog,
  on modules/serving/redshift.tf line 9, in resource "redshift_schema" "transform_external_from_glue_data_catalog":
   9: resource "redshift_schema" "transform_external_from_glue_data_catalog" {

@floojeda in your case, please request a lab refresh via this form.

@Amir_Zare I solved it but now I cant run glue jobs:

aws glue start-job-run --job-name glue_api_users_extract_job | jq -r '.JobRunId'

An error occurred (AccessDeniedException) when calling the StartJobRun operation: User: arn:aws:sts::339712957310:assumed-role/voclabs/user3823079=dyuqfeojgjei is not authorized to perform: glue:StartJobRun on resource: arn:aws:glue:us-east-1:339712957310:job/glue_api_users_extract_job because no identity-based policy allows the glue:StartJobRun action

@floojeda glue_api_users_extract_job is not the correct job name. You need to get the terraform output with the name glue_api_users_extract_job and use that as your job name when running the glue commands.

Hi, The VScode UI keeps disconnecting wih below error , have tried multiple times to complete but every time getting the same issue.

Hello @ami_il
Sorry for the inconvenience. There is a problem with the command you run before the terminal crashes. In order to find the source of the problem, you can record a screen cast before running the command, and see the video right before the terminal closes.

Hi,
Thank you for your response. The issue arises during the second Terraform execution. When I run terraform init , plan , and apply for the first time (which handles fetching data from the API and storing it in S3), everything works without any problems. However, when I run terraform init again for the transformation phase, that’s when I encounter the error mentioned earlier.