C3W2 Assignment terraform init plan causing error


When running terraform plan i lose connection to the lab, and it attempts to reconnect.

Hello @joaco368,
If you get that error you should choose reload window. That will reset your terraform variables and asks for inputs if you run terraform plan again and you need to run source scripts/setup.sh again and continue from the step you were before.

Note if there is some issue like when you uncomment main.tf and output.tf files you might disconnect again or get terminal crashes.

If that happens and you need to start fresh you can delete terraform folder and go to help to get latest version. Hope it helps

Hi mentor Georgios, i am facing the error too, everytime on terraform apply or plan the terminal will crash when this command is executed.

Hello! Just adding I am getting the same error. It crashes and the recommendations provided in the lab and above are not working to resolve the error.

I restarted the lab, and half an hour later worked correctly!

1 Like

Hello @htango, @David_Cox1,

I tried the lab and followed each step I couldn’t reproduce the issue. Since your issue is at the second time you run the terraform commands, I would check for any None values in step 4.2.3 in terraform/modules/transform_etl/glue.tf file.

After a crash of the terminal you need to run source scripts/setup.sh again and only uncomment exactly what is necessary in the outputs.tf and main.tf files. If it continues you could ask for a lab refresh with this form. Hope it helps

Hi, Georgios. Thanks for your reply. There are no None values in that file. I can’t get past “terraform plan” in step 3.2.2 as the terminal always crashes (see screenshot).

I tried asking for a lab refresh (which I was informed was completed). But I am still seeing my original files there. How can I get a clean refresh so I can start anew?

Hello @David_Cox1,

I could reproduce your issue if I only uncomment 1-17 lines in main.tf after I uncomment transform module in outputs.tf from previous try and keep it uncommented terraform plan will crash:

Solution above is to comment again the Module Transformation in outputs.tf (lines 15-25) to continue. Hope it helps

If you want to start fresh in labs for future reference you could delete the lab files you completed before and use get_latest_version from help. If you used terraform apply before and the lab server was rebooted without destroying terraform you need to go to AWS console or delete any existing resources manually using aws commands.

Thanks, Georgios! That worked. Thank you!

Now I am getting this error when trying to run the ratings-to-iceberg job. I’m at a bit of a loss where to go to identify the root cause.

Hello @David_Cox1,

Could you check in AWS console in de-c3w2a1-ratings-transformation-job and go to Runs tab. There is a SQL query you need to complete (step.4.2.1):

select * 
from ratings 
where ingest_ts = (select max(ingest_ts) from ratings)

Also you should have no None values in step 4.2.3 in terraform/modules/transform_etl/glue.tf. Hope it helps

That all looks good. Here is the SQL query in the file:

Here shows that succeeded in AWS:

And attached is the terraform/modules/transform_etl.glue.tf file showing no None values:
resource “aws_glue_job” “ratings_transformation_job” {
name = “${var.project}-ratings-transformation-job”
role_arn = var.glue_role_arn
glue_version = “4.0”

command {
name = “glueetl”
script_location = “s3://${var.scripts_bucket_name}/de_c3w2a1_json_transform.py”
python_version = 3
}

default_arguments = {
“–enable-job-insights” = “true”
“–job-language” = “python”
“–data_lake_bucket” = var.data_lake_name
“–database_name” = var.curated_db_name
“–ml_table” = var.curated_db_ml_table
“–lakehouse_path” = “s3://${var.data_lake_name}/”
“–datalake-formats” = “iceberg”
}

timeout = 7

number_of_workers = 2
worker_type = “G.1X”
}

resource “aws_glue_job” “csv_transformation_job” {
name = “${var.project}-csv-transformation-job”
role_arn = var.glue_role_arn
glue_version = “4.0”

command {
name = “glueetl”
script_location = “s3://${var.scripts_bucket_name}/de_c3w2a1_batch_transform.py”
python_version = 3
}

default_arguments = {
“–enable-job-insights” = “true”
“–job-language” = “python”
“–data_lake_bucket” = var.data_lake_name
}

timeout = 5

number_of_workers = 2
worker_type = “G.1X”
}

resource “aws_glue_job” “ratings_to_iceberg_job” {
name = “${var.project}-ratings-to-iceberg-job”
role_arn = var.glue_role_arn
glue_version = “4.0”

command {
name = “glueetl”
script_location = “s3://${var.scripts_bucket_name}/de_c3w2a1_ratings_to_iceberg.py”
python_version = 3
}

default_arguments = {
“–enable-job-insights” = “true”
“–job-language” = “python”
“–data_lake_bucket” = var.data_lake_name
“–database_name” = var.curated_db_name
“–table_name” = var.curated_db_ratings_table
“–lakehouse_path” = “s3://${var.data_lake_name}/”
“–datalake-formats” = “iceberg”
}

timeout = 7

number_of_workers = 2
worker_type = “G.1X”
}

Sorry I mean this job (de_c3w2a1_ratings_to_iceberg.py), what is the failed message :

AnalysisException: Column ‘ingst_ts’ does not exist. Did you mean one of the following? [ratings.ingest_ts, ratings.productCode, ratings.productRating, ratings.customerNumber]; line 4 pos 6;

@David_Cox1, it seems you have a typo in line 109, could you go to Scripts tab and edit it and run it, then try submitting to see if it pass, thanks:

Hello @David_Cox1

Or it could be line 159 in the the same Job (de_c3w2a1_ratings_to_iceberg.py). Thanks

I think I’m missing a connection somewhere. Both of those lines are spelled, “ingest_ts”.

Here are the screenshots:

What script exactly is being referenced with the end of this error message reporting this exists on line 4, position 6? AnalysisException: Column 'ingst_ts' does not exist. Did you mean one of the following? [ratings.ingest_ts, ratings.productCode, ratings.productRating, ratings.customerNumber]; line 4 pos 6;

Hello @David_Cox1,

Yes your scripts looks fine, I could reproduce this error after changing line 109, line 4 pos 30:

You could try deleting the file and use get_latest_version. But first try submitting to see if it pass. Hope it helps

Hello @David_Cox1,

Now I got the exact error, line 4 pos 6 after changing the first ingest_ts to ingst_ts in line 109:

Could you go to AWS console in de-c3w2a1-ratings-to-iceberg-job Scripts tab and check line 109 change it and use save then run:

Thanks for your reply, Georgios. I was able to get back into the lab today and start from the top and work down. For some reason it worked today? Maybe I had that error previously when I pushed the scripts to AWS. But, it ran through fine today so I think I am good.

Thanks for your help!

1 Like

I have the same problem here. No issue with any of my lab files but it asks me to manually pass variable values when I run terraform apply.

like pass value of var.password.

I should not that in the beginning of the lab my AWS Console link was returning an empty string. I had to rerun to get the link.