Hello @joaco368,
If you get that error you should choose reload window. That will reset your terraform variables and asks for inputs if you run terraform plan again and you need to run source scripts/setup.sh again and continue from the step you were before.
Note if there is some issue like when you uncomment main.tf and output.tf files you might disconnect again or get terminal crashes.
If that happens and you need to start fresh you can delete terraform folder and go to help to get latest version. Hope it helps
Hi mentor Georgios, i am facing the error too, everytime on terraform apply or plan the terminal will crash when this command is executed.
Hello! Just adding I am getting the same error. It crashes and the recommendations provided in the lab and above are not working to resolve the error.
I restarted the lab, and half an hour later worked correctly!
Hello @htango, @David_Cox1,
I tried the lab and followed each step I couldnât reproduce the issue. Since your issue is at the second time you run the terraform commands, I would check for any None
values in step 4.2.3 in terraform/modules/transform_etl/glue.tf
file.
After a crash of the terminal you need to run source scripts/setup.sh
again and only uncomment exactly what is necessary in the outputs.tf
and main.tf
files. If it continues you could ask for a lab refresh with this form. Hope it helps
Hi, Georgios. Thanks for your reply. There are no None
values in that file. I canât get past âterraform planâ in step 3.2.2 as the terminal always crashes (see screenshot).
I tried asking for a lab refresh (which I was informed was completed). But I am still seeing my original files there. How can I get a clean refresh so I can start anew?
Hello @David_Cox1,
I could reproduce your issue if I only uncomment 1-17 lines in main.tf
after I uncomment transform module in outputs.tf
from previous try and keep it uncommented terraform plan
will crash:
Solution above is to comment again the Module Transformation in outputs.tf
(lines 15-25) to continue. Hope it helps
If you want to start fresh in labs for future reference you could delete the lab files you completed before and use get_latest_version
from help. If you used terraform apply
before and the lab server was rebooted without destroying terraform you need to go to AWS console or delete any existing resources manually using aws commands.
Thanks, Georgios! That worked. Thank you!
Now I am getting this error when trying to run the ratings-to-iceberg job. Iâm at a bit of a loss where to go to identify the root cause.
Hello @David_Cox1,
Could you check in AWS console in de-c3w2a1-ratings-transformation-job
and go to Runs tab. There is a SQL query you need to complete (step.4.2.1):
select *
from ratings
where ingest_ts = (select max(ingest_ts) from ratings)
Also you should have no None
values in step 4.2.3 in terraform/modules/transform_etl/glue.tf
. Hope it helps
That all looks good. Here is the SQL query in the file:
Here shows that succeeded in AWS:
And attached is the terraform/modules/transform_etl.glue.tf
file showing no None values:
resource âaws_glue_jobâ âratings_transformation_jobâ {
name = â${var.project}-ratings-transformation-jobâ
role_arn = var.glue_role_arn
glue_version = â4.0â
command {
name = âglueetlâ
script_location = âs3://${var.scripts_bucket_name}/de_c3w2a1_json_transform.pyâ
python_version = 3
}
default_arguments = {
ââenable-job-insightsâ = âtrueâ
ââjob-languageâ = âpythonâ
ââdata_lake_bucketâ = var.data_lake_name
ââdatabase_nameâ = var.curated_db_name
ââml_tableâ = var.curated_db_ml_table
ââlakehouse_pathâ = âs3://${var.data_lake_name}/â
ââdatalake-formatsâ = âicebergâ
}
timeout = 7
number_of_workers = 2
worker_type = âG.1Xâ
}
resource âaws_glue_jobâ âcsv_transformation_jobâ {
name = â${var.project}-csv-transformation-jobâ
role_arn = var.glue_role_arn
glue_version = â4.0â
command {
name = âglueetlâ
script_location = âs3://${var.scripts_bucket_name}/de_c3w2a1_batch_transform.pyâ
python_version = 3
}
default_arguments = {
ââenable-job-insightsâ = âtrueâ
ââjob-languageâ = âpythonâ
ââdata_lake_bucketâ = var.data_lake_name
}
timeout = 5
number_of_workers = 2
worker_type = âG.1Xâ
}
resource âaws_glue_jobâ âratings_to_iceberg_jobâ {
name = â${var.project}-ratings-to-iceberg-jobâ
role_arn = var.glue_role_arn
glue_version = â4.0â
command {
name = âglueetlâ
script_location = âs3://${var.scripts_bucket_name}/de_c3w2a1_ratings_to_iceberg.pyâ
python_version = 3
}
default_arguments = {
ââenable-job-insightsâ = âtrueâ
ââjob-languageâ = âpythonâ
ââdata_lake_bucketâ = var.data_lake_name
ââdatabase_nameâ = var.curated_db_name
ââtable_nameâ = var.curated_db_ratings_table
ââlakehouse_pathâ = âs3://${var.data_lake_name}/â
ââdatalake-formatsâ = âicebergâ
}
timeout = 7
number_of_workers = 2
worker_type = âG.1Xâ
}
AnalysisException: Column âingst_tsâ does not exist. Did you mean one of the following? [ratings.ingest_ts, ratings.productCode, ratings.productRating, ratings.customerNumber]; line 4 pos 6;
@David_Cox1, it seems you have a typo in line 109, could you go to Scripts tab and edit it and run it, then try submitting to see if it pass, thanks:
Hello @David_Cox1
Or it could be line 159 in the the same Job (de_c3w2a1_ratings_to_iceberg.py
). Thanks
I think Iâm missing a connection somewhere. Both of those lines are spelled, âingest_tsâ.
Here are the screenshots:
What script exactly is being referenced with the end of this error message reporting this exists on line 4, position 6? AnalysisException: Column 'ingst_ts' does not exist. Did you mean one of the following? [ratings.ingest_ts, ratings.productCode, ratings.productRating, ratings.customerNumber]; line 4 pos 6;
Hello @David_Cox1,
Yes your scripts looks fine, I could reproduce this error after changing line 109, line 4 pos 30:
You could try deleting the file and use get_latest_version. But first try submitting to see if it pass. Hope it helps
Hello @David_Cox1,
Now I got the exact error, line 4 pos 6 after changing the first ingest_ts to ingst_ts in line 109:
Could you go to AWS console in de-c3w2a1-ratings-to-iceberg-job Scripts tab and check line 109 change it and use save then run:
Thanks for your reply, Georgios. I was able to get back into the lab today and start from the top and work down. For some reason it worked today? Maybe I had that error previously when I pushed the scripts to AWS. But, it ran through fine today so I think I am good.
Thanks for your help!
I have the same problem here. No issue with any of my lab files but it asks me to manually pass variable values when I run terraform apply.
like pass value of var.password.
I should not that in the beginning of the lab my AWS Console link was returning an empty string. I had to rerun to get the link.