While running the terraform jobs from 2.4 in Coure 4 Week 4 - Capstone Project part 2 I am receiving the following error:
(jupyterlab-venv) abc@328c49e1d593:~/workspace/terraform$ aws glue start-job-run --job-name glue_api_users_extract_job | jq
-r '.JobRunId'
An error occurred (AccessDeniedException) when calling the StartJobRun operation: User: arn:aws:sts::498941574322:assumed-role/VSCodeInstanceRole/i-0267184375b24efcf is not authorized to perform: glue:StartJobRun on resource: arn:aws:glue:us-east-1:498941574322:job/glue_api_users_extract_job because no identity-based policy allows the glue:StartJobRun action
(jupyterlab-venv) abc@328c49e1d593:~/workspace/terraform$
What is the reason? I don’t see where in the lab we were setting up the IAM.
I have gone to cloudshell and implemented the snippet under nano, however, nothing improves whatsoever. I am simply stuck on this part, and it is impacting my plans as I plan to graduate by no later the Jan 7th.
Hello @amiguel,
Sorry about this issue, it has been reported by others and I had my VSCode terminal crashed as well. Could you tell us which part of the Capstone 1 project it happens, since someone had the same issue in part 4.1.6 and 4.2.6 when running the terraform plan command and there was a typo in the configuration file (in step 4.2.3 "iceberg" is missing in "--datalake-formats" = "None").
Once it happens you don’t need to go to cloudshell whenever the terminal crashes, just try to open a new terminal instance. Thanks for your patience but we’re looking into the issue.
Could you check in terraform there are only two files you need to edit in the transform_job module, S3.tf and Glue.tf. In case the VSCode terminal keeps crashing after 4.2.6 there could be an issue in those files. In the terraform/modules/transform_job/glue.tf check you are using the correct ingestion_date (e.g. “2025-01-01”) and datalake-formats= “iceberg”.
You might get the terminal crash in the extract_job that and need to restart it (just open a new terminal an repeat the step 4.1.6). Thanks
Hi @Georgios - I think I am now on track and managed to bring under control the crashing side of the VSCode by swapping between terminals however, one thing I did notice though is that the job run status are very random and never deterministic, i.e., some times you get SUCCEDED in only a single job run and very often for others you get RUNNING and FAILED. I understand the grader is happy only to grading SUCCEDED job runs. Is there an approach to deterministically get SUCCEDED or it is just a matter of keeping trying until the lab assigned hours elapses?
You could check if you are using the correct API endpoint and todays date should be in Pacific time (not your local date). Also someone had his transform_songs job failing because of a typo in the ingest_on.
In order to get a better understanding of the issue, you can search for AWS Glue in the AWS console, choose ETL jobs from the menu on the left hand side, choose a job, and see the logs for the runs on those jobs.
Post a screenshot if you have a specific issue, thanks
Okay, I will make sure my time-zone dates fall within the PT time-zone. Moreover, mixing a bit on the topics here, for the capstone 1, I am facing the following issue as below. How shall I go about it?
Hello @amiguel,
Yes, please complete the form, I think you need to refresh your lab. Sorry for the inconvenience perhaps a staff @esanina could fix this for you. This might take more than usual after the holidays. Thank you
Hello @amiguel,
It looks like the landing zone ingestion job might be successful but something is not created within your bucket folder. This is the reason the transform job failing.
Could you check the API endpoint (in two places) since it changes every time you open the lab you might need to change that.
If you are stuck try to message me which part you need help and you are not sure. Thanks
