Terraform Glue Job continuous RUNNING

Course Q&A Data Engineering Introduction to Data Engineering
,
1

https://labs.vocareum.com/main/main.php?m=clabide&mode=s&asnid=2774164&stepid=2774165&hideNavBar=1

  • Description: Part 1.14 Check the status of the AWS Glue job exchanging the <JobRunID> with the output from the previous step:

Screenshot 2024-11-13 at 10.45.07 AM
Screenshot 2024-11-13 at 10.45.07 AM2091Ɨ1288 352 KB

Screenshot 2024-11-13 at 10.46.06 AM
Screenshot 2024-11-13 at 10.46.06 AM2131Ɨ1427 314 KB

It has been running for 40 minutes. Running out of lab time. Checked s3 buckets, de-c1w4--datalake, not found.

voclabs:~/environment/terraform $ aws glue start-job-run --job-name de-c1w4-etl-job | jq -r '.JobRunId'
jr_0201a2c6e68c6fc27e1ea953a138dd789725248d449699b5166c54ebedc39a45
voclabs:~/environment/terraform $ aws glue get-job-run --job-name de-c1w4-etl-job --run-id jr_0201a2c6e68c6fc27e1ea953a138dd789725248d449699b5166c54ebedc39a45 --output text --query "JobRun.JobRunState"
RUNNING
voclabs:~/environment/terraform $
1 Like
2

Hello @ty_pi,

If it takes too long alternatively, you can use the AWS glue console to start/stop your jobs. You could search for AWS glue then select ETL jobs, from there you can find all the jobs created and select your job. Select the Runs tab and see if you can stop it then rerun it. Thank you
1)

C1W4_select_job
C1W4_select_job1714Ɨ648 133 KB

2)
C1W4_stop_job
C1W4_stop_job1714Ɨ357 86.2 KB

1 Like
3

Awesome, thank you.

1 Like
4

Hello @ty_pi
Regarding the S3 buckets, you should replace the <PLACEHOLDER> with your account ID from the AWS console. I can see that you have 4 buckets, one of which should be the datalake one. I will ask the team to update the instructions about the placeholder.

image
image1895Ɨ823 142 KB

1 Like
5

Thanks, a bit confusing when glue job RUNNING, never switched to SUCCEEDED.
And Iā€™m back to this step, glue job RUNNING. Data lake bucket shows up(like before), but is empty. Which makes sense since glue job is still running.

1 Like
6

Thank you Georgios. I was still getting the RUNNING message in the terminal. By following your instructions I was able to see, in AWS Glue Studio, that the glue job Succeeded! And the data lake bucket had necessary folder(s).

2 Likes