AGAIN? other glitch in the testing?

Course Q&A Data Engineering Data Modeling, Transformation, and Serving
1

IM working on the capstone part 1, so the evaluation feedback are:

Glue Job de-c4w4a1-api-users-extract-job exists.
Glue Job de-c4w4a1-api-sessions-extract-job exists.
Glue Job de-c4w4a1-rds-extract-job exists.
Test 1 passed: All three extract jobs are correctly deployed.
========================================================================================================================
Glue Job de-c4w4a1-api-users-extract-job has at least one successful run.
Glue Job de-c4w4a1-api-sessions-extract-job has at least one successful run.
Glue Job de-c4w4a1-rds-extract-job has at least one successful run.
Test 2 passed: All extract jobs have at least one successful run.
========================================================================================================================
Glue Job de-c4w4a1-json-transform-job exists.
Glue Job de-c4w4a1-songs-transform-job exists.
Test 3 passed: Both transformation jobs are correctly deployed.
========================================================================================================================
Glue Job de-c4w4a1-json-transform-job has not completed a successful run.
Glue Job de-c4w4a1-songs-transform-job has not completed a successful run.
Test 4 failed: None of the transformation jobs have successful runs.

:rage: :rage: :rage: I thoght that it was a bad config on the tf files for the Transformation Section or maybe i didnt run the services well, the cli told me that:

nn
nn501Ă—248 47 KB

Dude i followed the instructions as the bible itself:

4.2.7. You will get some additional outputs, in particular, you require the following two: `glue_json_transformation_job` and `glue_songs_transformation_job`. Execute the two glue jobs, based on the name (`de-c4w4a1-json-transform-job` or `de-c4w4a1-songs-transform-job`). You can run those two jobs in parallel. Use the following command in the VSCode terminal:


aws glue start-job-run --job-name <JOB-NAME> | jq -r '.JobRunId'


And based on the job run ID track each job status by using this command:


aws glue get-job-run --job-name <JOB-NAME> --run-id <JobRunID> --output text --query "JobRun.JobRunState"


Wait until the jobs statuses change to `SUCCEEDED` (each job should take around 3 mins).

WHY my console doesnt says SUCCESS or Exists any way to see the running job ??? :rage: :rage: :rage:

2

UPDATE: I was reading other posts

  • Maybe the problem is originated from the glue.tf date format
'''terraform/modules/transform_job/glue.tf'''

    # Set "--ingest_date" to the server's current date in Pacific Time (UTC-7), in "yyyy-mm-dd" format.
    # (replace the placeholder `<PACIFIC-TIME-CURRENT-DATE>`)
    "--ingest_date"             = "yyyy-mm-dd"
    # Review the users source path
  • Perhaps the solution is to declare something into the terraform/assets/transform_job/*.py files to correctly declare the datetime format or something else (JUST a crazy hipothesis)
  • Or mayble use a crazy method from the terraform API

I will update later or you can update this post too :face_with_monocle:

3

Hello @Hvnt3rK3ys
In the part "--ingest_date" = "yyyy-mm-dd" of the glue.tf files, you are supposed to replace yyyy-mm-dd with the current pacific date. Did you do so?
If you want to see the status of your glue jobs in the UI, you can search for AWS Glue in the AWS Console, select ETL jobs from the left hand-side menu, choose a job, and choose a run for that job. There you can see the status of all the glue jobs’ runs.

1 Like
4

I was into the AWS gui for Glue looking for the Monitoring section:

ETL jobs
- Visual ETL
- Notebooks
- Job run monitoring <--- I think this option is the correct

Then, aws told me that:

Job runs summary
Error encountered
AccessDeniedException (status: 403): Unauthorized access for account id 719******

So in the IAM section appears:
Access denied
You do not have permission to iam:GetAccountSummary . To request access, copy the following text and send it to your AWS administrator. Learn more about troubleshooting access denied issues.

Now what ?? :upside_down_face:

5

FIXED: Its easier than it seems :woozy_face:

Into the datetime options

'''terraform/modules/transform_job/glue.tf'''

    # Set "--ingest_date" to the server's current date in Pacific Time (UTC-7), in "yyyy-mm-dd" format.
    # (replace the placeholder `<PACIFIC-TIME-CURRENT-DATE>`)
    "--ingest_date"             = "<PUT Todays date into the format>"
    # Review the users source path

To debug using de CLI:

## GET the current ID
aws glue start-job-run --job-name <JOB-NAME> | jq -r '.JobRunId'

## EXECUTE the order
aws glue get-job-run --job-name <JOB-NAME> --run-id <JobRunID> --output text --query "JobRun.JobRunState"

## CHECK if something explode 
aws glue get-job-run --job-name <JOB-NAME> --run-id <job_run_id>

Wait until you hace SUCCED test

❯ aws glue get-job-run --job-name de-c4w4a1-songs-transform-job --run-id jr_6981b5a4b5557140147044ed18c93da1f89a00c36d102c5426fd042b4acdfb9f
{
    "JobRun": {
        "Id": "jr_6981b5a4b5557140147044ed18c93da1f89a00c36d102c5426fd042b4acdfb9f",
        "Attempt": 0,
        "JobName": "de-c4w4a1-songs-transform-job",
        "JobMode": "SCRIPT",
        "JobRunQueuingEnabled": false,
        "StartedOn": 1735594749.885,
        "LastModifiedOn": 1735594873.402,
        "CompletedOn": 1735594873.402,
        "JobRunState": "SUCCEEDED",
        "PredecessorRuns": [],
        "AllocatedCapacity": 2,
        "ExecutionTime": 107,
        "Timeout": 5,
        "MaxCapacity": 2.0,
        "WorkerType": "G.1X",
        "NumberOfWorkers": 2,
        "LogGroupName": "/aws-glue/jobs",
        "GlueVersion": "4.0"
    }
}
6

@Hvnt3rK3ys happy to hear that the issue has been fixed. In order to see the runs status, you don’t need to go to the monitoring section in the UI. You can simply select the job, and there is a table including the job’s runs and their status.

1 Like