How to fix date format errors in de-c4w4a1-songs-transform-job and de-c4w4a1-json-transform-job?

Course Q&A Data Engineering Data Modeling, Transformation, and Serving
1

I completed the scripts for the two glue jobs (de-c4w4a1-songs-transform-job and de-c4w4a1-json-transform-job) but I run them, I got these errors (see them in the screeshots below). These errors come from the date format in the scripts but I don’t know how to fix them.
I put “yyyy-mm-dd” on the PACIFIC-TIME-CURRENT-DATE placeholder.

Could anyone help me fix these errors?

json_transform_job_error
json_transform_job_error632×184 16.7 KB

songs_tranform_job_error
songs_tranform_job_error627×151 13.6 KB

#W4-Capstone Part1 #de-c4w4a1-songs-transform-job #de-c4w4a1-json-transform-job

2

Hello @MartinAmouzou,
Could you check in parts
4.1.3. and 4.2.3. in the terraform/modules/extract_job/glue.tf and terraform/modules/transform_job/glue.tf files you need to replace placeholders with "actual dates" and use the currect date in Pacific time (e.g. “2025-01-08”) in two places. Thanks

3

Hi @Georgios ,

I’ve done the checks and modifications but they’re still failing.

Thanks.

4

Hello @MartinAmouzou,
You could check:

  1. rerun the part in 4.2.6 after you updated the glue.tf file
  2. if the correct folder is created in the data lake bucket from the landing jobs. If not check you have the correct API-Endpoint in the terraform/modules/extract_job/glue.tf
  3. you might need to run again the ingestion jobs if you make changes above.
  4. maybe on the of the py files in part 4.2.1 has any typos.
5

Hello @Georgios ,

I’ve finally fixed the issue. The problem was on the current paficic time and a method in one of the transformations glue job scripts.

Thank you !

1 Like
6

Hi @Georgios @MartinAmouzou , I’m having the same issue, would you care to elaborate how did you fix it ?

7

Hello @aurelio_gialluca,
I think what @MartinAmouzou had the wrong date in step 4.2.3 terraform/modules/transform_job/glue.tf file (replace with today’s date in Pacific time, just Google it e.g. “2025-01-12”)
The second tying he mentioned might be in one of the two py files in step 4.2.1 (a method like columns or ingest_on had a typo I guess). Hope it helped

8

Hello @aurelio_gialluca,

As said @Georgios, I had the wrong date and the second was on my two pyhon files.

Hope it helped.

9

Hi. Please help me. Wich is the error in the python files exactly? And, by the other hand, I change the pacific dates, but only in transformation… @Georgios

10

@paniJc,
Thats correct you change the dates in pacific time only in terraform/modules/transform_job/glue.tf in the terraform/modules/extract_job/glue.tf you keep the dates provided to you start:2020-01-01 to end 2020-01-31.

In the python files do you use the correct methods:
df.withColumn(“ingest_on”, F.to_date(F.lit(ingest_date), etc…
Hope it helps

11

Hi @Georgios ,
I got the problem with iam roles and can’t continue. How can I solve it. Or even can I start from begining. I have tried a lot of ways including deleting roles in IAM but I can not solve problem.

image
image1664×452 16.9 KB

12

Hello @JavidM,

The IdempotentParameterMismatchException that throws this error because those jobs already exist with a different configuration. Which should be able to fix with the lab refresh form. Are you in Vocareum or Coursera interface and could you wait and try in a new lab session. Hope it helps

1 Like