Sep 4 : Deployment 4.3.3 fails for de-c3w2a1-csv-transformation-job
KeyError: DoubleType()
What am I missing?
Hello @Shiji,
Could you check you succesfully run the landing zone jobs in part 3.2.3. Then you could move with the curated_zone and run the jobs in part 4.3.3. Hope it helps
Please let me know if you need any other information.
Yes, 3.2.3 jobs were successfully run. Only de-c3w2a1-csv-transformation-job in 4.3.3 failed
Hello @Shiji,
I could reproduce your error and my first idea was in part 4.1.1 that there was a typo in the script de_c3w2a1_batch_transform.py. It seems there is a bug in your code in the enforce_schema function. You should check in the second part of the code you used source_pd[field_type] instead of source_pd[field_name]. Thats the error I got after a few tries, thank you
Thanks for looking into it. I noticed it and fixed the issue. I was able to complete the assignment.
1 Like