In the transform glue job terraform files, it asks you to input a date in yyyy-mm-dd format in several spots. This is supposed to lookup data in your s3 bucket, which will have data stored in a file named something like “2024/10/29”…but that date will have been set by datetime.now() which on AWS will be the date in UTC. So if, like me, you’re on the east coast of the US and working at like 10 pm on 10/29, it’s actually 10/30 UTC, so you have to adjust the date to the day it is in UTC when you’re asked to manually enter it.
I hope this helps someone. This one had me stumped for a bit as I kept getting that my transform jobs were failing!