Hi - I’m trying to complete the capstone Project Part 1 - ETL and Data Modeling project, however, I keep getting this message:
Any ideas?
I’ve logged out, rebooted computer, etc., still it’s not loading.
Regards,
Matt.
I got this to work, however, when I progress down to running Terrform plan I get the following:
What value is expected here (i.e., Enter a value)?
Please note, this is the step before terraform apply.
Hello @MattP,
Your terminal probably crashed/restart and it asks to input the terraform variables. If the terminal crashes tou need to run source scripts/setup.sh before you go to the terraform folder and run the terraform commands from the beginning. Thank you
1 Like
Thanks, I’ve tried again and have had progress but when running the glue jobs (de-c4w4a1-api-users-extract-job and de-c4w4a1-api-sessions-extract-job) they keep incurring a timeout failure (as follows):
And
I’ve check the instructions in the scripts and my code updates, and I can’t see if I’ve done anything wrong.
Thoughts?
Regards,
Matt.
1 Like
Ignore that - I figured it out (I forgot to update the API endpoint).
1 Like
The server seems to keep crashing. I am was completing the Transform section and the server stopped. I’ve done a reboot, but when I run scripts I get the following error:
What’s seems to be the issue?
Regards,
Matt.
I think there may be something wrong with the backend of my setup.
I’ve run my extract scripts and while the sessions glue job works the users glue job is stating that the url is not found (however, this is the same url that the “sessions” glue job used and ran successfully).
And:
Sessions worked, however:
In the attached image (above), you can see that the api_url (for “users”) is missing the /users, however, in my glue script it is present (seems that even when I save the file is not refreshing or being uploaded).
What seems to be the issue with users glue job?
Regards,
Matt.
I managed to correct this (had to manually correct via the advanced properties tab within AWS Glue).
I’ve rerun my code and now the terminal keeps crashing at the Terraform apply segment of the project (within the transform glue scripts).
I’ve been on this most of the day, and it’s quite frustrating that the terminal keeps crashing.
Is there another way to complete this project outside of running the terminal?
Or, is there a way to enable the terminal to be more stable when running the Terraform scripts?
Regards,
Matt.
Hello @MattP , I’ve faced this issue. It’s possibly because the was credentials expired ( you probably run source scripts/setup.sh many times). You can use the refresh form to request for lab refresh.
link to form: Combined Data Engineering Lab Issue Report
Hello @MattP,
You figured it out, you need to update the API Endpoint every new lab session in the extract jobs and use /users or /sessions at the end.
Please submit this form as @Dan_m suggested for a lab refresh. As your lab probably exceeded limit, you could also check that in the vocapi_logger file from that form. Since you rebooted and probably getting a new AWS link no longer works you need to wait 2 business days to complete. I understand it’s frustrating for simple tasks terraform crashing but I think a new lab session can be more stable than the other. It took me more than an hour to complete the course 1 last assignment the other day to connect to the vector database. I hope you have better luck, thanks
Thanks - I’ve submitted the form.
Hi - I’ve got a new server it seems, but it just crashed again:
Also, this is now taking a lot longer than it needs to. I’ve just paid another month’s subscription (today) when I could have had this assignment closed out last week.
What is the underlying issue?
Until the server is stable (and stops crashing) I’m not going to be able to complete this course. It’s quite frustrating.
How can this be corrected?
Regards,
Matt.
Hello @MattP,
Honestly I am at my wits end after all the tries and lab refresh. My suggestion is to delete the whole terraform folder or make a backup folder as @hawraa.salami explains in this post. Then click help and use get latest version at the top right to update your lab. If it continues crashing in a specific point or takes longer after a new lab session since it supposed to take 5-10 minutes to load the resources. Please explain so maybe @hawraa.salami could check it. Hope it helps
Thanks. I finally got it to run this morning.
Thanks all for your help.
Warm regards,
Matt.
1 Like
Regarding section: 4.2.2. "The deployed DAG is not completed. Replace the following placeholders with the values of the Terraform outputs:
In terms of the replacement values are we to use:
data_lake_bucket (as in, is this what we write in, literally)
scripts_bucket (as in, is this what we write in, literally)
glue_role_arn (as in, is this what we write in, literally)?
Or, do we have to locate specific values (from the Terraform outputs) for:
data_lake_bucket, scripts_bucket and glue_role_arn?
If we need to add specific values (from the Terraform outputs) where does one locate those Terraform output values?
Thanks,
Matt.
Hello @MattP,
You could find those placeholders from the terraform output, should look like this, hope it helps:
- DATA-LAKE-BUCKET=“de-c4w4a2-[ACCOUNT ID]-us-east-1-data-lake”
- SCRIPTS-BUCKET =“de-c4w4a2-[ACCOUNT ID]-us-east-1-scripts”
- GLUE-EXECUTION-ROLE=“arn:aws:iam::[ACCOUNT ID]:role/de-c4w4a2-glue-role”