Programming Assignment: Assignment 5: Capstone Project Part 2 - Data Quality and Orchestration and Visualization

Hi.

I am facing another issue on 3.2.6, trying to execute

aws s3 cp $HOME/project/dbt_modeling/models/bi_views s3:///dbt_project/dbt_modeling/models/bi_views --recursive
aws s3 cp $HOME/project/dbt_modeling/dbt_project.yml s3:///dbt_project/dbt_modeling/dbt_project.yml

the terminal crash abruptly. I uploaded those files and folder manually on S3 bucket but even so the grade still not detecting it…

Hello @Jailson_Evora,

Could you check in the commands to copy the files you include the DBTBucket.:

aws s3 cp $HOME/project/dbt_modeling/models/bi_views s3://<DBTBucket>/dbt_project/dbt_modeling/models/bi_views --recursive
aws s3 cp $HOME/project/dbt_modeling/dbt_project.yml s3://<DBTBucket>/dbt_project/dbt_modeling/dbt_project.yml

Also I am not sure that uploading the files manually, even in the correct path without typos would work. Try to follow the instructions and successfully run the commands. Thank you

Hi, Georgios.

I am not able even to open the Go To AWS Console, neither executing the script. :neutral_face:

Regards.

Hello @Jailson_Evora,

Could you try rebooting might help:

  1. Click help on top right
  2. Click reboot
  3. If that doesn’t help your lab might exceeded limit and needs a lab refresh. Please submit this form, thank you

Thank you.

…didn’t work, I will fill the form again. It is been quite frustrating to finish this lab even following the tutorial. I’ve lost count of how many times I’ve filled out the form.

Regards.

Hello @Jailson_Evora,

Sorry for the inconvenience, I understand you spent a lot of time watching the tutorials and doing all the parts even running the DAG’s successfully. I hope the lab refresh won’t take a lot of time to complete. Perhaps @hawraa.salami could help with your issue so you can finish you assignment. Thank you

1 Like

I am also having an issue with this lab. The API Endpoint is not accessible and is refusing connection when I open in my browser: http://ec2-18-211-181-191.compute-1.amazonaws.com/.
This causes the extract jobs using the API endpoint to fail, so I can’t proceed. I did check that the VPC attached to the EC2 instance associated with the API endpoint has network connectivity and the route table is setup to allow all inbound traffic. Not sure what else to try on my end. Do I need to fill out a form?

Hello @jmacclure,

Thanks for submitting the form, hopefully that fix your exctract jobs. You could try in a new lab session since your API endpoint is different so you don’t need to wait for a lab refresh. What kind of error you get in your AWS glue job, just in case you pasted the API in glue.tf and didn’t include the /users or /sessions or you get a ConnectTimeout. Thanks:

Ok, thanks for all the help on the capstone, I was able to get everything to work and submitted successfully. :slight_smile:

1 Like

Hi, @Georgios.

I am still waiting for the lab refresh. Yesterday I receive an email that the issue was resolved but after accessing the lab …

I have replied the email with this issue and did not receive any feedback…

I would appreciate your help.

Regards.

Hello @Jailson_Evora,

I could reproduce the error with a low bandwidth (check background dowloadings)There was a learner I remember with a similar issue, post. Could you try updating to new a notebook:

  1. Issue I reproduced

  2. Rename your notebook to C4_W4_Assignment_2_copy.ipynb:

  1. Use get latest version from help:

  1. You should now have two files, one with a fresh notebook( you might need to reboot afterwards):

I don’t think the email can be replied back from labsupport at Deeplearning.ai. You could send any feedback here. Hope it helps

Hi, @Georgios thank you.

It works partially. I am able to open the file but when I try to open the AWS console nothing happen.

Hello @Jailson_Evora,

That is strange, I couldn’t reproduce the issue and even the airflow is working. Does the lab takes any longer to load or it works like before the lab refresh. Sorry for the inconvenience but If you have the same issue with source scripts/setup.sh again you might need to submit the form again. Hope it helps:

1 Like

Hi, @Georgios

Again even after another lab refresh, another issue that you are already aware of. I think if the lab is not stable you should drop out this for the assignment because it is been hurting the quality of the course overall. This situation it is been so unpleasant to keep fill the form over and over with the lab refresh being doing partially, letting some fixing to be done.

Error: creating Glue Connection (de-c4w4a2-connection-rds): operation error Glue: CreateConnection, https response error StatusCode: 400, RequestID: 471a2902-9632-41f2-9913-36356de4e3b8, AlreadyExistsException: Connection already exists.

with module.extract_job.aws_glue_connection.rds_connection,
on modules/extract_job/glue.tf line 2, in resource “aws_glue_connection” “rds_connection”:
2: resource “aws_glue_connection” “rds_connection” {

Error: creating IAM Role (de-c4w4a2-glue-role): operation error IAM: CreateRole, https response error StatusCode: 409, RequestID: efc29718-ef66-4898-aa2d-6872a780e50a, EntityAlreadyExists: Role with name de-c4w4a2-glue-role already exists.

with module.extract_job.aws_iam_role.glue_role,
on modules/extract_job/iam.tf line 1, in resource “aws_iam_role” “glue_role”:
1: resource “aws_iam_role” “glue_role” {

Hello @Jailson_Evora,

As a tester and mentor of the course I didn’t have any issues completing the assignments by myself. The lab support should be able to reset your lab for you but your error is because you used terraform apply when the lab server was rebooted without destroying terraform before. You need to delete them manually using aws commands. For example, with an existing connection, you can use ‘aws glue delete-connection --connection name ’. Thank you

  1. Go to AWS console> AWS glue > connections and delete it:
  2. For de-c4w4a2-glue-role> IAM> roles
1 Like

Reading through this thread and really empathizing with the situation. I spent two hours trying to get past the first step of the lab—initializing Terraform and applying changes. I just submitted this form to request a lab reset: Combined Data Engineering Lab Issue Report.

Whenever I run terraform init, the terminal crashes with the message:
The terminal process “/usr/bin/bash” terminated with exit code: 1.
Even when I follow the alternative instruction—
terraform apply -no-color 2> errors.txt—the same crash happens.
I couldn’t get past this first part of the lab. It’s very frustrating.

Hello @Lucas_Kawano_Mori,

Since you have all availabe resources and can run source scripts/setup.sh I am not sure why you request a lab refresh on the first place. It explains in Course 1 Week 2 [IMPORTANT] Guidelines before you start the labs in this course when you have Issues accessing your labs. Thanks for submitting the form because the terminal crashing on terraform init but I don’t think it will make a difference or help. A new lab session might fix that since terraform crashes from time to time. Hope it helps