C1W4 Lab: Missing Master Password Definition in Terraform Assignment

Course Q&A Data Engineering Introduction to Data Engineering
,
1

Hello,

I am encountering an issue in the Terraform assignment related to setting up an RDS instance. While there is a defined variable for the master username in variables.tf, there is no corresponding definition or resource for the master password. This omission is preventing me from completing the assignment successfully.

While the command terraform output vector_db_master_username outputs the default username postgres, the command terraform output vector_db_master_password fails, instead the output reads Error: Output "vector_db_master_password" not found.

So, I have confirmed that the username variable is present, but the password is not defined or generated anywhere in the configuration.

Please help me to fix this and finish the course.

2

Hello @maxhare,

I think you can find master_password in terraform/modules/vector-db in the rds.tf file when you create master_db instance (it is random and no need to change anything). There is also the master_username which is in the variables.tf as well (it is postgres by default).

It seems there was an error while creating the vector database could you try in a new lab session. I suggest you start fresh, delete the terraform folder, click help and get latest version
You should be able to start with a new folder with new files main.tf andoutput.tf, finally use reboot.

If you run in any errors, you could run terraform apply -no-color 2> errors.txt. Find which resources in AWS console already exist and delete them manually. Thank you

3

Hi georgios,

I tried the solution you mentioned above (deleting, updating, rebooting, trying to delete stuff) but am still facing the error… and cannot retrieve the master password… Can someone help me to reset my entire instance and clear everything in my AWS account for this C1W4 lab?? my user is zilingh… and my labid is wbmlaspwcepe… It’s been preventing my from completing the lab for a few weeks now. Thanks so much.

4

@Georgios

5

Hello @zilingh,

You could ask for a lab refresh, however the lab does a soft reset after a new lab session by itself(you will notice the difference since it takes a bit longer to load).

If you made any changes in the lab files you can delete those files and get latest version should refresh those files. What is the error from terraform output vector_db_master_password or you get no output. Hope it helps

6

Hi georgios,

This is the error msg i get:
coder@07dd16ab67e3:~/project/terraform$ terraform output vector_db_master_username
terraform output vector_db_master_password
“postgres”
╷
│ Error: Output “vector_db_master_password” not found
│
│ The output variable requested could not be found in the state file. If you recently added this to your configuration, be sure to run terraform apply, since the state won’t be updated with new output variables until that command is run.

Prior to that I deleted the terraform folder of the lab, and then pressed question mark and update, Reboot, and ran all of the steps again… and here are the errors i am getting mostly that the connection already exists…

Step 1.11 of tutorial notifications:
1.11

╷
│ Error: creating Glue Catalog Database (de-c1w4-ml-db): operation error Glue: CreateDatabase, https response error StatusCode: 400, RequestID: cabdc80d-4e35-4c50-9f57-96487cbbaefd, AlreadyExistsException: Database already exists.
│
│ with module.etl.aws_glue_catalog_database.ml_database,
│ on modules/etl/glue.tf line 1, in resource “aws_glue_catalog_database” “ml_database”:
│ 1: resource “aws_glue_catalog_database” “ml_database” {
│
╵
╷
│ Error: creating Glue Connection (de-c1w4-rds-connection): operation error Glue: CreateConnection, https response error StatusCode: 400, RequestID: c59d11e2-87c7-4e42-9b88-3feb966d4294, AlreadyExistsException: Connection already exists.
│
│ with module.etl.aws_glue_connection.rds_connection,
│ on modules/etl/glue.tf line 6, in resource “aws_glue_connection” “rds_connection”:
│ 6: resource “aws_glue_connection” “rds_connection” {
│
╵
╷
│ Error: creating IAM Role (de-c1w4-glue-role): operation error IAM: CreateRole, https response error StatusCode: 409, RequestID: a1d36d61-a543-466f-828a-85771c9b6f1e, EntityAlreadyExists: Role with name de-c1w4-glue-role already exists.
│
│ with module.etl.aws_iam_role.glue_role,
│ on modules/etl/iam_roles.tf line 1, in resource “aws_iam_role” “glue_role”:
│ 1: resource “aws_iam_role” “glue_role” {

Step 2.4 of Tutorial notifications:
2.4

module.etl.aws_glue_connection.rds_connection: Creating…
module.etl.aws_glue_catalog_database.ml_database: Creating…
module.etl.aws_iam_role.glue_role: Creating…
╷
│ Error: creating Glue Catalog Database (de-c1w4-ml-db): operation error Glue: CreateDatabase, https response error StatusCode: 400, RequestID: 51d7483b-8508-4f53-8214-ee4a27791718, AlreadyExistsException: Database already exists.
│
│ with module.etl.aws_glue_catalog_database.ml_database,
│ on modules/etl/glue.tf line 1, in resource “aws_glue_catalog_database” “ml_database”:
│ 1: resource “aws_glue_catalog_database” “ml_database” {
│
╵
╷
│ Error: creating Glue Connection (de-c1w4-rds-connection): operation error Glue: CreateConnection, https response error StatusCode: 400, RequestID: a4cc50ca-1043-4067-9f8e-356f62497821, AlreadyExistsException: Connection already exists.
│
│ with module.etl.aws_glue_connection.rds_connection,
│ on modules/etl/glue.tf line 6, in resource “aws_glue_connection” “rds_connection”:
│ 6: resource “aws_glue_connection” “rds_connection” {
│
╵
╷
│ Error: creating IAM Role (de-c1w4-glue-role): operation error IAM: CreateRole, https response error StatusCode: 409, RequestID: 244ec85b-60c4-41b5-81db-b9eb73b35c4a, EntityAlreadyExists: Role with name de-c1w4-glue-role already exists.
│
│ with module.etl.aws_iam_role.glue_role,
│ on modules/etl/iam_roles.tf line 1, in resource “aws_iam_role” “glue_role”:
│ 1: resource “aws_iam_role” “glue_role” {

7

I will try to wait 2 hours and try the lab again… to see if it helps… but i think the problem is that some of the database stuff already exists on s3 because i created it previously? even after two hours the s3 folders might still remain? I am not able to delete the s3 folders

8

Hello @zilingh,

Could you check in AWS console and delete them manually:

  1. de-c1w4-ml-db should be in AWS Glue> databases:

C1W4_glue_catalog_ml_db
C1W4_glue_catalog_ml_db1894×462 111 KB

  1. de-c1w4-rds-connection in AWS Glue > connections

C1W4_rds_connection
C1W4_rds_connection1893×696 190 KB

  1. For de-c1w4-glue-role go to IAM and then Roles:

C1W4_glue_role_iam
C1W4_glue_role_iam1781×717 212 KB

  1. If the issue continues you could wait for a new lab session cleanup (2 hours)
9

hi georgios, i encountered this error when i tried to delete and empty the ml artifacts folder. Would you recommend i wait two hours instead and see if the ml artifacts folder on s3 goes away

image
image1736×730 120 KB

?

10

also i deleted the others as you suggested. thank you for the screenshots!!

11

Hello @zilingh,

You could try terraform apply after you deleted them. No need to delete the s3 folder, you should be able to create the vector db. Hope it helps

12

i think i made it worse… i just saw your message that there is no need to delete the s3 folder a little too late…
│ Error: creating Glue Crawler (de-c1w4-ml-db-crawler): operation error Glue: CreateCrawler, https response error StatusCode: 400, RequestID: bc0ece05-2307-4fab-99b9-7a957d94eb09, InvalidInputException: S3 bucket de-c1w4-894662625660-us-east-1-datalake does not exist.
│
│ with module.etl.aws_glue_crawler.s3_crawler,
│ on modules/etl/glue.tf line 22, in resource “aws_glue_crawler” “s3_crawler”:
│ 22: resource “aws_glue_crawler” “s3_crawler” {
│

13

Hello,

You shoud find the de-c1w4-ml-db-crawler in AWS glue> crawlers. Finally run terraform apply again. Hope it helps

14

Hello @zilingh,

There should be a de-c1w4-ml-db-crawler in AWS glue (crawlers below connections) and delete it. Finally try terraform apply again to create the vector db

15

hi georgios, yes the crawler is there! however, the s3 data is not getting created from the aws glue job now because i mistakenly deleted the datalake bucket in s3 previously thinking it might help to reset things… S3 bucket de-c1w4-894662625660-us-east-1-datalake does not exist.

16

should i try to create an s3 bucket with that name?

17

@zilingh I don’t think you have permissions to create the s3bucket. You could try to run the AWS glue job again to try to create the files. If that fails you need to wait for a new S3 data lake bucket after a new lab session. Hope it helps