coder@6a00eee92bf3:~/project$ scripts/setup.sh
Terraform variables have been set
upload: terraform/assets/glue_job/de-c1w4-etl-job.py to s3://de-c1w4-905418484298-us-east-1-scripts/de-c1w4-etl-job.py
Glue script has been set in the bucket
Setup completed successfully. All environment variables and Terraform backend configurations have been set.
coder@6a00eee92bf3:~/project$ cd terraform
coder@6a00eee92bf3:~/project/terraform$ terraform init
Initializing the backend…
Initializing modules…
Initializing provider plugins…
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using previously-installed hashicorp/aws v5.98.0
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running “terraform plan” to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
coder@6a00eee92bf3:~/project/terraform$ terraform plan
module.etl.data.aws_security_group.db_sg: Reading…
module.etl.data.aws_subnet.public_a: Reading…
module.etl.data.aws_iam_policy_document.glue_base_policy: Reading…
module.etl.data.aws_iam_policy_document.glue_access_policy: Reading…
module.etl.data.aws_caller_identity.current: Reading…
module.etl.aws_glue_catalog_database.ml_database: Refreshing state… [id=905418484298:de-c1w4-ml-db]
module.etl.data.aws_iam_policy_document.glue_base_policy: Read complete after 0s [id=3940084333]
module.etl.data.aws_iam_policy_document.glue_access_policy: Read complete after 0s [id=2526717434]
module.etl.aws_iam_role.glue_role: Refreshing state… [id=de-c1w4-glue-role]
module.etl.data.aws_caller_identity.current: Read complete after 0s [id=905418484298]
module.etl.data.aws_security_group.db_sg: Read complete after 0s [id=sg-0b143884d09f45a8c]
module.etl.data.aws_subnet.public_a: Read complete after 0s [id=subnet-045def8bc9ff8ba28]
module.etl.aws_glue_connection.rds_connection: Refreshing state… [id=905418484298:de-c1w4-rds-connection]
module.etl.aws_iam_role_policy.task_role_policy: Refreshing state… [id=de-c1w4-glue-role:de-c1w4-glue-role-policy]
module.etl.aws_glue_crawler.s3_crawler: Refreshing state… [id=de-c1w4-ml-db-crawler]
module.etl.aws_glue_job.etl_job: Refreshing state… [id=de-c1w4-etl-job]
No changes. Your infrastructure matches the configuration.
Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.
coder@6a00eee92bf3:~/project/terraform$ terraform apply
module.etl.data.aws_subnet.public_a: Reading…
module.etl.data.aws_security_group.db_sg: Reading…
module.etl.data.aws_caller_identity.current: Reading…
module.etl.data.aws_iam_policy_document.glue_access_policy: Reading…
module.etl.data.aws_iam_policy_document.glue_base_policy: Reading…
module.etl.aws_glue_catalog_database.ml_database: Refreshing state… [id=905418484298:de-c1w4-ml-db]
module.etl.data.aws_iam_policy_document.glue_access_policy: Read complete after 0s [id=2526717434]
module.etl.data.aws_iam_policy_document.glue_base_policy: Read complete after 0s [id=3940084333]
module.etl.aws_iam_role.glue_role: Refreshing state… [id=de-c1w4-glue-role]
module.etl.data.aws_caller_identity.current: Read complete after 0s [id=905418484298]
module.etl.data.aws_subnet.public_a: Read complete after 0s [id=subnet-045def8bc9ff8ba28]
module.etl.data.aws_security_group.db_sg: Read complete after 0s [id=sg-0b143884d09f45a8c]
module.etl.aws_glue_connection.rds_connection: Refreshing state… [id=905418484298:de-c1w4-rds-connection]
module.etl.aws_iam_role_policy.task_role_policy: Refreshing state… [id=de-c1w4-glue-role:de-c1w4-glue-role-policy]
module.etl.aws_glue_crawler.s3_crawler: Refreshing state… [id=de-c1w4-ml-db-crawler]
module.etl.aws_glue_job.etl_job: Refreshing state… [id=de-c1w4-etl-job]
No changes. Your infrastructure matches the configuration.
Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Outputs:
data_lake_bucket_id = “de-c1w4-905418484298-us-east-1-datalake”
scripts_bucket_id = “de-c1w4-905418484298-us-east-1-scripts”
coder@6a00eee92bf3:~/project/terraform$ aws glue start-job-run --job-name de-c1w4-etl-job | jq -r ‘.JobRunId’
jr_81a2e2e3f1fa40c862995dde7b40092b3f8e12354aa297c86f91d61d5d34a6ca
coder@6a00eee92bf3:~/project/terraform$ aws glue get-job-run --job-name de-c1w4-etl-job --run-jr_81a2e2e3f1fa40c862995dde7b40092b3f8e12354aa297c86f91d61d5d34a6ca --output text --query “JobRun.JobRunState”
usage: aws [options] [ …] [parameters]
To see help text, you can run:
aws help
aws help
aws help
aws: error: the following arguments are required: --run-id
coder@6a00eee92bf3:~/project/terraform$ aws glue get-job-run --job-name de-c1w4-etl-job --run-id jr_81a2e2e3f1fa40c862995dde7b40092b3f8e12354aa297c86f91d61d5d34a6ca --output text --query “JobRun.JobRunState”
FAILED
coder@6a00eee92bf3:~/project/terraform$
I don’t understand why its failing