How to deal with Terraform prompting for variable input during terraform plan?

[Leb12]

Hi everyone,
I’m running Terraform in my workspace, and I encounter an issue where Terraform prompts for the value of a variable during the terraform plan execution. Here’s the sequence of commands I executed:
bash

abc@2b38ea732896:~/workspace$ cd terraform
abc@2b38ea732896:~/workspace/terraform$ terraform init

The initialization completes successfully:
code

Terraform has been successfully initialized!

However, when I run terraform plan, it asks me for the value of var.catalog_database like this:
code

var.catalog_database  
  Curated DB name  
  Enter a value:

I’ve already defined some variables in my configuration, and I’m not sure why Terraform is asking for this input interactively.
Context :
I’m using Terraform modules and variables.
I have a variables.tf file where catalog_database is declared.
I may have missed adding a default value or passing the variable via terraform.tfvars or the CLI.

Questions:

1. How can I avoid this interactive prompt?
2. What’s the best way to supply the value for var.catalog_database?
   Should I define it in a terraform.tfvars file, environment variables, or CLI arguments?
3. Are there any common pitfalls with variable management in Terraform that I might be missing?

I appreciate any advice on how to streamline this process!

Additional Concern: How to Save Successfully Run Glue Jobs?

I’ve successfully run the following Glue jobs:

glue_api_users_extract_job
glue_sessions_users_extract_job
glue_rds_extract_job

However, I often have to restart everything from scratch, which is time-consuming and difficult. I want to know:

1. How can I save the progress or output of these successfully run Glue jobs?
2. Is there a way to easily resume from where I left off, rather than starting over each time?

Any advice or best practices for managing this process would be greatly appreciated!

[Leb12]

Also, I’m working on a Terraform configuration for an AWS Glue job, and I need to set the --ingest_date dynamically to reflect the current server date in Pacific Time (UTC-7). Here’s what I’ve done so far:
code

 Replace the placeholder <PACIFIC-TIME-CURRENT-DATE>
"--ingest_date" = "${formatdate("yyyy-MM-dd", timeadd(timestamp(), "-7h"))}"

My Thought Process:

1. timestamp() gives me the current server time in UTC.
2. timeadd(timestamp(), "-7h") adjusts the time back by 7 hours to account for Pacific Time (UTC-7).
3. formatdate("yyyy-MM-dd", ...) formats the adjusted timestamp to the required yyyy-MM-dd format.

Questions on --ingest_date:

1. Does this approach correctly handle Pacific Time (UTC-7) year-round?
2. How does it behave during Daylight Saving Time (PDT vs. PST or EST)? Will I need further adjustments for accurate results?

[Amir_Zare]

Hello @Leb12
The terraform variables in this lab are defined in the scripts/setup_jupyter_env.sh file. So when you run source scripts/setup_jupyter_env.sh in step 6 of Vocareum instructions, these variables are initiated. The reason you are being prompted for terraform is that you probably had run the mentioned command in another terminal window, so the terminal you are running terraform plan command in doesn’t have access to this variables. I suggest using a single terminal instance all around.
About the Glue jobs, I’m afraid there isn’t a way of saving the progress. If you start a new lab session, you need to deploy the jobs using terraform and run them again.
Finally, about the --ingest-date variable, please search google for the current Pacific time and replace the placeholder. For instance, currently the variable should be set to 2024-12-18.

[Leb12]

Thanks! Please, l

look at that screenshot and give me your feedback. After password, I got: atisfied: types-python-dateutil>=2.8.10 in ./jupyterlab-venv/lib/python3.10/site-packages (from arrow>=0.15.0->isoduration->jsonschema[format-nongpl]>=4.18.0->jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->jupyterlab==4.0.9->-r /config/workspace/scripts/requirements.txt (line 8)) (2.9.0.20241206)
Requirements installed successfully
Total Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 12 100 12 0 0 88 0 --:–:-- --:–:-- --:–:-- 88
Jupyter is running at: http://35.171.41.59:8888/lab?token=c86dc93183ecc6d37a225e892714857f5f36dceddb65b58a
(jupyterlab-venv) abc@3b7bc91ebad2:~/workspace$

[Amir_Zare]

Hello again @Leb12
I don’t think the screenshot is from the same lab we were talking about in the first place. In module two labs, we don’t have setup.sh but setup_jupyter_env.sh.
However, you can look for lines like the following in the setup files.

Environment variables starting with TF_VAR_ are used by terraform as terraform variables. So, after you run these files in your terminal, terraform variables are given values, and you can continue with running terraform init, terraform plan, and terraform apply commands.

[Leb12]

Thanks for your support, @Amir_Zare . In the prompt:

'var.catalog_database
Curated DB name
Enter a value: ‘’

What value should I provide?
Any help or guidance is appreciated.
Thanks!

[Leb12]

default_arguments = {
“–enable-job-insights” = “true”
“–job-language” = “python”
# Set "--catalog_database" to aws_glue_catalog_database.transform_db.name
“–catalog_database” = aws_glue_catalog_database.transform_db.name
# Set “–ingest_date” to the server’s current date in Pacific Time (UTC-7), in “yyyy-mm-dd” format.
# (replace the placeholder <PACIFIC-TIME-CURRENT-DATE>)
#“–ingest_date” = “2024-12-20”
“–ingest_date” = “2024-12-20”

The issue remains. I got this error message: ValueError: time data ‘’ does not match format ‘%Y-%m-%d’.

[Amir_Zare]

Hello @Leb12
If you run the setup_jupyter_env.sh file, you won’t need to enter values for the variables after running the terraform commands. This is the standard procedure designed for these labs.
About your second issue, you are missing some of the placeholders. Please, make sure that you replace <PACIFIC-TIME-CURRENT-DATE> with its value in all the designated places. Moreover, please make sure that you save your changes to the files before running the terraform commands.