config: validate backend configuration can't contain interpolations

[mitchellh]

Validate that backend configuration doesn't contain interpolations.

[nanoz]

Do you think basic variables provided using a tfvars file could be interpolated at some point ?

It would be extremely useful to pipeline code through different environments (ref below) without requiring an external logic for -backend-config.

https://medium.com/@kief/https-medium-com-kief-using-pipelines-to-manage-environments-with-infrastructure-as-code-b37285a1cbf5#.ve3bqwrgw

[mitchellh]

@nanoz Yes its possible.

I find it important to start with a smaller surface area: you can always add things but you can't really take things away. For the variable issue, I'm concerned with restricting the TYPES of interpolations that are allowed in different areas of a Terraform config. My fear is that that leads to a future where a user is unclear what kind of interpolation they can use. You can claim (and I do) that we're already there with users unclear when they can/can't use interpolations. I view partial support as exacerbating that problem. I don't think there is an easy solution to it but without adding effectively unremovable features to the config, I'd rather ship with a more limited solution, consider this, and make it support more in the future.

All that is to say: I want to head into a future where almost all of Terraform is parametrizable, I really do. But I want to do so with some more thought first.

[apparentlymart]

@mitchellh I saw some initial work towards adding a first-class "environment" concept to Terraform a few weeks ago, but didn't yet see enough of it to have a good mental model for how that construct will work. My hope was that it would include a means to either specify a separate remote configuration per environment within the config or to use the environment name as an interpolation within a single configuration, both of which would go some way to address the issue raised here.

Currently at work we have a systematic way of constructing a remote state configuration given two pieces of information: a codebase name and an environment name. This allows us to centralize this convention in a single place, which for us is the wrapper script we use to run Terraform from a continuous integration/deployment tool. At the moment I find myself unsure how our current approach will adapt to the new architecture; if possible I'd like to avoid needing to hand-write the remote config for each environment+codebase pair into each repo, but it seems like for now that's going to be the recommended strategy. Is that right?

I'm sure I can make something work with generating on the fly an additional config file that gives the remote config before running terraform init, but I'd love to understand the design intent here so that I don't end up implementing something that goes against the grain of the intended workflow.

(I realize this is a bit of a big question to shove on the end of a closed issue. If there's documentation coming that will answer this question I'm happy to just wait and get my answers there... mainly just wanted to share the use-case.)

[mitchellh]

@apparentlymart Scripting terraform init is expected/recommended. The backend config file is HCL so you can just as easily generate JSON. And the backend config only needs to merge with what is in the actual TF config to make a full backend config, it doesn't need to specify everything.

[apparentlymart]

@mitchellh personally I agree that this is best kept for a later release, since this extra complexity should be approached with caution. In the long run it feels pragmatic to have three different possibilities for any configuration attribute:

• No expressions are allowed at all
• Only non-computed expressions are allowed
• Any valid expression is allowed.

The second case here is the awkward one, but when 0.9 ships we'll see with the count meta-attribute whether this situation is intuitive for users. I expect it'll take some iteration on UX and terminology to help users understand what really is valid in that scenario, and why.

I think the trick will be guiding users up the complexity curve so that they learn supporting concepts along the way. For example, a user should have learned what computed means before they are introduced to data sources, so that they can understand the ways in which a data source behaves differently than a resource.

Based on earlier questions on this subject it's felt to me like the term "computed" isn't resonating with all users, so it might be worth spending some time researching alternative terminology here that might resonate better.

With that said, none of the stuff in this comment is intended to disagree with this PR... just musing about possible future work. For now, dynamically generating a configuration file containing the backend config should work for anyone who was previously scripting calls to terraform remote config ....

[michalmedvecky]

@apparentlymart can you please give me some hint on what do you mean by 'generate one-off configuration file that contains the desired settings'?

I am experienced just with the local terraform.tfstate file (and the -state and -var-file parameters), but now I'm trying to migrate the workflow from locally-run terraform apply to CD pipeline. As this will run in an ephemeral environment (docker container) I surely need to hand off the state handling to a different backend.

From my little experience, when I configure the backend as mentioned in #12200 and run terraform init, it creates some .terraform/terraform.tfstate file with a reference to the swift container and offers me to copy the current terraform.tfstate there. Do I understand correctly that you're suggesting to generate this file automatically (out of terraform) every run of CD pipeline and refer to the correct container? If yes, what about lineage and hash parameters that look like some reference numbers which are absolutely unclear to me?

[apparentlymart]

The configuration I mean to generate is the terraform block that specifies the backend:

terraform {
  backend "foo" {
    # ...
  }
}

From 0.9 the settings that were formerly passed to terraform remote config must now be provided via configuration, so anyone that was scripting Terraform to mechanically configure remote state must replace the terraform remote config command with a call to generate a configuration file like above and then run terraform init.

[mitchellh]

@apparentlymart They don't have to specifically, they just need to generate terraform { backend "type" {}} and then pass in a -backend-config file that is just JSON K/V.

[nanoz]

I think #12182 (Environments) could actually mitigate my concern. Needs testing :)

[ozbillwang]

I will be happy if terraform init can support this

/bin/terraform init -backend=s3 -backend-config="bucket=com_mydomain_terraform_prod" -backend-config="key=iam/terraform.tfstate" -backend-config="region=us-east-1"

With that, I can avoid the interpolations limits.

[ozbillwang]

Vote down for this PR. Whatever, I hope to get interpolations support back. I like it work as old terraform remove config

And hope this type of PR can be reviewed not only one reviewer. It generates incompatible problem a lot.

[mitchellh]

@SydOps We never supported interpolations in backend configurations (new in 0.9) and this was protecting people from using a feature we never supported. :) As I said above, we'd love to extend this support in the future, but we didn't take anything away to give "back": it never worked!

In 0.9.0 we added more places you can use interpolations, we didn't roll any back at all, so I believe there is some confusion happening!

[mitchellh]

Just noticed you mentioned remote config. I think adding the ability to specify configurations on the CLI is something we can add back. Those aren't interpolations though. :)

[ozbillwang]

Thanks, @mitchellh

Since I can't make this work

terraform {
  backend "s3" {
   bucket = “terraform-xxxxxx"
   key = "${var.environment}/terraform.tfstate"
   encrypt = "true"
   kms_key_id = “xxx"
   region = “xxx"
 }
}

Then I prefer to feed the environment variable to terraform init directly via CLI. Otherwise, I have to prepare a backend configuration file every time.

[mitchellh]

Right, I think we'll bring back the CLI based config too, but you can also use partial configurations, documented here: https://www.terraform.io/docs/backends/config.html (still requires a file but makes it more parameterizable)

[ozbillwang]

Thanks, @mitchellh

Yes, that's what I did currently.

In main.tf:

terraform {
  backend "s3" {
    encrypt = "true"
    region = "xxx"
  }
}

Then prepare a backend.conf file as

bucket = "terraform-xxxxxx"
key = "dev1/terraform.tfstate"
kms_key_id = "xxx"

This backend.conf has to be generated every time in a wrap script before I run terraform init -backend-config=backend.conf

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.