Terraform init fails to create GCS bucket for backend

[lxkm]

Terraform Version

Terraform v0.11.7

Terraform Configuration Files

terraform {
    required_version = "0.11.7"

    backend "gcs" {
        project = "my-awesome-project"
        region  = "asia-northeast1"
        bucket  = "tf-state-myawesomeproject-bucket"
        prefix  = "testing/state"
    }
}

variable project_name {
    type        = "string"
    default     = "my-awesome-project"
}

provider google {
    project = "${var.project_name}"
    region  = "asia-northeast1"
    zone    = "asia-northeast1-a"
}

Debug Output

https://gist.github.com/lxkm/eeb076694df7108f67af874212a48829

Crash Output

N/A

Expected Behavior

Terraform should have created a new bucket in my project with name tf-state-myawesomeproject-bucket, containing the initial / basic state file.

Actual Behavior

Terraform claims to have configured the backend, but errors out saying the backend state file can't be found. On inspecting the GCP console, no new buckets have been created.

Steps to Reproduce

terraform init

Additional Context

I have reduced this down to what I think is the minimum working example and get the same response as in my larger projects.

Maybe of note is that I am using Application Default Credentials rather than supplying a credentials file. I have not yet tested if there is any difference between the two approaches.

If I disable/comment-out the backend configuration, I can create all of my resources without issue.

Also, if I manually create the bucket through either gsutil mb gs:https://tf-state-myawesomeproject-bucket/ or through the web console before attempting to initialise, the terraform init command proceeds as expected and creates the default state file in my bucket.

Pre-submit edit, trying to do my due-diligence :) ... looking into the master branch code base's tests for the GCS backend, it appears that most (maybe all?) tests manually create a bucket prior to running the code under test, via the setupBackend method. I'm fairly new to Go so take this with a pinch of salt (I expect I'm missing something) but I can't seem to see anything in the related non-test code to suggest it will ever attempt to create a new bucket if one doesn't already exist.

This seems to run counter to the documentation for the GCS backend and hence counter to the expectation I had. Snipped + highlighted the relevant section implying bucket can/will be automatically created (if not already existing):

Configuration variables
...
project / GOOGLE_PROJECT - (Optional) The project ID to which the bucket belongs. This is only used when creating a new bucket during initialization. Since buckets have globally unique names, the project ID is not required to access the bucket during normal operation.

References

None that I could find easily.

[jbardin]

Hi @lxkm,

Sorry you were confused by this, but it is expected behavior. Backend configurations are not resources themselves, and not directly managed by terraform. None of the remote state backends will create resources during the init process.

When using a backend that requires some pre-existing resources for operation, it's not uncommon for users to have a preliminary "bootstrap" configuration to create the necessary infrastructure.

[lxkm]

Interesting. Thanks for the speedy response!

I would like to suggest an update to the documentation to clarify that the bucket must exist prior to initialisation then, and changing the wording of the project argument since it isn't actually going to create one for us.

Similarly, I would think that the possible addition of a feature to create the bucket, if not already existing, could be very useful. Perhaps an additional config flag create_bucket, defaulting to false to keep current behaviour. At least in my usage case where we want to transition a large number of scripts to using GCS state storage this would be a very nice convenience, rather than needing to maintain potentially many pre-init scripts just to create state storage buckets.

While I think the former point could still be covered under this issue ticket(?), do you think the latter is a reasonable feature request? I could see this being useful for both the azurerm and s3 backends as well.

[jbardin]

@lxkm,

Thanks, after re-reading it, you're definitely correct that the documentation for project is misleading.
I'll leave this open to until we get the documentation updated.

As for the feature, it's been decided that terraform should not implicitly create resources that it cannot manage, and all backends should have the same overall behavior.

The prior discussions have generally concluded that while it could be convenient for first time users, any somewhat complex configuration and organization will require some sort of management for the state resources, and there will be many other pieces that need to be bootstrapped as well.

[lxkm]

WRT feature: sounds sensible :)

I'll leave this here then with another thank you for the fast responses! Loving my experience with Terraform so far.

[nhooyr]

According to https://www.terraform.io/docs/backends/types/gcs.html

project / GOOGLE_PROJECT - (Optional) The project ID to which the bucket belongs. This is only used when creating a new bucket during initialization. Since buckets have globally unique names, the project ID is not required to access the bucket during normal operation.

Docs say it creates the bucket if it doesn't already exist.

edit: my bad, this is already mentioned

[johntheo]

Hi, I fixed that in my project, putting the credentials variable. The value points to a json credential in the root path

[nikolay]

It's suboptimal that we all have to create a separate Terraform plan and store its state in a file just to create the bucket. I recognize the challenge that creating the bucket requires a lot of parameters, but it would be nice if there's a way to, let's say, embed resources underneath a backend. Also, it's sad that we cannot pass variables to a backend - I still get the challenge, but now we cannot keep our plans DRY.

[mildwonkey]

I am going to close this issue due to inactivity. The documentation has been updated since this was opened: https://www.terraform.io/docs/backends/types/gcs.html#configuration-variables

If there is still a question, I recommend the the community forum, where there are far more people available to help. If there is a bug or you would like to make a feature request, please open a new issue and fill out the template.
Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.