GCP SA key creation with "new value: Root resource was present, but now absent" #17332

mr-tko · 2024-02-21T08:47:21Z

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.
If an issue is assigned to a user, that user is claiming responsibility for the issue.
Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally
to expedite investigation and resolution of this issue.

Terraform Version

terraform --version
Terraform v1.7.3 
on linux_amd64
+ provider registry.terraform.io/hashicorp/google v5.17.0
+ provider registry.terraform.io/hashicorp/time v0.10.0

Affected Resource(s)

google_service_account_key

Terraform Configuration

resource "time_rotating" "mykey_rotation" {
  rotation_minutes = 1
}

resource "google_service_account" "myaccount" {
  count = 15
  account_id   = "test-sa-key-gen-issue-${count.index}"
  display_name = "Testing SA key issue [${count.index}]"
  project = var.project_id
}

resource "google_service_account_key" "mykey" {
  count = 15
  service_account_id = google_service_account.myaccount[count.index].name

  keepers = {
    rotation_time = time_rotating.mykey_rotation.rotation_rfc3339
  }
}

#!/bin/bash

i=0
while [ $i -lt 30 ]
do
   echo $i
   i=`expr $i + 1`
   terraform apply -auto-approve
   sleep 90
done

Debug Output

This issue occurs randomly, so I won't be pasting the code here.
Just run the script and it will output error - for me it did at the first run:

│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to google_service_account_key.mykey[5], provider
│ "provider[\"registry.terraform.io/hashicorp/google\"]" produced an unexpected new value: Root object was present,
│ but now absent.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

Expected Behavior

SA key rotated without and error

Actual Behavior

Randomly an error occurs

provider "provider[\"registry.terraform.io/hashicorp/google\"]" produced an unexpected new value: Root object was present, but now absent.

Steps to reproduce

sh ./test_script.sh > multiple_sa_key_issue.log

Important Factoids

Error occurs randomly, possibly when you make a lot of operations on the SA or SA keys.

References

Definitely can relate to this issue: #13364

b/327622972

The text was updated successfully, but these errors were encountered:

ggtisc · 2024-02-29T17:05:03Z

It looks similar to the issue #13364 so maybe you can try adding more time between the google_service_account and google_service_account_key or try with a google_cloud_scheduler_job https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloud_scheduler_job

evanyounggenesys · 2024-03-06T17:25:52Z

Is the delay to account for eventual consistency on the Google Cloud Platform side? Our team has encountered this a couple of times but I noticed the resourceGoogleServiceAccountKeyCreate uses a tail call to resourceGoogleServiceAccountKeyRead. It seems like that tail call read could be the issue with the eventually consistency which ends up removing the key from our state file.

I think a delay in the provider would be needed in that case as adding a delay as suggested would be too late and not help. Perhaps better yet would to be remove the tail call resourceGoogleServiceAccountKeyRead entirely and let any delay or validation happen for whatever reads the value later.

evanyounggenesys · 2024-03-15T20:08:28Z

@edwardmedia Can you take a look at this issue considering your delay suggestions from #13364? It seems like something that needs to be addressed by the provider if my rudimentary understanding of Terraform and Go is correct.

mcallaghan-geotab · 2024-05-11T11:27:17Z

we are also experiencing this intermittently

when it fails, it fails in similar way as OP

Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to
│ module.foo.module.workload_identity["iat"].google_service_account.cluster_service_account[0],
│ provider "provider[\"[registry.terraform.io/hashicorp/google\](http:https://registry.terraform.io/hashicorp/google%5C)"]" produced an
│ unexpected new value: Root resource was present, but now absent.
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.

—
but then we retry— and in it says the resource already exists


Error: Error creating service account: googleapi: Error 409: Service account foo-iat already exists within project projects/geotab-foobar.
│ Details:
│ [
│   {
│     "@type": "[type.googleapis.com/google.rpc.ResourceInfo](http:https://type.googleapis.com/google.rpc.ResourceInfo)",
│     "resourceName": "projects/foobar/serviceAccounts/[email protected]"
│   }
│ ]
│ , alreadyExists
│ 
│   with module.foo.module.workload_identity["iat"].google_service_account.cluster_service_account[0],
│   on .terraform/modules/foo.workload_identity/modules/workload-identity/[main.tf](http:https://main.tf/) line 42, in resource "google_service_account" "cluster_service_account":
│   42: resource "google_service_account" "cluster_service_account" {
│

mr-tko · 2024-05-20T05:13:20Z

I've got new facts: it's happening for sure, when you delete old and create a new key for the service account.
As a proof, I can only attach a small portion of obfuscated logs (as it is happening constantly on a production env).
First log says that key was present in the state file:

module.XYZ.google_service_account_key.sa_key: Refreshing state... [id=projects/XYZ/serviceAccounts/[email protected]/keys/KEY_ID_WAS_HERE]

This log presents the plan portion and later on the error message:

module.XYZ.google_service_account_key.sa_key must be replaced
+/- resource "google_service_account_key" "sa_key" {
      ~ id                 = "projects/XYZ/serviceAccounts/[email protected]/keys/KEY_ID_WAS_HERE" -> (known after apply)
      ~ keepers            = {
          - "rotation_time" = "2024-05-XXTXX:XX:XXZ"
        } -> (known after apply) # forces replacement
      ~ name               = "projects/XYZ/serviceAccounts/[email protected]/keys/KEY_ID_WAS_HERE" -> (known after apply)
...
.
.
.
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to
│ module.XYZ.google_service_account_key.sa_key,
│ provider "provider[\"registry.terraform.io/hashicorp/google\"]" produced an
│ unexpected new value: Root resource was present, but now absent.
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.

mr-tko added the bug label Feb 21, 2024

github-actions bot added forward/review In review; remove label to forward service/iam-serviceaccount labels Feb 21, 2024

c2thorn self-assigned this Feb 21, 2024

ggtisc self-assigned this Feb 29, 2024

ggtisc added waiting-response and removed forward/review In review; remove label to forward labels Feb 29, 2024

github-actions bot removed the waiting-response label Feb 29, 2024

modular-magician added the forward/linked label Feb 29, 2024

roaks3 added the waiting-response label Feb 29, 2024

roaks3 unassigned c2thorn Feb 29, 2024

github-actions bot removed the waiting-response label Mar 6, 2024

ggtisc added the waiting-response label Mar 6, 2024

github-actions bot removed the waiting-response label Mar 15, 2024

LujieZ mentioned this issue Jun 25, 2024

Add a sleep to reduce the likelihood of eventual consistency issue GoogleCloudPlatform/magic-modules#11044

Merged

rileykarson closed this as completed in GoogleCloudPlatform/magic-modules#11044 Jun 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GCP SA key creation with "new value: Root resource was present, but now absent" #17332

GCP SA key creation with "new value: Root resource was present, but now absent" #17332

mr-tko commented Feb 21, 2024 •

edited by modular-magician

Loading

ggtisc commented Feb 29, 2024

evanyounggenesys commented Mar 6, 2024

evanyounggenesys commented Mar 15, 2024

mcallaghan-geotab commented May 11, 2024 •

edited

Loading

mr-tko commented May 20, 2024

GCP SA key creation with "new value: Root resource was present, but now absent" #17332

GCP SA key creation with "new value: Root resource was present, but now absent" #17332

Comments

mr-tko commented Feb 21, 2024 • edited by modular-magician Loading

Community Note

Terraform Version

Affected Resource(s)

Terraform Configuration

Debug Output

Expected Behavior

Actual Behavior

Steps to reproduce

Important Factoids

References

ggtisc commented Feb 29, 2024

evanyounggenesys commented Mar 6, 2024

evanyounggenesys commented Mar 15, 2024

mcallaghan-geotab commented May 11, 2024 • edited Loading

mr-tko commented May 20, 2024

mr-tko commented Feb 21, 2024 •

edited by modular-magician

Loading

mcallaghan-geotab commented May 11, 2024 •

edited

Loading