Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Terraform console creates a lock that does not go away before it's closed #23655

Open
mhvelplund opened this issue Dec 12, 2019 · 14 comments · May be fixed by #35308
Open

Terraform console creates a lock that does not go away before it's closed #23655

mhvelplund opened this issue Dec 12, 2019 · 14 comments · May be fixed by #35308

Comments

@mhvelplund
Copy link

Terraform Version

Terraform v0.12.14

Expected Behavior

When i run "terraform console" I expect it to to only lock state remotely long enough to do data lookups. Once the prompt is displayed, I would expect the lock to be released.

Actual Behavior

When i run "terraform console" it creates a lock in the remote state that isn't released until I close the console.

Steps to Reproduce

  1. run terraform console on a project with shared state on machine A, leave the console open
  2. run terraform console on a project with shared state on machine B
  3. bow head in sadness
@Great-Stone
Copy link

Could I check how this issue proceed?

I have received similar issues from my customer when run terraform console


tfstate.lock.info

{"ID":"753dd145-4007-3fb5-00c9-58876ea44d8a","Operation":"OperationTypeInvalid","Info":"","Who":"WIN-5P0ODUDEVVF\Administrator@LGUPLUS-PC","Version":"0.14.2","Created":"2020-12-18T06:36:20.2570009Z","Path":"terraform.tfstate"}


$ terraform plan

Error: Error locking state: Error acquiring the state lock: Failed to read state file: The state file could not be read: read terraform.tfstate: The process cannot access the file because another process has locked a portion of the file.

Terraform acquires a state lock to protect the state from being written
by multiple users at the same time. Please resolve the issue above and try
again. For most commands, you can disable locking with the "-lock=false"
flag, but this is not recommended.

And then terraform force-unlock not working

$ terraform force-unlock 753dd145-4007-3fb5-00c9-58876ea44d8a
Local state cannot be unlocked by another process

@Starefossen
Copy link

Just got hit by this myself.... took a while to figure out 😅

@tringuyen-yw
Copy link

Best would be to add an argument --no-state-lock to the terraform console command.

@lmcgough-gc
Copy link

@tringuyen-yw better still to implement it as -lock=false just like terraform plan has

@philip-harvey
Copy link

Any update on this, it's been an open issue for a long time. When you strike the issue the error says to run with -lock=false. but this does not work so this makes it a bug.

@rwblokzijl
Copy link

rwblokzijl commented Jun 17, 2022

This would be a welcome feature. Our terraform developers dont have write access to the gcp bucket where we keep our tfstate. They run plan with -lock=false to see changes which will be applied by a CICD after a PR is merged. These developers will not be able to run terraform console for debugging because there is no -lock flag for terraform console

@philip-harvey
Copy link

Agree, this really needs to be fixed. Needs a -lock=false option

@kazuki-hanai
Copy link

Hi all. May I implement this improvement? I’m interested in terraform contribution and this issue is a good first one.

@crw
Copy link
Collaborator

crw commented Nov 2, 2022

Hi @kazuki-hanai, before you work on a pull request, please check out https://github.com/hashicorp/terraform/blob/main/.github/CONTRIBUTING.md#proposing-a-change - specifically:

In order to be respectful of the time of community contributors, we aim to discuss potential changes in GitHub issues prior to implementation. That will allow us to give design feedback up front and set expectations about the scope of the change, and, for larger changes, how best to approach the work such that the Terraform team can review it and merge it along with other concurrent work.

This would be the first step to making this contribution, so if you do decide to move forward please start with the discussion of a potential solution.

I raised this issue in a triage meeting in May 2022. That meeting did not have the benefit of the later use cases. Here is an overview of the feedback:

Broadly speaking, the console feature is intended for configuration language experimentation, rather than to be built into production workflows (with all respect to anyone who is using console in such a workflow). We want console to be as safe to use as possible. Another core assumption is that the console will not be left open indefinitely, unreasonably holding open locks.

One possible solution would be to acquire and release a lock on every statement execution. This solution would have a relatively high overhead, but is more feasible than a command-line option that would disable locks entirely.

In general, the team is not adding new command line options. So a solution that avoided adding a new argument would be preferred.

I hope this helps guide you in the right direction when proposing a solution!

@mhvelplund
Copy link
Author

I don't know why it should keep a lock. Last time I checked, it couldn't modify anything (may have changed).
What I, and my team mates used it for, was debugging the contents of calculated variables.

@klar42
Copy link

klar42 commented Jun 16, 2023

Hi @crw, I don't get your argumentation.

Following case:

  • Terraform was working for me in a toy project, workspace on NFS.
  • Because of changes in the NFS setup, it started to fail recently with "Error: Error acquiring the state lock" in commands which need a lock.

Where I don't doubt that it is a good thing to hold a lock, I'm a bit lost here:

  • terraform [plan|apply] -lock=false -- work as expected, even if potentially dangerous (not in my use case)
  • terraform console -- does not work, fails with "Error: Error acquiring the state lock"
  • terraform console -lock=false -- does not work, fails because it does not implement -lock=false

Now you argue to have "console as save to use as possible" and thus not want to implement -lock=false leaves the thing inconsistent.
The real dangerous "apply" allows -lock=false. The usually non-dangerous "console" does not, removing the claimed used as "language experimentation" prior to "apply".

Thanks for the great work 👍

@cwray
Copy link

cwray commented Aug 4, 2023

I'm also confused as to why the console would need to create a lock file if it can't actually change anything. From my understanding, it only reads the current state file from the defined backend and can not make changes to the current state. Correct me if I'm wrong here.

This debugging tool is super useful, and should never be used in a production pipeline. But when we are debugging with this tool and our pipeline runs while a developer is using the tool for debugging reasons, the lock file is present in remote storage and the pipeline fails. This should not be the intended function of a debugging tool.

@crw
Copy link
Collaborator

crw commented Dec 14, 2023

Apologies for the late response. The team would be happy to consider a PR that adds a -lock=false to terraform console. I was informed that you should be able to look at the implementation from the other sections of code where this exists for guidance on how to implement it. Thanks!

@phyzical phyzical linked a pull request Jun 7, 2024 that will close this issue
@phyzical
Copy link

phyzical commented Jun 7, 2024

Have given this a quick crack. hopefully itll get through soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.