Use xdg basedir spec on linux

[jleclanche]

Terraform Version

0.9.8

https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html

TLDR: Instead of ~/.terraform.d, configuration should be in the $XDG_CONFIG_HOME/terraform/ and cache (safely-deletable files) should be in $XDG_CACHE_HOME/terraform. If not defined, $XDG_CONFIG_HOME should be defaulted to $HOME/.config and $XDG_CACHE_HOME should be defaulted to $HOME/.cache.
I believe everything currently in .terraform.d is considered cache, therefore there's just $XDG_CACHE_HOME to worry about.

This is pretty standard and helps keep clutter out of the homedir :)

[apparentlymart]

Thanks for pointing this out, @jleclanche.

This seems like a reasonable idea as long as we can find a way to get there without breaking things for people already using the current paths, though should think about whether it makes sense to adopt this just for Linux (and thus have to document another platform-specific difference) or across all UNIX-flavored platforms (where it might seem more alien).

I'd like to let this one soak for a little while before we take any action. Thanks again for pointing it out!

[jleclanche]

👍
Out of curiosity is there anything that would actually break things if not given an upgrade path?
All I see in my dir is .terraform.d/checkpoint_cache .terraform.d/checkpoint_signature; both of which look to be cache for update checks.

[jleclanche]

Ah sorry, i see in config_unix.go that there's a .terraformrc file as well, I'm just not using it. Disregard.

[apparentlymart]

Yeah, indeed... seems worth thinking about splitting the cache-ish things from the settings-ish things too, like you said.

[jcrben]

Is there an environment variable to customize the location of this directory? I switch between OSX and Linux and I like to use environment variables to keep the location of config files consistent.

[jleclanche]

@apparentlymart I see you've been working on the config lately, any chance this is going in too?

[apparentlymart]

Hi again @jleclanche! Sorry for leaving this here so long.

We've been adding some more stuff into this dir in the mean time, such as more config files (.tfrc files) and a search location for plugins, so I think we do need to make sure to have a good upgrade path here, and indeed not everything in there is cache anymore. 😖

I do like the idea of following the relevant standards and of distinguishing config from cache, but I want to go into it with a stronger idea of how this changes all of our supported platforms, rather than just changing it for Linux, or indeed forcing Linux-specific conventions on other platforms.

I think to properly support this we need to alter our model to separately account for a config and cache dir on each platform, and then use the appropriate directory for each case. AFAICT on Mac OS X the appropriate directories are:

• Config: ~/Library/Preferences/
• Cache: ~/Library/Caches/

To get the migration path we need, and also to meet @jcrben's reasonable request at being able to use common directories across both OS X and Linux, I think we'd need to continue looking for config-ish things in ~/.terraform.d in addition to $XDG_CONFIG_HOME/terraform for pre-existing things (provider plugins and .tfrc files), but that doesn't seem like a big hardship.

AFAICT the BSDs also don't really use XDG and so continuing to use ~/.terraform.d alone is probably the most ergonomic thing there.

I wasn't able to get a read on whether Solaris users expect XDG support.

While we're doing this, we should also check if there's a config vs. cache dir distinction to be made for Windows systems as well. We currently use the same prefix for both on Windows.

I can't promise working on this soon since there are many other things competing for attention, but if we can get some consensus on what the most natural thing is on each platform then at least when someone has some time to look at this we'll have a reasonably-clear path forward.

[jleclanche]

If you want to look at how to implement it cross-platform, I recommend looking at the QStandardPaths implementation from Qt:

https://doc.qt.io/qt-5/qstandardpaths.html

This follows XDG standards on Linux and other platform-specific quirks for macOS and Windows.

[apparentlymart]

Awesome... thanks for that pointer, @jleclanche!

[jcrben]

In the case of a fair number of my tools, such as git, if I define the XDG variables, they are used on OSX as well. See https://git-scm.com/docs/git-config#git-config---global

A simpler place to start is to just let me place the entire directory into a different place.

[0az]

As a Mac user, I am against ~/Library/Preferences. According to rust-lang/rfcs#1615 (comment), the folder is for configuration plists for the "user defaults" subsystem, though I couldn't find an official source for this. Instead, there seems to be consensus in following XDG or providing an environment variable, both of which allow for neater, cleaner $HOMEs.

The actual change does not appear difficult, relative to the docs and migration. I might try submitting a PR for this once I'm more proficient at Go.

`rg 'terraform\.d' terraform/ -c` on 69c647c

terraform/plugins.go:1
terraform/config_windows.go:1
terraform/CHANGELOG.md:1
terraform/config_unix.go:1
terraform/command/command.go:1
terraform/website/upgrade-guides/0-7.html.markdown:1
terraform/website/guides/running-terraform-in-automation.html.md:1
terraform/website/docs/plugins/basics.html.md:2
terraform/website/docs/configuration/providers.html.md:4
terraform/website/docs/commands/cli-config.html.markdown:1
terraform/website/docs/commands/init.html.markdown:2

`rg 'terraform\.d' terraform/` on 69c647c

terraform/plugins.go
18:	// Look in ~/.terraform.d/plugins/ , or its equivalent on non-UNIX
terraform/CHANGELOG.md

488:* Load plugins from ~/.terraform.d/plugins/OS_ARCH/ and .terraformrc (#15769)
terraform/config_windows.go

33:	return filepath.Join(dir, "terraform.d"), nil
terraform/config_unix.go

27:	return filepath.Join(dir, ".terraform.d"), nil
terraform/command/command.go

28:const DefaultPluginVendorDir = "terraform.d/plugins/" + pluginMachineName
terraform/website/upgrade-guides/0-7.html.markdown

25:These binaries needed to all be extracted to somewhere in your $PATH or in the ~/.terraform.d directory for Terraform to work.
terraform/website/guides/running-terraform-in-automation.html.md

308:terraform.d/plugins/OS_ARCH directory, which will be searched before
terraform/website/docs/plugins/basics.html.md

42:directory, located at %APPDATA%\terraform.d\plugins on Windows and

43:~/.terraform.d/plugins on other systems.
terraform/website/docs/configuration/providers.html.md

214:Windows           | %APPDATA%\terraform.d\plugins

215:All other systems | ~/.terraform.d/plugins

286:plugin_cache_dir = "$HOME/.terraform.d/plugin-cache"

302:export TF_PLUGIN_CACHE_DIR="$HOME/.terraform.d/plugin-cache"
terraform/website/docs/commands/cli-config.html.markdown

43:plugin_cache_dir   = "$HOME/.terraform.d/plugin-cache"
terraform/website/docs/commands/init.html.markdown

125:~/.terraform.d/plugins on most operating systems and

126:%APPDATA%\terraform.d\plugins on Windows.

[apparentlymart]

Hi all,

We're intentionally not making any changes related to this right now because we're trying to wrap up development of the v0.12.0 release, which already has a large scope and so we'd prefer not to make changes to how Terraform interacts with the host system at the same time. However, I'd love to dive into this some more after that and see what makes sense here.

Given that there are a few different options, I think the best way to proceed here is to come to some consensus on what behavior is desirable first and only then move forward to implementation. I am open to using XDG on OS X if that is emerging as a de-facto standard in other tools, but I'd like to go into that with a fully-detailed idea of exactly what paths that will end up using on each of the different platforms Terraform supports, including figuring out what makes sense for all of the non-Linux and non-MacOS unix flavors, and seeing also if any changes are warranted on Windows at the same time so that we can make all necessary changes of this kind in a single release.

In the mean time it'd be helpful to collect here links to existing precedent for how other software is handling these concerns across platforms, since I think consistency with common behavior will be the primary deciding factor in what path we take here.

[rouing]

@apparentlymart

Check this out:

https://wiki.archlinux.org/index.php/XDG_Base_Directory

This is a list of Software Conforming to the XDG Base Directory Standard. If the Source was avail at the time in a git/svn repo, the pr/merge/commit is linked there.

Ex:

https://cgit.freedesktop.org/libreoffice/ure/commit/?id=a6f56f7
https://cgit.freedesktop.org/libreoffice/bootstrap/commit/?id=25bd2ee

Those have OSX Examples

https://src.chromium.org/viewvc/chrome/trunk/src/chrome/common/chrome_paths.cc?r1=23057&r2=23056&pathrev=23057
This one checks for linux.

https://gitlab.gnome.org/GNOME/gimp/commit/60e0cfe
In this case, they check for the .config directory.

I think that is the best approach here.

[philoserf]

In reference to macOS.

Since the ~/Library is by default hidden on Mac, the XDG should be used by command line applications leaving the ~/Library to native macOS applications .

[06kellyjac]

I've got plenty of programs that are as native as terraform that output to ~/Library and ~/Library/Application Support

• Jupyter
• nvm
• rustup
• texlive
• virtualbox
• virtualenv
• zsh

etc.

It doesn't really make sense to use the XDG standards for MacOS when MacOS already has it's own.

this Go XDG library accounts for Mac & Windows: OpenPeeDeeP/xdg
Rust has a similar one called dirs

https://stackoverflow.com/questions/3373948/equivalents-of-xdg-config-home-and-xdg-data-home-on-mac-os-x

https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/FileSystemProgrammingGuide/FileSystemOverview/FileSystemOverview.html#//apple_ref/doc/uid/TP40010672-CH2-SW1

[philoserf]

Three of those are gui apps or have a GUI over a cmdline executable.

I guess i am out of touch with the macOS community.

[0az]

I honestly find the basedirs more useful because of the reduced cognitive load when performing backup and/or synchronization. Instead of cherry-picking from `~/Library`, we just have the XDG directories.

[06kellyjac]

So instead of applications on MacOS conforming to the OS's official standards they should move to match XDG?...

The only reason for cognitive load is because of people not following the OS's standards. If all Linux programs followed XDG, Windows programs go to AppData etc, Mac go to ~/Library etc, the only cognitive load is knowing where the standard locations are.

There is more cognitive load due to the mishmash caused by ignoring the OS's standards and having them go in various different locations on one OS

The standard config for terraform should match the OS's standards, then env variables should just allow you to put the config in whatever non-standard location you like.

[philoserf]

Look in your ~/Library/Preferences

I looked at mine. Nearly only macOS GUI apps. Where are the rest?

dotfiles in ~/ or a few, that I think are well behaved, in ~/.config.

among those in ~/ items provided by Apple as it ships.

I return to where I started, macOS native apps in the ~/Library and command line executables following XDG.

[06kellyjac]

Well I listed both non native apps & some cli apps that follow that spec. Additionally my ZSH setup has XDG_CONFIG_HOME set to ~/.config so any cli programs that give priority to the env variable will have avoided ~/Library all together.

I'm going to start working on this to strictly respect XDG. Maybe later we can have a further chat about Mac standard dirs but even then XDG_... env vars would take priority.
At the end of the day I just want it out of my base home dir and also split config from the caches etc.

[06kellyjac]

In the PR I've explained the flow of priorities/preference. Please see the flows & the note in the first section and give me your thoughts.
Once we've decided on the order I'll get to work on fixing the docs etc. to match the changes

[0az]

As per the spec, the alternate directories should be ~/.config/* instead of ~/config/*. In addition, I believe the spec mandates $XDG_CONFIG_HOME/terraform/terraformrc instead of $XDG_CONFIG_HOME/terraform/.terraformrc (note the lack of full stop in the previous). Even if it isn't mandated, all other spec implementations I'm aware of drop the "dot".

I am strongly against $XDG_* taking lower priority, especially as other projects will gracefully fall back to the legacy locations.

[06kellyjac]

Where did I miss the dot in .config? I just had a check but line 54 in config_unix.go has the dot.

zsh keeps the dot in .zsh even if youre using ~/.config/zsh as your ZDOTDIR.
Im happy to swap it & I prefer it without the dot when outside of the home dir aesthetically but I went with the easy option of leaving it. Making it swap between dot for in home and no dot for elsewhere will take some extra logic & refactoring but I'll get on that soon.

The only issue with the env var taking highest priority is if people already have config at ~/.terraformrc and having XDG_CONFIG_HOME set and their config suddenly being ignored without much warning one update.
We can always have the legacy one be priority for now & log a warning to move it before X version where the env var can be Max priority.

[OJFord]

macOS's difference comes up every time in issues about adhering to XDG... essentially I think there's two camps:

• those that version (or otherwise curate) their dotfiles, and want everything with text configuration to be in their $XDG_CONFIG_HOME
• those that don't, and can't see why anybody wouldn't want to use ~/Library/Application\ Suppport/*/Content/Resources/MacOS or whatever it is

The really easy resolution is simply to respect $XDG_CONFIG_HOME iff it exists!

(resp. $XDG_CACHE_HOME, in the case of those checkpoint files)

[06kellyjac]

I've changed the code to only keep the . in-front of terraformrc in legacy locations & moved everything to work with the changes made in PR #22277

[WhyNotHugo]

Is there any known workaround for this?

It's very annoying to have .terraform.d showing up there between al my actual stuff in $HOME.

[jokerj40]

I would be great if you fixed this in nearest update

[rightaway]

@apparentlymart What is the latest thought on this? Now that 1.0 is out it should really be evaluated.

[jhogendorn]

Also interested in the latest movement on this, or a workaround until its correctly supported.

[eternaltyro]

Can we actually get this fixed on Linux while we continue to discuss the other platforms? The decision and solution seems really cut and dry when it comes to Linux.

[apparentlymart]

More recently we implemented a new directory structure for locally-mirrored provider plugins as part of introducing registry hostnames and namespaces to the provider addresses, and since we were effectively building on greenfield there we did adopt the XDG conventions on "Linux and other Unix-like systems" (as the documentation currently describes it), along with some paths on macOS and Windows that matched what we learned from researching those platforms' typical conventions.

That does at least mean that we now have some plumbing in place for implementing this sort of thing, via a "third-party" library (where the third party happens to be me wearing a different hat).

However, I believe there are some remaining design decisions for us to decide and so far we've not been able to prioritize this largely because there is already a solution for specifying CLI configuration files via the environment (the TF_CLI_CONFIG_FILE environment variable) and so switching to a less-custom one is definitely a nice to have (I'd love to do it, personally) but difficult to prioritize over solving new problems where there isn't already an answer.

From my memory (which is likely incomplete, since I'm just replying here off the top of my head), the two main decisions I remember us planning to research more here were:

• Since XDG is designed around the idea of scanning across multiple directories looking for many files, rather than finding a singleton "main" configuration file, we were initially unsure about how best to generalize the existing design of the .terraformrc file. I believe that's now largely resolved by the introduction of the ~/.terraform.d directory which can contain an arbitrary number of *.tfrc files, but we still have some internal work to do to teach Terraform to scan across multiple directories to find those (it currently expects exactly one, varying by platform) and to nail down exactly what the priority ordering for them should be.

  I don't think this is a particularly hard item to deal with, just needs some time to spend on it and think it out.

• The interaction with a possible upgrade of the configuration format grammar. We currently have some tech debt that the CLI configuration had to stay using the legacy version of HCL even after Terraform v0.12 because the main CLI config file is used by all versions of Terraform issued to date.

  However, that creates various weird inconsistencies with the main language grammar and tends to lead to weird quirks/bugs for any new feature we might add to the CLI configuration, and so we'd considered using the switch to a new search location (which older versions of Terraform would therefore not be able to "see") to also allow switching to a new underlying grammar, which would be broadly but not exactly compatible with the old one in some edge cases.

  Of course, that has the downside that it makes a relatively-simple problem of "just search some more directories in some defined preference order" block on a more messy problem of writing a new configuration parser, but there is no clear future event that would give an equivalent opportunity and so I think we've been holding to have some time to make a final decision on this either way.

With all of that said then, I would not expect to see Terraform start searching new directories on any platform in the immediate future, so that we can be sure to definitively answer these remaining questions (which requires research and design tradeoffs) before doing so. In the mean time, if you need to place the Terraform CLI configuration at a different location than Terraform will search by default, the TF_CLI_CONFIG_FILE is the existing available option for that, though I do acknowlege that it's significantly less convenient to have to set an environment variable specifically for one application than to just configure once for everything, and so I would still like to solve this eventually, once prioritization allows.

[OJFord]

Since XDG is designed around the idea of scanning across multiple directories looking for many files, rather than finding a singleton "main" configuration file, we were initially unsure about how best to generalize the existing design of the .terraformrc file. I believe that's now largely resolved by the introduction of the ~/.terraform.d directory which can contain an arbitrary number of *.tfrc files, but we still have some internal work to do to teach Terraform to scan across multiple directories to find those (it currently expects exactly one, varying by platform)

Please do correct me if I'm [I am] wrong @apparentlymart, but that's not my understanding at all - the base directory spec per my understanding is about locating where the 'terraform config' (for example) directory lives, whether it's ~/myconfs/terraform or ~/.config/terraform for example. In it's own words:

The XDG Base Directory Specification is based on the following concepts:

• There is a single base directory relative to which user-specific data files should be written. This directory is defined by the environment variable $XDG_DATA_HOME.
• There is a single base directory relative to which user-specific configuration files should be written. This directory is defined by the environment variable $XDG_CONFIG_HOME.
  [...]

[emphasis added]

edit: Oh! Sorry, I forgot about $XDG_CONFIG_DIRS and co. I would guess >95% generally-considered XDG-respecting programs do too, but certainly admirable to (want to) do it properly.

[eternaltyro]

More recently we implemented a new directory structure for locally-mirrored provider plugins as part of introducing registry hostnames and namespaces to the provider addresses, ...

That does at least mean that we now have some plumbing in place for implementing this sort of thing,

Which is really nice and I appreciate the work.

However, I believe there are some remaining design decisions for us to decide and so far we've not been able to prioritize this
...
With all of that said then, I would not expect to see Terraform start searching new directories on any platform in the immediate future, so that we can be sure to definitively answer these remaining questions (which requires research and design tradeoffs) before doing so.

I have a vested interest in not having dotfiles in the home directory because it messes with my backup plans (among other problems). However, I do understand that this issue might not sit high up in the priority list for hashicorp devs. But I appreciate you being open to looking into this and having this discussion continuing.

[apparentlymart]

In my earlier research I observed that some Linux distribution packages make use of the system-wide (rather than user-specific) directories to install system-level configuration files which the user-specific directories (including the "config home") can add to or override. I no longer have my notes about exactly which application packages they were, but I considered being a good citizen in Linux distributions to be one of the benefits of following the de-facto platform spec, and thus fully supporting XDG_CONFIG_DIRS feels like an important part of this story to me.

This may also prove useful for any organization building their own local system-level packages with Terraform providers and special configurations as part of a systematic configuration management strategy, though I don't have any specific data about that.

[ogarcia]

Both XDG_CONFIG_DIRS and XDG_DATA_DIRS are intended for system-wide configurations. This is interesting from a point of view of software developer or packager that can provide default config or data files for the users. For example GNOME uses XDG_CONFIG_DIRS (by default points to /etc/xdg) to add autostart .desktop files that all users need to run GNOME environment. Other example is systemd that uses it to store units that are provided by the system but must be runned by each user with their own permissions.

[rwxrob]

Go has a standardized way to get this information:

• https://pkg.go.dev/os#UserConfigDir
• https://pkg.go.dev/os#UserCacheDir

[tony-sol]

Sorry, if i'm wrong, but read whole thread and still don't get:
why not just add env variable like TF_HOME_DIR or similar and use it if set, otherwise, use default "${HOME}/.terraform.d"? e.g.:

diff --git a/internal/command/cliconfig/config_unix.go b/internal/command/cliconfig/config_unix.go
index 6dc1450b23..a0187c225e 100644
--- a/internal/command/cliconfig/config_unix.go
+++ b/internal/command/cliconfig/config_unix.go
@@ -20,6 +20,9 @@ func configFile() (string, error) {
 }
 
 func configDir() (string, error) {
+       if home := os.Getenv("TF_HOME_DIR"); home != "" {
+               return home, nil
+       }
        dir, err := homeDir()
        if err != nil {
                return "", err

Looks like it would solve people's pain, which is keeping $HOME clean from any dotfiles/dotdirs as possible, in the easiest way.

So, is any progress here anyway?

[06kellyjac]

you can try submit a PR, I'll take anything that gets it out of ~

[ninoCan]

@apparentlymart, please, could you recap what should (those) MacOS' users (who are also fans of XDG) should do to keep the home clean of the .terraform.d directory?

Would the following workaround do trick? cf. Terraform Docs

export TF_CLI_CONFIG_FILE=$XDG_CONFIG_HOME/terraform/terraform.rc

cat <<EOF > "$XDG_CONFIG_HOME/terraform/terraform.rc"
plugin_cache_dir   = "$XDG_CACHE_HOME/terraform/plugin-cache"
disable_checkpoint = true
EOF

Can we safely remove $HOME/.terraform.d afterwards?
Does it actually need the second line about checkpoints disabling?