pug --program=terragrunt does not find all modules

[ohmer]

Describe the bug

Terragrunt modules not loaded.

Setup (please complete the following information):

• OS: MacOS 14.5
• Shell: Fish
• Terminal Emulator: iTerm2
• Terminal Multiplexer: not in use

To Reproduce

Not sure how, sorry. I inherited of a fairly large and messy monorepo.
Opening this more as a troubleshooting request.

pug --program=terragrunt find only 77 workspaces while I have much more:

❯ echo **/terragrunt.hcl | tr ' ' '\n' | wc -l
    1245

The dependency crawling fails too with messages like that in the logs:

WARN   loading terragrunt dependencies error=<RELATIVE_PATH>: resource not found

Could not pin point the issue in the code. --debug did not lead to any other clue in message.log :-/

Expected behavior

1245 or more modules loaded.

Additional context

Terraform 1.7.5
Terragrunt 0.62.0
ASDF and direnv in use

Issue seen with both v0.4.2 and v0.4.3-0.20240723095144-361ae1859435

[ohmer]

I think I narrowed it down to a backend definition not found. With debug added:

diff --git a/internal/module/module.go b/internal/module/module.go
index 4ba5e14..59c7f17 100644
--- a/internal/module/module.go
+++ b/internal/module/module.go
@@ -81,20 +81,21 @@ func findModules(logger logging.Interface, workdir internal.Workdir) (modules []
 			}
 			return nil
 		}
 		if filepath.Ext(path) == ".tf" || d.Name() == "terragrunt.hcl" {
 			backend, found, err := detectBackend(path)
 			if err != nil {
 				logger.Error("reloading modules: parsing hcl", "path", path, "error", err)
 				return nil
 			}
 			if !found {
+				logger.Error("reloading modules: backend not found", "path", path, "error", err)
 				return nil
 			}
 			// Strip workdir from module path
 			stripped, err := filepath.Rel(workdir.String(), filepath.Dir(path))
 			if err != nil {
 				return err
 			}
 			modules = append(modules, Options{
 				Path:    stripped,
 				Backend: backend,

I get this in the logs:

ERROR  reloading modules: backend not found path=<ABSOLUTE_PATH>/terragrunt.hcl error=<nil>

There is a backend definition which is not found by the HCL parser. This terragrunt.hcl got it from an included file.

# /<ABSOLUTE_PATH>/terragrunt.hcl 
include {
  path = find_in_parent_folders("root.hcl")
}

# /root.hcl
remote_state {
  backend = "s3"
  config = {
    <OMITED>
  }
}

So I guess parsing HCL is not enough, it has to be evaluated so that Terragrunt's find_in_parent_folders is followed.
Do I get this right?

[leg100]

@ohmer Thanks for the write-up.

So I guess parsing HCL is not enough, it has to be evaluated so that Terragrunt's find_in_parent_folders is followed.
Do I get this right?

Yes. Thinking about this, the code does two things:

1. Detects a module
2. Detects the backend type, e.g. s3, etc., so it can add it to the modules listing for the user's information.

Really, detecting the presence of a terragrunt.hcl is sufficient to detect a module. Whereas detecting the backend type requires parsing the HCL, and, as you say, evaluating the terragrunt semantics. Which is not trivial.

Tell me if you think this is a good idea:

1. Change code to detect module via presence of terragrunt.hcl alone.
2. If the backend type cannot be parsed, don't try to evaluate further, and just say the backend type is "unknown".

I think this is good interim step until something more sophisticated can be implemented.

[ohmer]

evaluating the terragrunt semantics. Which is not trivial.

💯 and I don't care much about Terragrunt. In my opinion, it should be sent to a special place in DevOps hell. I replace it with Terramate wherever I can and it just works with pug thanks to its non intrusive integration with Terraform.

1. Change code to detect module via presence of terragrunt.hcl alone.

Seems sufficient to me. I fail to see a case where it wouldn't be working, as .terragrunt-cache directories are already excluded.

2. If the backend type cannot be parsed, don't try to evaluate further, and just say the backend type is "unknown".

The backend type information is a nice to have for me. I do not have a strong opinion about this one.