Detect-files Recursion

[HariSekhon]

The documentation is not clear on the recursion behaviour of pluto detect-files -d .

I've traced it like so on mac:

sudo dtruss -f -t open pluto detect-files -d .

and found that it seems to recurse 2 directory levels, but if there are yamls one subdirectory level down.

If you start at the top of a repo root and there are no yamls within 1 sub-level due to directory structures like app/base/*.yaml then pluto is not opening any yaml files and silently failing to scan.

Describe the solution you'd like

1. The recursion behaviour should be clearly documented either way
2. Ideally pluto detect-files -d . should recurse all subdirectory levels to find yamls in kubernetes repo trees
3. Optional: control recursion depth with a flag

Describe alternatives you've considered

Shell script recursing all directories and running pluto detect-files -d . at each level.

https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/kubernetes/pluto_detect_kustomize_materialize.sh

Additional Context

I'm trying to find deprecated API objects affecting my Kubernetes cluster upgrade that are inherited from Helm charts embedded in kustomization.yaml which are not currently detectable by Nova because the helm charts are not installed directy, and are not detectable by pluto unless I materialize the kustomization.yaml, as I do with this script:

https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/kubernetes/kustomize_materialize.sh

When I run pluto detect-files -d . on the resultant directory tree full of kustomization.materialized.yaml files, it fails to open and test any yamls unless I run pluto in each directory level.

I've worked around it for now using this script:

https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/kubernetes/pluto_detect_kustomize_materialize.sh

Somewhat related to #431

Pluto version

$ pluto version
Version:5.12.0 Commit:dc11ba13c840de0034510bff4712f651bf5ff45c

[sudermanjr]

detect-files is using filepath.Walk, which should follow the entire tree. It seems more like this is a bug related to our implementation.

pluto/pkg/finder/finder.go

Line 91 in 4c80d3a

err := filepath.Walk(dir.RootPath, func(path string, info os.FileInfo, err error) error {

[sudermanjr]

Actually, I can't seem to reproduce this. here's Pluto finding something several directories deep, with other yaml files above it.

▶ cat finder/testdata/dir1/dir2/deployment-extensions-v1beta1.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: two-directories-deep
  namespace: yaml-namespace
  labels:
    app: utilities
spec:
  replicas: 1
  selector:
    matchLabels:
      app: utilities
  template:
    metadata:
      labels:
        app: utilities
    spec:
      containers:
      - name: utilities
        image: quay.io/sudermanjr/utilities:latest
        command: [ "/bin/bash", "-c", "--" ]
        args: [ "while true; do sleep 30; done;" ]
        securityContext:
          readOnlyRootFilesystem: true
          allowPrivilegeEscalation: false
          runAsNonRoot: true
          runAsUser: 10324
          capabilities:
            drop:
              - ALL
        resources:
          requests:
            cpu: 30m
            memory: 64Mi
          limits:
            cpu: 100m
            memory: 128Mi
pluto/pkg   ☸ kind-kind  master [?]
▶ pluto detect-files -owide
NAME                   NAMESPACE        KIND         VERSION              REPLACEMENT   DEPRECATED   DEPRECATED IN   REMOVED   REMOVED IN
utilities              json-namespace   Deployment   extensions/v1beta1   apps/v1       true         v1.9.0          true      v1.16.0
utilities              yaml-namespace   Deployment   extensions/v1beta1   apps/v1       true         v1.9.0          true      v1.16.0
two-directories-deep   yaml-namespace   Deployment   extensions/v1beta1   apps/v1       true         v1.9.0          true      v1.16.0

[HariSekhon]

I managed to briefly recreate your recursion when using your CLI flags without -d . but this has since stopped working for me.

Unless dtruss is lying to me. Upon more repeated attempts it does start to show open() system calls to yamls buried further down, but this behaviour is not consistent between runs. Looks buggy. It's not clear if this is a bug in some underlying code or in dtruss.

[sudermanjr]

If you could provide an exact structure to reproduce the bug, we can look into it.