[BUG] Regression on "file.recurse" state after Salt 3006.0 when dealing with symlinks: Unable to manage file: [Errno 2] No such file or directory

[meaksh]

Description

We have identified a regression between Salt 3005.1 and Salt 3006.0 version for file.recurse state function when dealing with symlinks, leading to a failing state execution with the following error:

#### /tmp/test-path/symlink ####
              Unable to manage file: [Errno 2] No such file or directory: '/tmp/test-path/real_file'

Steps to Reproduce the behavior

Given an SLS file like at /srv/salt/test_symlink.sls:

test_symlink:
  file.recurse:
    - name: /tmp/test-path/
    - source: salt:https://test_symlink/
    - file_mode: keep
    - keep_symlinks: True

And the following structure as part of Salt "file roots":

/srv/salt/test_symlink/
/srv/salt/test_symlink/real_file
/srv/salt/test_symlink/symlink -> real_file (symlink to real_file)

Now, when we apply our SLS file, the state will fail:

# salt salttests state.apply test_symlink
salttests:
----------
          ID: test_symlink
    Function: file.recurse
        Name: /tmp/test-path/
      Result: False
     Comment: #### /tmp/test-path/symlink ####
              Unable to manage file: [Errno 2] No such file or directory: '/tmp/test-path/real_file'
     Started: 12:19:19.137984
    Duration: 271.621 ms
     Changes:   
              ----------
              /tmp/test-path/real_file:
                  ----------
                  diff:
                      New file
                  mode:
                      0644
              /tmp/test-path/symlink:
                  ----------
                  new:
                      /tmp/test-path/symlink

Summary for salttests
------------
Succeeded: 0 (changed=1)
Failed:    1
------------
Total states run:     1
Total run time: 271.621 ms
ERROR: Minions returned with non-zero exit code

Expected behavior
The state should run fine as it was in Salt versions < 3006.0

salttests:
----------
          ID: test_symlink
    Function: file.recurse
        Name: /tmp/test-path/
      Result: True
     Comment: Recursively updated /tmp/test-path/
     Started: 13:02:39.570411
    Duration: 165.118 ms
     Changes:   
              ----------
              /tmp/test-path/real_file:
                  ----------
                  diff:
                      New file
                  mode:
                      0644
              /tmp/test-path/symlink:
                  ----------
                  new:
                      /tmp/test-path/symlink

Summary for salttests
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time: 165.118 ms

Versions Report

salt --versions-report

Salt Version:
          Salt: 3006.1
 
Python Version:
        Python: 3.6.15 (default, Sep 23 2021, 15:41:43) [GCC]
 
Dependency Versions:
          cffi: 1.13.2
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.10.1
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: 0.38.0
          Mako: Not Installed
       msgpack: 0.5.6
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 21.3
     pycparser: 2.17
      pycrypto: Not Installed
  pycryptodome: 3.18.0
        pygit2: Not Installed
  python-gnupg: Not Installed
        PyYAML: 5.4.1
         PyZMQ: 25.1.0
        relenv: Not Installed
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: opensuse-leap 15.5 
        locale: UTF-8
       machine: x86_64
       release: 6.3.7-1-default
        system: Linux
       version: openSUSE Leap 15.5 

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at [email protected]. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[OrangeDog]

Have you checked 3006.1?

[meaksh]

Yes, I've reproduced this with 3006.1.

[twangboy]

I have confirmed this bug. The file and symlink are actually created, but the state returns an error. I tested on CentOS 8 Stream with Salt 3006.1

[vzhestkov]

I just found that this commit is causing this issue fd7e82f

@garethgreenaway could you please take a look to this part, not sure if I understand why it's required: https://github.com/saltstack/salt/pull/61279/files#diff-9440ccbdf1120d483596ef8c352b22f5805522bcd1dee629b5b33fd6ac19e2ea

The way to reproduce the issue suggested bu @meaksh is a bit flacky in sometimes it's not causing any issue, but fails in other cases, probably depending on which file is processed first.

[twangboy]

This issue seems to have been fixed in 3006.6

[mariushoch]

This issue seems to have been fixed in 3006.6

This seems to still be an issue for me on 3006.7 (Fedora 39 / Python 3.12).

[twangboy]

@mariushoch Could you provide details on how you were able to reproduce this?

[mariushoch]

@mariushoch Could you provide details on how you were able to reproduce this?

Sure! This can be reproduced in a fresh Fedora 38 container with salt 3007.0:

Containerized set up (all in the same folder)

$ cat Dockerfile 
FROM fedora:38

RUN rpm --import https://repo.saltproject.io/salt/py3/fedora/38/x86_64/SALT-PROJECT-GPG-PUBKEY-2023.pub
RUN curl -fsSL https://repo.saltproject.io/salt/py3/fedora/38/x86_64/latest.repo | sudo tee /etc/yum.repos.d/salt.repo
RUN dnf clean expire-cache
RUN dnf -y install salt-minion
COPY . /srv/salt

$ cat top.sls 
base:
  '*':
    - file-recurse-symlink

$ cat file-recurse-symlink.sls 
/etc/foo:
  file.recurse:
    - source: salt:https://source-folder/
    - keep_symlinks: True

$ ls -l source-folder/
total 4
-rw-r--r--. 1 marius marius 4 Apr  3 10:17 plainfile
lrwxrwxrwx. 1 marius marius 9 Apr  3 10:36 symlink -> plainfile

With podman build -f Dockerfile --tag salt-issue-64630, we will get (in maybe half the runs):

$ podman run --rm salt-issue-64630 salt-call --local state.apply
[ERROR   ] {'/etc/foo/symlink': {'new': '/etc/foo/symlink'}, '/etc/foo/plainfile': {'diff': 'New file', 'mode': '0644'}}
local:
----------
          ID: /etc/foo
    Function: file.recurse
      Result: False
     Comment: #### /etc/foo/symlink ####
              Unable to manage file: [Errno 2] No such file or directory: '/etc/foo/plainfile'
     Started: 08:40:38.905058
    Duration: 67.773 ms
     Changes:   
              ----------
              /etc/foo/plainfile:
                  ----------
                  diff:
                      New file
                  mode:
                      0644
              /etc/foo/symlink:
                  ----------
                  new:
                      /etc/foo/symlink

Summary for local
------------
Succeeded: 0 (changed=1)
Failed:    1
------------
Total states run:     1
Total run time:  67.773 ms

The file and the symlink are correctly created even if the state fails, it seems.

Versions Report

salt --versions-report

Salt Version:
          Salt: 3007.0
 
Python Version:
        Python: 3.10.13 (main, Feb 19 2024, 03:31:20) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.16.0
      cherrypy: 18.8.0
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.3
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 23.1
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.15.1
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.3.3
           ZMQ: 4.3.4
 
Salt Package Information:
  Package Type: onedir
 
System Versions:
          dist: fedora 38 
        locale: utf-8
       machine: x86_64
       release: 6.7.11-200.fc39.x86_64
        system: Linux
       version: Fedora Linux 38 
 

[twangboy]

Thank you. I'll try to reproduce again.