Bug: Symlinks in Testing

[eleanorjboyd]

Please see the first comment on this issue for the proposed solution.

Description of Issue:

In investigating the symlink issue affecting Python testing in VS Code, I uncovered several contributing factors. The problem begins during the test discovery phase: the Python extension initiates a subprocess which uses the symlink as its current working directory (since the symlink or at least a path relative to the symlink workspace is set as the cwd). Then when node calls the python subprocess which run pytest discovery, Node resolves the symlink cwd to target path. This means the subprocess which pytest run on is on the target path of the symlinked workspace. Consequently, when the PyTest extension is activated, it encounters already-resolved symlinks. This leads to the collection of test data with paths relative to the symlink's target value, not its relative location. These paths are then utilized to populate the Test Explorer in VS Code. As a result, when a user attempts to open a file from the Test Explorer, VS Code erroneously navigates to the target path which is incorrect for the workspace since it is workspace opened at the location of a symlink. Also each test then has its path set as the target path to its file, instead of the symlink, meaning the test is no longer associated with the given workspace it is in.

Given these findings, a design decision is needed to address and rectify the symlink issue in the context of testing within VS Code. This decision will determine the most effective solution or workaround to ensure accurate and efficient test handling.

I will explore how other symlink items are handled in vscode but some ideas include:

• passing the symlink into the subprocess and having the plugin handle returning data relative to the symlink
• editing resulting data on the extension side to substitute symlinks in for the target link if present.

Test Case Design:

Below are the steps to make this problem occur for testing and as an example:

• in terminal cd to parent folder parent
• create folder inside parent target_folder with tests such as test_example.py (a file with a test inside)
• from terminal create a symlink ln -s target_folder symlink_to_target_folder
• open vscode with the folder symlink_to_target_folder can be done with code symlink_to_target_folder in terminal
• run test discovery, test UI will be populated
• 🐛 the root folder will be target_folder instead of symlink_to_target_folder in the test explorer view
• 🐛 click on "go to file" on a test, the incorrect file will be opened
• 🐛 try to run tests, nothing will happen

Notable Code:

1. const proc = spawn(file, args, spawnOptions); when spawnOptions in rawProcessApis.ts has a value cwd. On this spawn call, the symlink is resolved.

2. here run requests are evaluated and tests included in the request and checked against known workspace folder to gather correct workspaces. This does not work correctly for symlinks. Code in controller.ts

if (request.include) {
            uniq(request.include.map((r) => this.workspaceService.getWorkspaceFolder(r.uri))).forEach((w) => {
                if (w) {
                    workspaces.push(w);
                }
            });

3. fs.lstatSync(workspaceFolderPath).isSymbolicLink(); a way to check if a given workspace folder is a symlink, possible addition to the code.

[eleanorjboyd]

Proposal for Fix (pytest)

On further investigation I have determined a potential fix to this issue. Below I have outlined how this fix may work as some decisions need to be made around the design and implementation.

As described in the issue description above, the problem stems from the extension running a python subprocess with the symlink as the cwd. The subprocess resolves the cwd to the target when it runs so the subprocess (pytest) has no knowledge it was even started in symlink directory.

A fix for this would be to used the --rootdir flag for pytest to provide the symlink path to the root. Given the rootdir, the pytest plugin can check to see if a --rootdir is being provided and is a symlink. If so it can then go through and edit all path values to make them relative to the symlink path of the root. This will cause all paths to be correct and all folder creation / nesting to be based on the symlink.

Specifics

1. I propose that before running the subprocess, the extension should check to see if the cwd is a symlink. If it is, then add the arg --rootdir to the arguments for the pytest request with the rootdir being set to the symlink of the cwd. A question here is what should be done if the --rootdir argument is already specified by the user. For this case I propose to always use the rootdir supplied by the user. The user could have already added a symlink root themselves, this would be fine and would work. There are cases where the user provides a rootdir that is not a symlink but if they have their workspace opened in a symlink then that is already not congruent and they probably expect the rootdir they provided. Given this weird case probably just throw a warning of this and continue with their provided value.

2. This extra work around symlinks should be notified to the user, a log should be included to state that the extension has a symlinked cwd and it is taking necessary steps.

3. Redoing all paths as we work through building the testing tree for discovery could add time. Since we are already doing the equivalent of a tree search to create "test nodes" for all files and parent folders to the session root, there should not be an additional time complexity but does add other computation item per node.

Outstanding questions:

These require exploration and invite user input.

1. What happens when the user is symlinked between a partitioned harddrive or another place where they do not have the same root? Currently I calculate the difference between the symlink and target link to know how to update paths.
2. Does this work on dev containers or ssh?
3. Are there any differences in how this will work on Linux vs Windows machines?
4. What happens if the symlinks are between machine types?

Required Engineering:

• Added check for symlinks on the extension side, with warning messages and notifications as reasonable
• Check for symlinks in the pytest plugin by checking if the cwd is a symlink. If so flag the workspace as symlinked and update all paths accordingly as the test nodes are built.
• Tests should be created which test:
  • symlinks no --rootdir specified by the user
  • symlinks with --rootdir already specified by the user
  • workspaces that contain --rootdir but are not symlinked workspaces
  • symlinks which have a different root (ie on a partitioned harddrive)

Final question how many people who face this issue use unittest and couldn't use pytest. Looking for numbers here to determine the priority of unittest vs pytest for symlinks. Unittest will require a different solution and I have not investigated the complexity of this.

[g24swint]

I followed the conversation in #22518 to this point, and this "symlink" issue seems to be my problem as well. Apart from a virtual "+1", I'm not sure I have much to add except that my symlink is outside / above the repository in which I'm working and is a symlink into an AWS EFS mount point. If I open VS code into my python package subdirectory of my repo using the symlink-to-EFS path, it will not run tests. If I open directly against the mount point in /mnt/efs, it does seem to run the tests correctly.

[alanwilter]

Testing vscode 1.86.1. The issue with Tests and symlink is still there.
Interestingly, vscode at least now shows the tests, but trying to run them and nothing happens.

[eleanorjboyd]

Hi @alanwilter, thanks for the update. The fix for symlinks hasn't been merged yet but it is almost done! I will let you know when it is merged in so you can try it in insiders, thanks!

[eleanorjboyd]

Hello everyone! I have made another fix to resolve this for scenarios where the workspace folder is not a symlink but the parent is. Please let me know if anyone sees problems if they try it on the newest version of insiders and the extension!