[Bug]: Evaluation of a project with an ProjectCachePlugin item impacts the next build, regardless of its involvement in the build.

[AArnott]

Issue Description

Merely evaluating a Project (even in a private ProjectCollection) that produces an ProjectCachePlugin item has the nasty side-effect of impacting the next build, even though that project is never built.
And in particular, it breaks attempts to programmatically enable the MSBuildCache plugin because it is very particular about only being initialized once, yet it is loaded twice in my testing because MSBuild allowed a single evaluation of an isolated project to also impact the build.

Steps to Reproduce

This code breaks the next build, yet is a natural way to reuse the plugin's defaults as defined in the plugin's nuget package:

buildParameters.ProjectCacheDescriptor = CreateCacheDescriptor();

private static ProjectCacheDescriptor? CreateCacheDescriptor()
{
    const string nugetPackageCache = @"C:\.tools\.nuget\packages";
    const string pluginPackageId = "microsoft.msbuildcache.local";
    const string pluginPackageVersion = "0.1.287-preview";
    ProjectCollection projectCollection = new();
    ProjectRootElement pre = ProjectRootElement.Create(projectCollection);
    pre.AddImport(Path.Combine(nugetPackageCache, pluginPackageId, pluginPackageVersion, "build", $"{pluginPackageId}.props"));
    pre.AddImport(Path.Combine(nugetPackageCache, pluginPackageId, pluginPackageVersion, "build", $"{pluginPackageId}.targets"));
    var project = Microsoft.Build.Evaluation.Project.FromProjectRootElement(pre, new ProjectOptions { });

    ProjectItem? pluginItem = project.GetItems("ProjectCachePlugin").FirstOrDefault();
    if (pluginItem is null)
    {
        return null;
    }

    string pluginPath = pluginItem.GetMetadataValue("FullPath");
    Dictionary<string, string> settings = new(StringComparer.OrdinalIgnoreCase);
    foreach(ProjectMetadata metadatum in pluginItem.DirectMetadata)
    {
        settings[metadatum.Name] = metadatum.EvaluatedValue;
    }

    settings["IdenticalDuplicateOutputPatterns"] = "**";

    return ProjectCacheDescriptor.FromAssemblyPath(pluginPath, settings);
}

Expected Behavior

A successful build.

Actual Behavior

The build fails with this message:

Another instance of MSBuildCache is already running in this build. This is typically due to a misconfiguration of the plugin settings, in particular different plugin settings across projects.

Analysis

@dfederm says msbuild stores the plugin in a mutable static (evil) just from seeing an ProjectCachePlugin item in any project evaluation.

Versions & Configurations

Dev17.12 (35228.240.main)

[maridematte]

@dfederm could you take a look at this issue?

[dfederm]

Yep, I'll look at some point. I worked with Andrew on this and am just using this issue for tracking so I don't forget.