Task randomly fails with 'Error code 5'

[dkaubar]

Description of the issue

Task randomly fails with 'Error code 5' when running a scan on bicep templates. I say randomly, since rerun of same task (identical configuration) can succeed, or fail a few more times before eventually succeeding.
Task runs for 3-5 mins before failing

Configuration
We do not run "Install PSRule module" task prior to this task. ps-rule-assert@1 is a first task in a job.
Job is running on Microsoft-hosted agent
Image: windows-latest

Yaml configuration of the extension.

      - task: ps-rule-assert@1
        displayName: "Run PSRule scan"
        inputs:
          path: "$(System.DefaultWorkingDirectory)"
          inputType: "inputPath"
          inputPath: "$(projectDirectory)/devops/infrastructure/bicep/*.bicep"
          source:
          modules: "PSRule.Rules.Azure"
          outputFormat: "NUnit3"
          outputPath: "$(Pipeline.Workspace)/PSRule/ps-rule-results.xml"

Expected behaviour

Task to succeed in providing scan results

Error output

Last logs before error:
From repository: {Repo-name}
on : refs/pull/6814/merge
at : 36f6dae96fa568f1c1349feb42dc7c3080cb8321

Then error:

##[error]Exit code 5 returned from process: file name 'C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe', arguments '-NoLogo -Sta -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". ([scriptblock]::Create('if ([Console]::InputEncoding -is [Text.UTF8Encoding] -and [Console]::InputEncoding.GetPreamble().Length -ne 0) { [Console]::InputEncoding = New-Object Text.UTF8Encoding $false } if (!$PSHOME) { $null = Get-Item -LiteralPath ''variable:PSHOME'' } else { Import-Module -Name ([System.IO.Path]::Combine($PSHOME, ''Modules\Microsoft.PowerShell.Management\Microsoft.PowerShell.Management.psd1'')) ; Import-Module -Name ([System.IO.Path]::Combine($PSHOME, ''Modules\Microsoft.PowerShell.Utility\Microsoft.PowerShell.Utility.psd1'')) }')) 2>&1 | ForEach-Object { Write-Verbose $_.Exception.Message -Verbose } ; Import-Module -Name 'D:\a\_tasks\ps-rule-assert_8804fc31-b62f-4d49-b2a1-c80dc0879dae\1.0.2201013\ps_modules\VstsTaskSdk\VstsTaskSdk.psd1' -ArgumentList @{ NonInteractive = $true } -ErrorAction Stop ; $VerbosePreference = 'SilentlyContinue' ; $DebugPreference = 'SilentlyContinue' ; Invoke-VstsTaskScript -ScriptBlock ([scriptblock]::Create('. ''D:\a\_tasks\ps-rule-assert_8804fc31-b62f-4d49-b2a1-c80dc0879dae\1.0.2201013\powershell.ps1'''))"'.

Task in use and version:

• Task: ps-rule-assert
• Version: SHOULD be 1.5.0, we have latest version installed. DevOps logs mention v 1.0.2201013, not sure what's that

From DevOps extension details:
Installed version
1.5.0 (Latest)

From task execution log:
Task : PSRule analysis
Description : Run analysis with PSRule.
Version : 1.0.2201013

Additional context

Additional logs:

 Checking module: PSRule.Rules.Azure
  - Installing module
  - Using version: 1.14.3

[info] Using Version: 2.0.1
[info] Using PWD: D:\a\_tasks\ps-rule-assert_8804fc31-b62f-4d49-b2a1-c80dc0879dae\1.0.2201013
[info] Using Path: D:\a\1\s
[info] Using Source: D:\a\1\s
[info] Using Baseline: 
[info] Using Conventions: 
[info] Using InputType: inputPath
[info] Using InputPath: D:\a\1\s\projectName\devops\infrastructure\bicep\*.bicep
[info] Using OutputFormat: NUnit3
[info] Using OutputPath: D:\a\1\PSRule\ps-rule-results.xml

[BernieWhite]

@dkaubar Thanks for logging the issue.

We might need to enable verbose logging to understand what error code 5 is related to, as it is not a PSRule specific error. It means access denied.

However I suspect that the issue may be related to the inputType parameter.

For bicep expansion to work, inputType needs to be set to repository.

For example:

      - task: ps-rule-assert@1
        displayName: "Run PSRule scan"
        inputs:
          path: "$(System.DefaultWorkingDirectory)"
          inputType: "repository"
          inputPath: "$(projectDirectory)/devops/infrastructure/bicep/*.bicep"
          modules: "PSRule.Rules.Azure"
          outputFormat: "NUnit3"
          outputPath: "$(Pipeline.Workspace)/PSRule/ps-rule-results.xml"

See Creating your pipeline for additional required configuration.

If that does not resolve the issue. Can you please enable verbose logging by setting the SYSTEM_DEBUG pipeline variable to true.

This should generate a lot of additional output and should give us an idea where the problem is.

[dkaubar]

will try that, thanks!

[dkaubar]

so far I was not able to reproduce the issue with inputType: repository

I would however like to ask a bit about this configuration:
what is the benefit of using repository as a value? is it only to allow multiple directories to be scanned? because if we set
input path to repo root, it takes way more time to execute. all solution files are unnecessarily scanned and skipped, until template files are found.
Is it a valid use to have it like

  inputType: "repository"
  inputPath: "$(projectDirectory)/devops/infrastructure/bicep/*.bicep"

where only inputType is different (inputPath -> repository) and the path itself points to the exact files we want to target (and not the repository)?

also, unrelated, but in logs we see warnings for every .bicep file, like:
##[warning]Target object 'Project/devops/infrastructure/bicep/function-app.bicep' has not been processed because no matching rules were found.
In a way it makes sense, rules are for ARM templates that are decompiled from bicep (for which they find and run rules), but it's still misleading to say that this template was not processed, no?

[BernieWhite]

@dkaubar That's great. Thanks for the feedback.

When inputType: inputPath is used a limited number of file formats are loaded from disk a read as objects. Supported formats are listed under the Input.Format.

For example, a JSON file:

[
  {
    "name": "object1"
  },
  {
    "name": "object2"
  }
]

Bicep and Azure template formats are a little tricky, they have parameters, variables, functions, loops, conditions, etc within their structure. Reading the objects is not a good indication of what resources would be deployed to Azure, or what their final state would be.

The repository option discovers the files and passed them through some additional functions to expand the Azure resources from the Bicep/ Template code.

---

With inputPath: "$(projectDirectory)/devops/infrastructure/bicep/*.bicep" only the files matching the path would be scanned regardless of if inputType set to inputPath or repository.

Additionally, with repository the Input.PathIgnore option can be set. See Configuring path exclusions and Input.PathIgnore.

In terms of the warning, yes that is expected. You are correct, the files are not being processed by any rule which is the reason for the warning. We could do better to hide this by default, however you can disable the warning by setting the Execution.NotProcessedWarning option to false.

I hope that helps.

[dkaubar]

Thanks for the detailed explanation and for the work you're doing on the tool/extension!
This helped a lot and I think that now we have a proper configuration in place