CI: Intermittent PSScriptAnalyzer crash in the Analyze task

[tablackburn]

The Analyze task in the repository's own build (psakeFile.ps1 line 19, Invoke-ScriptAnalyzer -Path $settings.ModuleOutDir -Recurse ...) intermittently crashes on the windows-latest CI leg with an internal PSScriptAnalyzer exception, failing the job before Pester ever runs. Re-running the job succeeds without any change.

Occurrences so far

Date	Run	Exception	Diff under test
2026-07-07	28897944004 (PR #133)	Object reference not set to an instance of an object.	Changelog text only
2026-07-09	29055849143 (PR #135)	Value cannot be null. (Parameter 'name')	Files under tests/ only

Why it is a flake, not a regression

• Both failing commits could not have influenced the Analyze task: it scans only the built module output (ModuleOutDir), and neither diff touched module source
• The identical merge commits passed Analyze on ubuntu and macOS in the same runs
• Both failures disappeared on re-run with no code change
• The two exceptions are different internal PSScriptAnalyzer errors at the same call site, which points at a non-deterministic crash inside PSScriptAnalyzer 1.25.0 (or its interaction with the windows-latest runner image) rather than at anything in this repository

Possible mitigations (not yet decided)

• Add a retry around Invoke-ScriptAnalyzer in the Analyze task (crude but effective for a known-flaky third-party call)
• Try a newer PSScriptAnalyzer release when one ships, in case the crash is fixed upstream
• Report upstream to PSScriptAnalyzer with the two stack signatures if it keeps recurring

Filed to track recurrence; if it bites again, that is three data points and worth acting on rather than re-running.

[HeyItsGilbert]

Let me know if you can figure this out. It's bitten me a few times on my repos.

[tablackburn]

Third occurrence, and this one breaks the windows-latest pattern.

Date	Run	Runner	Exception	Diff under test
2026-08-02	30755594401 (PR #146)	ubuntu-latest	Value cannot be null. (Parameter 'name')	.claude/settings.json only

Same call site (psakeFile.ps1:19), same exception string as the 2026-07-09 occurrence, same PSScriptAnalyzer 1.25.0 pin from requirements.psd1. Failed in 34s before any test ran, no testResults.xml produced, passed in 1m26s on re-run with zero changes to the commit.

Two new facts:

• Not windows-only. This leg was ubuntu-latest, while windows-latest, macOS-latest, and Windows PowerShell 5.1 all passed Analyze on the identical merge commit. The title's on windows-latest scoping is no longer accurate.
• Not one bad region. Failing runner reported Azure Region: centralus. An earlier cluster of these (2026-07-03, PR docs: initialize v0.8 to v1.0 migration guide skeleton #123) was correlated with eastus, so a single-region CDN/PSGallery theory does not cover both.

The diff under test was a Claude Code settings file — nothing the build reads, and Analyze only ever scans ModuleOutDir — so this is about as clean a "not the code" control as we are going to get.

That is three occurrences in the table plus the earlier #123/#129 sightings that predate this issue, across two runner OSes, two Azure regions, and two PSScriptAnalyzer versions (1.24.0 and 1.25.0). The common factor left is Invoke-ScriptAnalyzer itself. Per the bar set in the issue body, this seems worth acting on rather than re-running — the retry-around-Invoke-ScriptAnalyzer mitigation looks like the cheapest option, and @HeyItsGilbert reports hitting it on other repos too, so a fix here would not only help this one.

[HeyItsGilbert]

A retry is cheap and easy.

[tablackburn]

Root cause found: PSScriptAnalyzer runs rules in parallel over unsynchronised state

Short version: this is a thread-safety defect inside PSScriptAnalyzer, and the
suspicion in the issue body was right — nothing in this repository, on the runner, or in
the diff was ever causal.

PSScriptAnalyzer executes its script rules in parallel. Several of those rules reach
into PowerShell session state through a process-wide singleton whose mutable fields
are not synchronised. When two rule tasks touch that state at once, the loser sees
half-updated state and throws. Which exception surfaces depends only on where the
interleaving lands, which is exactly why we collected different error strings at one call
site.

The variables that actually matter are thread scheduling and how cold the PowerShell
module analysis cache is. That explains the whole pattern: fresh runner is worst case,
re-run always passes, OS and region are red herrings.

Evidence 1 — the rules really do run in parallel

Reflecting over Microsoft.Windows.PowerShell.ScriptAnalyzer 1.25.0.0, the closure that
ScriptAnalyzer.AnalyzeSyntaxTree builds is shaped for concurrency:

<>c__DisplayClass84_0
    ConcurrentBag`1      diagnostics
    ConcurrentBag`1      suppressed
    BlockingCollection`1 verboseOrErrors
<>c__DisplayClass84_1
    IScriptRule          scriptRule     <- one per parallel task
    method <AnalyzeSyntaxTree>b__2

You only need ConcurrentBag/BlockingCollection when multiple threads write. Every
crash stack I captured bottoms out in that <AnalyzeSyntaxTree>b__2() lambda.

Evidence 2 — the shared singleton is not thread-safe

Helper is a static singleton (static Helper instance) whose instance fields include:

Field	Type	Synchronised
invokeCommand	CommandInvocationIntrinsics	no
CmdletToAliasDictionary	Dictionary<,>	no
AliasToCmdletDictionary	Dictionary<,>	no
KeywordBlockDictionary	Dictionary<,>	no
VariableAnalysisDictionary	Dictionary<,>	no
_testModuleManifestLock	Object	a lock, for one call

That last field is the tell — upstream already hit thread-safety trouble here and guarded
exactly one call, leaving the plain dictionaries and invokeCommand open.

Evidence 3 — both of our error strings reproduce, from two different sites

Both signatures from the table reproduced locally. They turn out to be two distinct
crash sites sharing one root cause.

Object reference not set to an instance of an object.:

System.NullReferenceException
   at System.Management.Automation.CommandInfo.ResolveParameter(String name)
   at ...ScriptAnalyzer.Helper.GetExportedFunction(Ast ast)
   at ...BuiltinRules.ProvideCommentHelp.AnalyzeScript(Ast ast, String fileName)
   at ...ScriptAnalyzer.<>c__DisplayClass84_1.<AnalyzeSyntaxTree>b__2()

This appeared under two different rules — ProvideCommentHelp and
AvoidReservedCharInCmdlet. The rule is incidental; GetExportedFunction is the hazard.

Value cannot be null. (Parameter 'name'):

System.ArgumentNullException
   at System.ArgumentNullException.ThrowIfNull(Object argument, String paramName)
   at System.Management.Automation.CommandInfo..ctor(String name, CommandTypes type)
   at System.Management.Automation.PSModuleInfo.get_ExportedFunctions()
   at System.Management.Automation.AnalysisCache.CacheModuleExports(...)
   at Microsoft.PowerShell.Commands.ModuleCmdletBase.GetModule(...)
   at Microsoft.PowerShell.Commands.TestModuleManifestCommand.ProcessRecord()

Note this one runs through AnalysisCache, the module analysis cache — which is precisely
why a cold runner is the worst case.

Evidence 4 — a third variant, and proof the problem is engine-level

Analyzing a tree containing a module manifest also races inside Test-ModuleManifest,
reached from the MissingModuleManifestField rule, producing
Collection was modified; enumeration operation may not execute.

Taking PSScriptAnalyzer out of the picture entirely and calling Test-ModuleManifest
directly from several runspaces shows the cmdlet itself is not safe to run concurrently:

Mode	Failures
Serial (control)	0 / 60 (0%)
Concurrent ×4	3 / 60 (5%)

Higher concurrency also produced DirectoryNotFoundException: The specified RequiredModules entry 'BuildHelpers' ... is invalid, from concurrent RequiredModules probing. That is
four distinct exception types from one underlying race.

Minimisation

200 runs per configuration against the staged module output:

Configuration	Result
Full ModuleOutDir with -Recurse (what CI does)	fails
Only PowerShellBuild.psd1	0%
Full directory, PSMissingModuleManifestField excluded	0%
Full directory, .psd1 removed	0%

Both the manifest and the surrounding files are load-bearing — the race needs
concurrent rule work in flight to collide with.

Observed rates

Loop	Rate
Warm in-process loop	~2.5%
Cold fresh-process loop (CI-faithful)	~5%
Scanned tree fanned out to many manifests	100% (40/40)

Upstream status — "try a newer release" is a dead end

This is a known, open upstream defect, unfixed in any released version:

• PSScriptAnalyzer#1538 —
  the byte-identical ResolveParameter / GetExportedFunction stack in
  AvoidReservedCharInCmdlet. Open, deferred to 2.0.
• PSScriptAnalyzer#1351 —
  "Transient RULE_ERROR", matching the RULE_ERROR id these crashes carry. Open.
• PSScriptAnalyzer#1528 —
  the 2.0 roadmap lists making PSScriptAnalyzer thread-safe as a goal.

So of the three mitigations in the issue body, the middle one can be struck off.

The -Verbose workaround does not work

A community report on #1538 claims passing -Verbose suppresses the crash, and our call
site passes -Verbose:$false, so this looked like a cheap fix. Tested with an interleaved
A/B run over the cold loop, 120 iterations per arm:

Arm	Failures
-Verbose:$false (current)	8 / 120 (6.7%)
-Verbose (claimed workaround)	5 / 120 (4.2%)

The -Verbose arm still crashed five times with the identical NullReferenceException.
Two-proportion test gives p ≈ 0.39 — noise. Verbose output perturbs thread timing; it
synchronises nothing.

Reproducing it yourself

Self-contained, no extra tooling. Typically crashes within a few dozen iterations:

./build.ps1 -Task Build

$version  = (Import-PowerShellDataFile ./PowerShellBuild/PowerShellBuild.psd1).ModuleVersion
$target   = "./Output/PowerShellBuild/$version"
$settings = './PowerShellBuild/ScriptAnalyzerSettings.psd1'

foreach ($iteration in 1..400) {
    try {
        Invoke-ScriptAnalyzer -Path $target -Recurse -Verbose:$false -Settings $settings -ErrorAction Stop > $null
    } catch {
        "Crashed on iteration $iteration"
        $inner = $_.Exception
        while ($inner.InnerException) { $inner = $inner.InnerException }
        "  $($inner.GetType().Name): $($inner.Message)"
        ($inner.StackTrace -split [Environment]::NewLine) | Select-Object -First 4
        break
    }
}

Observed:

Crashed on iteration 20
  InvalidOperationException: Collection was modified; enumeration operation may not execute.
   at System.Collections.Generic.List`1.Enumerator.MoveNext()
   at Microsoft.PowerShell.Commands.ModuleCmdletBase.UpdateCommandCollection(...)
   at Microsoft.PowerShell.Commands.ModuleCmdletBase.LoadModuleManifest(...)
   at Microsoft.PowerShell.Commands.TestModuleManifestCommand.ProcessRecord()

Measured on Linux with PowerShell 7.6.4 and PSScriptAnalyzer 1.25.0. The mechanism is
engine-level rather than platform-specific, consistent with this hitting both
windows-latest and ubuntu-latest, but rates were not measured on Windows PowerShell 5.1.

Mitigation

Going with the retry, as agreed above. The Analyze task now retries the analyzer call up
to three times and rethrows on the last attempt.

The retry is deliberately unconditional rather than matched against the known crash
signatures. Those signatures are symptoms of an upstream race, not a fixed contract — four
distinct exception types turned up during this analysis alone — so matching on them would
let a new variant silently bypass the mitigation. The cost is that a genuine analyzer
failure takes three attempts before failing, which is a few seconds on a path already
headed for red.

Validated over 45 runs of the real Analyze task under the reproduction conditions: zero
build failures, with the crash occurring and being absorbed.

One thing this turned up

Test-PSBuildScriptAnalysis in the shipped module is exposed to the same race, but it
calls Invoke-ScriptAnalyzer without -ErrorAction Stop, so it behaves differently — it
does not fail. Filed separately as #147 rather than widening this issue.

---

Generated by Claude Code

[HeyItsGilbert]

@tablackburn you should see if you can create a fix for PSScriptAnalyzer. That would be nice.