[Zacgoose]

This pull request improves task cancellation and recovery behavior in the orchestration system, enhances diagnostics for blocked PowerShell workers, and updates dependencies. The main changes ensure that tasks left running by a cancelled or stopped process are properly retried and finished, and that logs now include detailed information about where a PowerShell worker is blocked. Several new tests have been added to verify these behaviors.

Orchestration cancellation and recovery improvements:

• Modified RunTaskAsync in OrchestratorService.cs to accept an attempt parameter and only finish as "Cancelled" if it is the first attempt, allowing retried tasks left running by a cancelled process to be executed again.
• Updated WorkPump.cs so that if a cancelled run has no pending tasks left to cancel, any running task left by a stopped process is reclaimed and run again, ensuring the run can finish properly.

PowerShell worker diagnostics:

• Enhanced the Abandon method in PowerShellWorker.cs to log the script call stack of a blocked pipeline, providing better visibility into where the worker is stuck. Added the BlockedCallStack helper method for this purpose. [1] [2]

Testing improvements:

• Added new tests in HungWorkerRecycleTests.cs, OrchestrationAzuriteTests.cs, and OrchestrationRecoveryTests.cs to verify correct behavior when cancelling runs with live tasks, reclaiming tasks from stopped processes, and logging of blocked statements. [1] [2] [3]
• Refactored test helpers to allow injection of a custom logger for better testability of log output. [1] [2]

Dependency updates:

• Updated the System.Security.Cryptography.Xml package to version 8.0.5 in Craft.csproj.