Add SDK-owned blob purge service contract and shared task integration - #805
wangbill (YunchuWang) wants to merge 21 commits into
Conversation
Reuse the existing purge tasks and bootstrap sequence with a task-hub-bound transport interface. Preserve standalone behavior and prove DTFx shim execution, failure handling, replay, and per-hub setup. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Reuse the existing native gRPC client for setup instead of adding an alternate bootstrap overload. Remove the unused setting facade and client shim tests while retaining actual DTFx purge task execution coverage. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Document normal function indexing, language-worker task execution, and the bound local-host transport. Preserve the existing SDK implementation and explicit bootstrap without claiming Functions end-to-end validation. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Clarify normal compiled-library Function indexing and opt-in package registration without a metadata transformer or generator. Preserve the base extension behavior and unchanged shared SDK tasks. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Describe explicit registration, normal function metadata and bindings without naming an unreleased package or prescribing its metadata implementation. Leave shared task APIs and bootstrap unchanged. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Describe the optional feature package's normal build-time Function discovery and generated invocation. Preserve shared SDK task APIs, explicit bootstrap behavior, and standalone operation. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Restore the constants file to its upstream scope after verifying final consumers use only the public activities and transport facade. Trim integration documentation to the current isolated Functions path. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Preserve the interface package identity and canonical SDK Client model references with an independent version. Add focused API and signing tests, Apache license attribution, solution discovery, and existing release signing coverage. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add the preserved package ID to the existing independent NuGet publication jobs, retaining the manual approval gate and symbol exclusion. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Explain the interface-only netstandard2.0 target alongside the matching Client model library. Use a typed xUnit assertion before validating activity deadlines. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Preserve both interface APIs and SDK model identities while removing the Blob implementation dependency from transport consumers. Add ownership and dependency regression coverage and package the existing Apache and MIT notices together. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use an extension-bearing package path so NuGet places the linked MIT license at the documented package root rather than treating it as a directory. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep the repository license linked without duplicating it and document the actual NuGet directory layout. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
halspang
left a comment
There was a problem hiding this comment.
A few questions and some concerns about the license. It seems like we did a lot of custom release stuff for this package. Is there a reason we didn't follow the same path as something like the ExportHistory which also releases under its own version but doesn't pull in its own licensing.
Normalize the package and assembly identity, inherit shared fetch/report operations, construct gRPC adapters in named worker factories, and retain normal SDK signing and release conventions. Align verified task-original interface and tests with the SDK MIT convention after scoped authorization. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Package licensing is inconsistent and publication does not enforce prerequisite package availability.
Review effort: Balanced
Findings: 3
Open (4)
What changed in this PR
Adds an independently versioned large-payload purge contract package for alternate host integrations.
Changes:
- Introduces service and transport interfaces using canonical SDK models.
- Exposes purge activities and adds a gRPC transport adapter.
- Adds integration tests, documentation, packaging, signing, and publication wiring.
| File | Description |
|---|---|
test/LargePayloadPurge.Abstractions.Tests/LargePayloadPurgeContractTests.cs |
Verifies contract API, identity, dependencies, and signing. |
test/LargePayloadPurge.Abstractions.Tests/LargePayloadPurge.Abstractions.Tests.csproj |
Defines the contract test project. |
test/Extensions/AzureBlobPayloads.Tests/PayloadStore/BlobPayloadStoreDeleteResponseTests.cs |
Uses the new gRPC adapter. |
test/Extensions/AzureBlobPayloads.Tests/AutoPurge/PurgeActivityDeadlineTests.cs |
Updates deadline tests for the adapter. |
test/Extensions/AzureBlobPayloads.Tests/AutoPurge/PurgeActivityBackendStatusTests.cs |
Updates status-handling tests. |
test/Extensions/AzureBlobPayloads.Tests/AutoPurge/AlternateHostPurgeTaskTests.cs |
Tests alternate-host execution and replay. |
src/LargePayloadPurge.Abstractions/RELEASENOTES.md |
Adds initial package release notes. |
src/LargePayloadPurge.Abstractions/README.md |
Documents contracts and dependencies. |
src/LargePayloadPurge.Abstractions/LargePayloadPurge.Abstractions.csproj |
Defines the contract package. |
src/LargePayloadPurge.Abstractions/IOrchestrationServiceLargePayloadPurgeClient.cs |
Adds the service capability interface. |
src/LargePayloadPurge.Abstractions/ILargePayloadPurgeClient.cs |
Adds the activity transport interface. |
src/Extensions/AzureBlobPayloads/DependencyInjection/DurableTaskWorkerBuilderExtensions.AzureBlobPayloads.cs |
Registers adapter-backed activities. |
src/Extensions/AzureBlobPayloads/AzureBlobPayloads.csproj |
References the contract project. |
src/Extensions/AzureBlobPayloads/AutoPurge/Client/GrpcLargePayloadPurgeClient.cs |
Implements the transport adapter. |
src/Extensions/AzureBlobPayloads/AutoPurge/Activities/ReportLargePayloadPurgeResultsActivity.cs |
Makes reporting publicly constructible. |
src/Extensions/AzureBlobPayloads/AutoPurge/Activities/GetLargePayloadTombstonesActivity.cs |
Makes fetching publicly constructible. |
README.md |
Documents isolated Functions integration. |
Microsoft.DurableTask.sln |
Adds contract and test projects. |
list-nuget-packages-links.ps1 |
Adds the package to generated links. |
eng/publish/publish.yml |
Adds a publication job. |
doc/release_process.md |
Documents independent release handling. |
CHANGELOG.md |
Records the new package and integration. |
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
Require successful approval, Abstractions publication and Client publication before publishing the contract. Document conservative skipping on prerequisite failure without changing independent package jobs. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Preserve both blob purge integration and upstream scheduler authentication documentation and release notes. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Microsoft.DurableTask.Extensions.AzureBlobPayloads depends on Microsoft.DurableTask.LargePayloadPurge.Abstractions at every target framework, but its publish job was only gated on approval. Depend on the contract publish job too, with succeeded() so a failed, skipped, or canceled contract prerequisite skips Blob instead of running against an unpublished dependency. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
IOrchestrationServiceLargePayloadPurgeClient was left in the DurableTask.LargePayloadPurge namespace, a Core-prototype leftover, while the shared transport contract it inherits (ILargePayloadPurgeClient) lives in Microsoft.DurableTask.AzureBlobPayloads. Move the service interface into that same namespace so both contracts share one namespace; this is a source-breaking move for the unreleased prototype, not a compatibility-preserving one, so no type forwarder is added. Remove the matching Core-prototype RootNamespace overrides in the contract and its test project so both fall back to the repo's standard Microsoft.DurableTask.* naming convention. Update the contract tests, README, and root README for the new namespace and full type name, and document the required cross-repository release order (Core, then this repo's SDK/contract/Blob, then Durable Functions, then the private provider) without inventing or bumping any committed package version. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Both ILargePayloadPurgeClient and IOrchestrationServiceLargePayloadPurgeClient moved to Microsoft.DurableTask.LargePayloadPurge.Abstractions, matching the contracts package's own name, instead of reusing Microsoft.DurableTask.AzureBlobPayloads -- the Azure Blob extension's own implementation namespace (BlobPurgeJobOrchestrator, the generated-client adapter, the purge activities) in a separate assembly. The contracts package is explicitly backend-neutral and does not depend on blob storage, gRPC, or worker implementations, so it should not appear to be owned by one specific backend's namespace. This moves the transport interface's namespace for the first time since it was split out of the Blob implementation; there is no type forwarder for either prior location since these are unreleased prototype types. The Blob project's activities and generated-client adapter now reference the contracts namespace explicitly via using, the same as any other consumer. Updated contract tests to assert the new namespace/full names and to negatively assert all three now-prior full names (the Core-prototype one and both AzureBlobPayloads ones) are absent. Updated root and package README accordingly. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Blob auto-purge: staged merge and release orderThis PR is stage 2 of the cross-repository integration. Review can proceed in parallel, but merge/release must follow the dependency order:
The SDK pipeline gates Client/Abstractions -> contracts -> Blob. Release operators must additionally verify the entire actual package dependency closure, including Grpc/Worker; those existing jobs are not all ordered by the new gate. Gate: completing four consecutive merges is not sufficient. Each upstream package must actually be published and its dependency closure restorable before the next consumer is merged. Local validation artifacts or matching version numbers alone are not release-readiness evidence. Keep auto-purge explicitly disabled until the compatible worker/provider/backend/storage rollout is verified. |
Remove LargePayloadPurge.Abstractions' independent VersionPrefix/VersionSuffix override so it inherits eng/targets/Release.props exactly like Client, Blob, and every other Microsoft.DurableTask.* package, rather than carrying its own 0.1.0 version. Rename the contract's signing test to assert equality with Client's assembly version instead of a hardcoded 0.1.0.0, and update the root/package README and release-process docs to describe the shared version and the existing Prepare Release workflow instead of an independent 0.1 tag. Add a regression test proving the real field-by-field mapping in GrpcLargePayloadPurgeClient: a non-empty, order-sensitive, opaque-token-like fetch response and a captured report request, asserted against the real adapter rather than a mock of ILargePayloadPurgeClient, so a swapped or dropped field/row or a constant-mapped disposition would fail. Copilot-Session: 883b4cbd-e93c-4d4e-8cb8-ffcf69525cfa Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>



Summary
What changed?
Microsoft.DurableTask.LargePayloadPurge.Abstractions, using the SDK's standard package and assembly naming, MIT license metadata, signing and approval-gated publication conventions.Microsoft.DurableTask.LargePayloadPurge.Abstractions.ILargePayloadPurgeClient.Microsoft.DurableTask.LargePayloadPurge.Abstractions.IOrchestrationServiceLargePayloadPurgeClientinherits it and adds only the explicit setting operation.Why is this change needed?
Issues / work items
Project checklist
release_notes.mdNo breaking change for released SDK packages: the new contracts and activity constructors are additive
relative to main. Canonical SDK models, serialization, protobuf definitions, explicit bootstrap, orchestration
and blob deletion behavior are unchanged.
Breaking Change (unreleased prototypes only)
Microsoft.DurableTask.LargePayloadPurge.Abstractions. The old prototype usedMicrosoft.Azure.DurableTask.LargePayloadPurge.Abstractionsand assemblyDurableTask.LargePayloadPurge.Abstractions. The provider contract now inherits the shared fetch/report interface rather than duplicating its operations. Both contract interfaces now share the package-ownedMicrosoft.DurableTask.LargePayloadPurge.Abstractionsnamespace, rather than the Core or Blob prototype namespaces.Microsoft.DurableTask.LargePayloadPurge.Abstractionsand rebuild SDK Blob, Functions, and provider consumers; Blob implementation types keep their original namespace. Replace providerGetLargePayloadsToPurgeAsyncimplementations/calls withGetLargePayloadTombstonesAsync, returningTask<List<LargePayloadTombstone>>. Report uses the inherited shared signature. No compatibility forwarders are added for unreleased binaries.The worker transport interface retains its exact fetch/report signatures and optional cancellation defaults.
The service contract declares only
SetLargePayloadAutoPurgeAsync(bool, DateTime, CancellationToken), whichdoes not schedule or stop a purge runner. The existing setting-only client infrastructure interface is unchanged.
AI-assisted code disclosure (required)
Was an AI tool used? (select one)
If AI was used:
AI verification (required if AI was used):
These first-person attestations remain for the human author.
Testing
Automated tests
dotnet test test\LargePayloadPurge.Abstractions.Tests\LargePayloadPurge.Abstractions.Tests.csprojdotnet test test\Extensions\AzureBlobPayloads.Tests\AzureBlobPayloads.Tests.csproj --filter "FullyQualifiedName~AutoPurge|FullyQualifiedName~DependencyInjection|FullyQualifiedName~BlobPayloadStoreDeleteResponseTests"Manual validation (only if runtime/behavior changed)
Notes for reviewers
eng/targets/Release.props, like Client and the Blob implementation; the normal Prepare Release workflow updates it automatically. Its assembly version follows the canonical SDK Client version and the signing key is unchanged. The standard signing pattern covers it without an exceptional job.RELEASENOTES.mdis consumed into NuGet metadata byRelease.targets; the root changelog is the persistent repository history. This package now uses the normal shared SDK release tag/link, not a separate 0.1.x release train.October 5 namespace alignment
Microsoft.DurableTask.LargePayloadPurge.Abstractions; both unreleased interface type names moved out of the Blob implementation namespace. Its package/assembly identity, SDK signing key, method signatures/defaults, Client model identities, and protobuf protocol are unchanged. Old prototype consumers must update imports and rebuild.Final namespace decision: the contracts own
Microsoft.DurableTask.LargePayloadPurge.Abstractions; Blob classes, stores, and activities ownMicrosoft.DurableTask.AzureBlobPayloadsand explicitly import the contracts. No prior interface alias or type forwarder is supplied. The coordinated namespace follow-up is not a production release or a fix for the private provider's separate dependency-pin/deadline review blockers.October 5 shared-version and adapter regression follow-up