Skip to content

fix(sdk): preserve error context and cover tunnel policy updates - #108

Merged
rcyw merged 3 commits into
inclusionAI:developfrom
WenYuLuo:test/tunnel-block-issue107
Oct 20, 2026
Merged

rcyw merged 3 commits into
inclusionAI:developfrom
WenYuLuo:test/tunnel-block-issue107

Conversation

@WenYuLuo

@WenYuLuo WenYuLuo commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Preserve structured backend failures at the public AKernel boundary and keep reverse tunnels usable across network-policy updates. Package the matching ADX 0.1.0rc2 platform/Execd release, Environment SDK and source revision.

  • Expose optional service codes, retry/outcome guidance, request identities, HTTP status and scheduling diagnostics on BackendOperationError, retaining the original cause. Cover confirmed cancellation, ambiguous create outcomes and unstructured errors.
  • Exercise creation-time block, live block/clear/reblock and consecutive policy updates through the same public Sandbox/tunnel handle with default and custom ports. Require recovery within 10 seconds and retain command, file, PTY, published HTTP, outbound-denial and cleanup checks.
  • Pin both release archives by SHA-256, use adx-environment==0.1.0rc2, and align src/agent-dx with the artifact source. Package contracts reject mismatched builds, mutable URLs and invalid checksums.

Fixes #105
Fixes #107

Consolidates #106. The server/SDK fixes are merged in ADX #69.

Release provenance:

  • Source/tag: b208c40964934ba24e04e80136d9f077bddb44f5 / v0.1.0rc2, containing merged ADX fix(sdk): make sandbox cleanup synchronous and best-effort #69. Buildkite #173 passed AMD64/ARM64 native builds, immutable OBS release publication, images and Kubernetes L0/auth/cleanup. Its original PyPI step lacked credentials; SDK Buildkite #13 published the original tested wheel/sdist to PyPI and verified identical hashes.
  • Downloaded both architecture release/Execd archives and verified SHA-256, commit/build identity, ELF architecture and embedded SDK hashes against the public manifests. The PyPI wheel/sdist match the same candidate; AKernel pins the AMD64 archives.

Validation at 082f7dd6074a85866d0afcf9a1d146cf8b51e7b0:

  • RC2 artifacts are mirrored at agent-dx OBS. All 26 artifact SHA-256/size pairs and 7 manifest/index objects were verified publicly; the original source commit, build identity and hashes are preserved. Both Dockerfile pins use this bucket with unchanged checksums.
  • Full GitHub CI passed: Python 3.10–3.14, SDK lint/type checks, clean wheel/sdist installation, deployment helpers and release/deployment contracts. The package contract suite includes 18 passing tests.
  • The real all-in-one build log confirms RC2 Platform/Execd downloads from the new bucket. The runsc image passed 10 examples, 7 integration tests and all 30 standalone tests. One additional OCI/Nydus private-write test skipped because AKERNEL_TEST_IMAGE was unset. All 20 tunnel recovery probes completed within 10 seconds; maximum 3.942 seconds. Standalone teardown passed. Docker Hub publication is skipped for PRs.

Expose service codes, retry and outcome guidance, request identities and
optional diagnostics on BackendOperationError. Preserve the original
cause and keep missing context optional for existing ADX SDKs and local
failures.

Cover confirmed scheduling cancellation, unknown create outcomes and
unstructured errors, and document the public exception contract.

Signed-off-by: robbluo <luo1442@gmail.com>
Add the standalone network-tunnel-block group for creation-time block,
live block/clear/reblock and consecutive policy replacements using the
same public tunnel handle with default and custom listener ports.

Require recovery probes within ten seconds and retain command, file, PTY,
published HTTP, outbound denial and cleanup checks. Document asynchronous
tunnel recovery and the matching ADX server and SDK release requirements.

Signed-off-by: robbluo <luo1442@gmail.com>
@WenYuLuo
WenYuLuo force-pushed the test/tunnel-block-issue107 branch from 5fa4cee to e54943e Compare October 19, 2026 12:52
@WenYuLuo WenYuLuo changed the title test(network): cover reverse tunnels under block policies fix(sdk): preserve error context and cover tunnel policy updates Oct 19, 2026
openjiuwen-release-bot Bot pushed a commit to openJiuwen-ai/agent-dx that referenced this pull request Oct 19, 2026
[mirror] #69: fix(platform): preserve admission diagnostics and tunnel recovery

Created-by: openjiuwen-sync
Commit-by: robbluo
Merged-by: robbluo
Description: Paired: [GitHub #69](#69) ↔ [GitCode !65](https://gitcode.com/openJiuwen/agent-dx/merge_requests/65)

<!-- bot1-mirror-meta -->
github_pr: 69
head_repo: WenYuLuo/agent-dx
head_ref: fix/tunnel-block-acl-issue107
head_sha: bd0ec63
base_repo: openJiuwen-ai/agent-dx
base_ref: master
github_repo: openJiuwen-ai/agent-dx
base_branch: master
base_sha: 1e9ae5f
github_user: WenYuLuo
<!-- /bot1-mirror-meta -->

GitHub PR mirror for CodeArts CI.

<!-- bot1-source-body -->
Preserve public Environment behavior when admission expires or a network policy changes. This PR contains two focused commits:

1. **Scheduling diagnostics.** A queue request confirmed cancelled before assignment returns bounded placement diagnostics and `DEADLINE_EXCEEDED / never / not_started`. JSON and SSE preserve the prepared Environment identity; the SDK retains diagnostic metadata and service exception types. Local ownership claims remain tracked while Redis writes run outside the scheduling lock, preserving unknown outcomes during uncertain assignment. Confirmed cancellations enter the normal completed replay cache.
2. **Reverse-tunnel recovery.** Persist server-owned listener configuration, validate reserved environment keys and listener collisions, and retain stateful WS ingress during creation and live policy replacement. After a successful acknowledgement, the owning SDK retires the captured old WS transport and reuses its reconnect loop, session and resumable requests. Other clients/raw HTTP updates retain heartbeat recovery. Policy installation and tunnel readiness complete asynchronously.

Validation:

- Full Environment SDK: 300 passed plus 100 subtests; direct transport regressions and 12 CI harness tests passed. AKernel against the matching wheel: 301 tests (6 skipped), 35 benchmark tests, Ruff and mypy passed.
- Native Linux AMD64 joint regression at ADX `bd0ec631b457e4470557a3cf982fd84170e8d724` and AKernel `e54943e5ab740fa97a765057d6445cb4dbc6e228`: 30/30 standalone tests passed; SDK integration 7 passed and 1 OCI/Nydus image-isolation test skipped because `AKERNEL_TEST_IMAGE` was unset. The 12-group campaign took 163.842 seconds. All 20 tunnel recovery probes completed within 10 seconds, maximum 3.894 seconds. Additional public AKernel CPU/memory oversubscription probes passed in 1.097/1.039 seconds, preserving HTTP 504, `DEADLINE_EXCEEDED / never / not_started`, request/Environment identities, scheduling diagnostics, original causes and resource baselines. Redis and backend inventory were clean; the task container and isolated build mount were removed.
- The unchanged Rust tree passed Linux six-crate tests (414 passed, 98 ignored), workspace/all-targets/all-features strict Clippy and rustfmt. The unchanged API fixture passed 32 Redis/mTLS/HTTPS api-control tests.
- Documentation consistency and diff checks passed. Added prose follows each document's existing language. The supplied Python static-analysis findings were fixed and locally rechecked; platform CodeCheck passed on the preceding code-identical revision `17fe441`. The current documentation-only revision's platform result is pending.

Scheduling diagnostics describe the current snapshot; per-resource maxima may belong to different workers, and scaling can change placement. Application requests may need retry while tunnels recover, and existing connection-failure backoff remains in effect.

This validates a source candidate on one standalone runsc node, with all nine ADX binaries rebuilt together and the matching SDK installed. All 1220 ADX and 297 AKernel tracked source files, and all 21 installed ADX SDK package files, were hash-verified. The fixture used sandboxd 0.1.0 and runsc `release-20260817.0-akernel.3-2-g7fdef444b108`. The candidate SDK retains source metadata 0.1.0rc1 and differs from the published artifact. Shipping requires a new immutable coherent ADX binary/SDK release, downstream pin updates and the full repository-image CI E2E. Kubernetes and multi-node deployment are separate gates.

Consolidates [#55](#55). Addresses [AKernel #105](inclusionAI/AKernel#105) and [#107](inclusionAI/AKernel#107). The companion public error adapter and maintained tunnel regressions are in [AKernel #108](inclusionAI/AKernel#108), which remains Draft until the release/pinned-image gates pass.
<!-- /bot1-source-body -->

See merge request: openJiuwen/agent-dx!65
@WenYuLuo
WenYuLuo marked this pull request as ready for review October 20, 2026 02:42
Consume the matching platform and Execd archives from the dedicated
agent-dx OBS bucket with fixed SHA-256 pins, advance the Environment SDK
dependency, and align the ADX source gitlink. This supplies the admission
diagnostics and tunnel recovery fixes exercised by the public AKernel
regression suite.

Accept versioned release URLs in the package contracts while retaining
matching-build and checksum validation. Update the dependency guides.

Signed-off-by: robbluo <luo1442@gmail.com>
@WenYuLuo
WenYuLuo force-pushed the test/tunnel-block-issue107 branch from 322ec94 to 082f7dd Compare October 20, 2026 03:11
@rcyw

rcyw commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator

Review of 082f7dd6074a85866d0afcf9a1d146cf8b51e7b0: I found one P1 to resolve before merging the ADX 0.1.0rc2 dependency update. The targeted fixes for #105 and #107 look consistent with their intended contracts; the finding concerns another behavior brought in by the dependency update.

P1: ordinary tenant callers can request arbitrary writable Local mounts

The previous ADX revision (af7d82f04a52d4604ea7ca4c550f09c6114dee4d) explicitly rejected public host_path mounts. In the new pinned revision, contract.rs:465-477 accepts an absolute path after lexical validation and turns it into StorageSource::Local, preserving caller-supplied mount options. This helper has no caller context. The public create route does not restrict Local mounts to administrators or enforce a tenant-specific source directory before preparing the request.

Offline reproduction: I built a small probe against this exact ADX revision and called the real public create_spec with CallerContext { tenant_id: "review-tenant-a", administrator: false }. All three source strings below were accepted as Local mounts, with ["rw"] and the caller's tenant preserved:

  • /
  • /mnt/sfs/workspaces/review-tenant-b
  • /var/lib/review-only-not-created

The parser-only input shape was:

{
  "image": "review-fixture:unused",
  "mounts": [
    {"type": "bind", "host_path": "/", "target": "/review-mount", "options": ["rw"]}
  ]
}

Verification boundary: the probe performed contract conversion only. It did not read, create, or mount any of these paths, pull an image, start a sandbox, or access a cluster. This confirms request acceptance, rather than claiming a live filesystem-access exploit.

Source inspection establishes the downstream consequence: ADXlet forwards Local sources and options directly as sandboxd HostPath mounts, and the sandboxd revision pinned by AKernel writes them into the OCI mount specification. Keeping the correct tenant_id on the environment does not authorize its mount source. An ordinary tenant can therefore request writable access to paths visible to the node runtime, including other tenants' storage where present. SDK-side type restrictions do not protect the raw HTTP API.

The new ADX Local mount contract explicitly leaves per-tenant path authorization to the deployment and requires restricting this capability to trusted management callers. AKernel's current public creation path does not implement that boundary.

Requested change: restore default rejection of Local mounts on public creation, or add explicit server-side management authorization and source-path/tenant restrictions before enabling them. Add negative coverage for non-admin callers, both host_path/hostPath spellings, and writable mounts. If the correction requires repackaging ADX, publish a new immutable version and align the platform/Execd archives, checksums, SDK dependency, and gitlink rather than replacing rc2 artifacts.

Checks completed

  • AKernel SDK: 301 unit tests and 35 benchmark-harness tests passed; Ruff and mypy passed.
  • Published adx-environment==0.1.0rc2: 10 targeted tests passed; its 21 package source/typing files match the pinned source. Six additional offline JSON/SSE probes through the real AKernel constructor confirmed public error fields/cause and terminal-versus-unknown retry behavior.
  • ADX Rust: 414 passed, 0 failed, 98 ignored; 97 ignored tests require isolated Redis/related services and one is a release benchmark.
  • Runtime/Helm/Terraform contracts: 18/27/3 passed. Both archives at the new agent-dx OBS URLs were downloaded and verified against the pinned SHA-256 values and embedded release metadata. Latest-head CI, including Standalone E2E, passed.

These checks support the #105/#107 fixes but do not cover the missing Local-source authorization. No product code, cloud resources, or cluster deployment was changed during this review.

@WenYuLuo

Copy link
Copy Markdown
Collaborator Author

@rcyw Thanks for the review. We confirmed that AKernel's public SDK at 082f7dd6074a85866d0afcf9a1d146cf8b51e7b0 already exposes only read-only Image/S3 mounts; it has no Local/HostPath source, writable mount options, or create_if_missing. We will keep Local/HostPath out of the AKernel SDK for now.

The server-side finding remains valid: raw HTTP requests can bypass those SDK restrictions. We are tracking the authorization correction separately in #110, which will remain open as follow-up work. Its acceptance criteria include the public/internal trust boundary, negative authorization coverage, immutable ADX release/dependency updates if needed, and joint E2E regression. This scope decision does not mean the server-side gap is fixed or mitigated by the SDK restriction; #108 will not close #110.

@rcyw rcyw left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving reviewed head 082f7dd6074a85866d0afcf9a1d146cf8b51e7b0 for integration into develop, following the explicit decision to accept the scoped #105/#107 fixes and track the Local/HostPath authorization correction separately in #110.

The P1 finding in the earlier review remains valid: raw HTTP callers can bypass the public SDK's Image/S3-only mount surface. The SDK restriction does not resolve or mitigate that service-side gap. #110 must stay open until the server-side authorization correction and its regression evidence are complete.

The reviewed source is unchanged, the relevant local checks passed with the previously documented Redis-dependent exclusions, and all current CI checks passed, including Standalone E2E. This approval records acceptance of the documented scope for develop integration; the unresolved Local mount authorization remains tracked as follow-up work.

@rcyw
rcyw merged commit 4e3dad0 into inclusionAI:develop Oct 20, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants