Skip to content

feat: sync API specs from infra and belt with Copybara - #1564

Merged
mishushakov merged 15 commits into
mainfrom
fetch-spec-via-copybara
Jul 24, 2026
Merged

feat: sync API specs from infra and belt with Copybara#1564
mishushakov merged 15 commits into
mainfrom
fetch-spec-via-copybara

Conversation

@mishushakov

@mishushakov mishushakov commented Jul 20, 2026

Copy link
Copy Markdown
Member

The specs in spec/ were copied from their source repos by hand and had drifted ~2,400 lines behind infra, so they are now imported with Copybara (copy.bara.sky, run in a pinned Docker image by scripts/fetch-spec.sh): make codegen re-fetches them at the commits pinned in spec/infra-ref and spec/belt-ref before generating, and the generated-files CI check fails if the tracked copies don't match the pins. Regenerating from the current pins picks up the accumulated spec changes in the generated JS/Python clients (renamed request schemas, SandboxNetworkConfig, SandboxIam workload identity, FILE_TYPE_SYMLINK, access-token auth deprecation, volume path-metadata tweaks). The one handwritten SDK change follows from that: the public FileType enums gain a SYMLINK member (JS and both Python surfaces) so entries envd reports as symlinks show up in files.list() and getInfo()/get_info() instead of being silently skipped as unknown types. The custom spec/remove_extra_tags.py tag-filtering script is replaced by Redocly CLI's filter-in decorator (redocly.yaml), which produces identical generated JS output; a filter-out decorator additionally drops any operation or component schema the upstream specs mark x-not-implemented: true (currently the SOCKS5 SandboxEgressProxyConfig/egressProxy surface, which infra flagged as spec-only); each SDK's bundle now goes to its own gitignored spec/openapi_generated.<api>.yml instead of both pipelines overwriting one shared file; Python client models now list fields in spec order instead of alphabetical (mechanical reordering only — construct models with keyword args). Spec fetches try whatever GitHub token is available and fall back to the tracked copies with a warning (the public infra specs also fetch anonymously); in CI a short-lived belt-scoped token is minted from the org-wide Autofixer GitHub App (no new secrets), so fork PRs simply fall back for the belt spec; the CI workflows also cache the Copybara image alongside the codegen image, and the previously ignored CODEGEN_IMAGE env is honored by the Makefile.

Usage

# update the specs: bump a pin, then regenerate
echo <infra-commit-sha> > spec/infra-ref
make codegen

# fetch a single spec without regenerating
pnpm fetch:api-spec     # spec/openapi.yml from infra
pnpm fetch:envd-spec    # spec/envd/ from infra
pnpm fetch:volume-spec  # spec/openapi-volumecontent.yml from belt

# try the latest spec without touching the pin
E2B_INFRA_REF=main pnpm fetch:api-spec

# change which endpoint tags an SDK exposes
$EDITOR redocly.yaml && make codegen
// symlinks are now visible in the filesystem API (JS; same shape in Python)
const entries = await sandbox.files.list('/home/user')
const link = entries.find((e) => e.type === FileType.SYMLINK)
console.log(link?.symlinkTarget)

🤖 Generated with Claude Code

@changeset-bot

changeset-bot Bot commented Jul 20, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: e1c667a

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages
Name Type
e2b Minor
@e2b/python-sdk Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@cursor

cursor Bot commented Jul 20, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Large generated-client churn and new spec-sync CI can break codegen on forks or pin drift; symlink and IAM surface changes affect runtime SDK behavior.

Overview
OpenAPI/envd/volume specs are no longer edited in-repo; make codegen runs fetch-specs (Copybara in copy.bara.sky / scripts/fetch-spec.sh) at pins in spec/infra-ref and spec/belt-ref, with CI building a Copybara image and minting a belt-scoped GitHub App token on non-fork PRs. Tag filtering and x-not-implemented stripping move from remove_extra_tags.py to redocly.yaml, with per-SDK bundles openapi_generated.js-sdk.yml / openapi_generated.python-sdk.yml. Generated REST clients pick up named timeout/snapshot/refresh bodies, SandboxIam, network config, v2 template list, and related spec drift; SOCKS5 egress marked not-implemented is omitted from clients. JS and Python sandbox filesystem layers add FileType.SYMLINK mapping and a JS unit test; Python API models use spec field order instead of alphabetical.

Reviewed by Cursor Bugbot for commit e1c667a. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions

github-actions Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Package Artifacts

Built from 23e0162. Download artifacts from this workflow run.

JS SDK (e2b@2.35.4-fetch-spec-via-copybara.0):

npm install ./e2b-2.35.4-fetch-spec-via-copybara.0.tgz

CLI (@e2b/cli@2.15.1-fetch-spec-via-copybara.0):

npm install ./e2b-cli-2.15.1-fetch-spec-via-copybara.0.tgz

Python SDK (e2b==2.34.0+fetch.spec.via.copybara):

pip install ./e2b-2.34.0+fetch.spec.via.copybara-py3-none-any.whl

Comment thread packages/js-sdk/src/envd/filesystem/filesystem_pb.ts

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9602b8a2ec

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread scripts/fetch-spec.sh Outdated
Comment thread packages/js-sdk/src/api/schema.gen.ts Outdated
Comment thread scripts/fetch-spec.sh Outdated
Comment thread .github/workflows/generated_files.yml Outdated
Comment thread .github/workflows/generated_files.yml
Comment thread .github/workflows/generated_files.yml
Comment thread scripts/fetch-spec.sh Outdated

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit d253e48. Configure here.

Comment thread scripts/fetch-spec.sh Outdated
mishushakov and others added 11 commits July 24, 2026 14:51
The specs in spec/ were copied from e2b-dev/infra and e2b-dev/belt by
hand and had drifted far behind. They are now imported with Copybara
(copy.bara.sky, run in a pinned Docker image by scripts/fetch-spec.sh):
make codegen re-fetches them at the commits pinned in spec/infra-ref
and spec/belt-ref before generating, and the generated-files CI check
fails if the tracked copies do not match the pins. Updating a spec is
now just bumping the pin and re-running make codegen.

Regenerating from the current pins picks up the accumulated spec
changes (renamed request schemas, SandboxNetworkConfig,
FILE_TYPE_SYMLINK, access-token auth deprecation, volume path
metadata tweaks) in the generated JS and Python clients.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
spec/remove_extra_tags.py filtered the REST spec down to the tags each
SDK exposes before client generation. Redocly CLI's filter-in decorator
(applyTo: Operation) has identical semantics — drop operations without
an allowlisted tag, drop untagged operations, drop emptied paths — so
the per-SDK tag lists now live in redocly.yaml and the pipelines run
`redocly bundle <api>` instead. Verified: identical generated JS output;
Python client models now list fields in spec order instead of the
alphabetical order imposed by the script's yaml.dump (mechanical
reordering, no behavior change). pyyaml is no longer needed by codegen.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the BELT_READ_TOKEN secret requirement in the generated-files
check with a short-lived app token scoped to e2b-dev/belt with
contents: read, minted from the org-wide Autofixer app credentials.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cp -R only overlays the staged Copybara output onto spec/, so a file
deleted or renamed upstream would linger locally with green CI. Each
spec now declares the paths its upstream owns and fetch-spec.sh removes
them before copying, so upstream deletions surface as git deletions
while repo-owned files in spec/ stay untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fork PRs can't access the Autofixer App secret, so the belt token mint
now runs only for same-repo PRs; forks fall back to the default
read-only token for the public infra fetches and skip the
volume-content spec via the new SKIP_VOLUME_SPEC Makefile flag. Also
add redocly.yaml to the generated-files paths filter so tag-filtering
changes trigger the freshness check.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rename SKIP_VOLUME_SPEC to FETCH_PRIVATE_SPECS (default 1) so the flag
covers any spec sourced from a private repository, not just the
volume-content one.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Generalize the fork-PR special case: make fetch-specs tries every spec
with whatever GitHub token is available (the public infra specs also
fetch anonymously) and warns and falls back to the tracked copy in
spec/ when a fetch fails, instead of gating the belt fetch behind
FETCH_PRIVATE_SPECS. Direct pnpm fetch:*-spec invocations still fail
loudly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The minted token is used for every fetch-spec.sh run in make codegen,
not just the belt one. Public-repo git reads happen to accept
out-of-scope tokens, but scope the token to both repos explicitly
rather than relying on that behavior.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both SDK generate pipelines bundled the spec to the same
spec/openapi_generated.yml, so the file on disk after a full codegen was
whichever SDK ran last (the python view, which lacks the auth-tagged
endpoints the js-sdk keeps for the CLI's /teams calls). Each pipeline
now writes spec/openapi_generated.<api>.yml, which removes the latent
race under parallel make and lets the two filtered views be inspected
side by side. The outputs stay gitignored; the old filename remains
ignored so stale copies in existing checkouts don't show up as
untracked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fetch-spec.sh removed the upstream-owned paths from spec/ before copying
the Copybara staging output over, so a fetch that produced an incomplete
staging tree (or a copy that failed midway) could leave spec/ missing
tracked files while `make fetch-specs` claimed to fall back to them. The
script now verifies every expected target exists in the staging dir
before touching spec/, and swaps each target in with mv — a same-
filesystem rename — instead of one big cp, so there is no window where
spec/ is half-populated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The upstream-owned spec/ paths were listed twice: as origin globs in
copy.bara.sky and again as TARGETS in fetch-spec.sh, which staged the
Copybara output and implemented the replace semantics itself. Each
workflow now declares its owned paths with destination_files and the
script points --folder-dir straight at spec/, so Copybara's folder
destination does the delete-and-replace natively (only after a
successful fetch, preserving the no-wipe-on-failure guarantee) and the
script is back to resolving pins and auth. The .copybara/ staging dir
is gone along with its gitignore entry.

Verified: all three fetches produce byte-identical trees, a planted
stale file under an owned path is deleted while repo-owned files
survive, a failed fetch (bad ref) leaves spec/ untouched, and
anonymous fetching still works with no token available.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mishushakov
mishushakov force-pushed the fetch-spec-via-copybara branch from afa391b to 95c37df Compare July 24, 2026 12:52
@linear-code

linear-code Bot commented Jul 24, 2026

Copy link
Copy Markdown

SDK-272

mishushakov and others added 2 commits July 24, 2026 15:31
…ients

The upstream specs can mark endpoints and component schemas that exist
in the spec but are not implemented with x-not-implemented: true. The
Redocly filter-out decorator now removes those nodes before client
generation. It intentionally has no applyTo so the flag works on
schemas as well as operations; a kept operation that still references
a dropped schema loses that media-type entry instead of keeping a
dangling $ref. No-op for the current specs, which carry no such flags.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pulls the latest infra spec: new SandboxIam workload-identity schemas
on NewSandbox and a 409 response on the node-status endpoint. The
upstream spec also marks the SOCKS5 egress-proxy config
x-not-implemented, so the new filter-out decorator drops
SandboxEgressProxyConfig and the egressProxy properties from both
generated clients — the first real exercise of that filter.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Upstream wrapped the egressProxy $ref in allOf so x-not-implemented is
a schema keyword instead of an ignored $ref sibling (infra#3383).
Purely structural: the filter-out decorator already dropped the old
shape, so the generated clients are unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mishushakov
mishushakov enabled auto-merge (squash) July 24, 2026 13:55
@mishushakov
mishushakov disabled auto-merge July 24, 2026 13:55
The regenerated envd protobufs added FILE_TYPE_SYMLINK, but the
hand-written FileType mappings in the JS and Python SDKs only knew FILE
and DIRECTORY, so files.list() silently dropped symlink entries and
getInfo()/get_info() returned an undefined/None type for them.

Also pin @redocly/cli@2.39.0 as a js-sdk devDependency (matching
codegen.Dockerfile) so local generate runs can't drift from CI, and
drop the stale `spec` entry from pnpm-workspace.yaml left behind by
the spec/package.json removal.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mishushakov
mishushakov merged commit 4fcf7cb into main Jul 24, 2026
48 of 51 checks passed
@mishushakov
mishushakov deleted the fetch-spec-via-copybara branch July 24, 2026 14:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants