fix(extensions,presets,workflows): resolve private GHES release assets via /api/v3 by HeroSizy · Pull Request #3157 · github/spec-kit

HeroSizy · 2026-06-25T07:58:01Z

Description

Fixes private GitHub Enterprise Server (GHES) release-asset downloads for specify extension add, specify preset add, and specify workflow add. Closes #3147 (approach proposed and discussed in #3147 (comment)).

Background. Catalog fetch from a private GHES host already works today via the opt-in ~/.specify/auth.json mechanism (#2393) — that was simply undocumented, so the issue assumed catalogs had to be public. The real gap is the release-asset download.

Root cause. Release-asset downloads first translate a browser release URL (https://<host>/<owner>/<repo>/releases/download/<tag>/<asset>) into the REST API asset URL and add Accept: application/octet-stream; otherwise a private repo's browser URL redirects to an SSO/HTML page instead of the binary. That translation lives in one shared helper, _github_http.resolve_github_release_asset_api_url, which hardcoded github.com/api.github.com and returned None for any GHES host — so for GHES the URL was never translated and the header never set, and the download was corrupt even with a valid token.

What this does.

Adds github_provider_hosts() — enumerates the hosts a user has listed under a github provider in auth.json.
Generalizes the resolver to build GHES REST API URLs ({scheme}://{host[:port]}/api/v3/...) for those hosts. Public github.com handling is unchanged (github_hosts=() reproduces prior behavior byte-for-byte).
Threads the host allowlist into all five call sites of the shared resolver (the extension/preset catalog wrappers, preset add --from, and both workflow add paths).
Documents the GHES auth.json recipe in docs/reference/authentication.md.

Design notes.

No new config surface — no env vars, no catalog-schema fields, no CLI flags. The same auth.json entry supplies the token and classifies the host as GitHub Enterprise.
Security: the auth.json allowlist is the anti-SSRF gate — only hosts the user explicitly trusts locally get /api/v3 treatment, so a remote catalog can never induce an API request to an arbitrary host. The host-classification list is injected into _github_http (which imports nothing from the auth layer), keeping that module's dependency boundary intact.

Scope: the issue's GH_HOST / GH_ENTERPRISE_TOKEN "zero-config" idea was intentionally not implemented — auth.json already carries the token and preserves the file-based opt-in model; reusing env vars would auto-send credentials based on environment alone. Easy to add later if there's demand.

Testing

Tested locally with uv run specify --help (and exercised the real specify preset add --from CLI in the manual test below)
Ran existing tests — full suite via .venv/bin/python -m pytest tests -q (the repo's recommended form over bare uv run pytest, per CONTRIBUTING/AGENTS.md). New: 6 resolver unit tests, github_provider_hosts() unit tests, and CLI/integration tests for every wired caller.
Tested with a sample project (manual smoke below)

Manual test results

Agent: Claude Code (Opus 4.8) | OS/Shell: macOS / zsh

Command tested	Notes
`specify preset add --from <private GHES release URL>`	PASS — verified against a real private GitHub Enterprise Server. Browser release URL resolved to `…/api/v3/repos/…/releases/assets/<id>`, authenticated with the `auth.json` bearer token, downloaded with `Accept: application/octet-stream`, preset installed (exit 0).
Private GHES extension release asset (`ExtensionCatalog` resolver path)	PASS (real GHES). Resolved via `/api/v3` and authenticated-downloaded a valid extension archive (verified valid zip + contents).
Negative — no `auth.json` (local mock)	PASS (gate holds). Host not classified as GHES → no `/api/v3` attempted → request 401s → command exits non-zero.
`specify extension add` · `specify preset add <id>` · `specify workflow add`	Same shared resolver path; additionally covered by automated `CliRunner` integration tests (GHES resolution + `Accept: application/octet-stream`).

Note on the suite: all new tests pass. Three failures present in my environment are pre-existing and unrelated to this branch: test_get_pack_info and test_default_active_catalogs come from a developer-global preset catalog leaking into tests that don't isolate ~/.specify (they pass once that global registry is moved aside); test_timestamp_branches…[go AI now] fails on pristine main as well. Happy to file separate issues for the test-isolation gaps.

AI Disclosure

I did not use AI assistance for this contribution
I did use AI assistance (describe below)

This change was developed with Claude Code (Anthropic), and the loop was AI-driven, disclosed as such:

Design: spec-driven, with me making the scope/approach decisions (e.g. rejecting the env-var and catalog-schema alternatives, choosing how far to wire the fix).
Implementation and the automated tests were performed by AI subagents (Claude Sonnet) under my direction; an AI code-review pass (Claude Opus) then found and fixed a real gap (3 unwired call sites).
I personally ran the manual end-to-end validation against my own private GHES instance — resolution plus authenticated download of real preset and extension release assets, and a full preset add --from install — and confirmed the results.
I reviewed the design, the scope decisions, and the results at each step; I have not hand-edited every line.
Every commit carries an Assisted-by: Claude Code (model: …, autonomous) trailer, and the issue comment proposing this approach was likewise AI-drafted and disclosed.

Happy to walk through any part in more detail.

Copilot

Pull request overview

This PR fixes authenticated downloads of private GitHub Enterprise Server (GHES) release assets across specify extension add, specify preset add, and specify workflow add by translating browser-style release download URLs into GHES REST API asset URLs under /api/v3 (with Accept: application/octet-stream), gated by the user’s opt-in ~/.specify/auth.json host allowlist.

Changes:

Add github_provider_hosts() to derive the GHES host allowlist from auth.json github provider entries.
Generalize resolve_github_release_asset_api_url() to support GHES REST API URL construction ({scheme}://{host[:port]}/api/v3/...) while preserving existing github.com behavior.
Wire GHES host allowlisting into all resolver call sites and add unit + CLI/integration tests, plus authentication docs updates.

Show a summary per file

File	Description
`src/specify_cli/authentication/http.py`	Adds `github_provider_hosts()` to expose configured `github` provider hosts for GHES classification.
`src/specify_cli/_github_http.py`	Extends release-asset URL resolution to support GHES `/api/v3` with an allowlisted host gate.
`src/specify_cli/extensions/__init__.py`	Threads `github_provider_hosts()` into extension catalog asset URL resolution.
`src/specify_cli/presets/__init__.py`	Threads `github_provider_hosts()` into preset catalog asset URL resolution.
`src/specify_cli/presets/_commands.py`	Ensures `preset add --from` uses GHES-aware resolver + octet-stream download behavior.
`src/specify_cli/__init__.py`	Ensures both `workflow add <url>` and `workflow add <id>` paths use GHES-aware resolver + octet-stream downloads.
`tests/test_github_http.py`	Adds resolver unit tests for GHES behavior, scheme/port preservation, and allowlist gating.
`tests/test_authentication.py`	Adds tests verifying `github_provider_hosts()` behavior.
`tests/test_presets.py`	Adds CLI-level GHES download test and end-to-end wiring test for preset catalog wrapper.
`tests/test_extensions.py`	Adds end-to-end wiring test for extension catalog wrapper GHES resolution.
`tests/test_workflows.py`	Adds CLI-level tests covering GHES release URL resolution for workflow add (URL + catalog paths).
`docs/reference/authentication.md`	Documents the GHES `auth.json` recipe and explains why the bare host must be listed.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 12/12 changed files
Comments generated: 1

mnriem · 2026-06-25T12:34:49Z

Please address Copilot feedback

Addresses Copilot review on PR github#3157: drop unnecessary __import__("io") in test_preset_add_from_ghes_release_url_resolves_via_api_v3 since io is already imported at module level.

…auth.json Assisted-by: Claude Code (model: claude-sonnet-4-6, autonomous)

Generalizes resolve_github_release_asset_api_url to GitHub Enterprise Server hosts (gated by auth.json github hosts), fixing private GHES extension/preset downloads. github#3147 Assisted-by: Claude Code (model: claude-sonnet-4-6, autonomous)

…olver Assisted-by: Claude Code (model: claude-sonnet-4-6, autonomous)

Assisted-by: Claude Code (model: claude-sonnet-4-6, autonomous)

…lease resolvers Wires preset add --from and workflow add through github_provider_hosts() so private GHES release assets resolve via /api/v3 there too. github#3147 Assisted-by: Claude Code (model: claude-sonnet-4-6, autonomous)

Addresses Copilot review on PR github#3157: drop unnecessary __import__("io") in test_preset_add_from_ghes_release_url_resolves_via_api_v3 since io is already imported at module level.

Copilot

Copilot's findings

Files reviewed: 12/12 changed files
Comments generated: 1

mnriem · 2026-06-25T14:17:02Z

Please address Copilot feedback

Addresses Copilot review on PR github#3157. A direct GHES /api/v3 release asset URL was only returned as already-resolved when its host was in the allowlist; otherwise the resolver returned None and the caller downloaded the same URL without 'Accept: application/octet-stream', fetching JSON metadata instead of the binary. Gate the passthrough on path shape alone, mirroring the github.com case. This is safe: passthrough returns the input URL unchanged and the caller fetches it either way, so no new request to an arbitrary host is induced; the token stays independently gated by auth.json in open_url. The allowlist remains the anti-SSRF gate on the tag-lookup resolving path. Add test_passthrough_for_unlisted_ghes_api_asset_url.

Copilot

Copilot's findings

Files reviewed: 12/12 changed files
Comments generated: 0 new

mnriem · 2026-06-25T15:44:37Z

Thank you!

Merge github/spec-kit 0.11.8 -> 0.11.9 (15 commits). Key upstream changes: - fix(extensions): tell agent to run mandatory hooks (github#2901) - fix(scripts): keep PowerShell branch-name acronym case-sensitive (github#3129) - fix(extensions,presets,workflows): resolve GHES private release assets (github#3157) - fix(catalog): companion docs, version-pinned URL, refreshed tags (github#2954) - Add community bundle submission path (github#3162) - Docs: document missing flags --force and --refresh-shared-infra (github#3179) Fork customizations preserved: - Phase A/B + Mission Brief in core specify.md - spec.* command aliases, orange theming - Bundled extensions/presets, enhanced hook dispatcher - Bundled-extension preferred update path (install_from_directory) - Git extension opt-in (aligned with upstream) Fork bug fixes: - Relative imports in extensions/ and presets/ __init__.py - registered.append guard in agents.py - Bundled update now uses install_from_directory, not catalog download

…wnloads Extends the GHES support pattern from extensions and presets (github#2855, github#3157) to the bundle manifest download path: resolve_github_release_asset_api_url now receives github_hosts=github_provider_hosts() so browser release URLs from GitHub Enterprise Server instances are resolved via /api/v3 rather than falling back to the unauthenticated download path. Also adds a contract test covering the GHES resolution path for _download_remote_manifest (analogous to the existing github.com tests). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…nloads (#3136) * fix: resolve GitHub release asset API URL for private repo bundle downloads For private/SSO-protected GitHub repos, browser release download URLs (https://github.com/<owner>/<repo>/releases/download/<tag>/<asset>) redirect to an HTML/SSO page instead of delivering the asset, causing bundle manifest downloads to fail. Extends the pattern from #2855 (presets/workflows) to cover the bundle manifest download path in _download_remote_manifest: - Resolves browser release URLs to GitHub REST API asset URLs via resolve_github_release_asset_api_url before downloading - Direct REST API asset URLs (api.github.com/repos/.../releases/assets/<id>) are passed through directly - Both cases use Accept: application/octet-stream so the API returns the binary payload rather than JSON metadata - The original catalog URL is used to determine artifact format (.zip vs YAML) since the resolved API URL does not carry the file extension Adds two CLI-level contract tests: - bundle info resolves browser release URL via GitHub tags API - bundle info passes direct API asset URL through with octet-stream Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: detect ZIP payload by magic bytes; add zip and API-asset tests Address Copilot review feedback on PR #3136: 1. Detect ZIP payloads by magic bytes (PK\x03\x04) in addition to the '.zip' URL suffix so that direct GitHub REST asset URLs — which carry no file extension — are correctly routed through the ZIP extraction path when the asset is a ZIP bundle artifact. 2. Add two new contract tests: - test_bundle_info_resolves_github_browser_release_url_zip: exercises the '.zip' browser release URL path end-to-end, verifying the tags API lookup fires, octet-stream header is used, and bundle.yml is successfully extracted from the ZIP payload. - test_bundle_info_api_asset_url_zip_detected_by_magic_bytes: verifies that a direct REST asset URL returning ZIP bytes is detected by magic and parsed correctly without a tags API call. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: improve error message, broaden ZIP magic, drop unused tmp_path Address second-round Copilot review feedback on PR #3136: - Error message: when the download fails, report the original catalog download_url so the user knows which entry to fix; include the resolved REST API URL when it differs for easier debugging. - ZIP detection: broaden the magic-bytes check from PK\x03\x04 to raw[:2] == b"PK", covering all valid ZIP variants (local-file header PK\x03\x04, empty-archive PK\x05\x06, spanned/split PK\x07\x08). - Tests: remove the unused tmp_path parameter from test_bundle_info_resolves_github_browser_release_url_zip. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: use full 4-byte ZIP signatures instead of 2-byte PK prefix Address Copilot feedback: raw[:2] == b"PK" is too broad and could misclassify any payload starting with ASCII "PK" as a ZIP, producing a confusing "not a valid bundle" error. Use the three specific 4-byte ZIP magic signatures instead: PK\x03\x04 — local file header (standard ZIP) PK\x05\x06 — end-of-central-directory (empty archive) PK\x07\x08 — data descriptor / spanning marker Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: harden _download_remote_manifest parsing and tighten tests - Promote _ZIP_SIGNATURES to module-level constant (was redefined per call) - Use PurePosixPath for URL path suffix extraction so query strings and fragments are ignored and URL paths are treated as POSIX on all OSes - Move yaml/BundleManifest imports to function top to flatten the previously nested try/except into a single handler with explicit except _yaml.YAMLError and except Exception clauses - Re-add None guard on _local_manifest_source return: the function is typed Optional[BundleManifest] and without the guard a None return propagates silently to callers that degrade gracefully rather than raising an actionable error; comment explains it is defensive not dead - Assert exact resolved asset URL in browser-URL download tests, not just the Accept header, so a regression where download uses the original URL instead of the resolved one would be caught - Add resolution-failure test: when tags API finds no matching asset the code falls back to the original URL and exits non-zero with Error: Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(bundle): pass github_provider_hosts() for GHES private release downloads Extends the GHES support pattern from extensions and presets (#2855, #3157) to the bundle manifest download path: resolve_github_release_asset_api_url now receives github_hosts=github_provider_hosts() so browser release URLs from GitHub Enterprise Server instances are resolved via /api/v3 rather than falling back to the unauthenticated download path. Also adds a contract test covering the GHES resolution path for _download_remote_manifest (analogous to the existing github.com tests). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test(bundle): remove unused ghes_entry variable from GHES contract test The dict was defined but never consumed — the test drives GHES host recognition entirely through the github_provider_hosts() patch. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(bundle): include source URL in remote manifest parse errors Thread the catalog URL (and resolved API URL when it differs) into the YAML parse, generic parse, and ZIP-extraction error paths of _download_remote_manifest so failures point at the offending source instead of an opaque temp path. Addresses PR review feedback. Assisted-by: GitHub Copilot (model: Claude Opus 4.8, autonomous) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Manfred Riem <15701806+mnriem@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

HeroSizy force-pushed the fix/3147-ghes-release-asset-download branch from 282b7ba to e2852e0 Compare June 25, 2026 08:02

HeroSizy marked this pull request as ready for review June 25, 2026 08:10

HeroSizy requested a review from mnriem as a code owner June 25, 2026 08:10

HeroSizy mentioned this pull request Jun 25, 2026

[Feature]: Support authentication for private catalogs and extensions #3147

Closed

2 tasks

mnriem requested a review from Copilot June 25, 2026 11:55

Copilot started reviewing on behalf of mnriem June 25, 2026 11:56 View session

Copilot AI reviewed Jun 25, 2026

View reviewed changes

Comment thread tests/test_presets.py Outdated

HeroSizy added 6 commits June 25, 2026 22:06

feat(auth): add github_provider_hosts() to enumerate GHES hosts from …

5e94111

…auth.json Assisted-by: Claude Code (model: claude-sonnet-4-6, autonomous)

fix(extensions,presets): pass auth.json github hosts into release res…

e7829b0

…olver Assisted-by: Claude Code (model: claude-sonnet-4-6, autonomous)

docs(auth): document GHES private catalog + release-asset auth

70fa6b6

Assisted-by: Claude Code (model: claude-sonnet-4-6, autonomous)

test(presets): use module-level io.BytesIO in GHES preset test

76c15ba

Addresses Copilot review on PR github#3157: drop unnecessary __import__("io") in test_preset_add_from_ghes_release_url_resolves_via_api_v3 since io is already imported at module level.

HeroSizy force-pushed the fix/3147-ghes-release-asset-download branch from 364be42 to 76c15ba Compare June 25, 2026 14:06

mnriem requested a review from Copilot June 25, 2026 14:10

Copilot started reviewing on behalf of mnriem June 25, 2026 14:11 View session

Copilot AI reviewed Jun 25, 2026

View reviewed changes

Comment thread src/specify_cli/_github_http.py Outdated

mnriem requested a review from Copilot June 25, 2026 15:14

Copilot started reviewing on behalf of mnriem June 25, 2026 15:15 View session

Copilot AI reviewed Jun 25, 2026

View reviewed changes

mnriem merged commit 1add203 into github:main Jun 25, 2026
12 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(extensions,presets,workflows): resolve private GHES release assets via /api/v3#3157

fix(extensions,presets,workflows): resolve private GHES release assets via /api/v3#3157
mnriem merged 7 commits into
github:mainfrom
HeroSizy:fix/3147-ghes-release-asset-download

HeroSizy commented Jun 25, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

mnriem commented Jun 25, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

mnriem commented Jun 25, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

mnriem commented Jun 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

HeroSizy commented Jun 25, 2026

Description

Testing

Manual test results

AI Disclosure

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

Uh oh!

mnriem commented Jun 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot's findings

Uh oh!

Uh oh!

mnriem commented Jun 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot's findings

Uh oh!

Uh oh!

mnriem commented Jun 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants