Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
251 changes: 251 additions & 0 deletions .agents/skills/release-captain/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,251 @@
---
name: release-captain
description: Run the weekly OpenShell release captain checklist. Monday checks that the current pre-release train is qualified, triages qualification, compatibility, and security reports, and prepares the release coordination post. Tuesday selects the final qualified pre-release, tags its commit as stable after maintainer approval, verifies publication, and starts the next train. Trigger keywords - release captain, weekly release, Tuesday release, cut the release, release checklist, pre-release status, release train, start next train, stable release.
metadata:
internal: true
---

# Release Captain

OpenShell publishes a stable release every Tuesday from a qualified
pre-release ([RFC 0014](../../../rfc/0014-release-stability/README.md)). The
release captain keeps the week's pre-release train green, coordinates the last
merges, and cuts the release. `CI.md` describes the release workflows,
qualification profile, and scanner policies. This skill covers what the captain
does with them.

## Guardrails

Without explicit approval from the release captain or a maintainer for that
specific action, never:

- push, move, or delete a tag;
- dispatch, cancel, or rerun any workflow;
- merge, approve, or enqueue a PR;
- post in Slack or comment on GitHub;
- install OpenShell or change the local machine.

Ask for approval with the exact command and target commit SHA. Every other
step in this skill is read-only.

Never repoint a stable tag at a different commit, and never re-dispatch a
release while an earlier run for the same tag might still start. `Release Tag`
has no concurrency guard, so two runs race to publish. Don't delete `-pre.N`
tags; the RFC's cleanup step is not implemented and tag deletion is restricted.

## How the train works

- A train has one base version, such as `v0.1.4`, fixed by its `-pre.1` tag. A
patch train can't take breaking changes to Stable interfaces. Those wait for a
minor train.
- `Release Auto-Tag` runs Monday through Friday at 14:00 Europe/Zurich. If
`main` has commits since the latest `-pre.N`, it tags the next one and
dispatches `Release Tag` (about 45 minutes). It skips after the base version
gets a stable tag, and fails if the latest pre-release isn't an ancestor of
`main`.
- Pushing a `-pre.*` tag does not start `Release Tag`. A pre-release tagged by
hand must be dispatched.
- Pushing a stable `vX.Y.Z` tag starts `Release Tag`, which rebuilds and
requalifies the commit. Codex Security runs only for pre-releases, so the
stable release relies on the selected pre-release's security result. Always
tag the commit of a pre-release that passed.

## Setup

Run from an OpenShell checkout with an authenticated `gh` CLI. Fetch tags with
`--force` so cached tags match the remote:

```shell
export GH_REPO=NVIDIA/OpenShell
git fetch origin main --tags --force
PREV=$(git tag -l 'v*' --sort=-v:refname | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' | head -1)
echo "Latest stable: $PREV"
git tag -l 'v*' --sort=-v:refname | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+-pre\.[0-9]+$' | head -5
```

Find the `Release Tag` run for a pre-release. Dispatched runs use `main`, so
their head commit is the tagged commit:

```shell
TAG=vX.Y.Z-pre.N
gh run list --workflow release-tag.yml --event workflow_dispatch --commit "$(git rev-parse "$TAG^{commit}")"
```

If that commit has several runs, the one whose Codex Security job name contains
`($TAG)` is the match. That job appears only after the images build.

## Monday: pre-flight

Finish these on Monday so Tuesday is only the cut.

1. **Check the train.** Confirm that the latest `-pre.N` targets the version you
plan to ship. Check that the most recent `Release Auto-Tag` run succeeded
(today's runs at 14:00 Europe/Zurich):

```shell
gh run list --workflow release-auto-tag.yml --limit 3
```

If the next version has no `-pre.1`, start the train (Tuesday step 6).

2. **Read the qualification summary for the latest attempt.** The summary's
`tag` and `source_sha` must match the candidate, and `current_profile_passed`
must be `true`:

```shell
RUN=<run-id>
ATTEMPT=$(gh run view "$RUN" --json attempt --jq .attempt)
dir=$(mktemp -d)
gh run download "$RUN" -n "qualification-$TAG-run-$RUN-attempt-$ATTEMPT" -D "$dir/q"
jq '{tag, source_sha, current_profile_passed, suites}' "$dir/q/qualification-summary.json"
git rev-parse "$TAG^{commit}"
```

For any failed suite, read the log with `gh run view "$RUN" --log-failed`.
`CI.md` lists which scanners block. Use the `watch-github-actions` skill for
CI diagnosis.

3. **Triage the advisory compatibility review.** It never blocks publication,
and a green job doesn't mean it found no breaks:

```shell
gh run download "$RUN" -n "compatibility-review-run-$RUN-attempt-$ATTEMPT" -D "$dir/c"
jq '{status, verdict: .review.verdict}' "$dir/c/report.json"
cat "$dir/c/report.md"
```

Only `complete` with `no_unaddressed_breaks` is a clean review. A missing
report or any other result means there's no assessment. For each finding,
decide whether it's a real break, a false positive, or unclear:

- **Real break:** needs a fix or revert before Tuesday, or must wait for a
minor train.
- **Unclear:** ask an owner of that code.
- **False positive:** send it to the owner of
`.github/prompts/compatibility-review.md` so the prompt can be tuned.

4. **Check the dev canary.** `Release Canary` smoke-tests `main` dev artifacts.
It doesn't gate the tag, but treat a red canary as a release risk. Use the
`test-release-canary` skill to diagnose it:

```shell
gh run list --workflow release-canary.yml --limit 3
```

5. **Plan fixes.** A transient infrastructure failure can be rerun after
approval. A code defect needs a merged fix and a new pre-release.

6. **Draft the coordination post.** Release coordination happens in a thread
in the OpenShell CNCF Slack channel, started on Monday. Draft it for the
captain to post:

- the target version and current candidate (tag, commit, run link);
- qualification and compatibility status, with an owner for each blocker;
- ready but unmerged PRs that should make the release, in merge order;
- the merge cutoff in UTC and the captain's time zone. Set it before
Tuesday's 14:00 Europe/Zurich Auto-Tag run so the last merges make that
pre-release.

Candidate PRs:

```shell
gh pr list --base main --state open --search "review:approved" --json number,title,author --jq '.[] | "#\(.number) \(.title) (@\(.author.login))"'
```

## Tuesday: cut the release

1. **Select the final candidate.** After the cutoff and Tuesday's Auto-Tag run,
pick the newest `-pre.N` whose qualification passed, and repeat Monday steps
2 and 3 for it. If the newest pre-release failed, either merge a fix and
qualify another pre-release, or pick an earlier one that passed and confirm
that leaving out the later commits is acceptable.

2. **Request approval.** Present the candidate tag, commit SHA, qualification
run, compatibility assessment, and the commits since `$PREV`:

```shell
git log --oneline --no-merges "$PREV..$TAG"
```

3. **Tag the stable release.** After approval, check the candidate, then tag its
exact commit and push only that tag:

```shell
VERSION=vX.Y.Z
SHA=$(git rev-parse "$TAG^{commit}")
[[ "$VERSION" == "${TAG%-pre.*}" ]] || echo "version does not match the train"
[[ "$SHA" == "$(jq -r .source_sha "$dir/q/qualification-summary.json")" ]] || echo "commit does not match qualification"
git tag --points-at "$SHA" | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' && echo "commit already has a stable tag"
git tag "$VERSION" "$SHA"
git push origin "refs/tags/$VERSION"
```

Stop if any check prints a message.

4. **Watch the release.** The tag push starts `Release Tag`. It runs for about
45 minutes, so poll the status, or run the watch in the background:

```shell
gh run list --workflow release-tag.yml --branch "$VERSION" --limit 1
gh run watch <run-id> --exit-status
```

If no run appears within a few minutes, or the run stays queued, see
Recovery.

5. **Verify publication.** Confirm that the GitHub Release exists as a
non-pre-release with assets and generated notes, and that no job failed:

```shell
gh release view "$VERSION" --json name,isPrerelease,publishedAt,assets --jq '{name, isPrerelease, publishedAt, assets: (.assets | length)}'
gh run view <run-id> --json jobs --jq '.jobs[] | select(.conclusion != "success" and .conclusion != "skipped") | "\(.conclusion)\t\(.name)"'
```

`Trigger Wheel Publish` only starts the GitLab pipeline, so confirm that the
Python wheel reached PyPI separately.

6. **Start the next train.** Auto-Tag waits until the next `-pre.1` exists.
Agree with the maintainers whether the next train is patch (the default) or
minor (notable features or planned breaking changes; see the
[version selection supplement](../../../rfc/0014-release-stability/release-version-selection.md)).

Seed the train on a `main` commit that:

- comes after `$VERSION`;
- has green branch checks;
- has no stable tag of its own.

Release versioning prefers a stable tag over a pre-release tag on the same
commit, which would mislabel the packages. If nothing has merged since the
release, wait. After approval:

```shell
NEXT=vX.Y.Z-pre.1
NEXT_SHA=<main commit>
git merge-base --is-ancestor "$VERSION" "$NEXT_SHA" || echo "commit does not contain the release"
git tag --points-at "$NEXT_SHA" | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' && echo "commit already has a stable tag"
git tag "$NEXT" "$NEXT_SHA"
git push origin "refs/tags/$NEXT"
gh workflow run release-tag.yml --ref main -f tag="$NEXT"
```

7. **Close the loop.** Draft a reply to the coordination thread with the release
link and any PRs deferred to the next train.

## Recovery

| Symptom | Action |
|---|---|
| Transient infrastructure failure in a pre-release | After approval, run `gh run rerun <run-id> --failed`, then recheck the new attempt's summary. |
| Code defect in a pre-release | Merge the fix. Let Auto-Tag create the next `-pre.N`, or, after approval, tag one by hand and dispatch it. |
| Stable run stuck queued | Don't re-dispatch or delete the tag. Check for a GitHub Actions incident and ask a maintainer to cancel the run (`gh run cancel <run-id>`, or force-cancel through the API). Only after it shows `cancelled` or `failed`, rerun it after approval. |
| Stable run fails before the `Release` job, transient | Nothing was published. After approval, run `gh run rerun <run-id> --failed`. |
| Stable run fails before the `Release` job, code defect | Nothing was published, and Auto-Tag won't create pre-releases for a base version that has a stable tag. Ask the maintainers to choose: remove the unpublished tag (needs a ruleset bypass) and recut from a qualified fix, or skip to the next patch version. |
| Stable run fails after the `Release` job | The GitHub Release exists. Check which publish jobs succeeded. A rerun of an npm or Snap publish fails if that version already went out. Rerun only jobs that didn't publish, after approval. |

## Related

- `watch-github-actions`: general `gh run` monitoring and log retrieval.
- `test-release-canary`: diagnose dev artifact smoke-test failures.
- `CI.md`: release workflows, qualification profile, and scanner policies.
- `rfc/0014-release-stability/`: cadence, qualification, and version selection.
1 change: 1 addition & 0 deletions .agents/skills/sync-agent-infra/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ Run after adding, removing, or renaming a skill or crate; changing issue or PR c
| Inference providers or native model endpoints | `debug-inference`, `openshell-cli` |
| TUI behavior | `tui-development` |
| Release artifacts or smoke coverage | `test-release-canary` |
| Release cadence, tagging, or qualification workflows | `release-captain` |
| CI diagnostics | `watch-github-actions` |
| Gator supervision | `launch-openshell-gator` |
| SBOM workflow | `sbom` |
Expand Down
1 change: 1 addition & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,7 @@ Contributor and maintainer skills live in `.agents/skills/`. They are marked int
| Platform | `build-openshell-mxc-windows` | Maintain and validate the build-only x64 and ARM64 Windows MSVC lane |
| Documentation | `update-docs-from-commits` | Scan recent commits and draft doc updates for user-facing changes |
| Maintenance | `sync-agent-infra` | Detect and fix drift across agent-first infrastructure files |
| Maintenance | `release-captain` | Run the weekly Monday pre-flight and Tuesday stable release checklist |
| Reference | `sbom` | Generate SBOMs and resolve dependency licenses |

### Issue Workflow
Expand Down
Loading