fix(server): supervise background service updates

[t3dotgg]

A boot-service self-update currently edits and restarts t3code.service from inside that service's own cgroup. systemd can terminate the restart child during teardown, which makes the request process restore the old unit and leaves remote clients waiting for a server that never reached the requested version.

This moves activation into a fixed-name transient systemd user service with its own cgroup. The helper stages the candidate, performs a blocking restart, verifies the exact version through the public readiness endpoint, and restores and verifies the previous unit on failure. Targets validate their migration identities against the database before the known-good server stops, and clients now distinguish a verified rollback from a restart timeout.

Focused verification:

• vp test run apps/server/src/cloud/selfUpdate.test.ts apps/server/src/cloud/systemdSelfUpdate.test.ts apps/server/src/persistence/Migrations.test.ts packages/client-runtime/src/state/server.test.ts apps/server/src/bin.test.ts apps/server/src/cli/service.test.ts
• vp test run apps/server/src/persistence/Migrations apps/server/src/persistence/Migrations.test.ts
• vp run --filter t3 typecheck
• vp run --filter @t3tools/client-runtime typecheck
• vp run --filter t3 build:bundle

Built by gpt-5.6-sol with the Codex harness in T3 Code.

---

Note

High Risk
Changes critical server self-update, systemd unit lifecycle, and database migration gating for remote activation; failures are mitigated by staged plans, helper rollback, and readiness probes, but mis-handoff or lock-release edge cases could still leave operators debugging receipt files under self-update.

Overview
Boot-service self-updates no longer rewrite and restart t3code.service from inside that service’s cgroup. The running server stages a plan and hands activation to a transient t3code-self-update.service via systemd-run, which applies the candidate unit, verifies readiness, and rolls back on failure.

Preflight and readiness: Before stop, the target build runs a read-only database migration frontier check (service _validate-update). A new /.well-known/t3/ready endpoint blocks until the runtime has published lifecycle ready (not merely HTTP up). Hidden CLI commands _apply-update and _validate-update support the helper and preflight.

Client: The update flow waits for disconnect/backoff before accepting a ready event, then validates the resumed version (ServerUpdateWrongVersionError on verified rollback).

Migrations: validateMigrationIdentities and validateAutomaticUpdateMigrationFrontier reject unsafe automatic activation when schema frontiers disagree.

^{Reviewed by Cursor Bugbot for commit da8ebd3. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Delegate server self-update activation to a supervised systemd transient unit with rollback

• Replaces the in-process systemd unit rewrite-and-restart path in selfUpdate.ts with a staged plan file handed off to a transient systemd-run helper unit (systemdSelfUpdate.ts) that handles install, restart, readiness polling, and rollback with receipt files.
• Adds a database compatibility preflight (service _validate-update) that runs against the target build before any restart, preventing unsafe schema migrations from being applied automatically.
• Adds GET /.well-known/t3/ready to the server HTTP API, backed by a new awaitReady signal on ServerRuntimeStartup; the systemd helper polls this endpoint to verify the resumed server version matches the expected target.
• Improves the client-side update flow to wait for an explicit transport disconnect before proceeding, fail fast with ServerUpdateWrongVersionError if the resumed server version differs from the requested target, and surface ServerUpdateRestartTimeoutError if no backoff signal is observed within 30 seconds.
• Risk: the new flow requires systemd user manager support; non-boot-service and non-systemd paths still use the legacy respawn approach but now also run the DB preflight.

^{Macroscope summarized da8ebd3.}

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: bd161d71-9273-4196-99c2-ca1699eb4b30

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[macroscopeapp]

One convention issue found in the new apps/server/src/cloud/systemdSelfUpdate.ts error model. Everything else (namespace imports, Migrations.ts / client-runtime error classes, CLI-boundary layer provision) matches the conventions.

^{Posted via Macroscope — Effect Service Conventions}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want fixes drafted automatically? Bugbot Autofix can create code changes for findings. A team admin can enable Autofix in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 70c2420. Configure here.}

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR introduces a new supervised systemd self-update mechanism with automatic rollback capability, database migration validation, and a new readiness endpoint. The scope and complexity of these infrastructure changes—new CLI commands, systemd orchestration logic, and cross-package modifications—warrant human review.

^{You can customize Macroscope's approvability policy. Learn more.}

[t3dotgg]

[gpt-5.6-sol] RESPONDING ON BEHALF OF THEO

Follow-up audit feedback identified two rollback gaps, both addressed in cd25a3d:

• Activation now probes /.well-known/t3/ready, which waits for the command runtime and lifecycle ready publication before returning the exact server version. Merely binding HTTP no longer passes health verification.
• The target-artifact preflight now requires the database and target to have the exact same migration frontier. Automatic downgrades and updates that would mutate the database are rejected while the known-good server is still running.

The systemd lifecycle is now covered by an opt-in acceptance test using uniquely named disposable host/helper units. It passed both a real successful handoff and a real rollback where the candidate bound HTTP but withheld runtime readiness (2 passed, 68.75s). The live t3code.service and live T3 data were not touched, and all disposable units/files were cleaned up.

Focused unit tests, server/contracts typechecks, formatting, and targeted lint also pass.