diff --git a/LifeOS/install/LIFEOS/DOCUMENTATION/Config/ConfigSystem.md b/LifeOS/install/LIFEOS/DOCUMENTATION/Config/ConfigSystem.md index 3f259cffcb..b3c54cd49c 100644 --- a/LifeOS/install/LIFEOS/DOCUMENTATION/Config/ConfigSystem.md +++ b/LifeOS/install/LIFEOS/DOCUMENTATION/Config/ConfigSystem.md @@ -1,5 +1,5 @@ --- -version: 1.2.5 +version: 1.2.6 --- # LifeOS Configuration @@ -81,6 +81,12 @@ The two trees are physically separate git repos: **Pre-flight refuses to proceed if the public LifeOS repo appears in either remote** — the workflow is explicitly for the two PRIVATE repos only. Public LifeOS release goes through the separate shadow-release pipeline with the 14 ShadowRelease gates. +### The unattended path (`BackupUserData.ts`) + +The `pre-push` hook above only fires when the principal pushes the SYSTEM tree, and it presupposes `~/.claude` is itself a git repo, which not every install has. `LIFEOS/TOOLS/BackupUserData.ts` is the trigger-independent half: a scheduled job that commits and pushes the USER tree whenever it changed, and no-ops when it didn't. It carries the same boundary gates in tool form (remote confirmed `PRIVATE` on every run, failing closed on any other answer; distribution-repo remotes rejected; staged files scanned for secrets before commit; post-push HEAD comparison) plus an exclusive lock so it can't race itself or a manual push. One JSONL line per run lands in `MEMORY/OBSERVABILITY/user-backup.jsonl`. + +**Scheduling is Linux-only.** The tool is platform-neutral, but the shipped schedule is a Linux user crontab and that is the only scheduler it has been verified on. On macOS, wrap the same command in a `launchd` agent instead. Full contract, restore procedure, and known limits: `LIFEOS/TOOLS/BackupUserData.README.md`. + ## Public releases The Shadow Release system (`skills/_LIFEOS/Tools/ShadowRelease.ts`) produces public staging at `~/.claude/LIFEOS/LIFEOS_RELEASES/{VERSION}/.claude/` via **containment** — clone the live tree, delete sensitive zones (USER, MEMORY, private underscore-prefixed skills), overlay fixed public templates from `skills/_LIFEOS/RELEASE_TEMPLATES/` (including `CLAUDE.public.md` + `settings.public.json`), run 14 gates (G1–G14: zone deletion, identity grep, CF ID grep, trufflehog, .env strays, private tokens, ref integrity, private-skill refs, username-path leak, staging boot, dashboard leak, template-only USER/MEMORY, hidden-file leakage, critical-artifact presence). Write `.shadow-state.json` report. `EmitSkill.ts` then reshapes this `.claude/` staging into the shippable `{VERSION}/LifeOS/` skill (and drops the staging clone) — the published distribution unit is that one self-contained skill, not the `.claude/` tree. Staging is isolated from `~/Projects/LIFEOS/`; public publish is a separate explicit step. diff --git a/LifeOS/install/LIFEOS/TOOLS/BackupUserData.README.md b/LifeOS/install/LIFEOS/TOOLS/BackupUserData.README.md new file mode 100644 index 0000000000..eaeb1e3563 --- /dev/null +++ b/LifeOS/install/LIFEOS/TOOLS/BackupUserData.README.md @@ -0,0 +1,119 @@ +# BackupUserData — hourly USER-tree backup to a private GitHub repo + +> Companion to `LIFEOS/TOOLS/BackupUserData.ts`. This is the recurring half of the +> two-repo sync specified in `LIFEOS/DOCUMENTATION/Config/ConfigSystem.md`. + +## What it backs up, and what it doesn't + +The USER tree — `~/.config/LIFEOS/USER`, mounted into the live tree as the symlink +`~/.claude/LIFEOS/USER` — is the principal's life-state: identity, DA identity, TELOS, +projects, operational rules, Conduit history, work config. It is the half of LifeOS that is +*yours*, and the half `BackupBeforeUpgrade.sh` deliberately does **not** capture (that +script archives the SYSTEM tree and stores `LIFEOS/USER` as a bare symlink entry). + +Excluded by `.gitignore` in the USER tree: + +- **Secrets** — `.env*`, `*.pem`, `*.key`, key material, `CONFIG/CREDENTIALS/*`. Credentials + live in `~/.claude/.env` by design and are never mirrored, private repo or not. +- **Regenerable caches** — `CACHE/`, `*-cache.json`, `*-cache.txt`. +- **Noise** — `node_modules/`, editor swap files. + +## How it runs + +An hourly user crontab entry (`17 * * * *`) invokes the tool. It checks for changes, +commits them, and pushes. When nothing changed it exits `0` with `status=noop` and creates +no commit, so the repo history is a record of real change, not of the clock. + +> **Scheduling is Linux-only.** The tool itself is platform-neutral: it shells out to `git` +> and the GitHub CLI and nothing else. The *schedule* below is a Linux user crontab, and +> that is the only scheduler this has been verified on. On macOS, cron is deprecated in +> favour of `launchd`; wrap the same command in a `launchd` agent (see +> `com.lifeos.pulse.plist` for the shape LifeOS already uses) rather than assuming the +> crontab path works. + +``` +SHELL=/bin/sh +PATH=/usr/bin:/bin +LIFEOS_GH_BIN= +17 * * * * /BackupUserData.ts >/dev/null 2>>/user-backup.err.log +``` + +Absolute paths are required: cron's environment has neither the login `PATH` nor the shell +profile. `LIFEOS_GH_BIN` exists because the GitHub CLI may be a wrapper rather than a plain +`gh` on `PATH` (e.g. when LifeOS runs as a system user distinct from the principal's own +account, bridging to the principal's `gh` config via `GH_CONFIG_DIR`). + +Every run appends one JSON line to `LIFEOS/MEMORY/OBSERVABILITY/user-backup.jsonl` with a +`status` of `pushed`, `noop`, `locked`, or a failure code. That file is the log to read when +asking "is my life-state actually backed up?". + +## The four refusals + +The tool fails closed. Each refusal is a non-zero exit with a logged reason, never a +silent skip: + +1. **Non-private remote.** Every run re-checks the remote's visibility via `gh repo view + --json visibility,isPrivate` and pushes only on a confirmed `PRIVATE`. `PUBLIC`, an API + error, an unparseable answer, or a missing `gh` all block the push. Privacy is verified + per-run rather than once at setup because a repo can be flipped public at any time. +2. **The LifeOS distribution repo.** A remote whose repo is named `LifeOS` (any case) or + whose owner is the public upstream is rejected outright. USER data never goes near the + public release path — that path is the shadow-release pipeline and its 14 gates. +3. **Secret material.** Staged files are scanned for private-key blocks, GitHub / OpenAI / + Anthropic / Slack / Google / AWS credentials, and JWTs before the commit is created. A + hit unstages everything and aborts. +4. **A concurrent run.** An exclusive lock at `.git/lifeos-backup.lock` means two overlapping + invocations produce one commit, not a race. Locks older than 30 minutes are treated as + stale and stolen. + +## Manual use + +```bash +bun BackupUserData.ts # the hourly path — commit + push if changed +bun BackupUserData.ts --dry-run # stage and scan, commit nothing +bun BackupUserData.ts --check-remote # test the distribution-repo guard +bun BackupUserData.ts --check-visibility # test the privacy gate against any repo +``` + +`LIFEOS_USER_DIR` overrides the USER-tree location; otherwise the tool resolves the +`~/.claude/LIFEOS/USER` symlink and falls back to `~/.config/LIFEOS/USER`. + +## Restore + +The mirror is a plain git repo — restore is a clone, not a tool: + +```bash +# 1. Stop the live service and quiesce any Claude Code session (same order as a +# BackupBeforeUpgrade restore — hooks write into this tree continuously). +sudo systemctl stop lifeos-pulse.service + +# 2. Clone the mirror to the USER-tree location. +git clone ~/.config/LIFEOS/USER + +# 3. Confirm the live tree's symlink resolves into it. +ls -la ~/.claude/LIFEOS/USER + +# 4. Re-source anything the mirror deliberately never carried: ~/.claude/.env, +# CONFIG/CREDENTIALS/*, and the CACHE/ tree (which regenerates on its own). + +# 5. Restart. +sudo systemctl start lifeos-pulse.service +``` + +Two things a restore does **not** give you: the SYSTEM tree (that's +`BackupBeforeUpgrade.sh`) and any secret (deliberately never captured, by the same doctrine +that keeps secrets out of the SYSTEM archive). + +## Known limits + +- **Torn writes.** `git add -A` snapshots whatever is on disk at that instant. A hook + mid-write into `CONDUIT/` can be captured half-written. Acceptable for a mirror — the next + hourly run captures the settled file — but it means a restored tree is + crash-consistent, not transaction-consistent. +- **Push rejection is not auto-resolved.** If the remote has commits the local tree lacks + (a second machine, or a web edit), the push fails loudly and the run exits non-zero rather + than rebasing unattended. Resolve it by hand; the next hourly run then proceeds. +- **The documented `pre-push` trigger is not installed.** `ConfigSystem.md` describes + `~/.claude/.git/hooks/pre-push` auto-pushing the USER tree before each SYSTEM-tree push. + That requires `~/.claude` to be a git repo. Where it isn't, cron is the only trigger and + this tool is the whole mechanism. diff --git a/LifeOS/install/LIFEOS/TOOLS/BackupUserData.ts b/LifeOS/install/LIFEOS/TOOLS/BackupUserData.ts new file mode 100644 index 0000000000..79805a261d --- /dev/null +++ b/LifeOS/install/LIFEOS/TOOLS/BackupUserData.ts @@ -0,0 +1,278 @@ +#!/usr/bin/env bun +/** + * BackupUserData.ts — mirror the USER tree to its private GitHub repo. + * + * The USER tree (`~/.config/LIFEOS/USER`, mounted as `~/.claude/LIFEOS/USER`) is the + * principal's life-state: identity, TELOS, projects, operational rules, Conduit history. + * `Config/ConfigSystem.md` § Two-repo sync specifies it as its own PRIVATE GitHub repo. + * This tool is the recurring half of that mechanism: it checks for changes, commits them, + * and pushes — designed to run unattended from cron. + * + * Guarantees (each is an ISC in MEMORY/WORK/20260727-083500_user-data-github-backup/ISA.md): + * - Never pushes to a remote it cannot confirm is PRIVATE (fails closed). + * - Never pushes to the public LifeOS repo. + * - Never commits recognizable secret material. + * - Never runs twice concurrently (exclusive lock). + * - No-ops silently when nothing changed. + * + * Usage: + * bun BackupUserData.ts # the hourly path + * bun BackupUserData.ts --dry-run # stage + scan, no commit/push + * bun BackupUserData.ts --check-remote # test the public-repo guard + * bun BackupUserData.ts --check-visibility # test the privacy gate + * + * Environment: + * LIFEOS_USER_DIR override the USER tree location + * LIFEOS_GH_BIN GitHub CLI binary/wrapper to use for the privacy check (default: gh) + */ + +import { appendFileSync, closeSync, existsSync, mkdirSync, openSync, readFileSync, realpathSync, statSync, unlinkSync, writeSync } from "fs" +import { dirname, join } from "path" +import { homedir } from "os" + +const HOME = process.env.HOME ?? homedir() +const GH_BIN = process.env.LIFEOS_GH_BIN ?? "gh" +const LOG_PATH = join(HOME, ".claude", "LIFEOS", "MEMORY", "OBSERVABILITY", "user-backup.jsonl") +const LOCK_STALE_MS = 30 * 60 * 1000 +const MAX_SCAN_BYTES = 1_000_000 + +/** Patterns that must never reach the mirror, private repo or not. */ +const SECRET_PATTERNS: Array<[string, RegExp]> = [ + ["private-key-block", /-----BEGIN (?:RSA |EC |OPENSSH |PGP )?PRIVATE KEY-----/], + ["github-token", /\bgh[pousr]_[A-Za-z0-9]{30,}/], + ["openai-key", /\bsk-[A-Za-z0-9_-]{32,}/], + ["anthropic-key", /\bsk-ant-[A-Za-z0-9_-]{32,}/], + ["slack-token", /\bxox[abposr]-[A-Za-z0-9-]{10,}/], + ["google-key", /\bAIza[0-9A-Za-z_-]{35}/], + ["aws-key-id", /\bAKIA[0-9A-Z]{16}\b/], + ["jwt", /\beyJ[A-Za-z0-9_-]{10,}\.eyJ[A-Za-z0-9_-]{10,}\./], +] + +type RunResult = { code: number; stdout: string; stderr: string } + +function run(cmd: string[], cwd?: string): RunResult { + const p = Bun.spawnSync(cmd, { cwd, stdout: "pipe", stderr: "pipe" }) + return { + code: p.exitCode ?? 1, + stdout: new TextDecoder().decode(p.stdout).trim(), + stderr: new TextDecoder().decode(p.stderr).trim(), + } +} + +function log(event: Record): void { + const line = JSON.stringify({ ts: new Date().toISOString(), tool: "BackupUserData", ...event }) + try { + mkdirSync(dirname(LOG_PATH), { recursive: true }) + appendFileSync(LOG_PATH, line + "\n") + } catch { + /* logging must never be the reason a backup fails */ + } + console.log(line) +} + +function die(status: string, message: string, extra: Record = {}): never { + log({ status, message, ...extra }) + process.exit(1) +} + +/** The USER tree, following the symlink the live tree mounts it through. */ +function resolveUserDir(): string { + const override = process.env.LIFEOS_USER_DIR + if (override) return override + const mounted = join(HOME, ".claude", "LIFEOS", "USER") + try { + return realpathSync(mounted) + } catch { + return join(HOME, ".config", "LIFEOS", "USER") + } +} + +/** + * Reject remotes that are, or could be, the public LifeOS repo. The mirror is for the two + * private repos only; public release goes through the shadow-release pipeline instead. + */ +function remoteIsForbidden(url: string): { forbidden: boolean; reason?: string } { + const slug = url + .replace(/\.git$/, "") + .replace(/^git@[^:]+:/, "") + .replace(/^(?:https?|ssh):\/\/[^/]+\//, "") + const [owner, repo] = slug.split("/") + if (!owner || !repo) return { forbidden: true, reason: `cannot parse owner/repo from remote "${url}"` } + if (/^lifeos$/i.test(repo)) return { forbidden: true, reason: `remote repo is named "${repo}" — that is the LifeOS distribution repo, not a USER-data mirror` } + if (/^danielmiessler$/i.test(owner)) return { forbidden: true, reason: `remote owner "${owner}" is the public LifeOS upstream` } + return { forbidden: false } +} + +function remoteSlug(url: string): string { + return url + .replace(/\.git$/, "") + .replace(/^git@[^:]+:/, "") + .replace(/^(?:https?|ssh):\/\/[^/]+\//, "") +} + +/** Fails closed: anything other than a confirmed PRIVATE answer blocks the push. */ +function assertPrivate(slug: string): { ok: boolean; visibility: string; detail?: string } { + const res = run([GH_BIN, "repo", "view", slug, "--json", "visibility,isPrivate"]) + if (res.code !== 0) return { ok: false, visibility: "unknown", detail: res.stderr || res.stdout } + try { + const parsed = JSON.parse(res.stdout) as { visibility?: string; isPrivate?: boolean } + const visibility = parsed.visibility ?? "unknown" + return { ok: parsed.isPrivate === true && visibility.toUpperCase() === "PRIVATE", visibility } + } catch { + return { ok: false, visibility: "unparseable", detail: res.stdout.slice(0, 200) } + } +} + +function scanStagedForSecrets(userDir: string): Array<{ file: string; pattern: string }> { + const staged = run(["git", "diff", "--cached", "--name-only", "--diff-filter=ACMR"], userDir) + if (staged.code !== 0 || !staged.stdout) return [] + const hits: Array<{ file: string; pattern: string }> = [] + for (const rel of staged.stdout.split("\n").filter(Boolean)) { + const abs = join(userDir, rel) + try { + if (statSync(abs).size > MAX_SCAN_BYTES) continue + const text = readFileSync(abs, "utf-8") + for (const [name, re] of SECRET_PATTERNS) { + if (re.test(text)) hits.push({ file: rel, pattern: name }) + } + } catch { + /* unreadable or binary — nothing to scan */ + } + } + return hits +} + +function acquireLock(userDir: string): string { + const lockPath = join(userDir, ".git", "lifeos-backup.lock") + for (let attempt = 0; attempt < 2; attempt++) { + try { + const fd = openSync(lockPath, "wx") + writeSync(fd, `${process.pid} ${new Date().toISOString()}\n`) + closeSync(fd) + return lockPath + } catch { + let ageMs = 0 + try { + ageMs = Date.now() - statSync(lockPath).mtimeMs + } catch { + continue // lock vanished between attempts — retry + } + if (ageMs > LOCK_STALE_MS) { + try { + unlinkSync(lockPath) + } catch { + /* someone else cleaned it */ + } + continue + } + log({ status: "locked", message: `another backup is running (lock age ${Math.round(ageMs / 1000)}s)` }) + process.exit(0) + } + } + die("lock-failed", "could not acquire the backup lock") +} + +function main(): void { + const args = process.argv.slice(2) + + // --check-remote: exercise the public-repo guard without touching the repo. + const remoteIdx = args.indexOf("--check-remote") + if (remoteIdx !== -1) { + const url = args[remoteIdx + 1] ?? "" + const verdict = remoteIsForbidden(url) + console.log(JSON.stringify({ url, ...verdict })) + process.exit(verdict.forbidden ? 1 : 0) + } + + // --check-visibility: exercise the privacy gate against an arbitrary repo. + const visIdx = args.indexOf("--check-visibility") + if (visIdx !== -1) { + const slug = args[visIdx + 1] ?? "" + const verdict = assertPrivate(slug) + console.log(JSON.stringify({ slug, ...verdict })) + process.exit(verdict.ok ? 0 : 1) + } + + const dryRun = args.includes("--dry-run") + const userDir = resolveUserDir() + + if (!existsSync(join(userDir, ".git"))) die("not-a-repo", `${userDir} is not a git repository`) + + const remote = run(["git", "remote", "get-url", "origin"], userDir) + if (remote.code !== 0) die("no-remote", `no "origin" remote in ${userDir}`, { detail: remote.stderr }) + + const forbidden = remoteIsForbidden(remote.stdout) + if (forbidden.forbidden) die("forbidden-remote", forbidden.reason ?? "remote rejected") + + const slug = remoteSlug(remote.stdout) + const privacy = assertPrivate(slug) + if (!privacy.ok) { + die("privacy-unconfirmed", `refusing to push: ${slug} is not confirmed PRIVATE (visibility=${privacy.visibility})`, { + detail: privacy.detail, + }) + } + + const lockPath = acquireLock(userDir) + try { + const branch = run(["git", "symbolic-ref", "--short", "HEAD"], userDir).stdout || "main" + const dirty = run(["git", "status", "--porcelain"], userDir) + if (dirty.code !== 0) die("status-failed", "git status failed", { detail: dirty.stderr }) + + // Unpushed commits from a previously failed push still need a push, even with a clean tree. + const ahead = run(["git", "rev-list", "--count", `origin/${branch}..HEAD`], userDir) + const unpushed = ahead.code === 0 ? parseInt(ahead.stdout || "0", 10) : 0 + + if (!dirty.stdout && unpushed === 0) { + log({ status: "noop", message: "no changes", repo: slug, branch }) + return + } + + let committed = 0 + if (dirty.stdout) { + const add = run(["git", "add", "-A"], userDir) + if (add.code !== 0) die("add-failed", "git add failed", { detail: add.stderr }) + + const secrets = scanStagedForSecrets(userDir) + if (secrets.length > 0) { + run(["git", "reset"], userDir) + die("secret-detected", `refusing to commit: ${secrets.length} secret pattern hit(s)`, { hits: secrets }) + } + + const files = run(["git", "diff", "--cached", "--name-only"], userDir).stdout.split("\n").filter(Boolean) + committed = files.length + if (dryRun) { + log({ status: "dry-run", message: `${committed} file(s) would be committed`, repo: slug, branch, files: files.slice(0, 20) }) + run(["git", "reset"], userDir) + return + } + + const stamp = new Date().toISOString().replace(/\.\d{3}Z$/, "Z") + const commit = run(["git", "commit", "-m", `backup: USER tree ${stamp} (${committed} file${committed === 1 ? "" : "s"})`], userDir) + if (commit.code !== 0) die("commit-failed", "git commit failed", { detail: commit.stderr || commit.stdout }) + } + + if (dryRun) { + log({ status: "dry-run", message: `${unpushed} unpushed commit(s) would be pushed`, repo: slug, branch }) + return + } + + const push = run(["git", "push", "origin", branch], userDir) + if (push.code !== 0) die("push-failed", "git push failed", { detail: push.stderr || push.stdout, repo: slug, branch }) + + const localHead = run(["git", "rev-parse", "HEAD"], userDir).stdout + const remoteHead = run(["git", "rev-parse", `origin/${branch}`], userDir).stdout + if (localHead !== remoteHead) { + die("push-unverified", "push reported success but local and remote HEAD differ", { localHead, remoteHead, repo: slug, branch }) + } + + log({ status: "pushed", message: `${committed} file(s) backed up`, repo: slug, branch, head: localHead }) + } finally { + try { + unlinkSync(lockPath) + } catch { + /* already gone */ + } + } +} + +main()