Repository navigation
Add production reliability guides for Harper operators - #671
Ethan-Arrowood wants to merge 22 commits into
Conversation
Ethan-Arrowood
added
the
content
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive set of administration guides and templates under the learn/administration/ directory, covering topics such as backup and recovery, replication, health checks, monitoring, and safe deployments, alongside updating the corresponding redirect rules. The review feedback focuses on improving documentation formatting and consistency, specifically recommending that critical caveats and limitations be wrapped in :::caution blocks, mid-sentence version badges be replaced with plain text, and API response fields in code examples use standard snake_case (e.g., renaming writtenAt to written_at).
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-671 This preview will update automatically when you push new commits. |
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-671 This preview will update automatically when you push new commits. |
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-671 This preview will update automatically when you push new commits. |
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-671 This preview will update automatically when you push new commits. |
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-671 This preview will update automatically when you push new commits. |
1 similar comment
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-671 This preview will update automatically when you push new commits. |
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-671 This preview will update automatically when you push new commits. |
hdbjeff
left a comment
hdbjeff
left a comment
There was a problem hiding this comment.
Inline Harper 5.2 technical review. Each requested change is attached to the specific affected line.
hdbjeff
left a comment
hdbjeff
left a comment
There was a problem hiding this comment.
Three targeted additions: one backup-program completeness note and two inline Mermaid recommendations.
hdbjeff
left a comment
hdbjeff
left a comment
There was a problem hiding this comment.
Readability pass across all changed files. Inline comments are limited to passages that can be materially shorter or more direct.
Takes the factual corrections from Jeff's review and leaves the prose, structure and diagram requests for the human editing pass. Each was re-verified at the v5.2.14 tag rather than origin/main, since the guides document released 5.2 and main has moved. - Replication port. `9933` is what a fresh Harper Pro install writes into `harper-config.yaml`, but it is configured rather than constant: with no `replication` block the secure port falls back to the Operations API secure port. An in-place upgrade never backfills that block, so an upgraded node and a fresh one can replicate on different ports. Says so, and the `cluster_status` sample no longer hard-codes the number. - `host` / `urlPath`. Routing by either has existed since v5.1.0; what v5.2.0 added is declaring it on the root-config entry and the `host` parameter on `deploy_component`. Those were collapsed into one badge. - `hdb_nodes`. The registry lives in `system` and propagates with it, which is what makes gossip discovery work. Keeping it per cluster is a deliberate per-route `excludeTables` setting, which the guide now covers. - Backup storage root. The rejection rule is a database whose tables resolve to more than one storage root. A per-table `path` landing on the same root is fine, so the blanket "per-table paths are unsupported" was wrong. - `restart: true`. Each node restarts independently as it finishes applying the deploy, unsequenced and free to overlap. The previous wording implied a coordinated simultaneous restart. - `HARPER_SAFE_MODE` skips your applications and components; Harper's built-in root component still loads. Also resolves the `get_configuration` placeholder link. The remaining `[TODO](TODO)` on the asynchronous-execution sentence is left in place because its target is a drafting decision, and it is the only thing keeping the build red. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
cc7e165 to
cc92816
Compare
… node out of rotation Add a Learn guide for applications on Fabric that feed their own health into the node's availability record, which Fabric's status-check component serves to the load balancer at GET /status. It covers when to do it, six safety rules (hysteresis; never undoing Fabric's or an operator's write, tracked by the record's __updatedtime__ in a non-replicated marker table and re-read before every write; never taking the last node out; a recheck right after going out; surviving restarts; warn logging and an off switch), a generic sample component, a local drill against a throwaway Harper with the real status-check component and the CLI's harper login, and caveats, including the narrow windows a set_status without compare-and-swap leaves open. Add a Fabric page for status-check: what GET /status answers, that POST and DELETE need a super user, that the record is per node, persisted and not replicated, and how to take a node out with harper login and harper set_status, locally or from GitHub Actions with OIDC, as a role whose operations list names only set_status and get_status (v5.3.0). It warns not to deploy the @harperdb/status-check npm package on Fabric. In the set_status reference, say that set_status stores only status, that Harper core does not act on status values, and that on Fabric the availability record decides routing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: OpenAI Codex <noreply@openai.com>
Co-Authored-By: OpenAI Codex <noreply@openai.com>
Co-Authored-By: OpenAI Codex <noreply@openai.com>
…mits Co-Authored-By: OpenAI Codex <noreply@openai.com>
Ethan-Arrowood
changed the title
Co-Authored-By: OpenAI Codex <noreply@openai.com>
Co-Authored-By: OpenAI Codex <noreply@openai.com>
Co-Authored-By: OpenAI Codex <noreply@openai.com>
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-671 This preview will update automatically when you push new commits. |
f35a6ce to
212c3b8
Compare
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-671 This preview will update automatically when you push new commits. |
| - How to separate build, deploy, activate, and expose into four decisions with four different control points | ||
| - The change loop, and what counts as evidence at each step | ||
| - Five rollout patterns and the Harper operations behind each | ||
| - How `"restart": true` and `"restart": "rolling"` differ, and how to verify completion | ||
| - How to classify a change so you reverse the right thing, and why reversing the wrong thing can cause data loss |
There was a problem hiding this comment.
This page talks about these concepts abstractly like an education paper might, with no practical examples. It also overstates what restart: rolling does by what it arguably should do, not by what it does do. It doesn't drain traffic [yet].
This page would be better off with some simple code examples that aim at our templates. It can talk about the different types of deployments you might do in various scenarios:
- A pure Harper backend that you can straight deploy and utilize
- A built vite React app that is compiled once, and its built artifacts are deployed. https://github.com/HarperFast/studio/blob/stage/.github/actions/studio-deploy/action.yaml#L86-L128
And then options for where to do deployments
- Locally after
harper login <your host name>you canharper deploy - In GHA or equivalent: harper setup cli example with oidc
Then rollback and such can be managed via Studio or via the CLI
Prefer examples that people can copy-paste into a terminal
Rework Safe Deployments and Rollback around commands operators can run: log in and deploy a create-harper backend, build a Vite frontend once and deploy only its output, deploy from GitHub Actions with OIDC, wait for the rolling job, and roll back by deployment_id from the CLI or Studio. State what restart=rolling actually does: it restarts nodes one at a time but leaves each node in rotation, runs no application checks between nodes, and (without canary certification) skips a node it cannot reach. Correct the "zero-downtime" and "keeps serving throughout" claims in the CLI reference and CI guide, and add a drain gate to the checklist. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: fix-Ethan-Arrowood_documentation_671-1627592f
- Staged frontend config: keep the project's own rest, schema, resource, and static entries (TypeScript templates load resources/*.ts) and drop only the plugins whose packages the release does not install. - Configuration operations reference: replace "the cluster stays available" with the same rolling-restart limit the guides now state. - Operating replication: define the exported OpsProbe table the convergence recipe writes, and name logging.auditRetention as the bound past which a peer sends a full copy. - Node rotation: note that a worker restart can briefly run two loops on Linux, and that it ends the "not going out again" state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: fix-Ethan-Arrowood_documentation_671-1627592f
…tabases A forced full copy is not yet verified to remove rows deleted while a node was away past retention, so state only that such a node needs a full copy. Write the convergence sentinel through databases.<name> so the recipe works for a database other than data. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: fix-Ethan-Arrowood_documentation_671-1627592f
Deletes older than logging.auditRetention are purged from history, so the full copy a long-absent node receives can leave rows that were deleted elsewhere. Qualify the returning-node row and the replication reference by that bound, and say to rebuild such a node from empty or replace it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: fix-Ethan-Arrowood_documentation_671-1627592f
…uild Rebuilding from empty would discard writes only that node took, so state both sides of the hazard and send operators to the restore guide's authority and reconciliation steps, noting that routes reconnect on start. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Dispatch-Task: fix-Ethan-Arrowood_documentation_671-1627592f
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-671 This preview will update automatically when you push new commits. |
dawsontoth
left a comment
dawsontoth
left a comment
There was a problem hiding this comment.
Way better! Left a few more comments, and then can you check the other files to see if the same concept of showing, copy-pasta examples, can help the readability?
| Set the cluster's Operations API URL once per shell, then log in with the cluster's admin username and password: | ||
|
|
||
| ```bash | ||
| export HARPER_CLI_TARGET=https://my-cluster.example.com:9925 |
There was a problem hiding this comment.
If we guide them to use a .env file, they can commit it, and not worry about setting it. That's what the templates do for you. Then login, deploy, etc all work magically without a target!
| rm -rf deploy && mkdir deploy | ||
| cp -R dist schemas resources deploy/ | ||
| cp deploy.config.yaml deploy/config.yaml | ||
| (cd deploy && harper deploy project=my-app restart=rolling json=true) > deploy.json |
There was a problem hiding this comment.
I feel like restart=true is going to be better and simpler for people, no job, and we can revisit when we implement real traffic drained deployments in the near future.
| Download a fresh snapshot from the configured node: | ||
|
|
||
| ```bash | ||
| harper get_backup database=data out=./data-$(date +%Y%m%dT%H%M%S).tar.gz |
There was a problem hiding this comment.
High: get_backup files cannot use restore_backup
This section recommends get_backup as the off-host copy, but the later restore procedure only covers restore_backup of a managed backup_id. A streamed snapshot has no backup_id; following those steps in an incident fails, or a RocksDB tar is unpacked without the extra blob restore, leaving records that point at missing files.
Suggested fix: State that restore_backup applies only to managed repository IDs. For a get_backup file, follow the manual snapshot restore in the backup overview with Harper stopped, including the RocksDB blob extra step, after traffic drain and replication isolation.
—
Reviewed 212c3b8
| - [ ] Error budget policy states what changes when the budget runs low | ||
| - [ ] Backup cadence reconciled against recovery point | ||
| - [ ] Restore time plus detection time reconciled against recovery time | ||
| - [ ] Convergence time reconciled against the admission gate |
There was a problem hiding this comment.
Medium: Launch checklist omits storage.writeAsync
The RPO guide says storage.writeAsync: true can drop acknowledged commits on power loss while backups still look current. This aggregated Objectives list never requires that check, so a launch review that uses only this page can accept a recovery point the storage path cannot meet.
Suggested fix: Add the RPO checklist item here: confirm writeAsync is off, or record the databases it is on beside their recovery point.
—
Reviewed 212c3b8
| ```bash | ||
| harper get_backup database=data target=https://operations.node-2.example.com out=./data.tar.gz | ||
| ``` | ||
|
|
There was a problem hiding this comment.
High: get_backup copy cannot be restored by this runbook
The guide prefers get_backup as the off-host copy and says turning that file into a live database is a separate restore step. The restore procedure that follows only documents managed restore_backup (online/offline table, jobs, target_database). Those operations cannot consume a get_backup tar.gz or .mdb. After losing the source node, operators following this track will have the recommended artifact and no matching restore steps.
Suggested fix: Link that sentence to the manual snapshot restore in the backup overview (RocksDB archive and LMDB .mdb plus blob roots) and state that restore_backup applies only to the managed repository path.
—
Reviewed 212c3b8
| - [ ] Every scenario in the scenario map has a named mechanism | ||
| - [ ] Availability target converted to minutes per month | ||
| - [ ] Error budget policy states what changes when the budget runs low | ||
| - [ ] Backup cadence reconciled against recovery point |
There was a problem hiding this comment.
High: launch checklist omits storage.writeAsync
The RPO guide treats storage.writeAsync as a setting that can silently invalidate the recovery point: acknowledged commits can be lost on power loss while backup timestamps still look current. This checklist claims to collect those launch gates, but the Objectives section never asks whether writeAsync is off or accepted per database. A launch review that uses only this page can sign off an RPO that the storage setting makes false.
Suggested fix: Add the RPO guide's writeAsync gate here: confirmed off, or the databases it is on recorded beside their recovery point.
—
Reviewed 212c3b8
| | Node registry (`hdb_nodes`) | Lives in `system`, so it propagates with it. Each node rewrites its own self-record from its `harper-config.yaml` routes on restart or component reload | Scoping applied through `add_node` alone does not survive a restart. Put durable constraints in config routes, and see below if you need the registry kept per cluster | | ||
| | Destructive schema changes | `drop_database` and `drop_table` replicate by default; `drop_attribute` does not | A drop is a cluster-wide event unless you pass `"replicated": false`. Verify the result on every node | | ||
|
|
||
| Measure incremental catch-up and full synchronization separately. A briefly offline node can catch up from retained transaction history, which its peers keep for [`logging.auditRetention`](/reference/v5/logging/configuration#loggingauditretention) (default `3d`). A node away longer than that gets a full copy of its peers' rows instead, even though it has synchronized before. Deletes older than the retention window are gone from that history, so the copy can leave rows on the returning node that were deleted elsewhere, while keeping writes that only that node took. Before such a node reconnects, decide which data is authoritative and how to reconcile it, as for [restoring in a replicated cluster](./backup-and-recovery.mdx#restoring-in-a-replicated-cluster). Configured routes reconnect as soon as Harper starts. |
There was a problem hiding this comment.
High: Retention-expired full copy is not an admission gate
When a node was away longer than logging.auditRetention, the full copy of peer rows still leaves locally deleted-elsewhere rows (and writes only this node took). cluster_status showing connected or copy-complete, and the OpsProbe sentinel, only prove that a new write arrived. Following the Health Checks return sequence after that can admit the node while users still read rows peers already deleted.
Suggested fix: In this paragraph, say to keep the node drained until reconciliation has actually removed extra rows (extract unique local writes if this node is authoritative, then wipe or replace if peers are). Do not treat full-copy completion or a new sentinel as the return gate; mirror that in the Health Checks return steps.
—
Reviewed 1ef7c4c
This branch was successfully deployed
4 participants
⊙ Problem
The Administration Learn track is a placeholder. Operators need practical guidance for capacity, traffic admission, replication, deployments, monitoring, and recovery, with measurable launch gates.
🤔 Keep this as one reliability track?
Yes: the guides share a service boundary, checklist, and plan template. Keeping them together gives operators a complete workflow; each guide can still be revised independently.
💡 Solution
Add ten linked Administration guides, including a production-readiness checklist and reusable reliability plan. Use direct instructions, concrete examples, and three diagrams for rollout decisions, draining a node, and recovery. Include Fabric availability and automated node-rotation guidance, with reference corrections checked against released source. Based on Jeff's field guide and Aleks's drafts.
🔧 Changes
Administration guides
isLeaderdoes not guarantee one copy source.Fabric and navigation
🤔 Should the last serving node remain available?
An external health controller can use the same status operation and runbook; the application loop is optional. Its default keeps the last node reachable, even when unhealthy. Services that must fail closed should choose another policy. Timestamp checks do not provide atomic ownership, so the guide documents the race windows and requires rehearsal.
Reference corrections
.mdbdownloads exclude external blob files. Table-filtered exports include audit data only when requested; recovering file-backed blobs requires a coordinated backup of the database and blob roots.harper runfor safe mode.isLeaderwithout promising an exclusive source, corrects status field names, and handles Copying values. Replication overview explains per-peer initial sync and operation defaults, including opt-in configuration updates and rolling restarts.✅ Verification
git diff --checkpassed. All three Mermaid diagrams parsed.cb59776c; the final version badge and sample-comment cleanup were checked locally, not re-reviewed. The unverified archived-v4 audit is tracked separately in Verify archived v4 status and replication documentation against released behavior #732.🤔 Is another full review worth the cost?
Continue with one closing pass: two full passes took roughly 45 minutes and found readiness and LMDB recovery hazards, plus smaller consistency issues. The latest fixes add 13 net lines across 12 files. Recovery safety warrants verifying those fixes; the unrelated v4 audit stays separate.
🤖 Generated by OpenAI Codex; posted via @Ethan-Arrowood.
Related PRs: #729 overlaps (Fabric/node-rotation work merged into this branch), 5 others independent
Complexity: complicated
Deferred findings
0d6e0904b0f240aaminor documentation —reference_versioned_docs/version-v4/operations-api/operations.md— Archived v4 contracts remain unverified — filed Verify archived v4 status and replication documentation against released behavior #732Review-Coverage: authored=codex; ran=claude,gemini; adjudicated=domain; blocked=cursor-grok(not-installed),cursor-composer(not-installed); declined=cursor-kimi,cursor-muse,muse; rounds=4; full=3 @ 212c3b8
Review-Attention: read ~8m (decisions: do-less-alternative, availability-ownership, prefer-serving-capacity, recovery-scope; raised: degraded review) @ 212c3b8