From 81d3ab0990bf5229edef897788c7dcfed16234a9 Mon Sep 17 00:00:00 2001 From: Ethan Arrowood Date: Mon, 14 Sep 2026 09:17:38 -0600 Subject: [PATCH 01/21] docs(learn): add Administration production reliability track Ten Learn > Administration guides derived from Jeff's "Operating Reliable Harper Applications" field guide, drafted by Aleks Haugom with Claude and moved into the repo largely as delivered. Replaces the Administration "Coming Soon" placeholder and redirects its old URL to the first guide. Guides, in sidebar order: 1. How Harper Runs in Production 2. Health Checks and Traffic Admission 3. Sizing a Harper Cluster 4. Operating Replication 5. Monitoring and Triage 6. Safe Deployments and Rollback 7. Backup and Recovery 8. Engineering RPO, RTO, and Uptime 9. Production Readiness Checklist 10. Reliability Plan Template Only content change from the delivered drafts: the get_components link in guide 1 pointed at an anchor that does not exist on the operations page; it now points at components/applications#get_components. Co-Authored-By: Aleks Haugom Co-Authored-By: Claude Fable 5.1 --- learn/administration/backup-and-recovery.mdx | 340 +++++++++++++++++ learn/administration/coming-soon.md | 1 - .../engineering-rpo-rto-and-uptime.mdx | 224 ++++++++++++ .../health-checks-and-traffic-admission.mdx | 314 ++++++++++++++++ .../how-harper-runs-in-production.mdx | 273 ++++++++++++++ .../administration/monitoring-and-triage.mdx | 293 +++++++++++++++ .../administration/operating-replication.mdx | 342 ++++++++++++++++++ .../production-readiness-checklist.mdx | 230 ++++++++++++ .../reliability-plan-template.mdx | 199 ++++++++++ .../safe-deployments-and-rollback.mdx | 328 +++++++++++++++++ .../sizing-a-harper-cluster.mdx | 261 +++++++++++++ redirects.ts | 1 + 12 files changed, 2805 insertions(+), 1 deletion(-) create mode 100644 learn/administration/backup-and-recovery.mdx delete mode 100644 learn/administration/coming-soon.md create mode 100644 learn/administration/engineering-rpo-rto-and-uptime.mdx create mode 100644 learn/administration/health-checks-and-traffic-admission.mdx create mode 100644 learn/administration/how-harper-runs-in-production.mdx create mode 100644 learn/administration/monitoring-and-triage.mdx create mode 100644 learn/administration/operating-replication.mdx create mode 100644 learn/administration/production-readiness-checklist.mdx create mode 100644 learn/administration/reliability-plan-template.mdx create mode 100644 learn/administration/safe-deployments-and-rollback.mdx create mode 100644 learn/administration/sizing-a-harper-cluster.mdx diff --git a/learn/administration/backup-and-recovery.mdx b/learn/administration/backup-and-recovery.mdx new file mode 100644 index 000000000..54bfa9f88 --- /dev/null +++ b/learn/administration/backup-and-recovery.mdx @@ -0,0 +1,340 @@ +--- +title: Backup and Recovery +sidebar_position: 7 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +Replication gives you availability. It does not give you recovery, and the reason +is worth stating plainly: replication faithfully propagates whatever you write, +including the bad write. A destructive operation, a schema mistake, or a release +that corrupted data reaches every peer at the speed of your convergence time. + +Backups cover the failure classes replication cannot. This guide is about +deciding what you need before you pick an operation, getting a copy somewhere the +node's failure cannot reach, and knowing your restore constraints before you are +in an incident rather than during one. + +## What You Will Learn + +- The five questions that determine your backup design, answered before any + command +- Which backup mechanism you actually have, since it depends on your storage + engine +- Why a copy on the node that created it is not a backup, and the specific way + copying one wrong breaks it +- What `verify_backup` does and does not check +- Which databases can be restored with the server running, and which cannot +- Why restoring a database to undo a code bug is usually the wrong move + +## Prerequisites + +- A cluster with a `super_user` credential, and CLI access to at least one node +- Knowledge of your storage engine per database, from your service boundary + inventory in + [How Harper Runs in Production](./how-harper-runs-in-production.mdx) +- [Operating Replication](./operating-replication.mdx), because a restore in a + replicated cluster is a replication event as much as a storage one +- A non-production database you are willing to destroy + +## Decide before you choose a command + +Answer these first. Every operation below is easy, and every one of them is the +wrong choice for some of these answers. + +1. **Which failure classes must you survive?** Node loss, storage loss, a bad + deployment, an accidental destructive operation, a corrupting application bug, + a site or region event, and a control-plane or credential loss are seven + different problems. Replication addresses the first one well and the sixth one + partially. It addresses none of the rest. +2. **What is the recovery point and recovery time per database?** Not per cluster. + A catalog table and an order ledger rarely deserve the same answer, and paying + ledger-grade backup cadence for catalog data is how backup cost becomes a + reason to reduce frequency. +3. **What granularity do you need?** Harper backs up and restores whole databases. + There is no per-table restore. If your recovery story requires restoring one + table, that requirement has to be met by database layout or by forward repair, + and it is far cheaper to learn that now. +4. **Where must copies live?** How many independent locations, under what + retention, and with what access control. +5. **Who is authorized to execute a restore, and how is that logged?** A restore + destroys current data by design. It deserves a named authority and an audit + trail. + +## Know which mechanism you have + +| Mechanism | Engines | What it is | +| ----------------------------------------------------- | --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | +| **Managed backups** | | Incremental, verifiable backups in a server-side repository under `storage.backupPath` | +| **Snapshot download** | | [`get_backup`](/reference/v5/backups/operations#get_backup) streams a snapshot over HTTP, with no server-side artifact | + +A backup is a whole-database copy: all tables, the transaction log, and any +file-backed blobs. So a restored database keeps its `read_audit_log` history as of +the backup point. + +Two limitations to check against your own deployment now rather than later: + +- **Managed backups require RocksDB.** For LMDB databases your options are + `get_backup` or volume snapshots. +- **One storage root per database.** A database whose tables use per-table `path` + storage configs spans multiple root stores and cannot be backed up with these + operations at all. Database-level custom storage paths are fine. If someone has + configured per-table paths, your backup strategy for that database does not + exist yet and you need to know that before launch. + +## Create and retain managed backups + + + + +```json +{ + "operation": "create_backup", + "database": "data" +} +``` + +Through a running server this returns a `job_id` immediately. Poll +[`get_job`](/reference/v5/operations-api/operations#get_job) for the outcome, +which includes the new `backup_id`, `size`, and `timestamp`. Treat the job result +as the completion signal, not the original response. + + + + +```bash +harper create_backup database=data +harper list_backups database=data +harper purge_backups database=data keep_count=7 +``` + +Every backup operation runs from the CLI under the same name. With the server +running the CLI forwards to the server; with it stopped the command operates +directly on the files. + + + + +Backups of the same database share unchanged RocksDB data files, so the first one +copies everything and later ones copy only what changed. Shared files are +reference-counted, so deleting a backup removes only files no remaining backup +references. + +:::warning +The incremental behavior applies to the RocksDB data files only. **The +transaction-log snapshot, and the blob snapshot for a database with file-backed +blobs, are copied in full on every backup.** With a large audit-retention window +or many blobs, frequent backups cost considerably more disk than the +data-only view suggests. Pass `exclude_blobs: true` to skip blobs when that is +appropriate, and size your backup volume against the full cost rather than the +incremental one. +::: + +Do not use `list_backups` sizes for capacity planning. The `size` and +`file_count` fields come from the RocksDB backup engine and exclude the +transaction-log and blob snapshots, so each entry undercounts, while the shared +files between entries mean summing them overcounts. Measure the repository +directory instead. + +## Get a copy off the node + +This is the step most likely to be missing, and the one where doing it slightly +wrong produces a copy that cannot be restored. + +**Managed backups live on the node that created them.** The repository is a local +directory, and RocksDB shares files across backup IDs, so **a backup ID is not a +self-contained folder.** Two consequences: + +- A disaster-recovery copy has to take the entire per-database repository, + `/`, not an individual backup. +- It has to do that while no backup operation is running, or from an atomic + filesystem snapshot. A live recursive copy can race `create_backup`, + `delete_backup`, or `purge_backups` and produce an unrestorable copy. + +An unrestorable copy is worse than no copy, because it will pass a "backups exist" +check and fail during an incident. + +The simpler off-host path, and the one to prefer unless you specifically need +retained managed backups off the node: + +```bash +# Pull a snapshot of the current state from a running node +harper get_backup database=data out=./data-$(date +%Y%m%dT%H%M%S).tar.gz + +# Or pull from another node, which also clones that node's database onto this one +harper get_backup database=data target=https://node-2.example.com:9925 out=./data.tar.gz +``` + +Note that `get_backup` always streams the current state. It cannot download a +historical managed backup, so it is a way to take a fresh off-host copy, not a way +to export your retention history. + +Whichever path you choose, the destination must not share a failure domain with +the source. A second directory on the same volume survives a deleted file and +nothing else. + +## Verification is not optional + +```json +{ + "operation": "verify_backup", + "database": "data", + "backup_id": 1, + "verify_checksum": true +} +``` + +`verify_backup` checks the RocksDB file sizes, their checksums when +`verify_checksum` is `true`, which is slower, and the framing of the +transaction-log snapshot, which is always checked. + +**The blob snapshot is not verified.** If your database has file-backed blobs, +verification does not tell you they are intact, and only a real restore does. + +More generally, a verified backup is a well-formed backup, not a proven recovery. +The only evidence that your recovery works is a restore you have actually +performed, which is why the drill at the end of this guide is the point of it. + +## Know your restore constraints before the incident + +RocksDB is single-writer, so an in-place restore requires the database to be fully +closed first. That produces hard constraints you cannot negotiate during an +outage: + +| Database | Online restore, server running | Offline restore, server stopped | +| ---------------------------------------------- | ------------------------------------ | ------------------------------- | +| A user database no loaded component holds open | Yes, restored in place | Yes | +| A user database a loaded component holds open | No, the job ends in `ERROR` | Yes | +| The `system` database | No, rejected before a job is created | Yes | + +Because the restore runs as a background job, a component holding the database +open does not fail your request. It fails inside the job, so you find out from +`get_job` rather than from the response. Harper does not track which component +uses which database, so it cannot selectively stop one. + +Two more constraints worth writing into your procedure: + +- **`target_database` requires the server stopped.** Restoring into a separate + database rather than overwriting the source is CLI-only with Harper down. The + target must not already exist or must be an empty directory. +- **An interrupted restore leaves the database unloadable.** On a crash or power + loss mid-restore, Harper marks the database as incompletely restored and refuses + to load it on the next start. Recover by rerunning `restore_backup` for the same + database and `backup_id`. Do not try to load or hand-repair the directory. + +## Restoring in a replicated cluster + +A restore is a point-in-time rollback of one node's data. In a cluster, that node +then has to rejoin peers that never rolled back, so sequence it deliberately: + +1. Take the node out of rotation with the availability flag, per + [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). + Never restore a node that is serving traffic. +2. Decide what should happen to replication for the restored database, and do that + deliberately. Whether the restored data should propagate, or be overwritten by + peers, is a decision with two very different outcomes, and it is not a decision + to improvise. +3. Perform the restore, online or offline according to the table above. +4. Verify data expectations locally before any peer sees the node. +5. Re-admit the node through the full return sequence, including convergence + verification. + +## What backups cannot fix + +If a release changed the meaning of persisted data, restoring the database is +usually the wrong response. A restore rolls back every write in the window, +including all the correct ones from the same period, so you can violate your +recovery point objective in the course of fixing a code bug. + +For that failure class, define forward repair instead: a targeted correction, or a +replay from the transaction log, that fixes the affected records and leaves the +rest alone. Decide which of your failure classes get restore and which get forward +repair while you are calm, and record the decision. See +[Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx) for how this +interacts with release reversal. + +### Prove it + +A backup you have never restored is a hypothesis. Run this on a non-production +cluster and record the numbers: + +1. Create a managed backup, then verify it with `verify_checksum: true`. +2. Take an off-host copy using the correct procedure for your mechanism, quiesced + or from an atomic snapshot if you are copying a managed repository. +3. Destroy the source database. +4. Restore it, taking the node out of rotation first, and time from decision to + restored service. That is your measured recovery time. +5. Determine how much data was actually lost against the backup timestamp. That is + your measured recovery point. +6. Validate correctness through the application, not just at the storage layer. + Row counts agreeing is not the same as the journey working. +7. Repeat for the `system` database specifically, offline, since it has different + constraints and it is the one people never rehearse. + +Both measured numbers feed +[Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx). If they +do not meet your stated targets, one of the two has to change, and it is better +that it is the target than a promise you cannot keep. + +## Operational notes + +- **Retention is a policy, not a side effect of disk space.** Use + `purge_backups` with an explicit `keep_count` that matches a written retention + decision. +- **Backup operations are `super_user` through the server, and filesystem + permissions offline.** An operator with shell access on a node can restore + without an API credential, so protect the host accordingly. +- **Schedule backups per database, matched to that database's recovery point.** + A single cluster-wide cadence overspends on some databases and underspends on + the ones that matter. +- **Alert on backup age, and on job failure.** A `create_backup` job that fails + silently produces a gap you will find at the worst possible time. Backup age + exceeding your recovery point objective is a pageable condition. +- **Record the storage engine per database in your inventory** and re-check after + migrations, since your entire mechanism choice depends on it. + +## Readiness checklist + +- [ ] Failure classes enumerated, with the mechanism that addresses each +- [ ] Recovery point and recovery time stated per database, not per cluster +- [ ] Storage engine confirmed per database, and the mechanism chosen accordingly +- [ ] No database in scope uses per-table storage paths, or its exclusion is known + and accepted +- [ ] Backup cadence matches the stated recovery point per database +- [ ] Backup volume sized against the full cost, including non-incremental + transaction-log and blob snapshots +- [ ] An off-host copy exists in a destination that does not share a failure + domain +- [ ] Managed repository copies take the whole `/` directory, + quiesced or from an atomic snapshot +- [ ] `verify_backup` runs on a schedule, with `verify_checksum` at least + periodically +- [ ] Blob integrity understood to be unverified by `verify_backup` +- [ ] Restore constraints documented for user databases, component-held databases, + and `system` +- [ ] Restore authority named, and restore execution logged +- [ ] Restore drill completed and dated, with measured recovery time and recovery + point +- [ ] `system` database restore rehearsed offline +- [ ] Forward repair defined for corruption caused by application code + +## Additional Resources + +- [Backups overview](/reference/v5/backups/overview) for how managed backups work, + the full limitation list, and manual restore examples +- [Backup operations](/reference/v5/backups/operations) for every parameter of + `create_backup`, `list_backups`, `verify_backup`, `delete_backup`, + `purge_backups`, `restore_backup`, and `get_backup` +- [Storage configuration](/reference/v5/configuration/options#storage) for + `storage.backupPath`, `storage.path`, and `storage.blobPaths` +- [Jobs](/reference/v5/operations-api/operations#jobs) and + [`get_job`](/reference/v5/operations-api/operations#get_job) for tracking + long-running backup operations +- [CLI operations](/reference/v5/cli/operations-api-commands) and + [remote operations](/reference/v5/cli/overview#remote-operations) for the + offline and `target=` forms +- [`read_audit_log`](/reference/v5/operations-api/operations#read_audit_log) for + transaction history usable in forward repair +- [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx) for + turning the measured numbers into commitments diff --git a/learn/administration/coming-soon.md b/learn/administration/coming-soon.md deleted file mode 100644 index d30962bdc..000000000 --- a/learn/administration/coming-soon.md +++ /dev/null @@ -1 +0,0 @@ -# Coming Soon diff --git a/learn/administration/engineering-rpo-rto-and-uptime.mdx b/learn/administration/engineering-rpo-rto-and-uptime.mdx new file mode 100644 index 000000000..958c625d5 --- /dev/null +++ b/learn/administration/engineering-rpo-rto-and-uptime.mdx @@ -0,0 +1,224 @@ +--- +title: Engineering RPO, RTO, and Uptime +sidebar_position: 8 +--- + +Recovery point, recovery time, and availability are usually written down once, in +a document nobody consults, as three numbers that were never checked against a +mechanism. This guide is about making them real: stating them per journey, +mapping each failure scenario to the mechanism that addresses it, and reconciling +the targets against numbers you have actually measured. + +If a target and a measurement disagree, one of them has to change. The point of +doing this deliberately is that you get to choose which, rather than finding out +during an incident that the answer was the target all along. + +## What You Will Learn + +- How to state recovery point, recovery time, and availability so they are + testable rather than aspirational +- A scenario-to-mechanism map, with what has to be proven for each to count +- How to convert an availability target into a monthly budget, and what each tier + demands of your automation +- How to reconcile stated targets against the numbers measured in earlier guides +- Why declaring your degraded modes is as important as declaring your targets + +## Prerequisites + +- Measured numbers from the earlier guides: per-node capacity, convergence time, + node re-entry time, release reversal time, and restore time +- [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) for the capacity model +- [Backup and Recovery](./backup-and-recovery.mdx) for the recovery mechanisms +- A named owner who can approve or reject a target, because this exercise produces + decisions rather than data + +## State the three per journey + +| Term | The question it answers | How to state it so it is testable | +| ------------------------ | ------------------------------------ | --------------------------------------------------------------------------- | +| Recovery point objective | How much recent data may be lost | Per database, in units of time, measured against your backup timestamp | +| Recovery time objective | How long until service is restored | Per journey, measured from detection to verified service, not from decision | +| Availability target | How much unavailability is permitted | Per journey, as minutes per month, with the measurement boundary named | + +Three details make the difference between a testable statement and a slogan. + +**Per journey, not per cluster.** A catalog browse and an order submission fail +differently, matter differently, and recover differently. One cluster-wide number +overcommits on the cheap journey and undercommits on the expensive one. + +**Recovery time starts at detection, not at decision.** The gap between something +breaking and someone knowing is part of the outage, and it is often the largest +part. Measuring from the moment a human decided to act produces a number that +flatters your automation and misleads your planning. + +**Name the measurement boundary.** Availability measured at your CDN edge and +availability measured at a Harper node are different quantities, and the +difference is exactly the part of the stack you may not control. Whichever you +choose, say so in the same sentence as the number. + +## Map scenarios to mechanisms + +For each scenario, know the mechanism, and know what must be proven for the +mechanism to count. The right-hand column is the whole exercise. + +| Scenario | Primary mechanism | What must be proven | +| ------------------------------------- | ------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | +| Single node loss | Peer capacity plus traffic removal | `(N - F)` capacity under load, measured time to remove traffic, no user-visible errors in transition | +| Node returns after a restart | Incremental catch-up plus an admission gate | Measured convergence time, and an admission gate that does not admit before convergence completes | +| Node replacement or scale-out | Full database synchronization to the new node | Measured full-sync duration at your data volume, and the load it puts on the source node | +| Failure domain loss | Topology spread across independent domains | No shared power or network fault, and surviving capacity inside the budget | +| Region loss | Multi-region topology plus route change | Data authority per region, route propagation time, and declared degraded behavior | +| Bad release, no data change | Release reversal | Measured release reversal time, and the previous artifact still addressable | +| Bad release that changed data meaning | Forward repair, not restore | A written repair or replay procedure, rehearsed on representative data | +| Accidental destructive operation | Restore from backup | Measured restore time and actual data loss, from a real drill | +| Storage loss on one node | Off-host backup copy | A restorable off-host copy, taken by the correct procedure for your mechanism | +| Corruption propagated by replication | Point-in-time restore plus a replication decision | A rehearsed sequence that includes what happens to peers | +| Replication certificate expiry | Certificate lifecycle management | Expiry dates tracked, with an alert far enough ahead to act | +| Downstream dependency outage | Declared degraded mode | The journey's behavior when the dependency is down, verified rather than assumed | + +Two rows in that table are the ones most often missing. Node replacement is +budgeted as though it were a restart, when it is a full synchronization and can be +much slower. And the downstream dependency row usually has no answer at all, which +means the answer is whatever the code happens to do. + +## Convert availability into a budget + +An availability target is a quantity of unavailability you may spend per month. + +| Monthly target | Approximate maximum unavailability | What the tier demands | +| -------------- | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | +| 99.9% | 43 min 50 sec | A human can be paged, diagnose, and act within some incidents. One untested maintenance window can consume the month | +| 99.95% | 21 min 55 sec | Automated traffic removal and a rehearsed release reversal become necessary rather than nice | +| 99.99% | 4 min 23 sec | Node failure has to be close to transparent. Detection, traffic removal, and validation must be automated, since no human response fits | + +Figures assume a 30.44 day month. Your contractual definition, exclusions, and +measurement boundary may differ. + +Read down that right-hand column rather than the middle one. The tier you choose +determines how much automation you are committing to build, not how many nodes you +buy. Adding nodes without automating detection and traffic removal moves you +across the table's rows without moving you down its column. + +Treat the budget as spendable. If the month's remaining budget is small, that is +an argument for pausing discretionary change, and if it is large, that is +permission to ship. An error budget that never changes anyone's behavior is +just a number in a document. + +## Reconcile targets against measurements + +This is the step that turns the exercise into engineering. Fill in both columns +and look for rows where they disagree. + +| Commitment | Measured from | Your target | Your measurement | +| --------------------------- | ---------------------------------------------------------------------------------------- | ----------- | ---------------- | +| Surviving capacity | The peak-load drain drill in [Sizing](./sizing-a-harper-cluster.mdx) | | | +| Time to remove traffic | The drain sequence in [Health Checks](./health-checks-and-traffic-admission.mdx) | | | +| Node re-entry time | The same drill, including convergence | | | +| Convergence time under load | The sentinel measurement in [Operating Replication](./operating-replication.mdx) | | | +| Full synchronization time | A node replacement at your data volume | | | +| Release reversal time | The stopped-rollout rehearsal in [Safe Deployments](./safe-deployments-and-rollback.mdx) | | | +| Restore time | The restore drill in [Backup and Recovery](./backup-and-recovery.mdx) | | | +| Actual data loss on restore | The same drill, against the backup timestamp | | | +| Time to detection | The fault injection drill in [Monitoring and Triage](./monitoring-and-triage.mdx) | | | + +Three reconciliations catch most problems: + +- **Backup cadence against recovery point.** If you back up every six hours, your + recovery point cannot be one hour, no matter what the document says. +- **Restore time plus detection time against recovery time.** Recovery time + includes noticing. A twenty minute restore behind a forty minute detection gap + is a one hour recovery. +- **Convergence time against your admission gate.** If your gate admits a node + faster than it converges, you are serving stale data on purpose and calling it + availability. + +Where a target and a measurement disagree, the resolution is one of three things: +invest in the mechanism, relax the target, or accept the gap explicitly with an +owner and a date. All three are legitimate. Leaving it unreconciled is not. + +## Declare your degraded modes + +Between fully working and fully down there is a range of states, and if you have +not decided what they should be, the code has decided for you. + +For each critical journey, write down what happens when a dependency is +unavailable, when a node is behind on replication, and when the cluster is below +its capacity budget. Reads served from slightly stale data may be entirely +acceptable for a catalog and entirely unacceptable for a balance check, and the +right answer differs by journey rather than by system. + +Then decide who can declare a degraded mode, and whether the declaration is +manual or automatic. A degraded mode nobody is authorized to invoke is not a +degraded mode. + +### Prove it + +Two passes, and both are needed. + +**Tabletop the whole scenario table.** For each row, walk through who detects it, +what they do, what mechanism carries the recovery, and what evidence confirms +success. Rows where the group cannot answer without speculating are the gaps, and +finding them costs an hour rather than an outage. + +**Then run one real drill per scenario class**, spread over a quarter rather than +attempted in a day. The classes are: node loss, node return, node replacement, +release reversal, restore, and dependency failure. Record the date, the measured +numbers, what surprised you, and what you changed as a result. + +The surprises are the deliverable. A drill that goes exactly as expected has +confirmed your documentation. A drill that does not has found the thing that +would have hurt you. + +## Operational notes + +- **Recovery numbers expire.** Every measurement here is a property of a Harper + version, a data volume, a topology, and a set of components. Re-measure after + version upgrades and significant data growth, and date every number you record. +- **Uptime and recovery targets belong in the same document as the mechanism.** A + target stored separately from its supporting evidence drifts from reality within + a quarter. +- **Do not let a single incident rewrite your targets.** Adjust targets from + measurement and business need, not from the last thing that went wrong. +- **Exclusions matter as much as the number.** Planned maintenance, third-party + dependency failures, and client-side problems are usually excluded from an + availability calculation. Whether yours excludes them changes the number + substantially, so it belongs in writing. +- **A target you cannot measure is not a target.** If nothing in your monitoring + produces the number in your commitment, the first investment is measurement, not + more nodes. + +## Readiness checklist + +- [ ] Recovery point stated per database, in time units +- [ ] Recovery time stated per journey, measured from detection +- [ ] Availability stated per journey, with the measurement boundary named +- [ ] Exclusions documented +- [ ] Every scenario in the map has a named mechanism +- [ ] Every mechanism has its "what must be proven" satisfied or explicitly + outstanding +- [ ] Availability target converted to minutes per month +- [ ] Error budget policy states what changes when the budget runs low +- [ ] Backup cadence reconciled against recovery point +- [ ] Restore time plus detection time reconciled against recovery time +- [ ] Convergence time reconciled against the traffic admission gate +- [ ] Every unreconciled gap has an owner and a date +- [ ] Degraded modes declared per journey, with authority to invoke them +- [ ] Tabletop completed and dated +- [ ] At least one real drill per scenario class completed and dated + +## Additional Resources + +- [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) for the capacity + invariant behind the surviving-capacity commitment +- [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx) + for drain, admission, and failback timing +- [Operating Replication](./operating-replication.mdx) for convergence measurement +- [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx) for release + reversal timing +- [Backup and Recovery](./backup-and-recovery.mdx) for restore timing and + mechanism limits +- [Monitoring and Triage](./monitoring-and-triage.mdx) for time to detection +- [Production Readiness Checklist](./production-readiness-checklist.mdx) for the + gate this feeds +- [Reliability Plan Template](./reliability-plan-template.mdx) for where to record + the results diff --git a/learn/administration/health-checks-and-traffic-admission.mdx b/learn/administration/health-checks-and-traffic-admission.mdx new file mode 100644 index 000000000..249e9ddf6 --- /dev/null +++ b/learn/administration/health-checks-and-traffic-admission.mdx @@ -0,0 +1,314 @@ +--- +title: Health Checks and Traffic Admission +sidebar_position: 2 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +The default health check on most load balancers asks one question: did something +answer on this port? A Harper node answers that question correctly while still +being the wrong place to send a user request, because a node can be running before +its databases have synchronized, while it is catching up on missed transactions, or +while a dependency it needs for one particular journey is down. + +This guide builds four separate signals so your traffic layer can tell the +difference between "the process is alive" and "this node will serve this journey +correctly," and then uses them to drain and return a node safely. + +## What You Will Learn + +- Why liveness, availability, readiness, and journey verification are four signals + rather than one, and which layer consumes each +- How to install and drive the `@harperdb/status-check` component to control + whether a node advertises itself as available +- How to write an application readiness route that checks only what the routed + journey needs +- The order of operations for draining a node and returning it to service +- What makes a readiness endpoint safe under load, and what makes one dangerous + +## Prerequisites + +- A Harper cluster with at least two nodes, so draining one leaves a service + behind ([Fabric](/fabric) or self-managed) +- A `super_user` credential for the Operations API +- A traffic layer you can configure health checks on: a load balancer, a CDN + origin group, or a service mesh +- [How Harper Runs in Production](./how-harper-runs-in-production.mdx), and your + ports written down + +## Four signals, not one + +| Signal | Where it lives | What it proves | Who consumes it | +| -------------------------- | ---------------------------------------------------------- | ----------------------------------------------------- | ---------------------------------- | +| Process liveness | Any trivial response on `9925` or `9926` | Harper is running | Process supervisor, restart policy | +| Node availability flag | `GET /status` on `9926`, from the `status-check` component | An operator or automation says this node should serve | Traffic layer health check | +| Application readiness | A component route you write on `9926` | This node can serve this journey right now | Traffic layer, or your own gating | +| Critical-journey synthetic | A real request through the public route | Users are actually being served | Your SLO and alerting | + +The distinction that does the most work is the second one. The availability flag is +not a measurement, it is a declaration. It exists so that you, or your automation, +can take a node out of rotation deliberately, before doing something to it, and +put it back afterwards. Nothing else in this list can be set by an operator, and +nothing else is safe to use as the primary routing signal. + +## Install the availability flag + +[`@harperdb/status-check`](https://github.com/HarperFast/status-check) is a Harper +component that adds a `/status` route on the application port. Deploy it like any +other component: + +```json +{ + "operation": "deploy_component", + "project": "status-check", + "package": "@harperdb/status-check", + "restart": true +} +``` + +Or declare it in your application's `config.yaml`, which is the better option +because it makes the availability contract part of your deployed artifact rather +than a manual step someone has to remember on a new node: + +```yaml +status-check: + package: '@harperdb/status-check' +``` + +Once deployed, the route answers with a status code rather than a body: + +- `GET /status` returns `200` when the node is available, `404` when it is not +- `POST /status` marks the node available (authenticated) +- `DELETE /status` marks the node unavailable (authenticated) + + + + +```bash +# Is this node advertising itself as available? +curl -s -o /dev/null -w '%{http_code}\n' https://my-node.example.com:9926/status + +# Take it out of rotation +curl -s -X DELETE https://my-node.example.com:9926/status -u 'admin:password' + +# Put it back +curl -s -X POST https://my-node.example.com:9926/status -u 'admin:password' +``` + + + + +```javascript +const base = 'https://my-node.example.com:9926/status'; +const auth = { Authorization: 'Basic ' + btoa('admin:password') }; + +const res = await fetch(base); +console.log(res.status); // 200 available, 404 unavailable + +await fetch(base, { method: 'DELETE', headers: auth }); // out of rotation +await fetch(base, { method: 'POST', headers: auth }); // back in +``` + + + + +Point your traffic layer's health check at `GET /status` on `9926`, not at the +Operations API and not at your application's root. A `404` is the node telling the +traffic layer to stop sending work, and it will keep saying so until something +sets it back. + +:::warning +Whatever mechanism you use to persist the availability flag, keep it out of +replication scope. A flag that replicates would let one node's maintenance state +propagate to its peers, which turns a routine drain into an outage. If you are +scoping replication by hand, confirm this explicitly rather than assuming it. See +[Operating Replication](./operating-replication.mdx). +::: + +### A note on `set_status` + +The Operations API also offers +[`set_status`, `get_status`, and `clear_status`](/reference/v5/operations-api/operations#set_status--get_status--clear_status) +for application-defined status values, with types for primary, maintenance, and +availability. + +These are a coordination primitive for your own automation, not a health report, +and not a substitute for a real readiness check. Nothing in Harper acts on a value +you set through them. Prefer the `status-check` component for traffic admission, +because its contract is an HTTP status code that a load balancer can consume +directly, and reach for `set_status` when you need to coordinate something between +your own scripts. + +## Write an application readiness route + +Liveness and the availability flag both answer questions about the node. +Readiness answers a question about the journey: if traffic arrives for this route +right now, will it succeed? + +The rule that keeps this useful is to check only the dependencies the routed +journey actually needs. A readiness route that checks everything will report a +node unready because of a subsystem that route never touches, and you will have +converted a partial degradation into a full outage yourself. + +Add a resource to your application's `resources.js`: + +```javascript +export class Readiness extends Resource { + static async get() { + const checks = {}; + + // A bounded read against the table this journey serves. + // Keep it to a single primary-key lookup, never a scan. + try { + await tables.Product.get('readiness-probe-sentinel'); + checks.data = 'ok'; + } catch (error) { + checks.data = 'failed'; + } + + // Only the downstream dependencies this route needs. + try { + const res = await fetch('https://pricing.internal.example.com/health', { + signal: AbortSignal.timeout(500), + }); + checks.pricing = res.ok ? 'ok' : 'failed'; + } catch (error) { + checks.pricing = 'failed'; + } + + const ready = Object.values(checks).every((v) => v === 'ok'); + + return new Response(JSON.stringify({ ready, checks, version: process.env.APP_VERSION }), { + status: ready ? 200 : 503, + headers: { 'Content-Type': 'application/json' }, + }); + } +} +``` + +Enable the `jsResource` plugin in `config.yaml` if it is not already, and the +route is served at `/Readiness` on the application port. See +[Harper Applications in Depth](../developers/harper-applications-in-depth.mdx) for +the resource and export mechanics. + +Two details in that example are the point of it. The timeout on the downstream +call means a slow dependency cannot make your readiness check hang, which would +make the node look dead to a probe rather than unready. And returning the version +means that when you are staring at a dashboard during a rollout, the readiness +response itself tells you which build answered. + +## Drain a node and bring it back + +The order matters, and the verification steps between them are the parts people +skip. + + + + +1. **Declare it unavailable.** `DELETE /status` on the target node. +2. **Verify traffic actually stopped.** Watch request volume on the target fall to + zero and rise on its peers. Do not trust the configured weight; check the + measured request count from + [analytics](/reference/v5/analytics/overview) or your traffic layer's own + metrics. Health check intervals, DNS TTLs, and client-side connection reuse all + add delay here, and the delay is yours to measure. +3. **Confirm the peers can carry it.** The remaining nodes have to be inside their + capacity budget with this node gone. If they are not, stop: see + [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx). +4. **Do the work.** Restart, upgrade, reconfigure, or investigate. + + + + +1. **Confirm the databases are current.** `cluster_status` should show the expected + peer sockets connected for every database this node serves, and convergence + should be complete rather than in progress. +2. **Confirm readiness passes locally.** Call your `/Readiness` route directly + against the node, bypassing the traffic layer. +3. **Run the journey synthetic against the node directly.** A real read, or a safe + write, through the same path a user would take. +4. **Declare it available.** `POST /status`. +5. **Hold before restoring full weight.** Give it a stability window at partial + traffic and watch error rate and latency against its peers before treating it as + fully back. + + + + +Step 5 on the return side is the one worth defending in a review. Failback is a +change like any other, and a node that has just synchronized under load is the +most likely one to surprise you. Design failback, do not just design failover. + +## Readiness hygiene + +- **Keep the response cheap and bounded.** A readiness check that performs a broad + scan or writes data will amplify an incident, because it runs at probe frequency + across every node at exactly the moment the system is already struggling. +- **Never make it the only routing signal.** Liveness plus availability plus + readiness, consumed at the right layers. +- **Probe from more than one location** where your traffic layer supports it. A + single probe point cannot distinguish a network path problem from a node problem. +- **Make it observable.** Record response code, latency, the reason for a failure, + and the node and component version. A readiness check whose failures you cannot + explain after the fact is a check you will end up ignoring. +- **Version the contract.** If you change what readiness means, that is a change + to the traffic admission policy, and it deserves the same care as a code + release. + +### Prove it + +On a non-production cluster, restart one node under representative read and write +load, using the full drain and return sequence above. Record three numbers: how +long from `DELETE /status` until measured traffic on that node reaches zero, +whether any user-visible errors occurred during the transition, and how long from +process start until the node legitimately passed all three return gates. + +That third number is your real node re-entry time, and it is almost always longer +than people assume, because it includes convergence rather than just startup. + +## Operational notes + +- **A returning node is not the same as a new node.** A node whose databases have + never synchronized downloads them in full. A node that was offline and comes back + catches up on the transactions it missed. Both need to finish before traffic + arrives, but they take very different amounts of time, so do not budget for the + first when you are planning a routine restart. +- **Set unavailable before recovery work, not after.** Any operation that touches + data on a node, including a restore, should happen with the node out of rotation. +- **Configuration changes need a restart to take effect**, so a node that has been + reconfigured but not restarted is running the old configuration while reporting + the new one. Sequence the restart into the same maintenance window. +- **Fabric provides its own cluster-level health and routing.** These signals still + matter, because the availability flag and your readiness route are what Fabric's + routing has to consult. + +## Readiness checklist + +- [ ] `@harperdb/status-check` deployed, declared in `config.yaml` rather than + deployed by hand +- [ ] Traffic layer health check points at `GET /status` on `9926` +- [ ] Availability flag confirmed to be outside replication scope +- [ ] An application readiness route exists, scoped to one journey's dependencies +- [ ] Every downstream call in the readiness route has a timeout +- [ ] Readiness response includes the component version +- [ ] A critical-journey synthetic runs against the public route, separately from + the liveness probe +- [ ] Drain sequence documented, with measured time-to-zero-traffic +- [ ] Return sequence documented, with a stability window before full weight +- [ ] Measured node re-entry time recorded, including convergence + +## Additional Resources + +- [`@harperdb/status-check`](https://github.com/HarperFast/status-check) component + source and options +- [Operations API operations](/reference/v5/operations-api/operations) for + `cluster_status`, `system_information`, and the status operations +- [Components overview](/reference/v5/components/overview) for the full list of + Harper-maintained components, including the Prometheus exporter +- [Analytics overview](/reference/v5/analytics/overview) for per-node request and + latency data +- [Harper Applications in Depth](../developers/harper-applications-in-depth.mdx) + for custom resources and the `jsResource` plugin +- [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) for whether your peers + can absorb a drained node diff --git a/learn/administration/how-harper-runs-in-production.mdx b/learn/administration/how-harper-runs-in-production.mdx new file mode 100644 index 000000000..f9690246e --- /dev/null +++ b/learn/administration/how-harper-runs-in-production.mdx @@ -0,0 +1,273 @@ +--- +title: How Harper Runs in Production +sidebar_position: 1 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +You have built an application, deployed it from a pipeline, and now real users are +going to depend on it. Operating Harper is not the same job as operating an +application tier in front of a database tier, because Harper does not have those +two tiers. One process holds your component code, the HTTP stack, the local +database, and the replication client. That shape removes several problems you may +be used to solving, and it changes where the remaining ones live. + +This guide is the map for the rest of the Administration track. It covers what a +single node actually contains, what that means for failure and scale, and how to +take an inventory of your own deployment before you design anything around it. + +## What You Will Learn + +- What a Harper node contains, and which port carries which kind of traffic +- Why the node is your unit of failure and your unit of scale, and what that + removes from your operating burden as well as what it adds +- The three ways a Harper runbook differs from a two-tier runbook +- How to inventory your service boundary with the Operations API, so later guides + have something concrete to work from + +## Prerequisites + +- A running Harper instance, either a [Harper Fabric](/fabric) cluster or a + [local installation](../getting-started/install-and-connect-harper.mdx) +- A `super_user` credential for the Operations API +- An application deployed to it + ([Create your First Application](../getting-started/create-your-first-application.mdx)) + +## What one node contains + +A Harper node is a single process running your components, an HTTP server, a local +storage engine, and peer replication. There is no network hop between your +application code and the data it reads, no separate cache tier to keep coherent, +and no connection pool to tune between tiers. + +Three ports carry the traffic you will operate around: + +| Port | Serves | Who should reach it | +| ------ | --------------------------------------------------------------------------- | --------------------------------- | +| `9926` | Application traffic: REST, WebSocket, MQTT-over-WebSocket, component routes | Your traffic layer and your users | +| `9925` | The [Operations API](/reference/v5/operations-api/overview) | Operators and your pipeline only | +| `9933` | Secure peer [replication](/reference/v5/replication/overview) | Other nodes in the cluster only | + +These are documented defaults, not guarantees about your cluster. Confirm the live +values rather than assuming them, because a replication port in particular can be +inherited from other configuration: + +```json +{ + "operation": "get_configuration" +} +``` + +Read back `http.port`, `operationsApi.network.port`, and the `replication` block. +Record what you find. Later guides in this track assume you know these numbers for +your own deployment. + +:::tip +Keep `9925` off any public route. The Operations API can deploy components, read +logs, and read configuration, so it is an administrative surface, not an +application one. See [security overview](/reference/v5/security/overview). +::: + +## The node is your unit of failure and your unit of scale + +Because one process holds the runtime and the local data together, there is no +internal application-to-database seam that can fail over independently. The node +is the practical unit of service failure. + +Start with what that removes, because it is the larger half of the trade: + +- No cross-tier network latency on data access, and no tail latency from a + saturated connection pool between tiers +- No cache invalidation problem between an application cache and a database of + record, because they are the same thing +- No partial-outage state where the application tier is healthy and the data tier + is not, which is the failure mode that produces the most confusing incidents +- One capacity number to measure and one thing to size + +Then the consequence: when you lose a node, you lose a whole slice of your +service, not one layer of it. This is why the first real design decision in +[Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) is not your peak +throughput. It is how many nodes you are willing to lose at once, and whether the +survivors can carry the load. + +Scale works on the same unit. Adding capacity means adding a node that carries +both request handling and data, so a scaling event is also a data movement event. +That is not a problem, but it is a thing with a duration, and traffic should not +arrive until it finishes. + +## Three differences that change your runbook + +### A process that is up is not a node that should take traffic + +A Harper process will answer a TCP connection and return an HTTP response before +it is a good place to send a user request. A new or replacement node has to +synchronize the databases it serves before its answers are correct. A returning +node has to catch up on transactions it missed. + +So liveness and admission are two different decisions, and a load balancer health +check that only proves liveness will route users to a node that is technically +running and functionally wrong. This is the whole subject of +[Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx), +and it is the single most common gap in a first production deployment. + +### Replication is peer-to-peer and scoped, so verify your own scope + +Harper peers exchange data over WebSockets with mTLS on the secure replication +port and discover each other through configured routes. There is no primary. Data +mutations and transactions replicate; some things do not, and the scope is +configurable per database and per table. + +The important operating habit is not memorizing a default. It is checking what +your cluster actually replicates, because the answer depends on your +configuration, your version, and whether anyone has scoped it since: + +```json +{ + "operation": "cluster_status" +} +``` + +The response lists each peer connection and, within it, one socket per database +per peer. That tells you which databases are actually flowing, which is the +question that matters during an incident. What is in scope, what is deliberately +out of it, and how to prove convergence rather than just connection are covered in +[Operating Replication](./operating-replication.mdx). + +### Reversal is a redeploy, not an infrastructure event + +Your application ships as a component, deployed with +[`deploy_component`](/reference/v5/operations-api/operations#deploy_component) +from an immutable reference. Rolling back means deploying the previous immutable +reference. There is no image to rebuild, no instance to replace, and no cluster to +rebuild to undo a bad release. + +That makes reversal fast enough to be a real option under pressure, which in turn +makes it worth designing for deliberately rather than improvising. It also means +code rollback, configuration rollback, and data recovery are three separate +actions with three different blast radii, and conflating them during an incident +is how a bad release becomes a data loss event. See +[Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx). + +## Inventory your service boundary + +Everything else in this track builds on knowing what you have. Run these four +operations against each node and write down the answers. + + + + +```bash +curl -s -X POST https://my-node.example.com:9925/ \ + -H 'Content-Type: application/json' \ + -u 'admin:password' \ + -d '{"operation":"system_information","attributes":["system","cpu","memory","disk","threads"]}' +``` + + + + +```javascript +await fetch('https://my-node.example.com:9925/', { + method: 'POST', + headers: { + 'Content-Type': 'application/json', + 'Authorization': 'Basic ' + btoa('admin:password'), + }, + body: JSON.stringify({ + operation: 'system_information', + attributes: ['system', 'cpu', 'memory', 'disk', 'threads'], + }), +}); +``` + + + + +:::warning +The `attributes` array silently drops names it does not recognize, so a typo +returns a smaller response rather than an error. The valid values are `system`, +`time`, `cpu`, `memory`, `disk`, `network`, `harperdb_processes`, `table_size`, +`metrics`, and `threads`. If a response is missing a section you asked for, check +the spelling before you check the node. +::: + +Then: + +- [`get_components`](/reference/v5/components/applications#get_components) for + what is deployed, including which extensions are present +- [`list_deployments`](/reference/v5/operations-api/operations#list_deployments) + for what changed and when +- `cluster_status` for the peers and databases actually connected + +Fill in a boundary record you can keep: + +| Item | Value for your deployment | +| ----------------------- | ----------------------------------------------------- | +| Harper version | From `system_information` | +| Node count and roles | From `cluster_status` and your topology intent | +| Ports in use | From `get_configuration` | +| Databases | Which exist, and which replicate | +| Storage engine | Per database, since it constrains your backup options | +| Components deployed | From `get_components`, with the version of each | +| Critical journeys | The user-facing paths that must work, named | +| Downstream dependencies | Anything Harper calls that can fail independently | + +The last two rows are the ones people skip and the ones that matter most. A +healthy Harper process cannot compensate for a failed downstream dependency or for +application logic returning wrong answers, so an operating model that only watches +Harper will miss the incidents your users actually notice. + +### Prove it + +Before moving on, confirm the picture is real rather than assumed. On a +non-production cluster, stop one node and watch what happens to the others: +whether peers keep serving, how long the remaining nodes take to show the change +in `cluster_status`, and what your traffic layer does about it. You are not +measuring anything precisely yet. You are checking that the boundary you wrote +down matches the system you have. + +## Operational notes + +- **Fabric and self-managed differ in what you own, not in how Harper behaves.** + On [Fabric](/fabric), cluster creation, certificates, and the metrics pipeline + are managed for you. The failure unit, the replication model, and the admission + problem are identical. +- **Version parity across nodes is an operating requirement, not a nicety.** + Mixed versions in a cluster change replication and deployment behavior. Record + the version per node in your boundary inventory and alert on drift. +- **Configuration changes made through the API take effect on restart.** A + `set_configuration` call that has not been followed by a restart or + `restart_service` leaves a node running something other than its stated + configuration. Track pending changes in your change record. +- **`get_status` reports a `restartRequired` flag, but it tracks component and + code restarts rather than configuration changes.** Do not rely on it to tell you + a configuration change is still pending. + +## Readiness checklist + +- [ ] Ports confirmed from `get_configuration`, not assumed from documentation +- [ ] Operations API on `9925` is not reachable from the public internet +- [ ] Harper version recorded per node, with an alert on drift +- [ ] Databases listed, with replication scope confirmed via `cluster_status` +- [ ] Storage engine recorded per database +- [ ] Components and versions recorded from `get_components` +- [ ] Critical user journeys named and written down +- [ ] Downstream dependencies named, with their own failure behavior understood + +## Additional Resources + +- [HTTP server reference](/reference/v5/http/overview) for the application port and + server architecture +- [Operations API overview](/reference/v5/operations-api/overview) and the + [full operation list](/reference/v5/operations-api/operations) +- [Replication overview](/reference/v5/replication/overview) for the peer model, + mTLS, routes, and scope +- [Components overview](/reference/v5/components/overview) for the component model + and the available extensions +- [Database overview](/reference/v5/database/overview) for storage engines and + transaction boundaries +- [Configuration overview](/reference/v5/configuration/overview) for + `harper-config.yaml` and restart requirements +- [Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx) for how + application code reaches these nodes diff --git a/learn/administration/monitoring-and-triage.mdx b/learn/administration/monitoring-and-triage.mdx new file mode 100644 index 000000000..f25621455 --- /dev/null +++ b/learn/administration/monitoring-and-triage.mdx @@ -0,0 +1,293 @@ +--- +title: Monitoring and Triage +sidebar_position: 5 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +A dashboard is useful when it lets an operator place a symptom. Users are seeing +errors: is it the edge, the traffic layer, one node, the replication path, a +downstream dependency, or the change someone shipped twenty minutes ago? A +dashboard that shows CPU across the cluster cannot answer that, which is why it +gets ignored during incidents. + +This guide builds monitoring around the boundaries you can actually act on, names +the specific Harper metrics worth alerting on, and gives you a triage sequence +short enough to run under pressure. + +## What You Will Learn + +- The layers a Harper symptom can live in, and the minimum signal for each +- Which Harper metrics are worth an alert, by name, and which are only worth a + dashboard +- How to get metrics out of Harper, on Fabric and self-managed +- How to write log entries that are still useful during an incident +- A triage sequence that narrows a Harper incident in about five minutes + +## Prerequisites + +- A cluster with your application deployed and taking traffic +- A `super_user` credential for the Operations API +- Somewhere to send metrics: [Grafana](/fabric/grafana-integration) on Fabric, or + a Prometheus-compatible system for self-managed +- [Operating Replication](./operating-replication.mdx), since replication signals + are half of what you will watch + +## Monitor at the boundaries you operate + +| Layer | Minimum signals | Where they come from | +| ------------------ | ------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------- | +| User and edge | Success, latency, correctness, broken out by geography and cohort | Your synthetic journey, real-user telemetry, CDN and traffic layer logs | +| HTTP and component | Request rate, error rate, latency percentiles, active component version, availability flag | `success` and `duration` metrics, `GET /status`, your readiness route | +| Node resources | CPU, memory, worker utilization, task queue latency, disk, restarts | `system_information`, `utilization`, `main-thread-utilization`, host or container metrics | +| Replication | Peer connections, per-database sockets, latency, convergence lag | `cluster_status`, `replication-latency`, `system.hdb_nodes` | +| Data | Read and write rate, transaction commit time, queue depth, freshness | `transaction-commit-time`, write and read transaction queue depth, your own sentinel | +| Storage | Database size, volume free space, table growth | `database-size`, `storage-volume`, `table-size` | +| Change | Component and config version, cohort, operator, start and end, outcome | `list_deployments`, `get_deployment`, `get_components`, `get_configuration` read-back | +| Recovery | Backup age, job state, verification result, last restore drill | `list_backups`, `verify_backup`, `get_job` | + +The Change row is the one teams leave out and the one that resolves incidents +fastest. Most production symptoms correlate with something a human did, so being +able to overlay deployments onto a latency graph is worth more than another +resource metric. + +## The signals worth an alert + +Harper records a large standard metric set automatically. Most of it belongs on a +dashboard. This much belongs on a pager: + +| Alert on | Metric or source | Why this one | +| -------------------------------------- | ------------------------------------------------------------------ | ---------------------------------------------------------------------------------------- | +| Journey success rate below SLO | `success`, by resource path and method | The only signal that maps directly to user impact | +| Journey latency at your SLO percentile | `duration`, by resource path and method | Alert on the percentile in your SLO, never on the mean | +| Worker saturation | `utilization`, per worker thread | Percentage of time the worker was processing requests. This is your real headroom signal | +| Event loop backpressure | `main-thread-utilization`, the `taskQueueLatency` attribute | Rises before throughput drops, so it is an early warning rather than a postmortem input | +| Replication convergence lag | `replication-latency`, or the receive-time gap in `cluster_status` | A node serving stale data looks healthy on every other signal | +| Write commit time | `transaction-commit-time` | Storage-level degradation shows here before it shows in request latency | +| Write queue depth growing | `write-transaction-queue-depth` | A growing queue means the node is accepting work faster than it can commit it | +| Storage headroom | `storage-volume` free, and `database-size` | Disk exhaustion is an outage with no graceful degradation | +| Missing peer | `cluster_status` | Covered in [Operating Replication](./operating-replication.mdx) | +| Backup age exceeding RPO | `list_backups` | The failure you will not notice until you need it | + +Two notes on how to set these. Alert on the percentile that appears in your SLO, +because a mean latency graph will look fine through an incident that is failing +your slowest ten percent of users. And set saturation thresholds from the +per-node capacity work in +[Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) rather than from a +generic number, since the point where latency leaves your SLO is specific to your +application and your data. + +## Get the metrics out + +Harper stores analytics locally in `hdb_raw_analytics` and aggregates them into +`hdb_analytics`. You can query those directly, but for a real monitoring setup +export them. + + + + +Use the [Grafana integration](/fabric/grafana-integration). It ships dashboards +over Harper's analytics without you building a pipeline, which is the fastest path +to the alert list above. + + + + +Deploy [`@harperdb/prometheus-exporter`](https://github.com/HarperFast/prometheus-exporter) +and scrape it into whatever you already run: + +```json +{ + "operation": "deploy_component", + "project": "prometheus-exporter", + "package": "@harperdb/prometheus-exporter", + "restart": true +} +``` + + + + +To find out what is actually available on your version rather than guessing from +documentation: + +```json +{ + "operation": "list_metrics", + "metric_types": ["builtin", "custom"] +} +``` + +Then [`describe_metric`](/reference/v5/analytics/operations#describe_metric) for +the shape of any one of them. + +Your application can add its own metrics with +[`server.recordAnalytics()`](/reference/v5/http/api#serverrecordanalyticsvalue-metric-path-method-type), +which is how you get business-level signals such as checkout completion onto the +same dashboard as node saturation. That correlation is usually what tells you +whether a technical symptom matters. + +:::tip +Check `analytics.replicate` in the +[analytics configuration](/reference/v5/analytics/overview#analytics-configuration) +before building dashboards. Whether metrics stay node-local or replicate changes +what a cluster-wide query means, and it is easier to decide that deliberately than +to discover it after building panels on the wrong assumption. +::: + +## Logs you can use during an incident + +Harper's `logger` global writes structured entries from component code: + +```javascript +logger.info('order submitted', { + component: 'orders-api', + version: process.env.APP_VERSION, + node: process.env.HARPER_NODE_NAME, + requestId: context.requestId, + operation: 'submit', + durationMs: elapsed, + outcome: 'ok', +}); +``` + +Include the component and its version, the node, a request or trace identifier, +the operation, its duration, the outcome, and a safe error class. That field set +is what lets you answer "was this only the new version" and "was this only one +node" without guessing, and those are the first two questions in almost every +incident. + +Levels run `trace`, `debug`, `info`, `warn`, `error`, `fatal`, and `notify`. The +default is `warn`, and `notify` is always logged regardless of level. Choose +levels deliberately: a production log at `debug` is a log nobody can read, and one +at `error` only has the incidents in it and none of the context. + +`console.log` output does not reach the log files unless `logging.console` is +enabled, so unstructured console output is not a production record. Centralize +logs off the node, because the node you most need logs from is the one you are +about to restart. + +Read logs through the API when you need them from a specific node: + +```json +{ + "operation": "read_log", + "limit": 200, + "level": "error" +} +``` + +## The five-minute triage sequence + +Run these in order. The goal is not diagnosis, it is narrowing. + +1. **Confirm and bound the impact.** What journey, starting exactly when, in which + geography or cohort, and which component version is live. Without a start time + you cannot correlate anything. + +2. **Compare the public route against a direct node call.** Request the same + journey through your traffic layer, then directly against each node on `9926`. + If direct calls succeed and the public route fails, you are looking at the + traffic layer or the availability flag, not at Harper. + +3. **Check admission state on every node.** `GET /status` on `9926` and your + readiness route. A node advertising unavailable is a node deliberately or + accidentally out of rotation, and finding that here saves a lot of time. + +4. **Compare a suspect node against a healthy peer.** Same call, both nodes, then + diff: + + ```json + { + "operation": "system_information", + "attributes": ["cpu", "memory", "threads", "harperdb_processes"] + } + ``` + + Differences between peers are more informative than absolute values, because + they tell you whether this is one node or the whole cluster. + +5. **Check the replication path.** `cluster_status` for the databases this journey + needs. Look for a missing peer, `connected: false`, or a widening gap between + `lastReceivedRemoteTime` and `lastReceivedLocalTime`, which means stale reads. + +6. **Correlate with the last change.** `list_deployments` for what shipped and + when, and `get_configuration` read back against what you believe is configured. + Remember that a configuration change applied without a restart leaves a node + running something other than its stated configuration. + +7. **Contain with the smallest reversible action.** Stop a ramp, drain one node, + deactivate a feature, restore prior traffic weights, or isolate suspected data. + Record the decision, who made it, and the next decision deadline. + +Two rules make this sequence work. Silence is a failed gate: if telemetry is +missing for the thing you are checking, treat that as a negative signal rather +than skipping the step. And containment comes before root cause. You can diagnose +after users are being served again. + +### Prove it + +Pick a fault and inject it on a non-production cluster, then time yourself. Good +candidates: saturate one node's workers, block the replication port on one peer, +make a downstream dependency return errors, or deploy a component that fails on +one route. + +Measure three things. How long until an alert fired. How long until an operator +following the sequence above could name the layer. And whether any step gave a +misleading answer, which is the most valuable output of the drill, because a +misleading signal during a real incident costs more than a missing one. + +## Operational notes + +- **Node-level dashboards, not cluster averages.** A cluster average conceals the + single node doing twice the work, which is the most common cause of a latency + complaint that looks like nothing on a dashboard. +- **Watch measured request distribution, not configured weights.** Per-node + request counts are the ground truth. Sticky sessions, DNS caching, and + connection reuse all skew actual distribution away from intent. +- **Annotate deployments onto your graphs.** If your monitoring supports + annotations, feed `list_deployments` into them. This single change resolves more + incidents faster than any additional metric. +- **`read_audit_log` needs transaction logging enabled** and is a heavier tool for + reconstructing what changed in a table. Know before an incident whether you have + it on, because turning it on afterwards does not help. +- **Protect your telemetry surfaces.** `read_log` and `get_components` can expose + configuration and source detail, so they are `super_user` operations for a + reason. Restrict and log their use. + +## Readiness checklist + +- [ ] A dashboard exists that can place a symptom at edge, traffic layer, node, + replication, data, or change +- [ ] Alerts on journey success and latency at the SLO percentile, not the mean +- [ ] Alerts on worker `utilization` and `taskQueueLatency`, thresholds derived + from measured per-node capacity +- [ ] Alerts on replication convergence lag and missing peers +- [ ] Alerts on storage headroom and backup age +- [ ] Metrics exported off the node, via Grafana on Fabric or the Prometheus + exporter self-managed +- [ ] `analytics.replicate` setting known and deliberate +- [ ] Structured logging includes component, version, node, request id, operation, + duration, and outcome +- [ ] Logs centralized off the node +- [ ] Deployments annotated onto dashboards +- [ ] Triage sequence written down where on-call can find it +- [ ] Fault injection drill completed, with time-to-detection recorded + +## Additional Resources + +- [Analytics overview](/reference/v5/analytics/overview) for the full standard + metric catalog and configuration options +- [Analytics operations](/reference/v5/analytics/operations) for `list_metrics` + and `describe_metric` +- [Grafana integration](/fabric/grafana-integration) for Fabric dashboards +- [`@harperdb/prometheus-exporter`](https://github.com/HarperFast/prometheus-exporter) + for self-managed metric scraping +- [Logging overview](/reference/v5/logging/overview), + [configuration](/reference/v5/logging/configuration), and + [operations](/reference/v5/logging/operations) +- [HTTP API reference](/reference/v5/http/api) for `server.recordAnalytics()` +- [Fabric logging](/fabric/logging) for log access on Fabric +- [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx) for the + gates these signals feed diff --git a/learn/administration/operating-replication.mdx b/learn/administration/operating-replication.mdx new file mode 100644 index 000000000..617f5c53f --- /dev/null +++ b/learn/administration/operating-replication.mdx @@ -0,0 +1,342 @@ +--- +title: Operating Replication +sidebar_position: 4 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +Replication is how a Harper cluster stays available: peers exchange data directly +over WebSockets, there is no primary, and a node that falls behind catches up on +its own. The [replication reference](/reference/v5/replication/overview) covers +how to configure that. This guide covers the part that only matters once real +users depend on it, which is how to tell whether replication is actually working. + +The distinction that runs through this guide is between a connection and +convergence. A connected socket proves two nodes can talk. It does not prove the +data a user is about to read is current. Those are different claims, and only one +of them is what your traffic admission decision depends on. + +## What You Will Learn + +- How to determine what your cluster actually replicates, rather than assuming a + default +- What moves between peers automatically, what does not, and which of those will + surprise you +- How to read `cluster_status` as an operator, including which timing field + actually indicates a node is behind +- How to prove convergence with a sentinel rather than inferring it from + connection state +- Which application behaviors replication cannot make safe, and what to do about + them instead + +## Prerequisites + +- A cluster of at least three nodes, so you can interrupt one peer and still + observe the others +- A `super_user` credential for the Operations API +- [How Harper Runs in Production](./how-harper-runs-in-production.mdx) and your + service boundary inventory +- Familiarity with how your cluster was joined, either through + `harper-config.yaml` routes or the + [clustering operations](/reference/v5/replication/clustering) + +## Know your own scope + +By default Harper replicates all data in all databases. Scope can be narrowed two +ways: per database in configuration, and per table in the schema. + +```yaml +replication: + databases: + - data + - system +``` + +```graphql +type LocalTableForNode @table(replicate: false) { + id: ID! + name: String! +} +``` + +All tables in a replicated database replicate unless the table opts out. So the +scope you are operating is the product of a config list, a set of schema +directives, and any directional routes someone added later. Do not reconstruct it +from memory. Read it off the running cluster: + + + + +```bash +curl -s -X POST https://my-node.example.com:9925/ \ + -H 'Content-Type: application/json' \ + -u 'admin:password' \ + -d '{"operation":"cluster_status"}' +``` + + + + +```javascript +await fetch('https://my-node.example.com:9925/', { + method: 'POST', + headers: { + 'Content-Type': 'application/json', + 'Authorization': 'Basic ' + btoa('admin:password'), + }, + body: JSON.stringify({ operation: 'cluster_status' }), +}); +``` + + + + +There is one socket per database per peer, so the sockets present in the response +are the ground truth about what is flowing. Run this from every node, not one. +Replication direction can be constrained per route, so node A's view of the +cluster is not necessarily node B's view, and a one-sided picture is how +directional configuration mistakes survive into production. + +:::warning +Whether the `system` database is in your replication scope is the highest-stakes +scoping question in a Harper cluster, because `system` holds `hdb_user`, +`hdb_role`, and `hdb_nodes`. Users and roles propagate only when `system` is +replicated, and a node that receives `system` must be trusted with its contents, +including encrypted secret rows. Confirm your own answer from `cluster_status` and +your configuration, and read +[replicating the system database with controlled flow](/reference/v5/replication/overview#replicating-the-system-database-with-controlled-flow) +before changing it. +::: + +## What moves automatically, and what does not + +| Object or change | Behavior | What it means for you | +| --------------------------- | ---------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | +| Data mutations | Insert, update, upsert, delete, and bulk load replicate for tables in scope | Monitor connections, latency, and convergence for the databases that serve critical journeys | +| Transactions | Replicated atomically, and a transaction may span multiple tables | Tables that need to commit together must live in the same database | +| A brand new node | Downloads the full database from the first node it connects to, then enters incremental replication | Budget the transfer, and expect load on the source node while it happens | +| A returning node | Resynchronizes automatically to catch up on the transactions it missed | This is not a full copy. A routine restart is much cheaper than a node replacement, so budget them separately | +| Component deployment | `deploy_component` supports `"replicated": true` for cluster-wide propagation | Useful once an artifact is trusted. Keep single-node deployment available for validation | +| Configuration | `set_configuration` supports `"replicated": true` | Never replicate node-local parameters. See the warning below | +| Users and roles | Propagate only when the `system` database is in replication scope | If `system` is out of scope, identity must be provisioned on every node by your own automation | +| Node registry (`hdb_nodes`) | Each node rewrites its own self-record from its `harper-config.yaml` routes on restart or component reload | Scoping applied through `add_node` alone does not survive a restart. Put durable constraints in config routes | +| Destructive schema changes | Behavior is operation and version dependent | Treat as high risk. Write an exact procedure, and verify the result on every node | + +The two rows most likely to bite you are the returning node and the node registry. + +On the returning node: it is common to see a routine restart budgeted as though it +were a full resynchronization, which makes teams avoid restarts they should be +comfortable with. A node whose databases have never synced does download them in +full. A node that was briefly offline catches up on what it missed. Measure both +on your own data volume once, and use the right number for the right situation. + +On the registry: a node's advertised record is derived from its configuration +file, and it replicates. That means a topology constraint you applied +imperatively is superseded the next time that node restarts or reloads +components, and the node quietly goes back to advertising itself more broadly +than you intended. + +:::danger +When replicating configuration, only send cluster-appropriate parameters. +Replicating a node-local value such as a port, `node.hostname`, a file path, TLS +material, or `replication.hostname`, `url`, or `routes` overwrites every peer's +own local value. To apply a cluster-wide change safely, use +`set_configuration` with `"replicated": true` for the parameter, then +`restart_service` with `"replicated": true`, which restarts nodes one at a time. +::: + +## Read `cluster_status` like an operator + +A trimmed response, with the fields that matter: + +```json +{ + "type": "cluster-status", + "node_name": "server-1.example.net", + "is_enabled": true, + "connections": [ + { + "url": "wss://server-2.example.net:9933", + "name": "server-2.example.net", + "database_sockets": [ + { + "database": "data", + "connected": true, + "latency": 0.7, + "lastCommitConfirmed": "Wed, 12 Feb 2025 19:09:34 GMT", + "lastReceivedRemoteTime": "Wed, 12 Feb 2025 16:49:29 GMT", + "lastReceivedLocalTime": "Wed, 12 Feb 2025 19:09:31 GMT" + } + ] + } + ] +} +``` + +What each field is telling you: + +- **`connected`** is the liveness of this one database's socket to this one peer. + A missing peer, or a peer present with `connected: false`, is actionable before + users notice. +- **`latency`** is the round trip to that peer in milliseconds. Alert on sustained + growth rather than on a single sample. +- **`lastCommitConfirmed`** is the last time this peer acknowledged receiving one + of your commits. If it stops advancing while you are still writing, your writes + are not landing on that peer. +- **`lastReceivedRemoteTime`** is the source node's timestamp on the newest + transaction you have received. +- **`lastReceivedLocalTime`** is your own clock when you received it. + +The last two are the pair that matters. **A widening gap between +`lastReceivedRemoteTime` and `lastReceivedLocalTime` means this node is behind and +working through a backlog.** That is the signal to alert on for convergence, and +it is the one that tells you a returning node is not ready for traffic yet. +`sendingMessage` appears while a transaction is actively being sent and is absent +when the socket is idle, so its absence is not a fault. + +Inventory `system.hdb_nodes` alongside this and compare it to your intended +topology. Configuration intent and live peer state should agree, and the node's +own row is in there too, not just its peers. + +### What to alert on + +- A peer missing entirely from `connections`, or `connected: false` on a database + that serves a critical journey +- Sustained growth in `latency`, judged against your own baseline +- `lastCommitConfirmed` not advancing on a peer while writes are occurring +- A `lastReceivedRemoteTime` to `lastReceivedLocalTime` gap exceeding your + admission budget +- Repeated reconnects, which are visible in the logs even when a point-in-time + status check looks healthy +- Version or configuration drift between peers + +## Prove convergence, not connection + +Socket state cannot tell you that a specific business record is current. For +anything where the answer matters, write a sentinel and read it back from the +peer. + +```javascript +// On the source node: write a sentinel with a known value +const marker = { id: 'convergence-probe', writtenAt: Date.now(), from: 'server-1' }; +await tables.OpsProbe.put(marker.id, marker); +``` + +```bash +# On the target node: read it back and compare writtenAt +curl -s https://server-2.example.net:9926/OpsProbe/convergence-probe \ + -u 'admin:password' +``` + +The interval between the write and the moment the peer returns the new value is +your measured convergence time for that database, under whatever load the cluster +is carrying at the time. Run it under load, not on an idle cluster, and record the +result. That number is what your node admission gate should be compared against, +and it is an input to the RPO work in +[Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx). + +The `replication-latency` metric from +[analytics](/reference/v5/analytics/overview#replication-metrics) gives you the +continuous version of the same measurement: the difference between the source +commit timestamp and local time, reported per node, database, and table. Use the +sentinel for a definitive answer during a change, and the metric for a dashboard. + +## Consistency belongs in the application design + +Replication makes data available on every peer. It does not make every peer agree +at every instant, and a distributed decision made on a node that has not yet +converged can observe stale state and act on it. + +This is not a Harper limitation to work around, it is the property that lets any +node serve any request without a coordinator. But it means a specific class of +operation is unsafe if you write it as a plain read followed by a write: + +- Claim-once actions: redeeming a code, assigning a unique handle, awarding a + one-per-customer offer +- Hard floors: inventory that must not go negative, a balance that must not + overdraw +- Global limits: a rate limit or quota enforced across the whole cluster +- State machine transitions where two nodes could both believe they are making + the same transition + +For each of these, choose one of three designs: route all decisions for a given +key to a single owner, use a serialization mechanism so the conflict is resolved +in one place, or delegate to an external coordinator. Then test your +read-after-write expectations through the actual public route rather than against +a single node, because a single node always looks consistent to itself. + +### Prove it + +On a non-production cluster under write load: + +1. Record baseline `cluster_status` timing fields on every node. +2. Interrupt replication to one peer, by stopping the node or blocking the + replication port, while writes continue elsewhere. +3. Watch what your monitoring reports, and how long it takes to say anything. Note + whether `connected` flipped, whether latency alerted, and how long until a + human would have known. +4. Verify your application's actual behavior on the isolated node. Does the + critical journey fail, serve stale data, or serve correctly? All three are + possible and you should know which. +5. Restore the peer, then measure convergence with the sentinel until it is + current. +6. Compare the convergence time against the admission gate in + [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). + If your gate would have admitted the node before step 5 finished, the gate is + wrong. + +## Operational notes + +- **mTLS is required on the secure replication port and cannot be disabled.** + Certificate expiry is therefore a replication outage, so treat certificate + lifetime as an operational deadline with its own alert. See + [certificate management](/reference/v5/security/certificate-management). +- **Gossip discovery means one route can join a whole cluster.** A node that + connects to one peer discovers the rest. That is convenient and it means an + accidental route can widen your topology more than you intended. +- **Sharding is a separate scope control.** If you use + [sharding](/reference/v5/replication/sharding), not every node holds every + record, so "this node is converged" and "this node can answer this query + locally" become different questions. +- **Analytics data has its own replication setting.** See `analytics.replicate` in + the [analytics configuration](/reference/v5/analytics/overview#analytics-configuration) + before you build dashboards that assume metrics are or are not cluster-wide. +- **Keep node-local operational state out of scope.** Availability flags, + maintenance markers, and anything else that describes one node rather than the + cluster should not replicate, or a routine drain propagates to peers. + +## Readiness checklist + +- [ ] Replication scope read off the running cluster, from every node, not + reconstructed from memory +- [ ] Whether the `system` database is in scope is a documented, deliberate + decision +- [ ] Identity provisioning procedure exists and matches that decision +- [ ] Tables requiring one transaction boundary confirmed to be in one database +- [ ] Durable topology constraints live in `harper-config.yaml` routes, not only in + `add_node` calls +- [ ] Measured convergence time recorded under load, per critical database +- [ ] Alerts configured on missing peers, sustained latency growth, stalled + `lastCommitConfirmed`, and the remote-to-local receive gap +- [ ] Non-commutative operations identified and given an owner, a serialization + point, or an external coordinator +- [ ] Replication certificate expiry dates tracked with an alert +- [ ] Replication interruption and recovery exercise completed and dated + +## Additional Resources + +- [Replication overview](/reference/v5/replication/overview) for routes, scope, + controlled flow, and securing connections +- [Clustering operations](/reference/v5/replication/clustering) for `add_node`, + `set_node`, and the `cluster_status` response in full +- [Sharding](/reference/v5/replication/sharding) for controlling how many nodes + hold a given record +- [Replication metrics](/reference/v5/analytics/overview#replication-metrics) for + `replication-latency` and byte counters +- [Certificate management](/reference/v5/security/certificate-management) and + [certificate verification](/reference/v5/security/certificate-verification) +- [Database schema](/reference/v5/database/schema) for the `replicate` table + directive +- [Monitoring and Triage](./monitoring-and-triage.mdx) for putting these signals + on a dashboard diff --git a/learn/administration/production-readiness-checklist.mdx b/learn/administration/production-readiness-checklist.mdx new file mode 100644 index 000000000..29cabc5aa --- /dev/null +++ b/learn/administration/production-readiness-checklist.mdx @@ -0,0 +1,230 @@ +--- +title: Production Readiness Checklist +sidebar_position: 9 +--- + +This is the aggregated launch gate for the whole Administration track. Every item +appears in one of the earlier guides, which is where the reasoning lives. This +page exists so you have one thing to work down before a launch, and one thing to +hand to a reviewer who asks how you operate Harper. + +Copy it into your own runbook and adapt it. It is a starting point for your gate, +not a substitute for having one. + +## How to use this + +Each item is a claim about your deployment. A claim counts when there is evidence: +a measured number with a date, a written procedure, a configured alert, or a +completed drill. "We know about that" is not evidence, and neither is a passing +intention. + +Some items will not apply to you, and that is fine. Mark them not applicable with +a reason rather than deleting them, so a reviewer can see the decision was made +rather than missed. + +Three things are worth deciding before you start: + +- **Who signs off.** A gate with no named approver is a document, not a gate. +- **What "outstanding" means.** Some items can be accepted as gaps with an owner + and a date. Decide in advance which ones cannot. +- **When you run it again.** This is not a one-time launch artifact. Re-run it + after a Harper version upgrade, a topology change, or significant data growth, + since most of the measured numbers expire. + +## Service definition + +From [How Harper Runs in Production](./how-harper-runs-in-production.mdx). + +- [ ] Critical user journeys named, with an owner for each +- [ ] Ports confirmed from `get_configuration` rather than assumed +- [ ] Operations API not reachable from the public internet +- [ ] Harper version recorded per node, with an alert on drift +- [ ] Databases listed, with replication scope confirmed from `cluster_status` +- [ ] Storage engine recorded per database +- [ ] Components and versions recorded from `get_components` +- [ ] Downstream dependencies named, with their failure behavior understood +- [ ] Fabric or self-managed responsibilities understood and divided + +## Capacity and topology + +From [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx). + +- [ ] `F`, the tolerated simultaneous node loss, declared explicitly and visible to + change operators +- [ ] Per-node throughput measured with your own application code, data shape, and + query mix +- [ ] Measurement taken with a peer synchronizing, not on an idle cluster +- [ ] Target utilization derived from latency at your SLO percentile +- [ ] `(N - F) x per-node x utilization >= peak` verified with real numbers +- [ ] Measurement conditions recorded alongside the number, and dated +- [ ] Topology's "what must be proven" satisfied, including independence of failure + domains +- [ ] Maintenance policy states what happens when a drain would breach `F` +- [ ] Storage growth thresholds set, separately from request-rate thresholds + +## Health and traffic admission + +From +[Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). + +- [ ] Liveness, availability flag, application readiness, and journey synthetic all + exist as separate signals +- [ ] `@harperdb/status-check` deployed, declared in `config.yaml` rather than by + hand +- [ ] Traffic layer health check points at `GET /status` on the application port +- [ ] Availability flag confirmed to be outside replication scope +- [ ] Readiness route scoped to one journey's dependencies, with a timeout on every + downstream call +- [ ] Readiness response includes the component version +- [ ] Drain sequence documented, with measured time to zero traffic +- [ ] Return sequence documented, including convergence verification and a + stability window before full weight +- [ ] Measured node re-entry time recorded + +## Replication + +From [Operating Replication](./operating-replication.mdx). + +- [ ] Replication scope read off the running cluster, from every node +- [ ] Whether the `system` database is in scope is a deliberate, documented + decision +- [ ] Identity provisioning procedure matches that decision +- [ ] Tables that must commit together confirmed to be in one database +- [ ] Durable topology constraints live in `harper-config.yaml` routes, not only in + `add_node` calls +- [ ] Measured convergence time recorded under load, per critical database +- [ ] Alerts on missing peers, sustained latency growth, stalled + `lastCommitConfirmed`, and the receive-time gap +- [ ] Non-commutative operations identified and given an owner, a serialization + point, or an external coordinator +- [ ] Replication certificate expiry tracked with an alert +- [ ] Replication interruption and recovery exercise completed and dated + +## Deployment and rollback + +From [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx). + +- [ ] Artifacts immutable, referenced by pinned version rather than by branch +- [ ] Deploy and expose are separate actions in the pipeline +- [ ] Previous known-good artifact addressable, and redeploying it rehearsed +- [ ] Cohort ladder defined, from smallest useful sample to full exposure +- [ ] Stop thresholds and hold times declared before a rollout begins +- [ ] Named decision owner for advance, hold, stop, and roll back +- [ ] Pipeline polls the deployment record and the restart job separately +- [ ] `deployment_timeout` and `ignore_replication_errors` set deliberately +- [ ] Configuration changes go through the same change record as code +- [ ] `get_configuration` read back after every configuration change +- [ ] Node-local parameters never replicated +- [ ] Schema changes follow expand then contract, with destructive operations + documented and verified per node +- [ ] Forward repair defined for changes that alter persisted meaning +- [ ] Measured release reversal time recorded from a rehearsal + +## Observability + +From [Monitoring and Triage](./monitoring-and-triage.mdx). + +- [ ] A dashboard exists that can place a symptom at edge, traffic layer, node, + replication, data, or change +- [ ] Alerts on journey success and latency at the SLO percentile, not the mean +- [ ] Alerts on worker utilization and task queue latency, with thresholds derived + from measured capacity +- [ ] Alerts on replication convergence lag +- [ ] Alerts on storage headroom and backup age +- [ ] Metrics exported off the node +- [ ] `analytics.replicate` setting known and deliberate +- [ ] Structured logs include component, version, node, request id, operation, + duration, and outcome +- [ ] Logs centralized off the node +- [ ] Deployments annotated onto dashboards +- [ ] Triage sequence written where on-call can find it +- [ ] Fault injection drill completed, with time to detection recorded + +## Backup and recovery + +From [Backup and Recovery](./backup-and-recovery.mdx). + +- [ ] Failure classes enumerated, each with the mechanism that addresses it +- [ ] Recovery point and recovery time stated per database +- [ ] Mechanism chosen per database according to its storage engine +- [ ] No database in scope uses per-table storage paths, or the exclusion is known + and accepted +- [ ] Backup cadence matches the stated recovery point +- [ ] Backup volume sized against the full cost, including the non-incremental + transaction-log and blob snapshots +- [ ] Off-host copy exists in a destination that does not share a failure domain +- [ ] Managed repository copies take the whole per-database directory, quiesced or + from an atomic snapshot +- [ ] `verify_backup` runs on a schedule +- [ ] Blob integrity understood to be unverified by `verify_backup` +- [ ] Restore constraints documented for user databases, component-held databases, + and `system` +- [ ] Restore authority named, and restore execution logged +- [ ] Restore drill completed and dated, with measured recovery time and actual + data loss +- [ ] `system` database restore rehearsed offline + +## Objectives + +From [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx). + +- [ ] Recovery point, recovery time, and availability stated per journey with the + measurement boundary named +- [ ] Exclusions documented +- [ ] Every scenario in the scenario map has a named mechanism +- [ ] Availability target converted to minutes per month +- [ ] Error budget policy states what changes when the budget runs low +- [ ] Backup cadence reconciled against recovery point +- [ ] Restore time plus detection time reconciled against recovery time +- [ ] Convergence time reconciled against the admission gate +- [ ] Every unreconciled gap has an owner and a date +- [ ] Degraded modes declared per journey, with authority to invoke them + +## Security and access + +- [ ] Least privilege enforced for operators, applications, and automation +- [ ] `super_user` credentials inventoried, with a rotation procedure +- [ ] Operations API restricted to administrative networks +- [ ] TLS material inventoried, with expiry alerts for both application and + replication certificates +- [ ] Deploy credentials held in a secret store, not in pipeline configuration +- [ ] Administrative access logged, including CLI access on the hosts +- [ ] Filesystem access on nodes understood to permit offline restore without an + API credential + +## Ownership and runbooks + +- [ ] On-call rotation and escalation path defined +- [ ] Runbooks exist for drain, return, deploy, reverse, restore, and cluster + expansion +- [ ] Every runbook has been executed by someone other than its author +- [ ] Change record location known, and used +- [ ] Incident and postmortem process defined +- [ ] A named owner for this checklist, and a date for the next review + +## Exercises completed + +Record the date of the most recent run of each. An undated exercise is an +undocumented one. + +| Exercise | Guide | Last run | Result | +| ----------------------------------------- | ------------------------------------------------------------------------ | -------- | ------ | +| Service boundary inventory | [How Harper Runs in Production](./how-harper-runs-in-production.mdx) | | | +| Drain and return under load | [Health Checks](./health-checks-and-traffic-admission.mdx) | | | +| Peak-load drain and rejoin | [Sizing](./sizing-a-harper-cluster.mdx) | | | +| Replication interruption and recovery | [Operating Replication](./operating-replication.mdx) | | | +| Fault injection and time to detection | [Monitoring and Triage](./monitoring-and-triage.mdx) | | | +| Stopped rollout and reversal | [Safe Deployments](./safe-deployments-and-rollback.mdx) | | | +| Restore drill, user database | [Backup and Recovery](./backup-and-recovery.mdx) | | | +| Restore drill, `system` database, offline | [Backup and Recovery](./backup-and-recovery.mdx) | | | +| Scenario tabletop | [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx) | | | + +## Additional Resources + +- [Reliability Plan Template](./reliability-plan-template.mdx) for the document + that holds the answers this checklist asks for +- Every guide in this section, linked per group above +- [Security overview](/reference/v5/security/overview) and + [certificate management](/reference/v5/security/certificate-management) +- [Configuration options](/reference/v5/configuration/options) for the settings + referenced throughout diff --git a/learn/administration/reliability-plan-template.mdx b/learn/administration/reliability-plan-template.mdx new file mode 100644 index 000000000..f580ef251 --- /dev/null +++ b/learn/administration/reliability-plan-template.mdx @@ -0,0 +1,199 @@ +--- +title: Reliability Plan Template +sidebar_position: 10 +--- + +The [Production Readiness Checklist](./production-readiness-checklist.mdx) asks +the questions. This is the document that holds the answers. + +Copy it, keep one per service, and keep it where your on-call can reach it during +an incident rather than in a wiki nobody remembers. It is deliberately short. +A reliability plan that takes a day to read is a reliability plan nobody consults +at 3am. + +Example values are included in italics to show the intended level of specificity. +Replace them. + +--- + +## 1. Service identity + +| Field | Value | +| ------------------ | ---------------------------------------------------------- | +| Service name | _orders-api_ | +| Business owner | _name, team_ | +| Technical owner | _name, team_ | +| On-call rotation | _link to schedule_ | +| Escalation path | _first, second, and decision authority for a stop_ | +| Harper deployment | _Fabric cluster, or self-managed, with cluster identifier_ | +| Harper version | _5.2.x, per node_ | +| Plan last reviewed | _date_ | +| Next review due | _date_ | + +## 2. Critical journeys + +The user-facing paths that must work. Everything else in this plan is stated per +journey, so this table defines the scope of all of it. + +| Journey | Description | Databases and tables involved | Downstream dependencies | Owner | +| ---------------- | ----------------------------- | ----------------------------- | ----------------------- | ----- | +| _submit order_ | _POST through the public API_ | _orders.orders, orders.items_ | _pricing, payments_ | | +| _browse catalog_ | _cached read_ | _catalog.products_ | _none_ | | + +## 3. Objectives + +| Journey | Availability target | Measurement boundary | Recovery time objective | Recovery point objective | Exclusions | +| ------- | ------------------- | -------------------- | ----------------------- | ------------------------ | ------------------------------------------ | +| | _99.95%_ | _at the CDN edge_ | _15 min from detection_ | _5 min_ | _planned maintenance, third-party outages_ | + +Monthly unavailability budget: _21 min 55 sec at 99.95%_ + +Error budget policy: _what changes when the remaining budget falls below a +threshold, and who decides_ + +## 4. Architecture and topology + +- Node count and `F`, the tolerated simultaneous loss: _5 nodes, F = 1_ +- Failure domain layout: _how nodes are distributed, and the proof that domains are independent_ +- Traffic layer and routing: _what fronts Harper, and how it selects nodes_ +- Replication scope: _which databases, which tables excluded, and whether `system` is in scope_ +- Sharding in use: _yes or no, and where_ +- Storage engine per database: _per database_ + +## 5. Capacity + +| Measurement | Value | Conditions | Date measured | +| -------------------------- | ----- | ------------------------------------------------ | ------------- | +| Peak required throughput | | _per journey_ | | +| Tested per-node throughput | | _data volume, query mix, peer synchronizing y/n_ | | +| Target utilization | | _derived from latency at the SLO percentile_ | | +| Surviving capacity at `F` | | _measured in the drain drill, not calculated_ | | + +Maintenance policy when a drain would breach `F`: _pause, add capacity, or accept +with documented duration_ + +## 6. Traffic admission + +- Liveness probe: _endpoint, interval, threshold_ +- Availability flag: _`status-check` deployed how, and who may set it_ +- Readiness route: _path, and the dependencies it checks_ +- Journey synthetic: _what it does, from where, how often_ +- Drain sequence: _link to runbook_ +- Return sequence, including stability window: _link to runbook_ +- Measured time to zero traffic after drain: _value, date_ +- Measured node re-entry time including convergence: _value, date_ + +## 7. Replication + +- Measured convergence time under load, per critical database: _value, date_ +- Measured full synchronization time for a new node: _value, date_ +- Admission gate threshold, and how it compares to convergence: _value_ +- Non-commutative operations and their handling: _operation, and the owner, serialization point, or coordinator_ +- Certificate expiry dates and alert lead time: _dates_ + +## 8. Deployment and change + +- Artifact source and reference format: _registry or git, pinned version format_ +- Cohort ladder: _one node, internal, 5 percent, 25 percent, full_ +- Stop thresholds and hold times: _stated numerically_ +- Decision authority for advance, hold, stop, roll back: _name or role_ +- Restart mode used, and how completion is confirmed: _`rolling`, polling the job_ +- Measured release reversal time: _value, date_ +- Forward repair procedures for data-affecting releases: _link_ +- Change record location: _link_ + +## 9. Observability + +- Dashboard location: _link_ +- Alerts configured, with thresholds and destinations: _table or link_ +- Metric export path: _Grafana on Fabric, or Prometheus scrape_ +- Log destination and retention: _system, retention period_ +- Deployment annotations enabled: _yes or no_ +- Triage runbook: _link_ +- Measured time to detection, per drill: _value, date_ + +## 10. Backup and recovery + +| Database | Mechanism | Cadence | Retention | Off-host destination | Last verified | Last restore drill | Measured restore time | Measured data loss | +| -------- | --------- | ------- | --------- | -------------------- | ------------- | ------------------ | --------------------- | ------------------ | +| | | | | | | | | | + +- Restore authority: _who may execute a restore_ +- Restore constraints per database: _online, or offline only because a component holds it open_ +- `system` database restore procedure: _offline only, link to runbook_ +- Failure classes assigned to restore versus forward repair: _table_ + +## 11. Scenario responses + +| Scenario | Mechanism | Runbook | What proves it works | Last exercised | +| ---------------------------------- | --------- | ------- | -------------------- | -------------- | +| _single node loss_ | | | | | +| _node return after restart_ | | | | | +| _node replacement_ | | | | | +| _failure domain loss_ | | | | | +| _region loss_ | | | | | +| _bad release, no data change_ | | | | | +| _bad release that changed data_ | | | | | +| _accidental destructive operation_ | | | | | +| _storage loss on one node_ | | | | | +| _certificate expiry_ | | | | | +| _downstream dependency outage_ | | | | | + +## 12. Degraded modes + +| Journey | Condition | Declared behavior | Who may invoke | Automatic or manual | +| ------- | ------------------------------- | ------------------------------------------- | -------------- | ------------------- | +| | _pricing service unavailable_ | _serve last known price, flag the response_ | | | +| | _node behind on replication_ | _remove from rotation_ | | | +| | _cluster below capacity budget_ | _shed non-critical traffic_ | | | + +## 13. Security and access + +- Operator roles and privileges: _who has what_ +- `super_user` credential inventory and rotation: _link, schedule_ +- Operations API network restriction: _how enforced_ +- Certificate inventory and expiry alerts: _link_ +- Deploy credential custody: _secret store, rotation method_ +- Administrative access logging, including host-level: _system_ + +## 14. Open risks + +Accepted gaps, each with an owner and a date. This section existing and being +honest is worth more than it being empty. + +| Risk | Impact if realized | Why accepted | Owner | Review date | +| ---- | ------------------ | ------------ | ----- | ----------- | +| | | | | | + +## 15. Revision history + +| Date | Author | Change | Trigger | +| ---- | ------ | ------ | ---------------------------------------------------- | +| | | | _launch, version upgrade, topology change, incident_ | + +--- + +## Keeping this current + +Most of the numbers in this plan are properties of a Harper version, a data +volume, a topology, and a set of components. All four change, so the numbers +expire. Re-measure and revise after: + +- A Harper version upgrade +- A topology change, including adding or removing nodes +- Significant data growth +- Any incident that produced a surprise + +Date every number. An undated measurement in a reliability plan is worse than a +blank field, because a blank field prompts someone to go and measure. + +## Additional Resources + +- [Production Readiness Checklist](./production-readiness-checklist.mdx) for the + gate that populates this plan +- [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx) for how + to derive sections 3 and 11 +- [Backup and Recovery](./backup-and-recovery.mdx) for section 10 +- [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx) for + section 8 +- [Monitoring and Triage](./monitoring-and-triage.mdx) for section 9 diff --git a/learn/administration/safe-deployments-and-rollback.mdx b/learn/administration/safe-deployments-and-rollback.mdx new file mode 100644 index 000000000..41310686c --- /dev/null +++ b/learn/administration/safe-deployments-and-rollback.mdx @@ -0,0 +1,328 @@ +--- +title: Safe Deployments and Rollback +sidebar_position: 6 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +[Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx) gets an +immutable artifact onto your cluster from a pipeline. This guide is about what +happens around that call: how to limit who sees a change, what evidence justifies +expanding it, and how to reverse it when the evidence says stop. + +Harper makes reversal unusually cheap, because your application is a component +deployed from an immutable reference, so rolling back is deploying the previous +reference. There is no image to rebuild and no instance to replace. That only +helps if the previous reference is still addressable and someone has done it +before under calm conditions, which is what makes rollback a designed capability +rather than a hope. + +## What You Will Learn + +- How to separate build, deploy, activate, and expose into four decisions with + four different control points +- The change loop, and what counts as evidence at each step +- Five rollout patterns and the Harper operations behind each +- What `"restart": true` and `"restart": "rolling"` actually do, which is not what + most people assume +- How to classify a change so you reverse the right thing, and why reversing the + wrong thing can cause data loss + +## Prerequisites + +- A cluster of at least three nodes, so you can hold one out of rotation and still + meet capacity +- A working pipeline that deploys from an immutable reference + ([Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx)) +- [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx), + since every pattern here depends on being able to move traffic +- [Monitoring and Triage](./monitoring-and-triage.mdx), since every gate here + depends on being able to compare versions + +## Separate four decisions + +Most bad deployments come from collapsing these into one action. + +| Decision | The question | Harper control point | +| ------------ | ----------------------------------------- | ----------------------------------------------------------------------------------------------- | +| **Build** | What immutable artifact exists? | A pinned package reference, a versioned tarball, a checksum, a dependency lock | +| **Deploy** | Which nodes have that artifact installed? | `deploy_component`, with `replicated` true or false | +| **Activate** | Which code path is actually enabled? | `urlPath` and `host` mounting, component configuration, a feature flag, a header or tenant rule | +| **Expose** | Which production traffic reaches it? | Traffic layer weights, the availability flag, cohort or geography targeting | + +Once these are separate you get options that do not exist otherwise. You can +install an artifact on every node and activate it nowhere. You can activate a +route and expose it only to internal traffic. And when something is wrong you can +reverse exposure in seconds without touching what is installed, which is the +fastest containment action available to you. + +`urlPath` mounts a component at an HTTP path. +`host` serves it on a virtual hostname. Both are +persisted on the component's root config entry, so they are part of the deployed +state rather than a runtime toggle. See +[HTTP middleware routing](/reference/v5/http/overview#middleware-routing). + +## The change loop + +1. **Preflight.** Confirm the target version, the component inventory from + `get_components`, `cluster_status` convergence, peer capacity with one node + held out, backup posture if data is at risk, and that the previous known-good + artifact is still addressable. + +2. **Limit.** Choose the smallest cohort that produces useful evidence. One + drained node, an internal cohort, a low-risk geography, a tenant set, or a + small weighted slice. Smaller is better right up until the sample is too small + to distinguish signal from noise. + +3. **Observe.** Compare the new and old versions on the same metrics: request + success, latency at your SLO percentile, worker saturation, logs and traces, + data correctness, replication behavior, and downstream errors. Comparison + against the other cohort is the point. Absolute numbers on the new version tell + you much less. + +4. **Decide.** Advance, hold, stop, or roll back, against thresholds declared + before you started, with a named owner. Missing telemetry is a failed gate, not + a pass. + +5. **Expand.** Increase exposure only after minimum sample and hold conditions + pass, and keep enough healthy capacity in the old version to reverse. + +6. **Close.** Verify uniform artifact and configuration across nodes, restore + intended traffic, record the actual outcome, and keep the evidence with the + change record. + +Declaring thresholds in step 4 before step 2 is the part that gets skipped and the +part that matters. A threshold invented while looking at a live graph is not a +threshold, it is a negotiation, and it always resolves toward shipping. + +## Five rollout patterns + +| Pattern | How Harper is used | Fits when, and watch out for | +| ----------------------------- | ------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------- | +| Isolated node validation | Drain one node, deploy with `"replicated": false`, validate, re-admit a small cohort | Strongest infrastructure and component validation. Requires N+1 capacity and node targeting | +| Replicated rolling deployment | `deploy_component` with `"replicated": true` and `"restart": "rolling"` | Efficient once you trust the artifact. Rolling alone is not progressive delivery, add traffic and telemetry gates | +| Feature-targeted release | Deploy compatible code broadly, activate by flag, header, tenant, or geography | Best when behavior can separate from code placement. Guard flag ownership and cleanup | +| Parallel environment | Deploy to a separate node pool, shift traffic after validation | Strong isolation and fast reversal. Write ownership and replication between pools need real design | +| Regional progression | Promote the same artifact region by region, exposing each after local gates | Good locality and blast radius control. One region does not predict every traffic shape | + +Isolated node validation, concretely: + + + + +```json +{ + "operation": "deploy_component", + "project": "orders-api", + "package": "https://artifacts.example.net/orders-api-2.4.1.tgz", + "replicated": false, + "restart": true +} +``` + +Drain the node first, deploy, run your readiness route and journey synthetic +against it directly, then admit a bounded cohort. + + + + +```json +{ + "operation": "deploy_component", + "project": "orders-api", + "package": "https://artifacts.example.net/orders-api-2.4.1.tgz", + "replicated": true, + "restart": "rolling" +} +``` + +Same immutable reference, now cluster-wide. Poll the returned `restartJobId` with +[`get_job`](/reference/v5/operations-api/operations#get_job), and poll +[`get_deployment`](/reference/v5/operations-api/operations#get_deployment) until +the deployment reports success rather than treating the response as completion. + + + + +Prefer a pinned version or an immutable tarball over a moving branch reference. +A branch moves, so you can neither audit what was running yesterday nor redeploy +it. + +## What restart actually does + +This is worth reading carefully, because the naming invites a wrong assumption. + +**`"restart": true`** restarts the HTTP worker threads on the node handling the +call, and waits for that restart to finish before responding. A successful +response therefore means every worker on that node is serving the new code. Until +a worker has been replaced it is still running the previous code, and on platforms +where replacements share a listening port it keeps accepting connections during +the changeover. The wait follows the restart's own progress rather than a fixed +timeout, so the response can take tens of seconds on a slow install with many +worker threads. + +**`"restart": "rolling"`** does not restart inline. It starts a replicated +`restart_service` job and returns a `restartJobId` for you to poll. Use this when +your caller has a short request timeout. + +Two consequences for your pipeline. A caller with an aggressive HTTP timeout +should use `"rolling"` and poll, because giving up on `true` does not stop the +restart, it just leaves you without the result. And a failed restart does not fail +the deploy: the component is installed and replicated either way, so your pipeline +needs to check both outcomes separately rather than assuming one implies the +other. + +Two parameters worth setting deliberately on replicated deploys: + +- `deployment_timeout` is how long a peer waits + for the replicated payload before failing, defaulting to 120000 ms. Raise it for + large components or slow links. +- `ignore_replication_errors` treats a peer that + fails to receive the deploy as non-fatal. By default a failed peer makes the + whole operation return a non-2xx status, while the component is still deployed + on the origin node. Decide which behavior you want before you need it, because + the default leaves you in a mixed-version state with a failed response, and that + is a confusing thing to reason about mid-incident. + +## Configuration changes are deployments too + +A configuration change carries the same risk as a code change and gets less +ceremony, which is backwards. + +`set_configuration` supports `"replicated": true` +to apply a change across the cluster in one call, with per-node outcomes in the +response. To finish the change cluster-wide, follow with `restart_service` using +`"replicated": true`, which restarts nodes one at a time. + +```json +{ + "operation": "set_configuration", + "logging_level": "info", + "replicated": true +} +``` + +:::danger +Only replicate cluster-appropriate parameters. Node-local values such as ports, +`node.hostname`, file paths, TLS material, and `replication.hostname`, `url`, or +`routes` would overwrite every peer's own values. Replicating one of these is a +cluster-wide outage delivered in a single API call. +::: + +Two more things to hold onto. A change takes effect only after a restart, so a +node that has been reconfigured and not restarted is running the old +configuration while reporting the new one. And `get_status` reports a +`restartRequired` flag, but it tracks component and code restarts rather than +configuration changes, so it will not tell you a configuration change is still +pending. Track pending configuration in your change record instead, and read back +`get_configuration` after the restart to confirm. + +## Rollback is a designed capability + +Classify the change before choosing a reversal path, because these have different +compatibility requirements, different authorities, and very different blast +radii: + +| Change type | Reversal | Watch out for | +| ---------------------- | ------------------------------------------------- | ------------------------------------------------------------------- | +| Component release | Deploy the previous immutable reference | The previous artifact must still be addressable | +| Feature behavior | Deactivate the flag, no deploy needed | Fastest reversal available. Requires the flag to have been built in | +| Traffic exposure | Restore prior weights or set the node unavailable | Fastest containment. Does not undo anything already written | +| Configuration | `set_configuration` back, then restart | Needs a restart, so it is not instant | +| Harper runtime version | Version-specific procedure | Mixed versions change replication and deploy behavior | +| Schema or data | Forward repair or restore | See the warning below | + +Practices that make each of these real: + +- **Keep the previous package addressable and rehearse redeploying it** before + launch, not during an incident. +- **Prefer expand-then-contract schema evolution.** Add the new shape, migrate, + then remove the old shape in a separate change. Destructive schema behavior is + operation and version dependent, so each destructive change needs an exact + written procedure and verification on every node. +- **Use the same gates for rollback as for forward movement.** Availability, + journey synthetic, peer stability, data validation, traffic reconciliation. A + rollback is a deployment and can fail like one. + +:::warning +If a release changed the meaning of persisted data, code rollback alone will not +fix it, and restoring a database to undo application code is usually the wrong +move. A restore rolls back every write in the window, including all the valid +ones, so it can violate your RPO in order to fix a code bug. Define forward repair +or replay for these cases instead. See +[Backup and Recovery](./backup-and-recovery.mdx). +::: + +### Prove it + +Rehearse a stopped rollout end to end on a non-production cluster: + +1. Declare a threshold before you start, for example "stop if journey success on + the new cohort is more than 0.5 percent below the control cohort over five + minutes." +2. Deploy a component that fails that threshold deliberately, to a bounded cohort. +3. Detect it through your dashboards rather than because you know what you did. +4. Reverse it, and time from decision to restored traffic. That number is your + release RTO, and it belongs in your reliability plan. +5. Verify uniform state afterwards: `get_components` on every node, and + `list_deployments` showing the reversal. + +The step people fail is 5. A partially reversed cluster looks fine on a dashboard +because the healthy majority dominates the average. + +## Operational notes + +- **Version parity is an operating requirement.** Mixed Harper versions in a + cluster change replication and deployment behavior, so a rollout that stalls + halfway is a state you want to detect and exit, not sit in. +- **Keep deploy credentials in your delivery platform's secret store**, use TLS + and least privilege, and retain the operation result as change evidence. See + [secrets](/reference/v5/security/secrets). +- **Component deployment can replicate, so a deploy is a cluster event.** Keep + isolated single-node deployment available for validation, because if the only + deployment path you have is replicated then you have no way to test anything on + one node. +- **Flag cleanup is part of the release.** A feature flag with no owner and no + removal date becomes permanent configuration that nobody understands, and it + will eventually be the thing nobody can explain during an incident. +- **Record who decided, not only what happened.** Named decision authority is what + makes a stop gate function under pressure, and it costs nothing to write down in + advance. + +## Readiness checklist + +- [ ] Artifacts are immutable and referenced by pinned version, never by branch +- [ ] Deploy and expose are separate actions in your pipeline +- [ ] The previous known-good artifact is addressable, and redeploying it has been + rehearsed +- [ ] Cohort ladder defined, from smallest useful sample to full exposure +- [ ] Stop thresholds and hold times declared before the rollout starts +- [ ] A named decision owner for advance, hold, stop, and roll back +- [ ] Pipeline polls `get_deployment` and the restart job separately, rather than + treating the deploy response as completion +- [ ] `deployment_timeout` and `ignore_replication_errors` set deliberately +- [ ] Configuration changes go through the same change record as code +- [ ] `get_configuration` read back after every configuration change +- [ ] Schema changes follow expand-then-contract, with destructive operations + documented per node +- [ ] Forward repair defined for changes that alter persisted meaning +- [ ] Measured release RTO recorded from a rehearsed reversal + +## Additional Resources + +- [Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx) for the + pipeline mechanics and deploy credentials +- [`deploy_component` reference](/reference/v5/operations-api/operations#deploy_component) + for every parameter including `urlPath`, `host`, and replication controls +- [Applications reference](/reference/v5/components/applications) for component + structure and the full component operation set +- [Deployment records](/reference/v5/operations-api/operations#deployment-operations) + for `list_deployments` and `get_deployment` +- [HTTP middleware routing](/reference/v5/http/overview#middleware-routing) for + `urlPath` and `host` activation +- [Configuration operations](/reference/v5/configuration/operations) for + `set_configuration` and restart requirements +- [Database schema](/reference/v5/database/schema) for schema evolution +- [Multiple Applications on One Cluster](../developers/multiple-applications.mdx) + for running more than one component side by side diff --git a/learn/administration/sizing-a-harper-cluster.mdx b/learn/administration/sizing-a-harper-cluster.mdx new file mode 100644 index 000000000..b06a86948 --- /dev/null +++ b/learn/administration/sizing-a-harper-cluster.mdx @@ -0,0 +1,261 @@ +--- +title: Sizing a Harper Cluster +sidebar_position: 3 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +Most clusters get sized by dividing expected peak throughput by measured per-node +throughput and rounding up. That arithmetic produces a cluster that is exactly +large enough to handle a good day, which means the first node failure during peak +traffic becomes a user-visible outage. + +Sizing Harper starts from a different question. Not "how much traffic do we +have," but "how many nodes are we willing to lose at once, and can the survivors +carry the load." This guide gives you the capacity rule that follows from that, +what to include when you measure a node, and how to turn an availability target +into an operating budget you can actually spend. + +## What You Will Learn + +- The capacity invariant that sizes for the failure state, with a worked example +- What to include in a per-node throughput measurement, and the three things + people leave out +- Why maintenance is a planned failure, and what that means for change windows +- How to choose a topology from your objectives rather than from a diagram +- How to convert an availability SLO into minutes per month, and what each tier + demands of your automation + +## Prerequisites + +- [How Harper Runs in Production](./how-harper-runs-in-production.mdx), and your + service boundary inventory +- [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx), + because capacity headroom is meaningless if traffic cannot be moved off a node +- A load generation tool and a non-production cluster you can push to saturation +- A stated peak throughput requirement for your critical journeys + +## Size for the failure state + +The rule: + +```text +(N - F) x tested per-node throughput x target utilization >= required peak throughput +``` + +Where `N` is your node count, `F` is the largest simultaneous node loss the design +tolerates, and target utilization is the fraction of measured capacity you are +willing to run at, which should leave room for latency to stay acceptable rather +than merely for requests to complete. + +Worked example. Suppose your critical journey needs 12,000 requests per second at +peak, a node sustains 5,000 requests per second in your own test with acceptable +latency, you will run at 70 percent utilization, and you want to survive losing +one node: + +```text +Required surviving capacity = 12,000 / 0.7 = 17,143 req/s +Surviving nodes needed = 17,143 / 5,000 = 3.43 -> 4 +N = 4 + F (1) = 5 nodes +``` + +Five nodes, not three. The naive calculation gives 12,000 / 5,000 = 2.4, rounded +up to 3, and that cluster degrades the moment anything goes wrong. + +Two things to notice. Increasing `F` from 1 to 2 costs you one more node here, not +double the cluster, so tolerating a second simultaneous failure is often cheaper +than people expect at this size. And the utilization factor is doing as much work +as `F` is: sizing to 100 percent of measured capacity means your "surviving" +nodes are at saturation, where latency degrades long before throughput does. + +## Measure a node honestly + +The per-node number in that formula is the one most likely to be wrong, because +benchmark conditions are kinder than production. Include all of this: + +- **Representative application code, data shape, and query mix.** A Harper node + runs your component logic in the same process as the data access, so your code + is part of the capacity measurement in a way it would not be for a standalone + database. A synthetic key-value benchmark tells you very little about the node's + capacity to serve your journey. +- **Realistic downstream latency.** If your resource calls an upstream pricing + service, its latency occupies worker capacity on the node. +- **Replication catch-up load.** A node that is feeding a recovering peer is doing + work that does not appear in its own request metrics. Measure with a peer + synchronizing, because that is precisely the state you will be in when you are + already down a node. +- **Traffic imbalance.** Configured weights express intent. Actual request + distribution is what consumes capacity, and it is rarely even. + +Measure each node separately rather than dividing a cluster total by node count. +An average conceals the one node that is about to tip. + + + + +```bash +curl -s -X POST https://my-node.example.com:9925/ \ + -H 'Content-Type: application/json' \ + -u 'admin:password' \ + -d '{"operation":"system_information","attributes":["cpu","memory","threads","harperdb_processes"]}' +``` + + + + +```javascript +await fetch('https://my-node.example.com:9925/', { + method: 'POST', + headers: { + 'Content-Type': 'application/json', + 'Authorization': 'Basic ' + btoa('admin:password'), + }, + body: JSON.stringify({ + operation: 'system_information', + attributes: ['cpu', 'memory', 'threads', 'harperdb_processes'], + }), +}); +``` + + + + +Watch CPU, memory, event loop and worker pressure from the `threads` attribute, +disk, and network, and pair them with request rate and latency from +[analytics](/reference/v5/analytics/overview). The number you want is not the +point where requests start failing. It is the point where your journey's latency +leaves your SLO, which arrives earlier. + +:::tip +Record the conditions alongside the number: Harper version, node size, data +volume, query mix, and whether a peer was synchronizing. A per-node capacity +figure without its conditions is not reusable, and six months later nobody will +remember whether the test included replication load. +::: + +## Maintenance is a planned failure + +Draining a node for a deploy, an upgrade, or an investigation consumes exactly the +same headroom that a node failure does. So a cluster sized for `F = 1` is running +at `F = 0` for the duration of every maintenance window, with no tolerance left. + +The operating rule that follows: if draining one node means the service can no +longer tolerate its declared `F`, pause the change. Either wait for lower traffic, +or add capacity for the window, or accept and document the reduced tolerance for a +bounded period. What you should not do is treat the maintenance window as free +because nothing is technically broken. + +This is also the argument for sizing `F` at 2 in a cluster that deploys +frequently. It is not paranoia about correlated hardware failure. It is that you +want to be able to deploy during business hours and still survive an incident. + +## Choose a topology from objectives + +Pick the pattern that matches the failures you actually need to survive, and know +what you have to prove for it to count. + +| Pattern | When it fits | What you must prove | +| ------------------------------------ | ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | +| Single region, multiple peers | Node and host failures dominate, and low inter-node latency matters | `N - F` capacity under load, peer catch-up time, traffic removal, backup independence | +| Multiple failure domains, one region | Zone or rack isolation is available from your provider | No shared power or network fault, acceptable replication latency across domains, balanced traffic | +| Multiple regions | Regional impairment or user locality is a real requirement | Data authority per region, route propagation time, convergence, degraded-mode behavior | +| Edge or on-premises peers | Data locality or residency constrains where data can live | Replication scope per location, identity provisioning, backup path from each site | + +The right-hand column is the useful one. A topology diagram is a claim, and the +claim is only true once you have exercised it. "Multiple failure domains" means +nothing if both domains draw from the same power feed, and you will not discover +that from your provider's documentation. + +Note that adding a node is a data movement event as well as a capacity event, +since a Harper node carries data along with request handling. A node whose +databases have never synchronized downloads them in full before it is useful, so +scaling out is not instantaneous and cannot be your response to an unexpected +traffic spike. Size ahead of demand. + +## Turn the SLO into a budget + +An availability target is a quantity of unavailability you are permitted to spend +per month. Written that way it becomes an operating constraint rather than an +aspiration. + +| Monthly SLO | Approximate maximum unavailability | What it demands | +| ----------- | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 99.9% | 43 min 50 sec | Manual diagnosis can fit inside some incidents, but one untested maintenance window can consume the month | +| 99.95% | 21 min 55 sec | Automated node removal and a rehearsed release reversal become necessary | +| 99.99% | 4 min 23 sec | Node failure must be close to transparent. Detection, traffic removal, and validation have to be automated, because no human response fits in the budget | + +Figures assume a 30.44 day month. Your contractual definition, exclusions, and +measurement boundary may differ, and the boundary matters more than the number: +availability measured at your CDN edge and availability measured at the Harper +node are different quantities. + +The practical consequence is that the SLO tier determines how much automation you +need, not how many nodes. At three nines a person can be paged, look at a +dashboard, and drain a node. At four nines that same sequence has already spent +the entire month's budget. + +### Prove it + +Run the failure state rather than calculating it. On a cluster carrying +representative load at your expected peak: + +1. Record per-node request rate, latency, and saturation as a baseline. +2. Drain one node using the sequence from + [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). +3. Measure what the survivors do: request rate, latency at your SLO percentile, + CPU and worker pressure, and whether any user-visible errors appeared during + the transition. +4. While still down a node, bring the drained node back and let it synchronize, so + the survivors are carrying peak traffic and feeding a recovering peer at the + same time. This is the real worst case and it is the one nobody tests. +5. Compare measured surviving capacity against your invariant. If step 4 pushed + latency out of SLO, your target utilization is too high or your `N` is too low. + +## Operational notes + +- **Utilization targets belong per journey, not per cluster.** A write-heavy + journey and a cached read journey consume very different resources on the same + node, so a single cluster-wide utilization figure will be wrong for both. +- **Re-measure after a version upgrade.** Per-node capacity is a property of a + Harper version, your component code, and your data volume. All three change. +- **Watch measured distribution, not configured weights.** Per-node request counts + from analytics are the ground truth. A misconfigured weight, a sticky session + policy, or DNS caching can leave one node doing far more work than the topology + claims. +- **Storage growth is a capacity dimension too.** Disk headroom, compaction + behavior, and backup space all scale with data volume rather than with request + rate, so they need their own thresholds. See + [compaction](/reference/v5/database/compaction). +- **On Fabric, cluster shape is managed but the invariant is unchanged.** You are + still choosing `N` and living with `F`. + +## Readiness checklist + +- [ ] `F` declared explicitly, and written down where change operators will see it +- [ ] Per-node throughput measured with your own application code and data shape +- [ ] Measurement taken with a peer synchronizing, not on an idle cluster +- [ ] Target utilization set from latency at your SLO percentile, not from request + success +- [ ] `(N - F) x per-node x utilization >= peak` verified with real numbers +- [ ] Measurement conditions recorded alongside the capacity figure +- [ ] Maintenance policy states what happens when a drain would breach `F` +- [ ] Topology's "what you must prove" column exercised, not assumed +- [ ] SLO converted to minutes per month, with the measurement boundary named +- [ ] Peak-load drain and rejoin exercise completed and dated + +## Additional Resources + +- [Analytics overview](/reference/v5/analytics/overview) and + [analytics operations](/reference/v5/analytics/operations) for per-node request + and latency metrics +- [Grafana integration](/fabric/grafana-integration) for dashboards on Fabric +- [`@harperdb/prometheus-exporter`](https://github.com/HarperFast/prometheus-exporter) + for self-managed metrics scraping +- [Storage tuning](/reference/v5/database/storage-tuning) for the durability and + throughput trade-offs available per database +- [Compaction](/reference/v5/database/compaction) for storage growth behavior +- [Replication overview](/reference/v5/replication/overview) for peer + synchronization behavior when adding or returning a node +- [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx) for + scenario-level targets built on this capacity model diff --git a/redirects.ts b/redirects.ts index c6eb20368..7e06c8f3b 100644 --- a/redirects.ts +++ b/redirects.ts @@ -358,6 +358,7 @@ const currentRedirects: RedirectRule[] = [ // ── Learn ───────────────────────────────────────────────────────────────── { from: '/learn/developers/coming-soon', to: '/learn' }, + { from: '/learn/administration/coming-soon', to: '/learn/administration/how-harper-runs-in-production' }, // ── Fabric ──────────────────────────────────────────────────────────────── { from: '/fabric/rest-api', to: '/fabric' }, From fa96fddb579cb53de69bd67556e6861469f0ebc9 Mon Sep 17 00:00:00 2001 From: Ethan Arrowood Date: Tue, 15 Sep 2026 08:58:55 -0600 Subject: [PATCH 02/21] docs: correct Administration guides against harper/harper-pro source Technical verification pass over the ten Administration guides against harper and harper-pro at origin/main (v5.2.12 current). Most of the track held up; these are the claims that did not, plus the reference fixes they turned up. Guides: - Rewrite "What restart actually does" for released 5.2 behavior. The awaited `restart: true` is v5.3.0 only, so the section now describes the fire-and-forget behavior operators actually have, with the wait moved to its own subsection badged `changed in v5.3.0`. - Warn that a replicated deploy with `restart: true` restarts every node at once, and prefer `rolling` on a cluster. Matches the guidance already in the shipped CI/CD guide. - Document that `replicated` is opt-out: omitting it replicates. - Correct the new-node row. A joining node requests a full copy from every peer it has no resume cursor for, and established peers full-copy from it in turn, so a join into N peers is N inbound plus N outbound transfers. - `system` is in replication scope by default, so users and roles already propagate unless someone narrowed it. - `drop_database` and `drop_table` replicate by default; `drop_attribute` does not. - `get_backup target=` writes a file; it does not clone a database. - Blob snapshots are hard-linked, so they are near-free on the default layout and a full copy only across filesystems. The transaction-log snapshot is the part that genuinely costs disk every backup. - `verify_backup` checks that a blob snapshot is present, not that blob contents are intact. - The availability flag is already `replicate: false` in both mechanisms, so the guidance is to preserve that rather than to go establish it. - `/status` returns a body as well as a status code, and the component entry belongs in the root harper-config.yaml, not an application's config.yaml, where it would replace the default component config. - Readiness sample: a missing record resolves rather than throws, so check the result; import from 'harper'; and note the route is unauthenticated because a static handler bypasses the authorization gate. - Logging sample: there is no node-name environment variable, so use server.hostname. `context.requestId` needs http.logging.id, which is off by default. Harper's logger renders text via Node's Console, not JSON. - Badge `write-transaction-queue-depth` v5.2.0, and note the Prometheus exporter does not cover the saturation, commit-time, or storage metrics. Reference: - State the `replicated` default across the component operations; it was the only optional parameter whose default went unstated. - `drop_database`/`drop_table` replicate by default; only `drop_attribute` is per-node. - Users and roles do propagate by default, which contradicted both the code and this file's own statement that all databases replicate by default. Co-Authored-By: Claude Opus 5 --- learn/administration/backup-and-recovery.mdx | 46 +++++--- .../health-checks-and-traffic-admission.mdx | 67 ++++++++--- .../administration/monitoring-and-triage.mdx | 61 +++++++++- .../administration/operating-replication.mdx | 77 ++++++++++--- .../production-readiness-checklist.mdx | 3 +- .../safe-deployments-and-rollback.mdx | 108 +++++++++++++----- reference/components/applications.md | 6 +- reference/operations-api/operations.md | 4 +- reference/replication/overview.md | 9 +- 9 files changed, 295 insertions(+), 86 deletions(-) diff --git a/learn/administration/backup-and-recovery.mdx b/learn/administration/backup-and-recovery.mdx index 54bfa9f88..8e0a286ad 100644 --- a/learn/administration/backup-and-recovery.mdx +++ b/learn/administration/backup-and-recovery.mdx @@ -123,12 +123,22 @@ references. :::warning The incremental behavior applies to the RocksDB data files only. **The -transaction-log snapshot, and the blob snapshot for a database with file-backed -blobs, are copied in full on every backup.** With a large audit-retention window -or many blobs, frequent backups cost considerably more disk than the -data-only view suggests. Pass `exclude_blobs: true` to skip blobs when that is -appropriate, and size your backup volume against the full cost rather than the -incremental one. +transaction-log snapshot is copied in full on every backup.** With a large +audit-retention window that is real, recurring disk cost that the data-only view +hides, so size your backup volume against it rather than against the incremental +figure. + +Blobs behave differently and are easy to over-budget. Each backup captures a full +set of blobs, but the files are hard-linked rather than copied when the backup +directory and the blob storage share a filesystem, which is the default layout. In +that case the extra space is near zero. They become genuine full copies only when +the two live on different filesystems, so putting backups on a separate volume is +a real cost decision rather than a free one. Pass `exclude_blobs: true` to skip +blobs when that is appropriate. + +One consequence for disaster recovery: because the blob snapshot is hard-linked, +copying a backup repository with an ordinary recursive copy materializes every +blob as a full, separate file at the destination. ::: Do not use `list_backups` sizes for capacity planning. The `size` and @@ -162,11 +172,16 @@ retained managed backups off the node: # Pull a snapshot of the current state from a running node harper get_backup database=data out=./data-$(date +%Y%m%dT%H%M%S).tar.gz -# Or pull from another node, which also clones that node's database onto this one +# Or pull a snapshot of another node, writing it to a file here harper get_backup database=data target=https://node-2.example.com:9925 out=./data.tar.gz ``` -Note that `get_backup` always streams the current state. It cannot download a +`target=` only changes which node the snapshot is read from. It writes a file and +nothing else: it does not restore, import, or clone that node's database onto the +local one. Turning the file into a live database is a separate, deliberate restore +step, performed with Harper stopped. + +Note also that `get_backup` always streams the current state. It cannot download a historical managed backup, so it is a way to take a fresh off-host copy, not a way to export your retention history. @@ -189,8 +204,11 @@ nothing else. `verify_checksum` is `true`, which is slower, and the framing of the transaction-log snapshot, which is always checked. -**The blob snapshot is not verified.** If your database has file-backed blobs, -verification does not tell you they are intact, and only a real restore does. +**Blob contents are not verified.** Verification confirms that a backup which +recorded blobs still has its blob snapshot present, and fails the backup as corrupt +if that snapshot is missing. It does not check the individual blob files for size, +checksum, or readability. So verification can pass on a backup whose blobs are +damaged, and only a real restore proves they are intact. More generally, a verified backup is a well-formed backup, not a proven recovery. The only evidence that your recovery works is a restore you have actually @@ -302,15 +320,17 @@ that it is the target than a promise you cannot keep. - [ ] No database in scope uses per-table storage paths, or its exclusion is known and accepted - [ ] Backup cadence matches the stated recovery point per database -- [ ] Backup volume sized against the full cost, including non-incremental - transaction-log and blob snapshots +- [ ] Backup volume sized against the non-incremental transaction-log snapshot, + and against full blob copies if the backup and blob paths are on different + filesystems - [ ] An off-host copy exists in a destination that does not share a failure domain - [ ] Managed repository copies take the whole `/` directory, quiesced or from an atomic snapshot - [ ] `verify_backup` runs on a schedule, with `verify_checksum` at least periodically -- [ ] Blob integrity understood to be unverified by `verify_backup` +- [ ] Blob contents understood to be unchecked by `verify_backup`, which confirms + only that the blob snapshot is present - [ ] Restore constraints documented for user databases, component-held databases, and `system` - [ ] Restore authority named, and restore execution logged diff --git a/learn/administration/health-checks-and-traffic-admission.mdx b/learn/administration/health-checks-and-traffic-admission.mdx index 249e9ddf6..fcd9524b3 100644 --- a/learn/administration/health-checks-and-traffic-admission.mdx +++ b/learn/administration/health-checks-and-traffic-admission.mdx @@ -67,21 +67,36 @@ other component: } ``` -Or declare it in your application's `config.yaml`, which is the better option -because it makes the availability contract part of your deployed artifact rather -than a manual step someone has to remember on a new node: +Or declare it in the root `harper-config.yaml`, so the component is part of the +node's configuration rather than something an operator has to remember to deploy: ```yaml status-check: package: '@harperdb/status-check' ``` -Once deployed, the route answers with a status code rather than a body: +:::note +That entry belongs in the root `harper-config.yaml` (in the Harper `rootPath`, +typically `~/hdb`), not in an application's own `config.yaml`. The two files look +alike but behave differently: in the root config the entry name is free-form, while +in a component's `config.yaml` it must match a `package.json` dependency. A +component `config.yaml` also **replaces** Harper's default component +configuration outright instead of merging with it, so a file containing only this +entry would switch off the `rest`, `graphqlSchema`, `jsResource`, and +`fastifyRoutes` defaults your application relies on. See +[applications](/reference/v5/components/applications). +::: + +Once deployed, the route's contract is its status code, which is what lets a load +balancer consume it without parsing anything: - `GET /status` returns `200` when the node is available, `404` when it is not - `POST /status` marks the node available (authenticated) - `DELETE /status` marks the node unavailable (authenticated) +It does also return a body, which is useful when you are checking by hand: a short +message on `200`, and an RFC 9457 problem-details document on `404`. + @@ -118,11 +133,16 @@ Operations API and not at your application's root. A `404` is the node telling t traffic layer to stop sending work, and it will keep saying so until something sets it back. -:::warning -Whatever mechanism you use to persist the availability flag, keep it out of -replication scope. A flag that replicates would let one node's maintenance state -propagate to its peers, which turns a routine drain into an outage. If you are -scoping replication by hand, confirm this explicitly rather than assuming it. See +:::note +The availability flag is node-local out of the box. The component stores it in a +table declared `replicate: false`, and the Operations API's own status values are +stored the same way, so neither propagates to peers. + +That is the property you want, and it is worth knowing why: a flag that replicated +would let one node's maintenance state reach its peers, turning a routine drain +into a cluster-wide outage. So the rule is to preserve it rather than to establish +it. If you fork the component or persist the flag some other way, keep +`replicate: false` on whatever holds it. See [Operating Replication](./operating-replication.mdx). ::: @@ -154,15 +174,19 @@ converted a partial degradation into a full outage yourself. Add a resource to your application's `resources.js`: ```javascript +import { Resource, tables } from 'harper'; + export class Readiness extends Resource { static async get() { const checks = {}; // A bounded read against the table this journey serves. // Keep it to a single primary-key lookup, never a scan. + // A missing record resolves to undefined rather than throwing, + // so check the result as well as catching a storage fault. try { - await tables.Product.get('readiness-probe-sentinel'); - checks.data = 'ok'; + const sentinel = await tables.Product.get('readiness-probe-sentinel'); + checks.data = sentinel ? 'ok' : 'failed'; } catch (error) { checks.data = 'failed'; } @@ -187,11 +211,25 @@ export class Readiness extends Resource { } ``` -Enable the `jsResource` plugin in `config.yaml` if it is not already, and the -route is served at `/Readiness` on the application port. See +The `jsResource` plugin is enabled by default, so the route is served at +`/Readiness` on the application port as soon as the class is exported. See [Harper Applications in Depth](../developers/harper-applications-in-depth.mdx) for the resource and export mechanics. +:::note +Because the handler is a `static` method, it replaces Harper's built-in dispatch +along with the authorization check that lives inside it, so `GET /Readiness` is +unauthenticated. That is what a load balancer probe needs, and it is why the +endpoint must not return anything you would not publish. Keep the response to +check names and outcomes, never connection strings, credentials, or internal +hostnames. + +If you rewrite this as an instance `get()` instead, authorization comes back and +defaults to `super_user` only, which will make your probe start failing with an +authorization error rather than a readiness one. In that form you need +`allowRead() { return true; }` to keep it reachable. +::: + Two details in that example are the point of it. The timeout on the downstream call means a slow dependency cannot make your readiness check hang, which would make the node look dead to a probe rather than unready. And returning the version @@ -288,7 +326,8 @@ than people assume, because it includes convergence rather than just startup. - [ ] `@harperdb/status-check` deployed, declared in `config.yaml` rather than deployed by hand - [ ] Traffic layer health check points at `GET /status` on `9926` -- [ ] Availability flag confirmed to be outside replication scope +- [ ] Availability flag storage still declares `replicate: false`, if you forked the + component or persist it yourself - [ ] An application readiness route exists, scoped to one journey's dependencies - [ ] Every downstream call in the readiness route has a timeout - [ ] Readiness response includes the component version diff --git a/learn/administration/monitoring-and-triage.mdx b/learn/administration/monitoring-and-triage.mdx index f25621455..7549ccc58 100644 --- a/learn/administration/monitoring-and-triage.mdx +++ b/learn/administration/monitoring-and-triage.mdx @@ -65,7 +65,7 @@ dashboard. This much belongs on a pager: | Event loop backpressure | `main-thread-utilization`, the `taskQueueLatency` attribute | Rises before throughput drops, so it is an early warning rather than a postmortem input | | Replication convergence lag | `replication-latency`, or the receive-time gap in `cluster_status` | A node serving stale data looks healthy on every other signal | | Write commit time | `transaction-commit-time` | Storage-level degradation shows here before it shows in request latency | -| Write queue depth growing | `write-transaction-queue-depth` | A growing queue means the node is accepting work faster than it can commit it | +| Write queue depth growing | `write-transaction-queue-depth` | A growing queue means the node is accepting work faster than it can commit it | | Storage headroom | `storage-volume` free, and `database-size` | Disk exhaustion is an outage with no graceful degradation | | Missing peer | `cluster_status` | Covered in [Operating Replication](./operating-replication.mdx) | | Backup age exceeding RPO | `list_backups` | The failure you will not notice until you need it | @@ -102,10 +102,30 @@ and scrape it into whatever you already run: "operation": "deploy_component", "project": "prometheus-exporter", "package": "@harperdb/prometheus-exporter", - "restart": true + "restart": "rolling" } ``` +Once deployed, the scrape endpoint is `//metrics` on the application +port, so the call above exposes it at `/prometheus-exporter/metrics`. The route is +authorized, so give your scraper a credential rather than expecting it to be open. + +:::warning +**The exporter does not cover every metric in the alert table above.** It +translates the request and replication metrics, including `success`, `duration`, +and `replication-latency`, but it does not currently export +`main-thread-utilization` and its `taskQueueLatency` attribute, +`transaction-commit-time`, `write-transaction-queue-depth`, `database-size`, +`storage-volume`, or `table-size`. The `utilization` it does expose is a thread +utilization figure drawn from `system_information`, which is not the same as the +analytics `utilization` metric. + +For the saturation, commit-time, and storage rows you will need to query +`hdb_analytics` directly or collect `system_information` on your own schedule. +Check what your version actually exports before you build a dashboard on the +assumption that everything above arrives in Prometheus. +::: + @@ -138,14 +158,15 @@ to discover it after building panels on the wrong assumption. ## Logs you can use during an incident -Harper's `logger` global writes structured entries from component code: +Harper's `logger` global takes a message plus a context object from component +code: ```javascript logger.info('order submitted', { component: 'orders-api', - version: process.env.APP_VERSION, - node: process.env.HARPER_NODE_NAME, - requestId: context.requestId, + version: process.env.APP_VERSION, // whatever your build injects + node: server.hostname, + requestId: context.requestId, // requires http.logging.id, see below operation: 'submit', durationMs: elapsed, outcome: 'ok', @@ -158,6 +179,34 @@ is what lets you answer "was this only the new version" and "was this only one node" without guessing, and those are the first two questions in almost every incident. +Three details in that example need care: + +- **The node name comes from `server.hostname`**, not an environment variable. + Harper does not set a node-name variable in the process environment, and + `server.hostname` is the same identity analytics uses, so it is what correlates + with your metrics. +- **`context.requestId` is only populated when `http.logging.id` is enabled**, and + HTTP request logging is off by default. Without it the field is `undefined` and + you silently lose your correlation id. Either enable it in + [logging configuration](/reference/v5/logging/configuration) or generate an id + in your own code. +- **`APP_VERSION` is yours to inject.** Harper does not provide it. Set it in your + deployment so the log line can name the build. + +:::warning +Harper's logger is built on Node's `Console`, so this renders as a formatted text +line, not JSON. The context object is inspected into the message rather than +emitted as separate fields: + +```text +[main/3] [info]: order submitted { component: 'orders-api', version: '2.4.1', ... } +``` + +That is fine for reading during an incident, but a log pipeline cannot key on +`component` or `outcome` without parsing the line. If you need queryable fields, +serialize the context yourself and log a single JSON string. +::: + Levels run `trace`, `debug`, `info`, `warn`, `error`, `fatal`, and `notify`. The default is `warn`, and `notify` is always logged regardless of level. Choose levels deliberately: a production log at `debug` is a log nobody can read, and one diff --git a/learn/administration/operating-replication.mdx b/learn/administration/operating-replication.mdx index 617f5c53f..93baa86c7 100644 --- a/learn/administration/operating-replication.mdx +++ b/learn/administration/operating-replication.mdx @@ -55,7 +55,7 @@ replication: ```graphql type LocalTableForNode @table(replicate: false) { - id: ID! + id: ID! @primaryKey name: String! } ``` @@ -101,29 +101,32 @@ directional configuration mistakes survive into production. :::warning Whether the `system` database is in your replication scope is the highest-stakes scoping question in a Harper cluster, because `system` holds `hdb_user`, -`hdb_role`, and `hdb_nodes`. Users and roles propagate only when `system` is -replicated, and a node that receives `system` must be trusted with its contents, -including encrypted secret rows. Confirm your own answer from `cluster_status` and -your configuration, and read +`hdb_role`, and `hdb_nodes`. **It is in scope by default**, since the default +replication scope is every database, so unless someone has narrowed it your users +and roles already propagate and every node that receives `system` must be trusted +with its contents, including encrypted secret rows. That is usually what you want, +but it should be a decision rather than a surprise. Confirm your own answer from +`cluster_status` and your configuration, and read [replicating the system database with controlled flow](/reference/v5/replication/overview#replicating-the-system-database-with-controlled-flow) before changing it. ::: ## What moves automatically, and what does not -| Object or change | Behavior | What it means for you | -| --------------------------- | ---------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | -| Data mutations | Insert, update, upsert, delete, and bulk load replicate for tables in scope | Monitor connections, latency, and convergence for the databases that serve critical journeys | -| Transactions | Replicated atomically, and a transaction may span multiple tables | Tables that need to commit together must live in the same database | -| A brand new node | Downloads the full database from the first node it connects to, then enters incremental replication | Budget the transfer, and expect load on the source node while it happens | -| A returning node | Resynchronizes automatically to catch up on the transactions it missed | This is not a full copy. A routine restart is much cheaper than a node replacement, so budget them separately | -| Component deployment | `deploy_component` supports `"replicated": true` for cluster-wide propagation | Useful once an artifact is trusted. Keep single-node deployment available for validation | -| Configuration | `set_configuration` supports `"replicated": true` | Never replicate node-local parameters. See the warning below | -| Users and roles | Propagate only when the `system` database is in replication scope | If `system` is out of scope, identity must be provisioned on every node by your own automation | -| Node registry (`hdb_nodes`) | Each node rewrites its own self-record from its `harper-config.yaml` routes on restart or component reload | Scoping applied through `add_node` alone does not survive a restart. Put durable constraints in config routes | -| Destructive schema changes | Behavior is operation and version dependent | Treat as high risk. Write an exact procedure, and verify the result on every node | - -The two rows most likely to bite you are the returning node and the node registry. +| Object or change | Behavior | What it means for you | +| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | +| Data mutations | Insert, update, upsert, delete, and bulk load replicate for tables in scope | Monitor connections, latency, and convergence for the databases that serve critical journeys | +| Transactions | Replicated atomically, and a transaction may span multiple tables | Tables that need to commit together must live in the same database | +| A brand new node | Requests a full copy of each in-scope database from every peer it has no resume cursor for, then enters incremental replication | Budget N transfers, not one. See the note below before sizing a join | +| A returning node | Resynchronizes automatically to catch up on the transactions it missed | This is not a full copy. A routine restart is much cheaper than a node replacement, so budget them separately | +| Component deployment | `deploy_component` replicates by default; pass `"replicated": false` to hold it to one node | Cluster-wide is the default, so single-node validation is the case you must ask for | +| Configuration | `set_configuration` supports `"replicated": true` | Never replicate node-local parameters. See the warning below | +| Users and roles | Propagate when the `system` database is in replication scope, which is the default | If you narrow scope to exclude `system`, identity must be provisioned on every node by your own automation | +| Node registry (`hdb_nodes`) | Each node rewrites its own self-record from its `harper-config.yaml` routes on restart or component reload | Scoping applied through `add_node` alone does not survive a restart. Put durable constraints in config routes | +| Destructive schema changes | `drop_database` and `drop_table` replicate by default; `drop_attribute` does not | A drop is a cluster-wide event unless you pass `"replicated": false`. Verify the result on every node | + +The three rows most likely to bite you are the returning node, the new node, and +the node registry. On the returning node: it is common to see a routine restart budgeted as though it were a full resynchronization, which makes teams avoid restarts they should be @@ -131,6 +134,21 @@ comfortable with. A node whose databases have never synced does download them in full. A node that was briefly offline catches up on what it missed. Measure both on your own data volume once, and use the right number for the right situation. +On the new node, the cost is larger than "one copy from one peer," because the +full-copy decision is made per peer and per database rather than once against a +bootstrap source. A joining node requests a full copy from every peer it has no +resume cursor for. And because replication is bidirectional, each established peer +independently decides it has no cursor for the newcomer and requests a full copy +_from_ it as well. Joining a cluster of N peers is therefore N inbound transfers +plus N outbound transfers of the new node's own (empty) databases, not a single +stream from one source. + +The practical consequences are worth planning around. Load lands on every existing +peer at once rather than on one, so a join during peak traffic is a capacity event +for the whole cluster. If you want a single designated source instead, `add_node` +accepts `isLeader: true`, which tells the joining node to request its full copy +from that peer alone. + On the registry: a node's advertised record is derived from its configuration file, and it replicates. That means a topology constraint you applied imperatively is superseded the next time that node restarts or reloads @@ -195,6 +213,21 @@ it is the one that tells you a returning node is not ready for traffic yet. `sendingMessage` appears while a transaction is actively being sent and is absent when the socket is idle, so its absence is not a fault. +Two things will mislead you if you automate on that pair without knowing them. +The gap only means "behind" while transactions are actually arriving: on an idle +source both stamps freeze, and the gap sits at whatever constant it last reached +rather than signalling lag. And because the two stamps come from two different +clocks, skew between peers is added to the gap directly, so calibrate your +threshold against a healthy baseline rather than treating the raw number as +replication delay. + +:::tip +While a database is taking a full copy, these timing fields render as the literal +string `"Copying"` rather than a date. Anything that parses them as timestamps +will fail on a joining node, which is exactly the node you most want to be +watching. Handle that value explicitly. +::: + Inventory `system.hdb_nodes` alongside this and compare it to your intended topology. Configuration intent and live peer state should agree, and the node's own row is in there too, not just its peers. @@ -206,7 +239,7 @@ own row is in there too, not just its peers. - Sustained growth in `latency`, judged against your own baseline - `lastCommitConfirmed` not advancing on a peer while writes are occurring - A `lastReceivedRemoteTime` to `lastReceivedLocalTime` gap exceeding your - admission budget + admission budget, evaluated only while writes are flowing - Repeated reconnects, which are visible in the logs even when a point-in-time status check looks healthy - Version or configuration drift between peers @@ -242,6 +275,12 @@ continuous version of the same measurement: the difference between the source commit timestamp and local time, reported per node, database, and table. Use the sentinel for a definitive answer during a change, and the metric for a dashboard. +Note that `replication-latency` is not recorded for the `system` database. Since +`system` is in replication scope by default and carries your users and roles, a +dashboard built only on this metric will show nothing for the database that +propagates identity. Use the sentinel, or the `cluster_status` receive-time pair, +if you need to watch `system` convergence. + ## Consistency belongs in the application design Replication makes data available on every peer. It does not make every peer agree diff --git a/learn/administration/production-readiness-checklist.mdx b/learn/administration/production-readiness-checklist.mdx index 29cabc5aa..0cbeac7f7 100644 --- a/learn/administration/production-readiness-checklist.mdx +++ b/learn/administration/production-readiness-checklist.mdx @@ -72,7 +72,8 @@ From - [ ] `@harperdb/status-check` deployed, declared in `config.yaml` rather than by hand - [ ] Traffic layer health check points at `GET /status` on the application port -- [ ] Availability flag confirmed to be outside replication scope +- [ ] Availability flag storage still declares `replicate: false`, if you forked the + component or persist it yourself - [ ] Readiness route scoped to one journey's dependencies, with a timeout on every downstream call - [ ] Readiness response includes the component version diff --git a/learn/administration/safe-deployments-and-rollback.mdx b/learn/administration/safe-deployments-and-rollback.mdx index 41310686c..92e62b3e4 100644 --- a/learn/administration/safe-deployments-and-rollback.mdx +++ b/learn/administration/safe-deployments-and-rollback.mdx @@ -47,7 +47,7 @@ Most bad deployments come from collapsing these into one action. | Decision | The question | Harper control point | | ------------ | ----------------------------------------- | ----------------------------------------------------------------------------------------------- | | **Build** | What immutable artifact exists? | A pinned package reference, a versioned tarball, a checksum, a dependency lock | -| **Deploy** | Which nodes have that artifact installed? | `deploy_component`, with `replicated` true or false | +| **Deploy** | Which nodes have that artifact installed? | `deploy_component`, with `"replicated": false` to hold it to one node | | **Activate** | Which code path is actually enabled? | `urlPath` and `host` mounting, component configuration, a feature flag, a header or tenant rule | | **Expose** | Which production traffic reaches it? | Traffic layer weights, the availability flag, cohort or geography targeting | @@ -57,6 +57,19 @@ route and expose it only to internal traffic. And when something is wrong you ca reverse exposure in seconds without touching what is installed, which is the fastest containment action available to you. +:::danger +**`replicated` is opt-out, not opt-in.** A `deploy_component` call with no +`replicated` field replicates to every peer. So does `add_component`, +`drop_component`, `set_component_file`, `set_env_value`, `delete_env_value`, and +the destructive `drop_database` and `drop_table`. Only `"replicated": false` +changes anything; `"replicated": true` is the default spelled out. + +This is the opposite of what most operators assume, and it is why the single-node +validation pattern below has to say `false` explicitly. If you intend to touch one +node, you must say so. ([`set_configuration`](/reference/v5/configuration/operations) +is the deliberate exception: it replicates only when you ask.) +::: + `urlPath` mounts a component at an HTTP path. `host` serves it on a virtual hostname. Both are persisted on the component's root config entry, so they are part of the deployed @@ -137,10 +150,13 @@ against it directly, then admit a bounded cohort. } ``` -Same immutable reference, now cluster-wide. Poll the returned `restartJobId` with -[`get_job`](/reference/v5/operations-api/operations#get_job), and poll -[`get_deployment`](/reference/v5/operations-api/operations#get_deployment) until -the deployment reports success rather than treating the response as completion. +Same immutable reference, now cluster-wide. The deploy itself is finished when the +call returns, so the thing still outstanding is the restart: poll the returned +`restartJobId` with +[`get_job`](/reference/v5/operations-api/operations#get_job). Read +[`get_deployment`](/reference/v5/operations-api/operations#get_deployment) for the +per-peer detail the response summarizes away, including which peers received the +artifact and which failed. @@ -149,29 +165,66 @@ Prefer a pinned version or an immutable tarball over a moving branch reference. A branch moves, so you can neither audit what was running yesterday nor redeploy it. +:::tip +A pinned registry version such as `@my-org/orders-api@2.4.1` is the +best-travelled form. If you pin a tarball URL instead, note that a bare `https://` +URL pointing at `github.com`, `gitlab.com`, or `bitbucket.org` is treated as a git +clone rather than a download, so a GitHub release asset URL will not behave the +way the example above does. Host release tarballs somewhere neutral, or use a +registry version. +::: + ## What restart actually does This is worth reading carefully, because the naming invites a wrong assumption. -**`"restart": true`** restarts the HTTP worker threads on the node handling the -call, and waits for that restart to finish before responding. A successful -response therefore means every worker on that node is serving the new code. Until -a worker has been replaced it is still running the previous code, and on platforms -where replacements share a listening port it keeps accepting connections during -the changeover. The wait follows the restart's own progress rather than a fixed -timeout, so the response can take tens of seconds on a slow install with many -worker threads. - -**`"restart": "rolling"`** does not restart inline. It starts a replicated -`restart_service` job and returns a `restartJobId` for you to poll. Use this when -your caller has a short request timeout. - -Two consequences for your pipeline. A caller with an aggressive HTTP timeout -should use `"rolling"` and poll, because giving up on `true` does not stop the -restart, it just leaves you without the result. And a failed restart does not fail -the deploy: the component is installed and replicated either way, so your pipeline -needs to check both outcomes separately rather than assuming one implies the -other. +**`"restart": true`** starts a restart of the HTTP worker threads on the node +handling the call and returns immediately, without waiting for it. A `200` means +the deploy succeeded and a restart has been requested. It does not mean the new +code is serving yet. Until a worker has been replaced it is still running the +previous code, and on platforms where replacements share a listening port it keeps +accepting connections during the changeover. + +**`"restart": "rolling"`** does not restart inline either, but it is observable. +It starts a replicated `restart_service` job and returns a `restartJobId` you can +poll with [`get_job`](/reference/v5/operations-api/operations#get_job). That job +walks the cluster **one node at a time**, waiting for each node to come back +before starting the next, so the cluster keeps serving throughout. + +:::warning +**On a cluster, prefer `"rolling"`.** `"restart": true` is forwarded verbatim to +every peer, and peers are dispatched in parallel, so a replicated deploy with +`"restart": true` restarts every node in the cluster at roughly the same moment. +That is a cluster-wide availability event, not a node-local one. Reserve +`"restart": true` for a single-node or development instance, and use `"rolling"` +anywhere you care about staying up. See +[Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx). +::: + +Two more consequences for your pipeline. Neither restart mode reports completion +in the deploy response, so if you need to know the new code is live, poll the +rolling restart job or probe the nodes themselves. And a failed restart does not +fail the deploy: the component is installed and replicated either way, so your +pipeline needs to check both outcomes separately rather than assuming one implies +the other. + +### Waiting for the restart + + + +From v5.3.0, `"restart": true` waits for the worker restart to finish before +responding, rather than returning as soon as it has been requested. The wait +follows the restart's own progress rather than a fixed timeout, so the call can +take tens of seconds on a slow install with many worker threads, with a hard +ceiling of ten minutes. + +Two things do not change, and both matter more than the wait itself. A caller that +gives up early does not stop the restart, it only loses the result. And the +response still does not carry the restart's outcome: a restart that stalls, times +out, or leaves workers on the old code is reported in the node's log, not to you. +So even on v5.3.0, treat a `200` as "the deploy landed and a restart ran," and +confirm the version actually serving through your readiness route or journey +synthetic rather than through the deploy response. Two parameters worth setting deliberately on replicated deploys: @@ -299,8 +352,11 @@ because the healthy majority dominates the average. - [ ] Cohort ladder defined, from smallest useful sample to full exposure - [ ] Stop thresholds and hold times declared before the rollout starts - [ ] A named decision owner for advance, hold, stop, and roll back -- [ ] Pipeline polls `get_deployment` and the restart job separately, rather than - treating the deploy response as completion +- [ ] Pipeline polls the restart job rather than treating the deploy response as + proof the new code is serving +- [ ] `"replicated": false` used deliberately wherever a change is meant to reach + one node only, since replication is the default +- [ ] Cluster deploys use `"restart": "rolling"`, not `"restart": true` - [ ] `deployment_timeout` and `ignore_replication_errors` set deliberately - [ ] Configuration changes go through the same change record as code - [ ] `get_configuration` read back after every configuration change diff --git a/reference/components/applications.md b/reference/components/applications.md index 2cf06274a..1f6c84d10 100644 --- a/reference/components/applications.md +++ b/reference/components/applications.md @@ -466,7 +466,7 @@ Creates a new component project in the component root directory using a template - `install_command` _(optional)_ — Install command. Defaults to `npm install` - `install_timeout` _(optional)_ — Install timeout in milliseconds. Defaults to `300000` (5 minutes) - `install_allow_scripts` _(optional)_ — Allow install scripts to run. Defaults to `false`, which causes `--ignore-scripts` to be passed to the install command (this is ignored with `install_command`). -- `replicated` _(optional)_ — Replicate to all cluster nodes +- `replicated` _(optional)_ — Replicate to all cluster nodes. Defaults to `true`; pass `false` to apply on this node only ```json { @@ -570,7 +570,7 @@ On a `.env` file it replaces the whole file with the payload, so saving back the - `file` _(required)_ — Path relative to project folder - `payload` _(required)_ — File content to write - `encoding` _(optional)_ — File encoding. Defaults to `utf8` -- `replicated` _(optional)_ — Replicate update to all cluster nodes +- `replicated` _(optional)_ — Replicate the update to all cluster nodes. Defaults to `true`; pass `false` to apply on this node only ```json { @@ -593,7 +593,7 @@ For deploying from private repositories, SSH keys must be registered on the Harp - `host` _(required)_ — Host alias for SSH config (used in `package` URL): a single alias, not a pattern - `hostname` _(required)_ — Actual domain (e.g., `github.com`) - `known_hosts` _(optional)_ — Public SSH keys of the host. Auto-retrieved for `github.com` -- `replicated` _(optional)_ — Replicate to all cluster nodes +- `replicated` _(optional)_ — Replicate to all cluster nodes. Defaults to `true`; pass `false` to apply on this node only A `key`, `host` or `hostname` that ssh couldn't use is refused with a `400` naming the problem; see [what `key`, `host` and `hostname` must be](../operations-api/operations.md#what-key-host-and-hostname-must-be). Each key's settings live in a block of `/ssh/config` that Harper owns; you can edit the file outside those blocks (see [the key's block in the ssh config](../operations-api/operations.md#the-keys-block-in-the-ssh-config)). diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index b31687fb0..8734bb923 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -116,7 +116,7 @@ Creates a new database. ### `drop_database` -Drops a database and all its tables/records. Supports `"replicated": true` to propagate to all cluster nodes. +Drops a database and all its tables/records. **Replicates to all cluster nodes by default** — pass `"replicated": false` to drop it on this node only. ```json { "operation": "drop_database", "database": "dev" } @@ -137,7 +137,7 @@ Creates a new table. Optional fields: `database` (defaults to `data`), `attribut ### `drop_table` -Drops a table and all associated records. Supports `"replicated": true`. +Drops a table and all associated records. **Replicates to all cluster nodes by default** — pass `"replicated": false` to drop it on this node only. ```json { "operation": "drop_table", "database": "dev", "table": "dog" } diff --git a/reference/replication/overview.md b/reference/replication/overview.md index 107e52ecf..bec57e419 100644 --- a/reference/replication/overview.md +++ b/reference/replication/overview.md @@ -384,9 +384,14 @@ The following data operations are replicated across the cluster: - Delete - Bulk loads (CSV data load, CSV file load, CSV URL load, import from S3) -**Destructive schema operations are not replicated**: `drop_database`, `drop_table`, and `drop_attribute` must be run on each node independently. +**Destructive schema operations differ from each other, so check before you run one:** -Users and roles are not replicated across the cluster by default. They do propagate when the `system` database (where `hdb_user` and `hdb_role` live) is included in replication; as of v5.2 this no longer forces a full mesh — see [Replicating the `system` database with controlled flow](#replicating-the-system-database-with-controlled-flow). +- `drop_database` and `drop_table` **replicate to the whole cluster by default**, like the other operations that accept a `replicated` flag. Pass `"replicated": false` to drop on one node only. +- `drop_attribute` is not replicated and must be run on each node independently. + +Note that `replicated` is opt-out rather than opt-in across the operations that support it: omitting the flag replicates. [`set_configuration`](../configuration/operations.md#set-configuration) is the deliberate exception, replicating only when you pass `"replicated": true`, because configuration bodies routinely carry node-local values. + +Users and roles **do** propagate by default. They live in the `system` database, and the default replication scope is every database, so `system` is in scope unless you have narrowed it. As of v5.2 replicating `system` no longer forces a full mesh — see [Replicating the `system` database with controlled flow](#replicating-the-system-database-with-controlled-flow). If you narrow the scope to exclude `system`, users and roles stop propagating and must be provisioned on every node by your own automation. Certain management operations — including component deployment and rolling restarts — can also be replicated across the cluster. From 6339ad4a9c1f4f79b93697d21f7eb4f195ec2658 Mon Sep 17 00:00:00 2001 From: Ethan Arrowood Date: Tue, 15 Sep 2026 09:18:46 -0600 Subject: [PATCH 03/21] docs(reference): correct six claims contradicted by harper source Follow-on from the Administration verification pass. Each of these was checked against harper/harper-pro at origin/main and is wrong on the live site today. - `set_status` / `get_status` / `clear_status` are not in-memory. They persist to the `hdb_status` table in the `system` database, which is declared `replicate: false`, so values survive a restart and stay node-local. Adds what `restartRequired` on a bare `get_status` actually tracks. - `system_information` no longer advertises a `replication` attribute, which does not exist. Lists the ten real attribute names and notes that unrecognized names are dropped silently rather than rejected, so a typo returns a smaller response instead of an error. - The `cluster_status` sample showed `lastSendTime`, a field present in neither repo, and `thread_id` where the code emits `threadId`. Also notes that the timing fields hold the literal string "Copying" during an initial full copy, which breaks anything parsing them as dates. - `logging.console` defaults to `false`, not `true`. The two reference pages disagreed; the runtime default is `logOptions.console ?? false`. Core's own config schema description carries the same stale "Default: true" and should be fixed separately. - `get_backup target=` reads a snapshot from another node into a local file. It does not clone that node's database onto the current one. - `verify_backup` confirms a recorded blob snapshot is present and fails the backup as corrupt when it is missing, but does not check individual blob files, so verification can pass on damaged blob contents. Co-Authored-By: Claude Opus 5 --- reference/backups/operations.md | 2 +- reference/backups/overview.md | 2 +- reference/configuration/options.md | 2 +- reference/operations-api/operations.md | 28 +++++++++++++++++--------- reference/replication/clustering.md | 6 ++++-- 5 files changed, 25 insertions(+), 15 deletions(-) diff --git a/reference/backups/operations.md b/reference/backups/operations.md index 3abe60d37..d736a460c 100644 --- a/reference/backups/operations.md +++ b/reference/backups/operations.md @@ -68,7 +68,7 @@ harper list_backups database=data -Verifies a managed backup's RocksDB file sizes — and their checksums when `verify_checksum` is `true` (slower) — together with the framing of its transaction-log snapshot (always checked). The blob snapshot is not verified. Through a running server this runs as a background [job](../operations-api/operations.md#jobs). `backup_id` is required. +Verifies a managed backup's RocksDB file sizes — and their checksums when `verify_checksum` is `true` (slower) — together with the framing of its transaction-log snapshot (always checked). For a backup that recorded blobs, verification confirms the blob snapshot is still present and fails the backup as corrupt if it is missing, but it does not check individual blob files for size, checksum, or readability — so verification can pass on a backup whose blob contents are damaged. Through a running server this runs as a background [job](../operations-api/operations.md#jobs). `backup_id` is required. ```json { "operation": "verify_backup", "database": "data", "backup_id": 1, "verify_checksum": true } diff --git a/reference/backups/overview.md b/reference/backups/overview.md index 3a0ab21dd..4db448213 100644 --- a/reference/backups/overview.md +++ b/reference/backups/overview.md @@ -93,7 +93,7 @@ For a RocksDB database the stream is a `tar` archive, gzipped by default. It con ```bash harper get_backup database=data out=./data.tar.gz -# or pull from another node — this clones that node's database onto the current one +# or read the snapshot from another node, still writing it to a file here harper get_backup database=data target=https://node-2.example.com:9925 out=./data.tar.gz ``` diff --git a/reference/configuration/options.md b/reference/configuration/options.md index d12128410..65b2601e8 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -243,7 +243,7 @@ logging: - `root` — Log directory; _Default_: `/log` - `path` — Explicit log file path (overrides `root`) - `stdStreams` — Write to stdout/stderr; _Default_: `false` -- `console` — Include `console.*` output; _Default_: `true` +- `console` — Include `console.*` output in the log file; _Default_: `false` - `auditLog` — Enable table transaction audit logging; _Default_: `false` - `auditRetention` — Audit log retention duration; _Default_: `3d` - `external` — Logging for components using the logger API; sub-options: `level`, `path` diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 8734bb923..cec727bb7 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -1730,14 +1730,14 @@ Detailed documentation: [WAF Operations and Rule Schema](../web-application-fire Operations for restarting Harper and managing system state. -| Operation | Description | Role Required | -| -------------------- | ----------------------------------------------------- | ------------- | -| `restart` | Restarts the Harper instance | super_user | -| `restart_service` | Restarts a specific Harper service | super_user | -| `system_information` | Returns detailed host system metrics | super_user | -| `set_status` | Sets an application-specific status value (in-memory) | super_user | -| `get_status` | Returns one status value, or the instance summary | super_user | -| `clear_status` | Removes a status entry | super_user | +| Operation | Description | Role Required | +| -------------------- | ------------------------------------------------- | ------------- | +| `restart` | Restarts the Harper instance | super_user | +| `restart_service` | Restarts a specific Harper service | super_user | +| `system_information` | Returns detailed host system metrics | super_user | +| `set_status` | Sets an application-specific status value | super_user | +| `get_status` | Returns one status value, or the instance summary | super_user | +| `clear_status` | Removes a status entry | super_user | ### `restart` @@ -1767,7 +1767,11 @@ Set `scope` to an [isolated application's name](../components/applications.md#is ### `system_information` -Returns system metrics including CPU, memory, disk, network, and Harper process info. Optionally filter by `attributes` array (e.g., `["cpu", "memory", "replication"]`). With `attributes: ["threads"]`, the response includes a `threads` array of workers. Each entry has a `threadId`; a dedicated isolated worker also has `application: ""`. These describe the node that answers, not its peers. +Returns system metrics including CPU, memory, disk, network, and Harper process info. Optionally filter by an `attributes` array (e.g., `["cpu", "memory", "threads"]`). With `attributes: ["threads"]`, the response includes a `threads` array of workers. Each entry has a `threadId`; a dedicated isolated worker also has `application: ""`. These describe the node that answers, not its peers. + +Valid attribute names are `system`, `time`, `cpu`, `memory`, `disk`, `network`, `harperdb_processes`, `table_size`, `metrics`, and `threads`. Omitting `attributes` returns all of them. + +> **Unrecognized attribute names are silently ignored**, not rejected. A typo produces a smaller response rather than an error, so if a section you asked for is missing, check the spelling before you check the node. ```json { "operation": "system_information" } @@ -1775,7 +1779,11 @@ Returns system metrics including CPU, memory, disk, network, and Harper process ### `set_status` / `get_status` / `clear_status` -Manage in-memory application status values. Status types: `primary`, `maintenance`, `availability` (availability only accepts `'Available'` or `'Unavailable'`). Status is not persisted across restarts. +Manage application-defined status values. Status types: `primary`, `maintenance`, `availability` (availability only accepts `'Available'` or `'Unavailable'`). + +Values are **persisted**, stored in the `hdb_status` table in the `system` database, so they survive a restart. That table is declared `replicate: false`, so a status set on one node stays on that node and does not propagate to peers — which is what makes these values usable as node-local coordination markers. + +These are a coordination primitive for your own automation: Harper stores and returns them but does not change its own behavior based on them, so they are not a substitute for a real health check. A `get_status` call with no `id` additionally returns Harper-derived state, including a `restartRequired` flag that tracks pending component and code restarts (not configuration changes). ```json { "operation": "set_status", "id": "primary", "status": "active" } diff --git a/reference/replication/clustering.md b/reference/replication/clustering.md index 194418bb5..3d4760670 100644 --- a/reference/replication/clustering.md +++ b/reference/replication/clustering.md @@ -172,12 +172,12 @@ Returns an array of status objects from the cluster, including active WebSocket "database": "data", "connected": true, "latency": 0.7, - "thread_id": 1, + "threadId": 1, "nodes": ["server-2.domain.com"], "lastCommitConfirmed": "Wed, 12 Feb 2025 19:09:34 GMT", "lastReceivedRemoteTime": "Wed, 12 Feb 2025 16:49:29 GMT", "lastReceivedLocalTime": "Wed, 12 Feb 2025 16:50:59 GMT", - "lastSendTime": "Wed, 12 Feb 2025 16:50:59 GMT" + "sendingMessage": "Wed, 12 Feb 2025 16:50:59 GMT" } ] } @@ -196,6 +196,8 @@ Returns an array of status objects from the cluster, including active WebSocket | `lastReceivedLocalTime` | Local time when the last transaction was received. A gap between this and `lastReceivedRemoteTime` suggests the node is catching up | | `sendingMessage` | Timestamp of the transaction actively being sent. Absent when waiting for the next transaction | +While a database is taking its initial full copy, these timing fields hold the literal string `"Copying"` rather than a timestamp. Anything that parses them as dates needs to handle that value, since a node taking a full copy is exactly the node you are most likely to be watching. + --- ### Configure Cluster From 4bb49ff30e26e302097970f8f2300a276fc154a0 Mon Sep 17 00:00:00 2001 From: Ethan Arrowood Date: Tue, 15 Sep 2026 09:32:14 -0600 Subject: [PATCH 04/21] docs(learn): close coverage gaps against the original field guide Read Jeff's "Operating Reliable Harper Applications" doc and compared it against the ten guides. Coverage was broadly good, but three documented capabilities were dropped entirely and one CLI claim was wrong. - `get_backup` has no offline CLI form. Guide 7 told readers every backup operation works with the server stopped; `get_backup` is absent from the offline command switch and fails as an unknown command. That matters because a stopped node is exactly when an operator reaches for it. - `storage.writeAsync` was missing from the RPO guide. It defaults off, and turning it on disables fsync so acknowledged commits can be lost on power loss or an OS crash. Nothing else in the track surfaces it: backup timestamps still look right and no alert fires, so a stated recovery point can be quietly false. Now framed as an RPO decision with the cases where enabling it is defensible. - `threads.preload` and `threads.preloadRequire` were missing from the observability guide. They are how an APM or tracing agent loads ahead of the code it instruments. Includes the split-entry case, which fails confusingly: with only the loader hooks preloaded a tracer emits no-op spans and exports nothing, so instrumentation looks installed while the collector stays empty. - `HARPER_SAFE_MODE` was missing from the backup guide. Guide 7 explains that a component holding a database open blocks an online restore but never mentioned the escape hatch. Safe mode starts Harper without loading components, so the restore can run, and it is also how you reach a node a broken component stopped. Also corrects the safe mode example in the configuration reference, which invoked `harperdb` rather than `harper`. Co-Authored-By: Claude Opus 5 --- learn/administration/backup-and-recovery.mdx | 25 +++++++++++ .../engineering-rpo-rto-and-uptime.mdx | 29 ++++++++++++ .../administration/monitoring-and-triage.mdx | 44 +++++++++++++++++++ reference/configuration/overview.md | 2 +- 4 files changed, 99 insertions(+), 1 deletion(-) diff --git a/learn/administration/backup-and-recovery.mdx b/learn/administration/backup-and-recovery.mdx index 8e0a286ad..cb13c67aa 100644 --- a/learn/administration/backup-and-recovery.mdx +++ b/learn/administration/backup-and-recovery.mdx @@ -113,6 +113,12 @@ Every backup operation runs from the CLI under the same name. With the server running the CLI forwards to the server; with it stopped the command operates directly on the files. +`get_backup` is the exception. It streams from a running server and has no offline +form, so with Harper stopped it fails as an unknown command. If you need a copy +off a node that is down, start it, or take the backup directory or a volume +snapshot instead. Worth knowing before an incident, because a stopped node is +exactly when you reach for it. + @@ -231,6 +237,23 @@ open does not fail your request. It fails inside the job, so you find out from `get_job` rather than from the response. Harper does not track which component uses which database, so it cannot selectively stop one. +:::tip +That last constraint has an escape hatch worth knowing before you need it. Setting +`HARPER_SAFE_MODE` to any value starts Harper without loading any applications or +components, while the database, Operations API, and HTTP server come up normally: + +```bash +HARPER_SAFE_MODE=1 harper +``` + +Because no component loads, nothing holds a user database open, so a restore that +would fail inside its job can run online. It is also how you get back in when a +broken component is what stopped Harper in the first place, since the Operations +API is available to inspect, repair, or remove it. Traffic should stay off the +node either way. See +[safe mode](/reference/v5/configuration/overview#harper_safe_mode). +::: + Two more constraints worth writing into your procedure: - **`target_database` requires the server stopped.** Restoring into a separate @@ -334,6 +357,8 @@ that it is the target than a promise you cannot keep. - [ ] Restore constraints documented for user databases, component-held databases, and `system` - [ ] Restore authority named, and restore execution logged +- [ ] Safe mode known as the way to restore a database a component holds open, + and to reach a node a broken component stopped - [ ] Restore drill completed and dated, with measured recovery time and recovery point - [ ] `system` database restore rehearsed offline diff --git a/learn/administration/engineering-rpo-rto-and-uptime.mdx b/learn/administration/engineering-rpo-rto-and-uptime.mdx index 958c625d5..b1d7043e2 100644 --- a/learn/administration/engineering-rpo-rto-and-uptime.mdx +++ b/learn/administration/engineering-rpo-rto-and-uptime.mdx @@ -21,6 +21,7 @@ during an incident that the answer was the target all along. - How to convert an availability target into a monthly budget, and what each tier demands of your automation - How to reconcile stated targets against the numbers measured in earlier guides +- Why one storage setting can silently invalidate your stated recovery point - Why declaring your degraded modes is as important as declaring your targets ## Prerequisites @@ -56,6 +57,32 @@ availability measured at a Harper node are different quantities, and the difference is exactly the part of the stack you may not control. Whichever you choose, say so in the same sentence as the number. +### One storage setting can override your recovery point + +Your recovery point objective is a statement about how much acknowledged data you +can afford to lose. One configuration option can quietly make that statement +false. + +[`storage.writeAsync`](/reference/v5/database/storage-tuning#storagewriteasync) +defaults to `false`. Set it to `true` and Harper stops calling `fsync` on commit, +so a write returns as soon as it is queued to the operating system's page cache. +Throughput on write-heavy workloads improves substantially. The cost is that a +power loss or OS crash between the commit and the flush to disk loses the most +recent transactions, even though your application was told they succeeded. The +database stays structurally consistent; only the newest writes are gone. + +The trap is that this is invisible to everything else in this guide. Backup +timestamps look right, replication looks healthy, and no alert fires. You discover +it the first time a node loses power, which is also the first time the gap between +your stated recovery point and your real one matters. + +So treat it as an RPO decision rather than a performance one. Enabling it is +defensible when the data is reproducible from an upstream source, when the +workload is bulk ingest that can simply be re-run, or when peers acknowledge +writes before the loss window opens. Outside those cases, leave it off. If it is +on anywhere, record which databases and say so next to the recovery point in your +plan, because the number no longer means what it appears to. + ## Map scenarios to mechanisms For each scenario, know the mechanism, and know what must be proven for the @@ -199,6 +226,8 @@ would have hurt you. - [ ] Availability target converted to minutes per month - [ ] Error budget policy states what changes when the budget runs low - [ ] Backup cadence reconciled against recovery point +- [ ] `storage.writeAsync` confirmed off, or the databases it is on for recorded + alongside their recovery point - [ ] Restore time plus detection time reconciled against recovery time - [ ] Convergence time reconciled against the traffic admission gate - [ ] Every unreconciled gap has an owner and a date diff --git a/learn/administration/monitoring-and-triage.mdx b/learn/administration/monitoring-and-triage.mdx index 7549ccc58..ef33fffa0 100644 --- a/learn/administration/monitoring-and-triage.mdx +++ b/learn/administration/monitoring-and-triage.mdx @@ -22,6 +22,7 @@ short enough to run under pressure. - Which Harper metrics are worth an alert, by name, and which are only worth a dashboard - How to get metrics out of Harper, on Fabric and self-managed +- How to preload an APM or tracing agent so it can instrument your application - How to write log entries that are still useful during an incident - A triage sequence that narrows a Harper incident in about five minutes @@ -156,6 +157,47 @@ what a cluster-wide query means, and it is easier to decide that deliberately th to discover it after building panels on the wrong assumption. ::: +## Run an APM or tracing agent + +Harper's own metrics tell you how the node is behaving. They do not give you +distributed traces across your application code and its downstream calls, which is +what you want when a journey is slow and you need to know where the time went. +That comes from an instrumentation agent, and an agent has to load before the code +it instruments. + +Two configuration keys put a module on each worker thread's startup, ahead of +Harper's own modules and yours: + +```yaml +threads: + preloadRequire: dd-trace/init # the entry that calls init() + preload: dd-trace/register.js # ESM loader hooks for automatic instrumentation +``` + +[`threads.preload`](/reference/v5/configuration/options#threads) + + loads a module via Node's `--import`, which is how an agent installs the loader hooks +that let it instrument modules imported later. [`threads.preloadRequire`](/reference/v5/configuration/options#threads) + uses `--require`, which runs the module body and is typically how an agent's +initialization entry actually starts it. + +Installing loader hooks and starting the agent are two different jobs, and which +of your agent's entry points does which is specific to that agent. Some ship one +entry that does both; others split them, in which case set both keys. The example +above is the split-entry case, and it is the one that produces the most confusing +failure: with only the hooks loaded, a tracer will hand out spans with plausible +trace ids that are no-ops and export nothing, so your instrumentation looks +installed and your collector stays empty. + +Two constraints worth knowing before you plan around this. Both keys apply to +worker threads only, and neither works under Bun. And bare specifiers resolve +against the `node_modules` of your installed components, so an agent can ship as a +dependency of a deployed component rather than as a host-level install. + +Whatever agent you use, follow its own worker-thread documentation, and finish by +confirming spans actually arrive at your collector rather than assuming the +configuration took. + ## Logs you can use during an incident Harper's `logger` global takes a message plus a context object from component @@ -316,6 +358,8 @@ misleading signal during a real incident costs more than a missing one. - [ ] Alerts on storage headroom and backup age - [ ] Metrics exported off the node, via Grafana on Fabric or the Prometheus exporter self-managed +- [ ] If you run an APM agent, both `threads.preload` and `threads.preloadRequire` + set as that agent requires, with spans confirmed at the collector - [ ] `analytics.replicate` setting known and deliberate - [ ] Structured logging includes component, version, node, request id, operation, duration, and outcome diff --git a/reference/configuration/overview.md b/reference/configuration/overview.md index 112327b24..51284de3f 100644 --- a/reference/configuration/overview.md +++ b/reference/configuration/overview.md @@ -299,7 +299,7 @@ logging: Setting `HARPER_SAFE_MODE` to any value starts Harper without loading user applications or components. Harper's core services (database, operations API, HTTP server) start normally, but no applications from the components directory are loaded and no package-based extensions are initialized. ```bash -HARPER_SAFE_MODE=1 harperdb +HARPER_SAFE_MODE=1 harper ``` This is useful when a broken or misbehaving component prevents Harper from starting. Safe mode lets you access the operations API to inspect, repair, or remove the problematic component without needing to manually edit files on disk. From ef9223afa904c1e791a2a74c5bab525a87bd8c6b Mon Sep 17 00:00:00 2001 From: Ethan Arrowood Date: Tue, 15 Sep 2026 13:23:45 -0600 Subject: [PATCH 05/21] docs: address Gemini review feedback Three of the seven review comments were worth taking. - Move `` out of prose and list items in guide 6, per CONTRIBUTING's "reserve VersionBadge for standalone placement after headings". Uses the repo's own inline form, `(Added in: vX.Y.Z)`, rather than the bare `(v5.2.0)` the review suggested. Also applies it to the two list-item badges the review did not flag, so the guide is internally consistent. Table-cell badges are left alone, which is established practice across the reference. - Promote the two managed-backup limitations to an admonition, since a database on per-table storage paths silently has no backup strategy at all. Uses `:::warning` rather than `:::caution` to match the dominant type in this repo. The other four were declined; see the PR threads for reasoning. In short: the snake_case suggestion would have made the sentinel example inconsistent with every shipped Learn guide, and the three remaining suggestions wrapped multi-paragraph body prose in admonitions, one of them quoting text that had already been corrected. Co-Authored-By: Claude Opus 5 --- learn/administration/backup-and-recovery.mdx | 3 +++ learn/administration/safe-deployments-and-rollback.mdx | 8 ++++---- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/learn/administration/backup-and-recovery.mdx b/learn/administration/backup-and-recovery.mdx index cb13c67aa..ab87c0a08 100644 --- a/learn/administration/backup-and-recovery.mdx +++ b/learn/administration/backup-and-recovery.mdx @@ -73,6 +73,7 @@ A backup is a whole-database copy: all tables, the transaction log, and any file-backed blobs. So a restored database keeps its `read_audit_log` history as of the backup point. +:::warning Two limitations to check against your own deployment now rather than later: - **Managed backups require RocksDB.** For LMDB databases your options are @@ -83,6 +84,8 @@ Two limitations to check against your own deployment now rather than later: configured per-table paths, your backup strategy for that database does not exist yet and you need to know that before launch. +::: + ## Create and retain managed backups diff --git a/learn/administration/safe-deployments-and-rollback.mdx b/learn/administration/safe-deployments-and-rollback.mdx index 92e62b3e4..baf18bffc 100644 --- a/learn/administration/safe-deployments-and-rollback.mdx +++ b/learn/administration/safe-deployments-and-rollback.mdx @@ -71,7 +71,7 @@ is the deliberate exception: it replicates only when you ask.) ::: `urlPath` mounts a component at an HTTP path. -`host` serves it on a virtual hostname. Both are +`host` (Added in: v5.2.0) serves it on a virtual hostname. Both are persisted on the component's root config entry, so they are part of the deployed state rather than a runtime toggle. See [HTTP middleware routing](/reference/v5/http/overview#middleware-routing). @@ -228,10 +228,10 @@ synthetic rather than through the deploy response. Two parameters worth setting deliberately on replicated deploys: -- `deployment_timeout` is how long a peer waits +- `deployment_timeout` (Added in: v5.1.4) is how long a peer waits for the replicated payload before failing, defaulting to 120000 ms. Raise it for large components or slow links. -- `ignore_replication_errors` treats a peer that +- `ignore_replication_errors` (Added in: v5.1.4) treats a peer that fails to receive the deploy as non-fatal. By default a failed peer makes the whole operation return a non-2xx status, while the component is still deployed on the origin node. Decide which behavior you want before you need it, because @@ -243,7 +243,7 @@ Two parameters worth setting deliberately on replicated deploys: A configuration change carries the same risk as a code change and gets less ceremony, which is backwards. -`set_configuration` supports `"replicated": true` +`set_configuration` supports `"replicated": true` (Added in: v5.2.0) to apply a change across the cluster in one call, with per-node outcomes in the response. To finish the change cluster-wide, follow with `restart_service` using `"replicated": true`, which restarts nodes one at a time. From 6febc5ee48959536610bf4421fd7d2f18bb81a48 Mon Sep 17 00:00:00 2001 From: Ethan Arrowood Date: Tue, 15 Sep 2026 14:42:08 -0600 Subject: [PATCH 06/21] docs: fixes from an independent Gemini review pass Ran the PR diff through Gemini (gemini-3.1-pro-high) file by file. Of 52 findings, 12 held up against the harper/harper-pro sources; the rest were contradicted by code I had already verified. The ones that held up cluster almost entirely in internal consistency, which is what a diff-only reviewer can actually check. Bugs this pass caught, several of my own making: - `HARPER_SAFE_MODE=1 harper` does not start the server. Bare `harper` falls through to the CLI-operations branch; starting it needs `harper run`. I introduced this when correcting the reference's `harperdb`. - The drain sequence checked peer capacity at step 3, after the node was already drained and its traffic had moved. Moved the check before the drain and added the abort path, which was missing entirely. - The readiness sample could never pass. After I changed it to test the sentinel record, nothing told the reader to create one, so the probe would return 503 forever. Added the seeding step. - Guide 8 contradicted itself on recovery time: one place measured from detection, another counted a detection gap inside the total. Settled on recovery time starting when the journey breaks, detection included. - The availability flag does not stop replication, so the restore sequence implied an isolation it never provided. An online restore on a drained but still-connected node exposes rolled-back data to peers before validation. Isolation is now its own step with a danger note. - `system_information` reports node and npm versions, not Harper's. The service inventory now points at `registration_info`. - The sizing sample requested neither disk nor network while the prose told the reader to watch both. - A Prettier wrap left a link orphaned as its own paragraph, stranding the sentence that followed it. - Smaller: "point-in-time restore" implied continuous PITR rather than restoring a snapshot; two checklist items still named `config.yaml` after the prose moved to the root `harper-config.yaml`; the status types are fixed while only their values are application-defined; and one leftover opt-in phrasing sat next to the corrected opt-out default. Co-Authored-By: Claude Opus 5 --- learn/administration/backup-and-recovery.mdx | 38 +++++++++++--- .../engineering-rpo-rto-and-uptime.mdx | 51 ++++++++++--------- .../health-checks-and-traffic-admission.mdx | 36 ++++++++++--- .../how-harper-runs-in-production.mdx | 5 +- .../administration/monitoring-and-triage.mdx | 12 ++--- .../production-readiness-checklist.mdx | 6 ++- .../sizing-a-harper-cluster.mdx | 4 +- reference/configuration/overview.md | 2 +- reference/operations-api/operations.md | 2 +- reference/replication/overview.md | 2 +- 10 files changed, 104 insertions(+), 54 deletions(-) diff --git a/learn/administration/backup-and-recovery.mdx b/learn/administration/backup-and-recovery.mdx index ab87c0a08..9b8419730 100644 --- a/learn/administration/backup-and-recovery.mdx +++ b/learn/administration/backup-and-recovery.mdx @@ -246,7 +246,7 @@ That last constraint has an escape hatch worth knowing before you need it. Setti components, while the database, Operations API, and HTTP server come up normally: ```bash -HARPER_SAFE_MODE=1 harper +HARPER_SAFE_MODE=1 harper run ``` Because no component loads, nothing holds a user database open, so a restore that @@ -275,15 +275,35 @@ then has to rejoin peers that never rolled back, so sequence it deliberately: 1. Take the node out of rotation with the availability flag, per [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). Never restore a node that is serving traffic. -2. Decide what should happen to replication for the restored database, and do that - deliberately. Whether the restored data should propagate, or be overwritten by - peers, is a decision with two very different outcomes, and it is not a decision - to improvise. -3. Perform the restore, online or offline according to the table above. -4. Verify data expectations locally before any peer sees the node. -5. Re-admit the node through the full return sequence, including convergence +2. **Isolate it from its peers.** This is a separate action from step 1, and the + step most likely to be skipped. See the warning below. +3. Decide what should happen to the restored data: whether it should propagate to + peers, or be overwritten by them. Those are two very different outcomes and it + is not a decision to improvise once the node is back on the mesh. +4. Perform the restore, online or offline according to the table above. +5. Verify data expectations locally before any peer sees the node. +6. Re-admit the node through the full return sequence, including convergence verification. +:::danger +**The availability flag does not stop replication.** It tells your traffic layer +to stop sending user requests. Replication is a separate path: peer connections +stay up, and the node keeps exchanging transactions the whole time it is "out of +rotation." + +So a node that is drained but still connected is not isolated. Restore it in place +and the rolled-back database is live on the mesh immediately, before you have +validated anything. Two things can then happen, both bad: peers overwrite your +restored data with newer transactions, or your rolled-back state propagates +outward to peers that were never affected. + +Isolation is its own step. Stopping Harper and restoring offline is the +unambiguous way to get it, which is a further argument for the offline path in a +cluster even where the table above permits an online restore. If you restore +online, you must have removed the node from the replication topology first, and +put it back deliberately in step 6. +::: + ## What backups cannot fix If a release changed the meaning of persisted data, restoring the database is @@ -360,6 +380,8 @@ that it is the target than a promise you cannot keep. - [ ] Restore constraints documented for user databases, component-held databases, and `system` - [ ] Restore authority named, and restore execution logged +- [ ] Restore procedure isolates the node from replication, not just from user + traffic, since the availability flag does not stop peer exchange - [ ] Safe mode known as the way to restore a database a component holds open, and to reach a node a broken component stopped - [ ] Restore drill completed and dated, with measured recovery time and recovery diff --git a/learn/administration/engineering-rpo-rto-and-uptime.mdx b/learn/administration/engineering-rpo-rto-and-uptime.mdx index b1d7043e2..af0a44dd7 100644 --- a/learn/administration/engineering-rpo-rto-and-uptime.mdx +++ b/learn/administration/engineering-rpo-rto-and-uptime.mdx @@ -35,11 +35,11 @@ during an incident that the answer was the target all along. ## State the three per journey -| Term | The question it answers | How to state it so it is testable | -| ------------------------ | ------------------------------------ | --------------------------------------------------------------------------- | -| Recovery point objective | How much recent data may be lost | Per database, in units of time, measured against your backup timestamp | -| Recovery time objective | How long until service is restored | Per journey, measured from detection to verified service, not from decision | -| Availability target | How much unavailability is permitted | Per journey, as minutes per month, with the measurement boundary named | +| Term | The question it answers | How to state it so it is testable | +| ------------------------ | ------------------------------------ | -------------------------------------------------------------------------------------------------- | +| Recovery point objective | How much recent data may be lost | Per database, in units of time, measured against your backup timestamp | +| Recovery time objective | How long until service is restored | Per journey, measured from the moment the journey starts failing, not from the moment someone acts | +| Availability target | How much unavailability is permitted | Per journey, as minutes per month, with the measurement boundary named | Three details make the difference between a testable statement and a slogan. @@ -47,10 +47,12 @@ Three details make the difference between a testable statement and a slogan. differently, matter differently, and recover differently. One cluster-wide number overcommits on the cheap journey and undercommits on the expensive one. -**Recovery time starts at detection, not at decision.** The gap between something -breaking and someone knowing is part of the outage, and it is often the largest -part. Measuring from the moment a human decided to act produces a number that -flatters your automation and misleads your planning. +**Recovery time starts when the journey breaks, not when someone acts on it.** +Detection is inside the clock, not before it, and the gap between something +breaking and someone knowing is often the largest part of an outage. Starting the +count at the moment a human decided to act, or even at the moment an alert fired, +produces a number that flatters your automation and misleads your planning. Your +users were already down for all of it. **Name the measurement boundary.** Availability measured at your CDN edge and availability measured at a Harper node are different quantities, and the @@ -88,20 +90,20 @@ plan, because the number no longer means what it appears to. For each scenario, know the mechanism, and know what must be proven for the mechanism to count. The right-hand column is the whole exercise. -| Scenario | Primary mechanism | What must be proven | -| ------------------------------------- | ------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | -| Single node loss | Peer capacity plus traffic removal | `(N - F)` capacity under load, measured time to remove traffic, no user-visible errors in transition | -| Node returns after a restart | Incremental catch-up plus an admission gate | Measured convergence time, and an admission gate that does not admit before convergence completes | -| Node replacement or scale-out | Full database synchronization to the new node | Measured full-sync duration at your data volume, and the load it puts on the source node | -| Failure domain loss | Topology spread across independent domains | No shared power or network fault, and surviving capacity inside the budget | -| Region loss | Multi-region topology plus route change | Data authority per region, route propagation time, and declared degraded behavior | -| Bad release, no data change | Release reversal | Measured release reversal time, and the previous artifact still addressable | -| Bad release that changed data meaning | Forward repair, not restore | A written repair or replay procedure, rehearsed on representative data | -| Accidental destructive operation | Restore from backup | Measured restore time and actual data loss, from a real drill | -| Storage loss on one node | Off-host backup copy | A restorable off-host copy, taken by the correct procedure for your mechanism | -| Corruption propagated by replication | Point-in-time restore plus a replication decision | A rehearsed sequence that includes what happens to peers | -| Replication certificate expiry | Certificate lifecycle management | Expiry dates tracked, with an alert far enough ahead to act | -| Downstream dependency outage | Declared degraded mode | The journey's behavior when the dependency is down, verified rather than assumed | +| Scenario | Primary mechanism | What must be proven | +| ------------------------------------- | --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | +| Single node loss | Peer capacity plus traffic removal | `(N - F)` capacity under load, measured time to remove traffic, no user-visible errors in transition | +| Node returns after a restart | Incremental catch-up plus an admission gate | Measured convergence time, and an admission gate that does not admit before convergence completes | +| Node replacement or scale-out | Full database synchronization to the new node | Measured full-sync duration at your data volume, and the load it puts on the source node | +| Failure domain loss | Topology spread across independent domains | No shared power or network fault, and surviving capacity inside the budget | +| Region loss | Multi-region topology plus route change | Data authority per region, route propagation time, and declared degraded behavior | +| Bad release, no data change | Release reversal | Measured release reversal time, and the previous artifact still addressable | +| Bad release that changed data meaning | Forward repair, not restore | A written repair or replay procedure, rehearsed on representative data | +| Accidental destructive operation | Restore from backup | Measured restore time and actual data loss, from a real drill | +| Storage loss on one node | Off-host backup copy | A restorable off-host copy, taken by the correct procedure for your mechanism | +| Corruption propagated by replication | Restore the last good backup, plus a replication decision | A rehearsed sequence that includes what happens to peers | +| Replication certificate expiry | Certificate lifecycle management | Expiry dates tracked, with an alert far enough ahead to act | +| Downstream dependency outage | Declared degraded mode | The journey's behavior when the dependency is down, verified rather than assumed | Two rows in that table are the ones most often missing. Node replacement is budgeted as though it were a restart, when it is a full synchronization and can be @@ -217,7 +219,8 @@ would have hurt you. ## Readiness checklist - [ ] Recovery point stated per database, in time units -- [ ] Recovery time stated per journey, measured from detection +- [ ] Recovery time stated per journey, measured from the start of failure and + including detection time - [ ] Availability stated per journey, with the measurement boundary named - [ ] Exclusions documented - [ ] Every scenario in the map has a named mechanism diff --git a/learn/administration/health-checks-and-traffic-admission.mdx b/learn/administration/health-checks-and-traffic-admission.mdx index fcd9524b3..da0339e74 100644 --- a/learn/administration/health-checks-and-traffic-admission.mdx +++ b/learn/administration/health-checks-and-traffic-admission.mdx @@ -230,6 +230,21 @@ authorization error rather than a readiness one. In that form you need `allowRead() { return true; }` to keep it reachable. ::: +The sentinel is a record you create once, on purpose, and leave alone. Seed it on +every node before you point anything at this route, or readiness will report +`failed` forever and you will have built a probe that never passes: + +```bash +curl -s -X PUT https://my-node.example.com:9926/Product/readiness-probe-sentinel \ + -H 'Content-Type: application/json' \ + -u 'admin:password' \ + -d '{"name":"readiness probe sentinel","description":"do not delete"}' +``` + +Give it a name that says what it is, because the next person to find it will be +deciding whether it is safe to delete. If your table replicates, seeding it once +is enough; if it does not, seed it per node. + Two details in that example are the point of it. The timeout on the downstream call means a slow dependency cannot make your readiness check hang, which would make the node look dead to a probe rather than unready. And returning the version @@ -244,17 +259,22 @@ skip. -1. **Declare it unavailable.** `DELETE /status` on the target node. -2. **Verify traffic actually stopped.** Watch request volume on the target fall to +1. **Confirm the peers can carry it, before you drain anything.** The remaining + nodes have to stay inside their capacity budget with this node gone. If they + cannot, stop here and do not drain: see + [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx). Checking this after + the traffic has already moved is how a maintenance window becomes an incident. +2. **Declare it unavailable.** `DELETE /status` on the target node. +3. **Verify traffic actually stopped.** Watch request volume on the target fall to zero and rise on its peers. Do not trust the configured weight; check the measured request count from [analytics](/reference/v5/analytics/overview) or your traffic layer's own metrics. Health check intervals, DNS TTLs, and client-side connection reuse all add delay here, and the delay is yours to measure. -3. **Confirm the peers can carry it.** The remaining nodes have to be inside their - capacity budget with this node gone. If they are not, stop: see - [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx). -4. **Do the work.** Restart, upgrade, reconfigure, or investigate. +4. **Watch the peers absorb it.** If they are going outside budget despite step 1, + abort: `POST /status` to put this node back in rotation, and re-plan the change + for a lower-traffic window or with added capacity. +5. **Do the work.** Restart, upgrade, reconfigure, or investigate. @@ -323,8 +343,8 @@ than people assume, because it includes convergence rather than just startup. ## Readiness checklist -- [ ] `@harperdb/status-check` deployed, declared in `config.yaml` rather than - deployed by hand +- [ ] `@harperdb/status-check` deployed, declared in the root `harper-config.yaml` + rather than deployed by hand - [ ] Traffic layer health check points at `GET /status` on `9926` - [ ] Availability flag storage still declares `replicate: false`, if you forked the component or persist it yourself diff --git a/learn/administration/how-harper-runs-in-production.mdx b/learn/administration/how-harper-runs-in-production.mdx index f9690246e..81e6160df 100644 --- a/learn/administration/how-harper-runs-in-production.mdx +++ b/learn/administration/how-harper-runs-in-production.mdx @@ -198,13 +198,16 @@ Then: what is deployed, including which extensions are present - [`list_deployments`](/reference/v5/operations-api/operations#list_deployments) for what changed and when +- [`registration_info`](/reference/v5/operations-api/operations#registration_info) + for the Harper version, which `system_information` does not report (it returns + the node and npm versions, not Harper's) - `cluster_status` for the peers and databases actually connected Fill in a boundary record you can keep: | Item | Value for your deployment | | ----------------------- | ----------------------------------------------------- | -| Harper version | From `system_information` | +| Harper version | From `registration_info` | | Node count and roles | From `cluster_status` and your topology intent | | Ports in use | From `get_configuration` | | Databases | Which exist, and which replicate | diff --git a/learn/administration/monitoring-and-triage.mdx b/learn/administration/monitoring-and-triage.mdx index ef33fffa0..d51d8b89b 100644 --- a/learn/administration/monitoring-and-triage.mdx +++ b/learn/administration/monitoring-and-triage.mdx @@ -174,12 +174,12 @@ threads: preload: dd-trace/register.js # ESM loader hooks for automatic instrumentation ``` -[`threads.preload`](/reference/v5/configuration/options#threads) - - loads a module via Node's `--import`, which is how an agent installs the loader hooks -that let it instrument modules imported later. [`threads.preloadRequire`](/reference/v5/configuration/options#threads) - uses `--require`, which runs the module body and is typically how an agent's -initialization entry actually starts it. +`threads.preload` (Added in: v5.2.0) loads a module via Node's `--import`, which is +how an agent installs the loader hooks that let it instrument modules imported +later. `threads.preloadRequire` (Added in: v5.2.0) uses `--require`, which runs the +module body and is typically how an agent's initialization entry actually starts +it. Both are documented under +[threads configuration](/reference/v5/configuration/options#threads). Installing loader hooks and starting the agent are two different jobs, and which of your agent's entry points does which is specific to that agent. Some ship one diff --git a/learn/administration/production-readiness-checklist.mdx b/learn/administration/production-readiness-checklist.mdx index 0cbeac7f7..45d1165ba 100644 --- a/learn/administration/production-readiness-checklist.mdx +++ b/learn/administration/production-readiness-checklist.mdx @@ -69,8 +69,8 @@ From - [ ] Liveness, availability flag, application readiness, and journey synthetic all exist as separate signals -- [ ] `@harperdb/status-check` deployed, declared in `config.yaml` rather than by - hand +- [ ] `@harperdb/status-check` deployed, declared in the root `harper-config.yaml` + rather than by hand - [ ] Traffic layer health check points at `GET /status` on the application port - [ ] Availability flag storage still declares `replicate: false`, if you forked the component or persist it yourself @@ -161,6 +161,8 @@ From [Backup and Recovery](./backup-and-recovery.mdx). - [ ] Restore constraints documented for user databases, component-held databases, and `system` - [ ] Restore authority named, and restore execution logged +- [ ] Restore procedure isolates the node from replication, not just from user + traffic - [ ] Restore drill completed and dated, with measured recovery time and actual data loss - [ ] `system` database restore rehearsed offline diff --git a/learn/administration/sizing-a-harper-cluster.mdx b/learn/administration/sizing-a-harper-cluster.mdx index b06a86948..9a226224f 100644 --- a/learn/administration/sizing-a-harper-cluster.mdx +++ b/learn/administration/sizing-a-harper-cluster.mdx @@ -98,7 +98,7 @@ An average conceals the one node that is about to tip. curl -s -X POST https://my-node.example.com:9925/ \ -H 'Content-Type: application/json' \ -u 'admin:password' \ - -d '{"operation":"system_information","attributes":["cpu","memory","threads","harperdb_processes"]}' + -d '{"operation":"system_information","attributes":["cpu","memory","disk","network","threads","harperdb_processes"]}' ``` @@ -113,7 +113,7 @@ await fetch('https://my-node.example.com:9925/', { }, body: JSON.stringify({ operation: 'system_information', - attributes: ['cpu', 'memory', 'threads', 'harperdb_processes'], + attributes: ['cpu', 'memory', 'disk', 'network', 'threads', 'harperdb_processes'], }), }); ``` diff --git a/reference/configuration/overview.md b/reference/configuration/overview.md index 51284de3f..c2fc1e65b 100644 --- a/reference/configuration/overview.md +++ b/reference/configuration/overview.md @@ -299,7 +299,7 @@ logging: Setting `HARPER_SAFE_MODE` to any value starts Harper without loading user applications or components. Harper's core services (database, operations API, HTTP server) start normally, but no applications from the components directory are loaded and no package-based extensions are initialized. ```bash -HARPER_SAFE_MODE=1 harper +HARPER_SAFE_MODE=1 harper run ``` This is useful when a broken or misbehaving component prevents Harper from starting. Safe mode lets you access the operations API to inspect, repair, or remove the problematic component without needing to manually edit files on disk. diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index cec727bb7..4dd8ee857 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -1779,7 +1779,7 @@ Valid attribute names are `system`, `time`, `cpu`, `memory`, `disk`, `network`, ### `set_status` / `get_status` / `clear_status` -Manage application-defined status values. Status types: `primary`, `maintenance`, `availability` (availability only accepts `'Available'` or `'Unavailable'`). +Manage application status values. The status types are fixed — `primary`, `maintenance`, `availability` — while the value you store against each is yours to define (availability only accepts `'Available'` or `'Unavailable'`). Values are **persisted**, stored in the `hdb_status` table in the `system` database, so they survive a restart. That table is declared `replicate: false`, so a status set on one node stays on that node and does not propagate to peers — which is what makes these values usable as node-local coordination markers. diff --git a/reference/replication/overview.md b/reference/replication/overview.md index bec57e419..c79bbf6aa 100644 --- a/reference/replication/overview.md +++ b/reference/replication/overview.md @@ -393,7 +393,7 @@ Note that `replicated` is opt-out rather than opt-in across the operations that Users and roles **do** propagate by default. They live in the `system` database, and the default replication scope is every database, so `system` is in scope unless you have narrowed it. As of v5.2 replicating `system` no longer forces a full mesh — see [Replicating the `system` database with controlled flow](#replicating-the-system-database-with-controlled-flow). If you narrow the scope to exclude `system`, users and roles stop propagating and must be provisioned on every node by your own automation. -Certain management operations — including component deployment and rolling restarts — can also be replicated across the cluster. +Certain management operations — including component deployment and rolling restarts — are replicated across the cluster as well, following the same opt-out rule described above. ## Inspecting Cluster Configuration From 10f15ae52d52f19b1101ec6b959923140a1afe28 Mon Sep 17 00:00:00 2001 From: Ethan Arrowood Date: Tue, 15 Sep 2026 15:46:26 -0600 Subject: [PATCH 07/21] docs(learn): unwrap hard-wrapped prose in the Administration guides The ten guides arrived hard-wrapped at 80 columns, which nothing else in the repo does: existing Learn and reference pages run one line per paragraph, with prose lines routinely past 500 characters. Prettier is configured without proseWrap, so it preserves whatever it is given and never enforced either style. Wrapping is undone across all ten. Paragraphs and list items are now one logical line each. Code fences, tables, frontmatter, JSX, admonitions, headings, blockquotes, and import lines are untouched, and a content fingerprint was compared before and after each rewrite to confirm nothing but line breaks changed. Verified per file: identical structural element counts, identical token streams, and a clean build. Also carries an edit to the opening paragraph of how-harper-runs-in-production.mdx that was made by hand in the worktree, not by the rewrap. It is the one token-level change in the diff. Co-Authored-By: Claude Opus 5 --- learn/administration/backup-and-recovery.mdx | 342 +++++------------- .../engineering-rpo-rto-and-uptime.mdx | 181 +++------ .../health-checks-and-traffic-admission.mdx | 238 +++--------- .../how-harper-runs-in-production.mdx | 173 +++------ .../administration/monitoring-and-triage.mdx | 261 ++++--------- .../administration/operating-replication.mdx | 282 ++++----------- .../production-readiness-checklist.mdx | 117 ++---- .../reliability-plan-template.mdx | 39 +- .../safe-deployments-and-rollback.mdx | 277 ++++---------- .../sizing-a-harper-cluster.mdx | 178 +++------ 10 files changed, 523 insertions(+), 1565 deletions(-) diff --git a/learn/administration/backup-and-recovery.mdx b/learn/administration/backup-and-recovery.mdx index 9b8419730..164222592 100644 --- a/learn/administration/backup-and-recovery.mdx +++ b/learn/administration/backup-and-recovery.mdx @@ -6,24 +6,15 @@ sidebar_position: 7 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -Replication gives you availability. It does not give you recovery, and the reason -is worth stating plainly: replication faithfully propagates whatever you write, -including the bad write. A destructive operation, a schema mistake, or a release -that corrupted data reaches every peer at the speed of your convergence time. +Replication gives you availability. It does not give you recovery, and the reason is worth stating plainly: replication faithfully propagates whatever you write, including the bad write. A destructive operation, a schema mistake, or a release that corrupted data reaches every peer at the speed of your convergence time. -Backups cover the failure classes replication cannot. This guide is about -deciding what you need before you pick an operation, getting a copy somewhere the -node's failure cannot reach, and knowing your restore constraints before you are -in an incident rather than during one. +Backups cover the failure classes replication cannot. This guide is about deciding what you need before you pick an operation, getting a copy somewhere the node's failure cannot reach, and knowing your restore constraints before you are in an incident rather than during one. ## What You Will Learn -- The five questions that determine your backup design, answered before any - command -- Which backup mechanism you actually have, since it depends on your storage - engine -- Why a copy on the node that created it is not a backup, and the specific way - copying one wrong breaks it +- The five questions that determine your backup design, answered before any command +- Which backup mechanism you actually have, since it depends on your storage engine +- Why a copy on the node that created it is not a backup, and the specific way copying one wrong breaks it - What `verify_backup` does and does not check - Which databases can be restored with the server running, and which cannot - Why restoring a database to undo a code bug is usually the wrong move @@ -31,36 +22,19 @@ in an incident rather than during one. ## Prerequisites - A cluster with a `super_user` credential, and CLI access to at least one node -- Knowledge of your storage engine per database, from your service boundary - inventory in - [How Harper Runs in Production](./how-harper-runs-in-production.mdx) -- [Operating Replication](./operating-replication.mdx), because a restore in a - replicated cluster is a replication event as much as a storage one +- Knowledge of your storage engine per database, from your service boundary inventory in [How Harper Runs in Production](./how-harper-runs-in-production.mdx) +- [Operating Replication](./operating-replication.mdx), because a restore in a replicated cluster is a replication event as much as a storage one - A non-production database you are willing to destroy ## Decide before you choose a command -Answer these first. Every operation below is easy, and every one of them is the -wrong choice for some of these answers. - -1. **Which failure classes must you survive?** Node loss, storage loss, a bad - deployment, an accidental destructive operation, a corrupting application bug, - a site or region event, and a control-plane or credential loss are seven - different problems. Replication addresses the first one well and the sixth one - partially. It addresses none of the rest. -2. **What is the recovery point and recovery time per database?** Not per cluster. - A catalog table and an order ledger rarely deserve the same answer, and paying - ledger-grade backup cadence for catalog data is how backup cost becomes a - reason to reduce frequency. -3. **What granularity do you need?** Harper backs up and restores whole databases. - There is no per-table restore. If your recovery story requires restoring one - table, that requirement has to be met by database layout or by forward repair, - and it is far cheaper to learn that now. -4. **Where must copies live?** How many independent locations, under what - retention, and with what access control. -5. **Who is authorized to execute a restore, and how is that logged?** A restore - destroys current data by design. It deserves a named authority and an audit - trail. +Answer these first. Every operation below is easy, and every one of them is the wrong choice for some of these answers. + +1. **Which failure classes must you survive?** Node loss, storage loss, a bad deployment, an accidental destructive operation, a corrupting application bug, a site or region event, and a control-plane or credential loss are seven different problems. Replication addresses the first one well and the sixth one partially. It addresses none of the rest. +2. **What is the recovery point and recovery time per database?** Not per cluster. A catalog table and an order ledger rarely deserve the same answer, and paying ledger-grade backup cadence for catalog data is how backup cost becomes a reason to reduce frequency. +3. **What granularity do you need?** Harper backs up and restores whole databases. There is no per-table restore. If your recovery story requires restoring one table, that requirement has to be met by database layout or by forward repair, and it is far cheaper to learn that now. +4. **Where must copies live?** How many independent locations, under what retention, and with what access control. +5. **Who is authorized to execute a restore, and how is that logged?** A restore destroys current data by design. It deserves a named authority and an audit trail. ## Know which mechanism you have @@ -69,20 +43,13 @@ wrong choice for some of these answers. | **Managed backups** | | Incremental, verifiable backups in a server-side repository under `storage.backupPath` | | **Snapshot download** | | [`get_backup`](/reference/v5/backups/operations#get_backup) streams a snapshot over HTTP, with no server-side artifact | -A backup is a whole-database copy: all tables, the transaction log, and any -file-backed blobs. So a restored database keeps its `read_audit_log` history as of -the backup point. +A backup is a whole-database copy: all tables, the transaction log, and any file-backed blobs. So a restored database keeps its `read_audit_log` history as of the backup point. :::warning Two limitations to check against your own deployment now rather than later: -- **Managed backups require RocksDB.** For LMDB databases your options are - `get_backup` or volume snapshots. -- **One storage root per database.** A database whose tables use per-table `path` - storage configs spans multiple root stores and cannot be backed up with these - operations at all. Database-level custom storage paths are fine. If someone has - configured per-table paths, your backup strategy for that database does not - exist yet and you need to know that before launch. +- **Managed backups require RocksDB.** For LMDB databases your options are `get_backup` or volume snapshots. +- **One storage root per database.** A database whose tables use per-table `path` storage configs spans multiple root stores and cannot be backed up with these operations at all. Database-level custom storage paths are fine. If someone has configured per-table paths, your backup strategy for that database does not exist yet and you need to know that before launch. ::: @@ -98,10 +65,7 @@ Two limitations to check against your own deployment now rather than later: } ``` -Through a running server this returns a `job_id` immediately. Poll -[`get_job`](/reference/v5/operations-api/operations#get_job) for the outcome, -which includes the new `backup_id`, `size`, and `timestamp`. Treat the job result -as the completion signal, not the original response. +Through a running server this returns a `job_id` immediately. Poll [`get_job`](/reference/v5/operations-api/operations#get_job) for the outcome, which includes the new `backup_id`, `size`, and `timestamp`. Treat the job result as the completion signal, not the original response. @@ -112,70 +76,37 @@ harper list_backups database=data harper purge_backups database=data keep_count=7 ``` -Every backup operation runs from the CLI under the same name. With the server -running the CLI forwards to the server; with it stopped the command operates -directly on the files. +Every backup operation runs from the CLI under the same name. With the server running the CLI forwards to the server; with it stopped the command operates directly on the files. -`get_backup` is the exception. It streams from a running server and has no offline -form, so with Harper stopped it fails as an unknown command. If you need a copy -off a node that is down, start it, or take the backup directory or a volume -snapshot instead. Worth knowing before an incident, because a stopped node is -exactly when you reach for it. +`get_backup` is the exception. It streams from a running server and has no offline form, so with Harper stopped it fails as an unknown command. If you need a copy off a node that is down, start it, or take the backup directory or a volume snapshot instead. Worth knowing before an incident, because a stopped node is exactly when you reach for it. -Backups of the same database share unchanged RocksDB data files, so the first one -copies everything and later ones copy only what changed. Shared files are -reference-counted, so deleting a backup removes only files no remaining backup -references. +Backups of the same database share unchanged RocksDB data files, so the first one copies everything and later ones copy only what changed. Shared files are reference-counted, so deleting a backup removes only files no remaining backup references. :::warning -The incremental behavior applies to the RocksDB data files only. **The -transaction-log snapshot is copied in full on every backup.** With a large -audit-retention window that is real, recurring disk cost that the data-only view -hides, so size your backup volume against it rather than against the incremental -figure. - -Blobs behave differently and are easy to over-budget. Each backup captures a full -set of blobs, but the files are hard-linked rather than copied when the backup -directory and the blob storage share a filesystem, which is the default layout. In -that case the extra space is near zero. They become genuine full copies only when -the two live on different filesystems, so putting backups on a separate volume is -a real cost decision rather than a free one. Pass `exclude_blobs: true` to skip -blobs when that is appropriate. - -One consequence for disaster recovery: because the blob snapshot is hard-linked, -copying a backup repository with an ordinary recursive copy materializes every -blob as a full, separate file at the destination. +The incremental behavior applies to the RocksDB data files only. **The transaction-log snapshot is copied in full on every backup.** With a large audit-retention window that is real, recurring disk cost that the data-only view hides, so size your backup volume against it rather than against the incremental figure. + +Blobs behave differently and are easy to over-budget. Each backup captures a full set of blobs, but the files are hard-linked rather than copied when the backup directory and the blob storage share a filesystem, which is the default layout. In that case the extra space is near zero. They become genuine full copies only when the two live on different filesystems, so putting backups on a separate volume is a real cost decision rather than a free one. Pass `exclude_blobs: true` to skip blobs when that is appropriate. + +One consequence for disaster recovery: because the blob snapshot is hard-linked, copying a backup repository with an ordinary recursive copy materializes every blob as a full, separate file at the destination. ::: -Do not use `list_backups` sizes for capacity planning. The `size` and -`file_count` fields come from the RocksDB backup engine and exclude the -transaction-log and blob snapshots, so each entry undercounts, while the shared -files between entries mean summing them overcounts. Measure the repository -directory instead. +Do not use `list_backups` sizes for capacity planning. The `size` and `file_count` fields come from the RocksDB backup engine and exclude the transaction-log and blob snapshots, so each entry undercounts, while the shared files between entries mean summing them overcounts. Measure the repository directory instead. ## Get a copy off the node -This is the step most likely to be missing, and the one where doing it slightly -wrong produces a copy that cannot be restored. +This is the step most likely to be missing, and the one where doing it slightly wrong produces a copy that cannot be restored. -**Managed backups live on the node that created them.** The repository is a local -directory, and RocksDB shares files across backup IDs, so **a backup ID is not a -self-contained folder.** Two consequences: +**Managed backups live on the node that created them.** The repository is a local directory, and RocksDB shares files across backup IDs, so **a backup ID is not a self-contained folder.** Two consequences: -- A disaster-recovery copy has to take the entire per-database repository, - `/`, not an individual backup. -- It has to do that while no backup operation is running, or from an atomic - filesystem snapshot. A live recursive copy can race `create_backup`, - `delete_backup`, or `purge_backups` and produce an unrestorable copy. +- A disaster-recovery copy has to take the entire per-database repository, `/`, not an individual backup. +- It has to do that while no backup operation is running, or from an atomic filesystem snapshot. A live recursive copy can race `create_backup`, `delete_backup`, or `purge_backups` and produce an unrestorable copy. -An unrestorable copy is worse than no copy, because it will pass a "backups exist" -check and fail during an incident. +An unrestorable copy is worse than no copy, because it will pass a "backups exist" check and fail during an incident. -The simpler off-host path, and the one to prefer unless you specifically need -retained managed backups off the node: +The simpler off-host path, and the one to prefer unless you specifically need retained managed backups off the node: ```bash # Pull a snapshot of the current state from a running node @@ -185,18 +116,11 @@ harper get_backup database=data out=./data-$(date +%Y%m%dT%H%M%S).tar.gz harper get_backup database=data target=https://node-2.example.com:9925 out=./data.tar.gz ``` -`target=` only changes which node the snapshot is read from. It writes a file and -nothing else: it does not restore, import, or clone that node's database onto the -local one. Turning the file into a live database is a separate, deliberate restore -step, performed with Harper stopped. +`target=` only changes which node the snapshot is read from. It writes a file and nothing else: it does not restore, import, or clone that node's database onto the local one. Turning the file into a live database is a separate, deliberate restore step, performed with Harper stopped. -Note also that `get_backup` always streams the current state. It cannot download a -historical managed backup, so it is a way to take a fresh off-host copy, not a way -to export your retention history. +Note also that `get_backup` always streams the current state. It cannot download a historical managed backup, so it is a way to take a fresh off-host copy, not a way to export your retention history. -Whichever path you choose, the destination must not share a failure domain with -the source. A second directory on the same volume survives a deleted file and -nothing else. +Whichever path you choose, the destination must not share a failure domain with the source. A second directory on the same volume survives a deleted file and nothing else. ## Verification is not optional @@ -209,25 +133,15 @@ nothing else. } ``` -`verify_backup` checks the RocksDB file sizes, their checksums when -`verify_checksum` is `true`, which is slower, and the framing of the -transaction-log snapshot, which is always checked. +`verify_backup` checks the RocksDB file sizes, their checksums when `verify_checksum` is `true`, which is slower, and the framing of the transaction-log snapshot, which is always checked. -**Blob contents are not verified.** Verification confirms that a backup which -recorded blobs still has its blob snapshot present, and fails the backup as corrupt -if that snapshot is missing. It does not check the individual blob files for size, -checksum, or readability. So verification can pass on a backup whose blobs are -damaged, and only a real restore proves they are intact. +**Blob contents are not verified.** Verification confirms that a backup which recorded blobs still has its blob snapshot present, and fails the backup as corrupt if that snapshot is missing. It does not check the individual blob files for size, checksum, or readability. So verification can pass on a backup whose blobs are damaged, and only a real restore proves they are intact. -More generally, a verified backup is a well-formed backup, not a proven recovery. -The only evidence that your recovery works is a restore you have actually -performed, which is why the drill at the end of this guide is the point of it. +More generally, a verified backup is a well-formed backup, not a proven recovery. The only evidence that your recovery works is a restore you have actually performed, which is why the drill at the end of this guide is the point of it. ## Know your restore constraints before the incident -RocksDB is single-writer, so an in-place restore requires the database to be fully -closed first. That produces hard constraints you cannot negotiate during an -outage: +RocksDB is single-writer, so an in-place restore requires the database to be fully closed first. That produces hard constraints you cannot negotiate during an outage: | Database | Online restore, server running | Offline restore, server stopped | | ---------------------------------------------- | ------------------------------------ | ------------------------------- | @@ -235,176 +149,96 @@ outage: | A user database a loaded component holds open | No, the job ends in `ERROR` | Yes | | The `system` database | No, rejected before a job is created | Yes | -Because the restore runs as a background job, a component holding the database -open does not fail your request. It fails inside the job, so you find out from -`get_job` rather than from the response. Harper does not track which component -uses which database, so it cannot selectively stop one. +Because the restore runs as a background job, a component holding the database open does not fail your request. It fails inside the job, so you find out from `get_job` rather than from the response. Harper does not track which component uses which database, so it cannot selectively stop one. :::tip -That last constraint has an escape hatch worth knowing before you need it. Setting -`HARPER_SAFE_MODE` to any value starts Harper without loading any applications or -components, while the database, Operations API, and HTTP server come up normally: +That last constraint has an escape hatch worth knowing before you need it. Setting `HARPER_SAFE_MODE` to any value starts Harper without loading any applications or components, while the database, Operations API, and HTTP server come up normally: ```bash HARPER_SAFE_MODE=1 harper run ``` -Because no component loads, nothing holds a user database open, so a restore that -would fail inside its job can run online. It is also how you get back in when a -broken component is what stopped Harper in the first place, since the Operations -API is available to inspect, repair, or remove it. Traffic should stay off the -node either way. See -[safe mode](/reference/v5/configuration/overview#harper_safe_mode). +Because no component loads, nothing holds a user database open, so a restore that would fail inside its job can run online. It is also how you get back in when a broken component is what stopped Harper in the first place, since the Operations API is available to inspect, repair, or remove it. Traffic should stay off the node either way. See [safe mode](/reference/v5/configuration/overview#harper_safe_mode). ::: Two more constraints worth writing into your procedure: -- **`target_database` requires the server stopped.** Restoring into a separate - database rather than overwriting the source is CLI-only with Harper down. The - target must not already exist or must be an empty directory. -- **An interrupted restore leaves the database unloadable.** On a crash or power - loss mid-restore, Harper marks the database as incompletely restored and refuses - to load it on the next start. Recover by rerunning `restore_backup` for the same - database and `backup_id`. Do not try to load or hand-repair the directory. +- **`target_database` requires the server stopped.** Restoring into a separate database rather than overwriting the source is CLI-only with Harper down. The target must not already exist or must be an empty directory. +- **An interrupted restore leaves the database unloadable.** On a crash or power loss mid-restore, Harper marks the database as incompletely restored and refuses to load it on the next start. Recover by rerunning `restore_backup` for the same database and `backup_id`. Do not try to load or hand-repair the directory. ## Restoring in a replicated cluster -A restore is a point-in-time rollback of one node's data. In a cluster, that node -then has to rejoin peers that never rolled back, so sequence it deliberately: - -1. Take the node out of rotation with the availability flag, per - [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). - Never restore a node that is serving traffic. -2. **Isolate it from its peers.** This is a separate action from step 1, and the - step most likely to be skipped. See the warning below. -3. Decide what should happen to the restored data: whether it should propagate to - peers, or be overwritten by them. Those are two very different outcomes and it - is not a decision to improvise once the node is back on the mesh. +A restore is a point-in-time rollback of one node's data. In a cluster, that node then has to rejoin peers that never rolled back, so sequence it deliberately: + +1. Take the node out of rotation with the availability flag, per [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). Never restore a node that is serving traffic. +2. **Isolate it from its peers.** This is a separate action from step 1, and the step most likely to be skipped. See the warning below. +3. Decide what should happen to the restored data: whether it should propagate to peers, or be overwritten by them. Those are two very different outcomes and it is not a decision to improvise once the node is back on the mesh. 4. Perform the restore, online or offline according to the table above. 5. Verify data expectations locally before any peer sees the node. -6. Re-admit the node through the full return sequence, including convergence - verification. +6. Re-admit the node through the full return sequence, including convergence verification. :::danger -**The availability flag does not stop replication.** It tells your traffic layer -to stop sending user requests. Replication is a separate path: peer connections -stay up, and the node keeps exchanging transactions the whole time it is "out of -rotation." - -So a node that is drained but still connected is not isolated. Restore it in place -and the rolled-back database is live on the mesh immediately, before you have -validated anything. Two things can then happen, both bad: peers overwrite your -restored data with newer transactions, or your rolled-back state propagates -outward to peers that were never affected. - -Isolation is its own step. Stopping Harper and restoring offline is the -unambiguous way to get it, which is a further argument for the offline path in a -cluster even where the table above permits an online restore. If you restore -online, you must have removed the node from the replication topology first, and -put it back deliberately in step 6. +**The availability flag does not stop replication.** It tells your traffic layer to stop sending user requests. Replication is a separate path: peer connections stay up, and the node keeps exchanging transactions the whole time it is "out of rotation." + +So a node that is drained but still connected is not isolated. Restore it in place and the rolled-back database is live on the mesh immediately, before you have validated anything. Two things can then happen, both bad: peers overwrite your restored data with newer transactions, or your rolled-back state propagates outward to peers that were never affected. + +Isolation is its own step. Stopping Harper and restoring offline is the unambiguous way to get it, which is a further argument for the offline path in a cluster even where the table above permits an online restore. If you restore online, you must have removed the node from the replication topology first, and put it back deliberately in step 6. ::: ## What backups cannot fix -If a release changed the meaning of persisted data, restoring the database is -usually the wrong response. A restore rolls back every write in the window, -including all the correct ones from the same period, so you can violate your -recovery point objective in the course of fixing a code bug. +If a release changed the meaning of persisted data, restoring the database is usually the wrong response. A restore rolls back every write in the window, including all the correct ones from the same period, so you can violate your recovery point objective in the course of fixing a code bug. -For that failure class, define forward repair instead: a targeted correction, or a -replay from the transaction log, that fixes the affected records and leaves the -rest alone. Decide which of your failure classes get restore and which get forward -repair while you are calm, and record the decision. See -[Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx) for how this -interacts with release reversal. +For that failure class, define forward repair instead: a targeted correction, or a replay from the transaction log, that fixes the affected records and leaves the rest alone. Decide which of your failure classes get restore and which get forward repair while you are calm, and record the decision. See [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx) for how this interacts with release reversal. ### Prove it -A backup you have never restored is a hypothesis. Run this on a non-production -cluster and record the numbers: +A backup you have never restored is a hypothesis. Run this on a non-production cluster and record the numbers: 1. Create a managed backup, then verify it with `verify_checksum: true`. -2. Take an off-host copy using the correct procedure for your mechanism, quiesced - or from an atomic snapshot if you are copying a managed repository. +2. Take an off-host copy using the correct procedure for your mechanism, quiesced or from an atomic snapshot if you are copying a managed repository. 3. Destroy the source database. -4. Restore it, taking the node out of rotation first, and time from decision to - restored service. That is your measured recovery time. -5. Determine how much data was actually lost against the backup timestamp. That is - your measured recovery point. -6. Validate correctness through the application, not just at the storage layer. - Row counts agreeing is not the same as the journey working. -7. Repeat for the `system` database specifically, offline, since it has different - constraints and it is the one people never rehearse. - -Both measured numbers feed -[Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx). If they -do not meet your stated targets, one of the two has to change, and it is better -that it is the target than a promise you cannot keep. +4. Restore it, taking the node out of rotation first, and time from decision to restored service. That is your measured recovery time. +5. Determine how much data was actually lost against the backup timestamp. That is your measured recovery point. +6. Validate correctness through the application, not just at the storage layer. Row counts agreeing is not the same as the journey working. +7. Repeat for the `system` database specifically, offline, since it has different constraints and it is the one people never rehearse. + +Both measured numbers feed [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx). If they do not meet your stated targets, one of the two has to change, and it is better that it is the target than a promise you cannot keep. ## Operational notes -- **Retention is a policy, not a side effect of disk space.** Use - `purge_backups` with an explicit `keep_count` that matches a written retention - decision. -- **Backup operations are `super_user` through the server, and filesystem - permissions offline.** An operator with shell access on a node can restore - without an API credential, so protect the host accordingly. -- **Schedule backups per database, matched to that database's recovery point.** - A single cluster-wide cadence overspends on some databases and underspends on - the ones that matter. -- **Alert on backup age, and on job failure.** A `create_backup` job that fails - silently produces a gap you will find at the worst possible time. Backup age - exceeding your recovery point objective is a pageable condition. -- **Record the storage engine per database in your inventory** and re-check after - migrations, since your entire mechanism choice depends on it. +- **Retention is a policy, not a side effect of disk space.** Use `purge_backups` with an explicit `keep_count` that matches a written retention decision. +- **Backup operations are `super_user` through the server, and filesystem permissions offline.** An operator with shell access on a node can restore without an API credential, so protect the host accordingly. +- **Schedule backups per database, matched to that database's recovery point.** A single cluster-wide cadence overspends on some databases and underspends on the ones that matter. +- **Alert on backup age, and on job failure.** A `create_backup` job that fails silently produces a gap you will find at the worst possible time. Backup age exceeding your recovery point objective is a pageable condition. +- **Record the storage engine per database in your inventory** and re-check after migrations, since your entire mechanism choice depends on it. ## Readiness checklist - [ ] Failure classes enumerated, with the mechanism that addresses each - [ ] Recovery point and recovery time stated per database, not per cluster - [ ] Storage engine confirmed per database, and the mechanism chosen accordingly -- [ ] No database in scope uses per-table storage paths, or its exclusion is known - and accepted +- [ ] No database in scope uses per-table storage paths, or its exclusion is known and accepted - [ ] Backup cadence matches the stated recovery point per database -- [ ] Backup volume sized against the non-incremental transaction-log snapshot, - and against full blob copies if the backup and blob paths are on different - filesystems -- [ ] An off-host copy exists in a destination that does not share a failure - domain -- [ ] Managed repository copies take the whole `/` directory, - quiesced or from an atomic snapshot -- [ ] `verify_backup` runs on a schedule, with `verify_checksum` at least - periodically -- [ ] Blob contents understood to be unchecked by `verify_backup`, which confirms - only that the blob snapshot is present -- [ ] Restore constraints documented for user databases, component-held databases, - and `system` +- [ ] Backup volume sized against the non-incremental transaction-log snapshot, and against full blob copies if the backup and blob paths are on different filesystems +- [ ] An off-host copy exists in a destination that does not share a failure domain +- [ ] Managed repository copies take the whole `/` directory, quiesced or from an atomic snapshot +- [ ] `verify_backup` runs on a schedule, with `verify_checksum` at least periodically +- [ ] Blob contents understood to be unchecked by `verify_backup`, which confirms only that the blob snapshot is present +- [ ] Restore constraints documented for user databases, component-held databases, and `system` - [ ] Restore authority named, and restore execution logged -- [ ] Restore procedure isolates the node from replication, not just from user - traffic, since the availability flag does not stop peer exchange -- [ ] Safe mode known as the way to restore a database a component holds open, - and to reach a node a broken component stopped -- [ ] Restore drill completed and dated, with measured recovery time and recovery - point +- [ ] Restore procedure isolates the node from replication, not just from user traffic, since the availability flag does not stop peer exchange +- [ ] Safe mode known as the way to restore a database a component holds open, and to reach a node a broken component stopped +- [ ] Restore drill completed and dated, with measured recovery time and recovery point - [ ] `system` database restore rehearsed offline - [ ] Forward repair defined for corruption caused by application code ## Additional Resources -- [Backups overview](/reference/v5/backups/overview) for how managed backups work, - the full limitation list, and manual restore examples -- [Backup operations](/reference/v5/backups/operations) for every parameter of - `create_backup`, `list_backups`, `verify_backup`, `delete_backup`, - `purge_backups`, `restore_backup`, and `get_backup` -- [Storage configuration](/reference/v5/configuration/options#storage) for - `storage.backupPath`, `storage.path`, and `storage.blobPaths` -- [Jobs](/reference/v5/operations-api/operations#jobs) and - [`get_job`](/reference/v5/operations-api/operations#get_job) for tracking - long-running backup operations -- [CLI operations](/reference/v5/cli/operations-api-commands) and - [remote operations](/reference/v5/cli/overview#remote-operations) for the - offline and `target=` forms -- [`read_audit_log`](/reference/v5/operations-api/operations#read_audit_log) for - transaction history usable in forward repair -- [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx) for - turning the measured numbers into commitments +- [Backups overview](/reference/v5/backups/overview) for how managed backups work, the full limitation list, and manual restore examples +- [Backup operations](/reference/v5/backups/operations) for every parameter of `create_backup`, `list_backups`, `verify_backup`, `delete_backup`, `purge_backups`, `restore_backup`, and `get_backup` +- [Storage configuration](/reference/v5/configuration/options#storage) for `storage.backupPath`, `storage.path`, and `storage.blobPaths` +- [Jobs](/reference/v5/operations-api/operations#jobs) and [`get_job`](/reference/v5/operations-api/operations#get_job) for tracking long-running backup operations +- [CLI operations](/reference/v5/cli/operations-api-commands) and [remote operations](/reference/v5/cli/overview#remote-operations) for the offline and `target=` forms +- [`read_audit_log`](/reference/v5/operations-api/operations#read_audit_log) for transaction history usable in forward repair +- [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx) for turning the measured numbers into commitments diff --git a/learn/administration/engineering-rpo-rto-and-uptime.mdx b/learn/administration/engineering-rpo-rto-and-uptime.mdx index af0a44dd7..6b1cc11d0 100644 --- a/learn/administration/engineering-rpo-rto-and-uptime.mdx +++ b/learn/administration/engineering-rpo-rto-and-uptime.mdx @@ -3,35 +3,25 @@ title: Engineering RPO, RTO, and Uptime sidebar_position: 8 --- -Recovery point, recovery time, and availability are usually written down once, in -a document nobody consults, as three numbers that were never checked against a -mechanism. This guide is about making them real: stating them per journey, -mapping each failure scenario to the mechanism that addresses it, and reconciling -the targets against numbers you have actually measured. +Recovery point, recovery time, and availability are usually written down once, in a document nobody consults, as three numbers that were never checked against a mechanism. This guide is about making them real: stating them per journey, mapping each failure scenario to the mechanism that addresses it, and reconciling the targets against numbers you have actually measured. -If a target and a measurement disagree, one of them has to change. The point of -doing this deliberately is that you get to choose which, rather than finding out -during an incident that the answer was the target all along. +If a target and a measurement disagree, one of them has to change. The point of doing this deliberately is that you get to choose which, rather than finding out during an incident that the answer was the target all along. ## What You Will Learn -- How to state recovery point, recovery time, and availability so they are - testable rather than aspirational +- How to state recovery point, recovery time, and availability so they are testable rather than aspirational - A scenario-to-mechanism map, with what has to be proven for each to count -- How to convert an availability target into a monthly budget, and what each tier - demands of your automation +- How to convert an availability target into a monthly budget, and what each tier demands of your automation - How to reconcile stated targets against the numbers measured in earlier guides - Why one storage setting can silently invalidate your stated recovery point - Why declaring your degraded modes is as important as declaring your targets ## Prerequisites -- Measured numbers from the earlier guides: per-node capacity, convergence time, - node re-entry time, release reversal time, and restore time +- Measured numbers from the earlier guides: per-node capacity, convergence time, node re-entry time, release reversal time, and restore time - [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) for the capacity model - [Backup and Recovery](./backup-and-recovery.mdx) for the recovery mechanisms -- A named owner who can approve or reject a target, because this exercise produces - decisions rather than data +- A named owner who can approve or reject a target, because this exercise produces decisions rather than data ## State the three per journey @@ -43,52 +33,25 @@ during an incident that the answer was the target all along. Three details make the difference between a testable statement and a slogan. -**Per journey, not per cluster.** A catalog browse and an order submission fail -differently, matter differently, and recover differently. One cluster-wide number -overcommits on the cheap journey and undercommits on the expensive one. +**Per journey, not per cluster.** A catalog browse and an order submission fail differently, matter differently, and recover differently. One cluster-wide number overcommits on the cheap journey and undercommits on the expensive one. -**Recovery time starts when the journey breaks, not when someone acts on it.** -Detection is inside the clock, not before it, and the gap between something -breaking and someone knowing is often the largest part of an outage. Starting the -count at the moment a human decided to act, or even at the moment an alert fired, -produces a number that flatters your automation and misleads your planning. Your -users were already down for all of it. +**Recovery time starts when the journey breaks, not when someone acts on it.** Detection is inside the clock, not before it, and the gap between something breaking and someone knowing is often the largest part of an outage. Starting the count at the moment a human decided to act, or even at the moment an alert fired, produces a number that flatters your automation and misleads your planning. Your users were already down for all of it. -**Name the measurement boundary.** Availability measured at your CDN edge and -availability measured at a Harper node are different quantities, and the -difference is exactly the part of the stack you may not control. Whichever you -choose, say so in the same sentence as the number. +**Name the measurement boundary.** Availability measured at your CDN edge and availability measured at a Harper node are different quantities, and the difference is exactly the part of the stack you may not control. Whichever you choose, say so in the same sentence as the number. ### One storage setting can override your recovery point -Your recovery point objective is a statement about how much acknowledged data you -can afford to lose. One configuration option can quietly make that statement -false. - -[`storage.writeAsync`](/reference/v5/database/storage-tuning#storagewriteasync) -defaults to `false`. Set it to `true` and Harper stops calling `fsync` on commit, -so a write returns as soon as it is queued to the operating system's page cache. -Throughput on write-heavy workloads improves substantially. The cost is that a -power loss or OS crash between the commit and the flush to disk loses the most -recent transactions, even though your application was told they succeeded. The -database stays structurally consistent; only the newest writes are gone. - -The trap is that this is invisible to everything else in this guide. Backup -timestamps look right, replication looks healthy, and no alert fires. You discover -it the first time a node loses power, which is also the first time the gap between -your stated recovery point and your real one matters. - -So treat it as an RPO decision rather than a performance one. Enabling it is -defensible when the data is reproducible from an upstream source, when the -workload is bulk ingest that can simply be re-run, or when peers acknowledge -writes before the loss window opens. Outside those cases, leave it off. If it is -on anywhere, record which databases and say so next to the recovery point in your -plan, because the number no longer means what it appears to. +Your recovery point objective is a statement about how much acknowledged data you can afford to lose. One configuration option can quietly make that statement false. + +[`storage.writeAsync`](/reference/v5/database/storage-tuning#storagewriteasync) defaults to `false`. Set it to `true` and Harper stops calling `fsync` on commit, so a write returns as soon as it is queued to the operating system's page cache. Throughput on write-heavy workloads improves substantially. The cost is that a power loss or OS crash between the commit and the flush to disk loses the most recent transactions, even though your application was told they succeeded. The database stays structurally consistent; only the newest writes are gone. + +The trap is that this is invisible to everything else in this guide. Backup timestamps look right, replication looks healthy, and no alert fires. You discover it the first time a node loses power, which is also the first time the gap between your stated recovery point and your real one matters. + +So treat it as an RPO decision rather than a performance one. Enabling it is defensible when the data is reproducible from an upstream source, when the workload is bulk ingest that can simply be re-run, or when peers acknowledge writes before the loss window opens. Outside those cases, leave it off. If it is on anywhere, record which databases and say so next to the recovery point in your plan, because the number no longer means what it appears to. ## Map scenarios to mechanisms -For each scenario, know the mechanism, and know what must be proven for the -mechanism to count. The right-hand column is the whole exercise. +For each scenario, know the mechanism, and know what must be proven for the mechanism to count. The right-hand column is the whole exercise. | Scenario | Primary mechanism | What must be proven | | ------------------------------------- | --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | @@ -105,10 +68,7 @@ mechanism to count. The right-hand column is the whole exercise. | Replication certificate expiry | Certificate lifecycle management | Expiry dates tracked, with an alert far enough ahead to act | | Downstream dependency outage | Declared degraded mode | The journey's behavior when the dependency is down, verified rather than assumed | -Two rows in that table are the ones most often missing. Node replacement is -budgeted as though it were a restart, when it is a full synchronization and can be -much slower. And the downstream dependency row usually has no answer at all, which -means the answer is whatever the code happens to do. +Two rows in that table are the ones most often missing. Node replacement is budgeted as though it were a restart, when it is a full synchronization and can be much slower. And the downstream dependency row usually has no answer at all, which means the answer is whatever the code happens to do. ## Convert availability into a budget @@ -120,23 +80,15 @@ An availability target is a quantity of unavailability you may spend per month. | 99.95% | 21 min 55 sec | Automated traffic removal and a rehearsed release reversal become necessary rather than nice | | 99.99% | 4 min 23 sec | Node failure has to be close to transparent. Detection, traffic removal, and validation must be automated, since no human response fits | -Figures assume a 30.44 day month. Your contractual definition, exclusions, and -measurement boundary may differ. +Figures assume a 30.44 day month. Your contractual definition, exclusions, and measurement boundary may differ. -Read down that right-hand column rather than the middle one. The tier you choose -determines how much automation you are committing to build, not how many nodes you -buy. Adding nodes without automating detection and traffic removal moves you -across the table's rows without moving you down its column. +Read down that right-hand column rather than the middle one. The tier you choose determines how much automation you are committing to build, not how many nodes you buy. Adding nodes without automating detection and traffic removal moves you across the table's rows without moving you down its column. -Treat the budget as spendable. If the month's remaining budget is small, that is -an argument for pausing discretionary change, and if it is large, that is -permission to ship. An error budget that never changes anyone's behavior is -just a number in a document. +Treat the budget as spendable. If the month's remaining budget is small, that is an argument for pausing discretionary change, and if it is large, that is permission to ship. An error budget that never changes anyone's behavior is just a number in a document. ## Reconcile targets against measurements -This is the step that turns the exercise into engineering. Fill in both columns -and look for rows where they disagree. +This is the step that turns the exercise into engineering. Fill in both columns and look for rows where they disagree. | Commitment | Measured from | Your target | Your measurement | | --------------------------- | ---------------------------------------------------------------------------------------- | ----------- | ---------------- | @@ -152,85 +104,50 @@ and look for rows where they disagree. Three reconciliations catch most problems: -- **Backup cadence against recovery point.** If you back up every six hours, your - recovery point cannot be one hour, no matter what the document says. -- **Restore time plus detection time against recovery time.** Recovery time - includes noticing. A twenty minute restore behind a forty minute detection gap - is a one hour recovery. -- **Convergence time against your admission gate.** If your gate admits a node - faster than it converges, you are serving stale data on purpose and calling it - availability. +- **Backup cadence against recovery point.** If you back up every six hours, your recovery point cannot be one hour, no matter what the document says. +- **Restore time plus detection time against recovery time.** Recovery time includes noticing. A twenty minute restore behind a forty minute detection gap is a one hour recovery. +- **Convergence time against your admission gate.** If your gate admits a node faster than it converges, you are serving stale data on purpose and calling it availability. -Where a target and a measurement disagree, the resolution is one of three things: -invest in the mechanism, relax the target, or accept the gap explicitly with an -owner and a date. All three are legitimate. Leaving it unreconciled is not. +Where a target and a measurement disagree, the resolution is one of three things: invest in the mechanism, relax the target, or accept the gap explicitly with an owner and a date. All three are legitimate. Leaving it unreconciled is not. ## Declare your degraded modes -Between fully working and fully down there is a range of states, and if you have -not decided what they should be, the code has decided for you. +Between fully working and fully down there is a range of states, and if you have not decided what they should be, the code has decided for you. -For each critical journey, write down what happens when a dependency is -unavailable, when a node is behind on replication, and when the cluster is below -its capacity budget. Reads served from slightly stale data may be entirely -acceptable for a catalog and entirely unacceptable for a balance check, and the -right answer differs by journey rather than by system. +For each critical journey, write down what happens when a dependency is unavailable, when a node is behind on replication, and when the cluster is below its capacity budget. Reads served from slightly stale data may be entirely acceptable for a catalog and entirely unacceptable for a balance check, and the right answer differs by journey rather than by system. -Then decide who can declare a degraded mode, and whether the declaration is -manual or automatic. A degraded mode nobody is authorized to invoke is not a -degraded mode. +Then decide who can declare a degraded mode, and whether the declaration is manual or automatic. A degraded mode nobody is authorized to invoke is not a degraded mode. ### Prove it Two passes, and both are needed. -**Tabletop the whole scenario table.** For each row, walk through who detects it, -what they do, what mechanism carries the recovery, and what evidence confirms -success. Rows where the group cannot answer without speculating are the gaps, and -finding them costs an hour rather than an outage. +**Tabletop the whole scenario table.** For each row, walk through who detects it, what they do, what mechanism carries the recovery, and what evidence confirms success. Rows where the group cannot answer without speculating are the gaps, and finding them costs an hour rather than an outage. -**Then run one real drill per scenario class**, spread over a quarter rather than -attempted in a day. The classes are: node loss, node return, node replacement, -release reversal, restore, and dependency failure. Record the date, the measured -numbers, what surprised you, and what you changed as a result. +**Then run one real drill per scenario class**, spread over a quarter rather than attempted in a day. The classes are: node loss, node return, node replacement, release reversal, restore, and dependency failure. Record the date, the measured numbers, what surprised you, and what you changed as a result. -The surprises are the deliverable. A drill that goes exactly as expected has -confirmed your documentation. A drill that does not has found the thing that -would have hurt you. +The surprises are the deliverable. A drill that goes exactly as expected has confirmed your documentation. A drill that does not has found the thing that would have hurt you. ## Operational notes -- **Recovery numbers expire.** Every measurement here is a property of a Harper - version, a data volume, a topology, and a set of components. Re-measure after - version upgrades and significant data growth, and date every number you record. -- **Uptime and recovery targets belong in the same document as the mechanism.** A - target stored separately from its supporting evidence drifts from reality within - a quarter. -- **Do not let a single incident rewrite your targets.** Adjust targets from - measurement and business need, not from the last thing that went wrong. -- **Exclusions matter as much as the number.** Planned maintenance, third-party - dependency failures, and client-side problems are usually excluded from an - availability calculation. Whether yours excludes them changes the number - substantially, so it belongs in writing. -- **A target you cannot measure is not a target.** If nothing in your monitoring - produces the number in your commitment, the first investment is measurement, not - more nodes. +- **Recovery numbers expire.** Every measurement here is a property of a Harper version, a data volume, a topology, and a set of components. Re-measure after version upgrades and significant data growth, and date every number you record. +- **Uptime and recovery targets belong in the same document as the mechanism.** A target stored separately from its supporting evidence drifts from reality within a quarter. +- **Do not let a single incident rewrite your targets.** Adjust targets from measurement and business need, not from the last thing that went wrong. +- **Exclusions matter as much as the number.** Planned maintenance, third-party dependency failures, and client-side problems are usually excluded from an availability calculation. Whether yours excludes them changes the number substantially, so it belongs in writing. +- **A target you cannot measure is not a target.** If nothing in your monitoring produces the number in your commitment, the first investment is measurement, not more nodes. ## Readiness checklist - [ ] Recovery point stated per database, in time units -- [ ] Recovery time stated per journey, measured from the start of failure and - including detection time +- [ ] Recovery time stated per journey, measured from the start of failure and including detection time - [ ] Availability stated per journey, with the measurement boundary named - [ ] Exclusions documented - [ ] Every scenario in the map has a named mechanism -- [ ] Every mechanism has its "what must be proven" satisfied or explicitly - outstanding +- [ ] Every mechanism has its "what must be proven" satisfied or explicitly outstanding - [ ] Availability target converted to minutes per month - [ ] Error budget policy states what changes when the budget runs low - [ ] Backup cadence reconciled against recovery point -- [ ] `storage.writeAsync` confirmed off, or the databases it is on for recorded - alongside their recovery point +- [ ] `storage.writeAsync` confirmed off, or the databases it is on for recorded alongside their recovery point - [ ] Restore time plus detection time reconciled against recovery time - [ ] Convergence time reconciled against the traffic admission gate - [ ] Every unreconciled gap has an owner and a date @@ -240,17 +157,11 @@ would have hurt you. ## Additional Resources -- [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) for the capacity - invariant behind the surviving-capacity commitment -- [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx) - for drain, admission, and failback timing +- [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) for the capacity invariant behind the surviving-capacity commitment +- [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx) for drain, admission, and failback timing - [Operating Replication](./operating-replication.mdx) for convergence measurement -- [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx) for release - reversal timing -- [Backup and Recovery](./backup-and-recovery.mdx) for restore timing and - mechanism limits +- [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx) for release reversal timing +- [Backup and Recovery](./backup-and-recovery.mdx) for restore timing and mechanism limits - [Monitoring and Triage](./monitoring-and-triage.mdx) for time to detection -- [Production Readiness Checklist](./production-readiness-checklist.mdx) for the - gate this feeds -- [Reliability Plan Template](./reliability-plan-template.mdx) for where to record - the results +- [Production Readiness Checklist](./production-readiness-checklist.mdx) for the gate this feeds +- [Reliability Plan Template](./reliability-plan-template.mdx) for where to record the results diff --git a/learn/administration/health-checks-and-traffic-admission.mdx b/learn/administration/health-checks-and-traffic-admission.mdx index da0339e74..a4d48196e 100644 --- a/learn/administration/health-checks-and-traffic-admission.mdx +++ b/learn/administration/health-checks-and-traffic-admission.mdx @@ -6,36 +6,24 @@ sidebar_position: 2 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -The default health check on most load balancers asks one question: did something -answer on this port? A Harper node answers that question correctly while still -being the wrong place to send a user request, because a node can be running before -its databases have synchronized, while it is catching up on missed transactions, or -while a dependency it needs for one particular journey is down. +The default health check on most load balancers asks one question: did something answer on this port? A Harper node answers that question correctly while still being the wrong place to send a user request, because a node can be running before its databases have synchronized, while it is catching up on missed transactions, or while a dependency it needs for one particular journey is down. -This guide builds four separate signals so your traffic layer can tell the -difference between "the process is alive" and "this node will serve this journey -correctly," and then uses them to drain and return a node safely. +This guide builds four separate signals so your traffic layer can tell the difference between "the process is alive" and "this node will serve this journey correctly," and then uses them to drain and return a node safely. ## What You Will Learn -- Why liveness, availability, readiness, and journey verification are four signals - rather than one, and which layer consumes each -- How to install and drive the `@harperdb/status-check` component to control - whether a node advertises itself as available -- How to write an application readiness route that checks only what the routed - journey needs +- Why liveness, availability, readiness, and journey verification are four signals rather than one, and which layer consumes each +- How to install and drive the `@harperdb/status-check` component to control whether a node advertises itself as available +- How to write an application readiness route that checks only what the routed journey needs - The order of operations for draining a node and returning it to service - What makes a readiness endpoint safe under load, and what makes one dangerous ## Prerequisites -- A Harper cluster with at least two nodes, so draining one leaves a service - behind ([Fabric](/fabric) or self-managed) +- A Harper cluster with at least two nodes, so draining one leaves a service behind ([Fabric](/fabric) or self-managed) - A `super_user` credential for the Operations API -- A traffic layer you can configure health checks on: a load balancer, a CDN - origin group, or a service mesh -- [How Harper Runs in Production](./how-harper-runs-in-production.mdx), and your - ports written down +- A traffic layer you can configure health checks on: a load balancer, a CDN origin group, or a service mesh +- [How Harper Runs in Production](./how-harper-runs-in-production.mdx), and your ports written down ## Four signals, not one @@ -46,17 +34,11 @@ correctly," and then uses them to drain and return a node safely. | Application readiness | A component route you write on `9926` | This node can serve this journey right now | Traffic layer, or your own gating | | Critical-journey synthetic | A real request through the public route | Users are actually being served | Your SLO and alerting | -The distinction that does the most work is the second one. The availability flag is -not a measurement, it is a declaration. It exists so that you, or your automation, -can take a node out of rotation deliberately, before doing something to it, and -put it back afterwards. Nothing else in this list can be set by an operator, and -nothing else is safe to use as the primary routing signal. +The distinction that does the most work is the second one. The availability flag is not a measurement, it is a declaration. It exists so that you, or your automation, can take a node out of rotation deliberately, before doing something to it, and put it back afterwards. Nothing else in this list can be set by an operator, and nothing else is safe to use as the primary routing signal. ## Install the availability flag -[`@harperdb/status-check`](https://github.com/HarperFast/status-check) is a Harper -component that adds a `/status` route on the application port. Deploy it like any -other component: +[`@harperdb/status-check`](https://github.com/HarperFast/status-check) is a Harper component that adds a `/status` route on the application port. Deploy it like any other component: ```json { @@ -67,8 +49,7 @@ other component: } ``` -Or declare it in the root `harper-config.yaml`, so the component is part of the -node's configuration rather than something an operator has to remember to deploy: +Or declare it in the root `harper-config.yaml`, so the component is part of the node's configuration rather than something an operator has to remember to deploy: ```yaml status-check: @@ -76,26 +57,16 @@ status-check: ``` :::note -That entry belongs in the root `harper-config.yaml` (in the Harper `rootPath`, -typically `~/hdb`), not in an application's own `config.yaml`. The two files look -alike but behave differently: in the root config the entry name is free-form, while -in a component's `config.yaml` it must match a `package.json` dependency. A -component `config.yaml` also **replaces** Harper's default component -configuration outright instead of merging with it, so a file containing only this -entry would switch off the `rest`, `graphqlSchema`, `jsResource`, and -`fastifyRoutes` defaults your application relies on. See -[applications](/reference/v5/components/applications). +That entry belongs in the root `harper-config.yaml` (in the Harper `rootPath`, typically `~/hdb`), not in an application's own `config.yaml`. The two files look alike but behave differently: in the root config the entry name is free-form, while in a component's `config.yaml` it must match a `package.json` dependency. A component `config.yaml` also **replaces** Harper's default component configuration outright instead of merging with it, so a file containing only this entry would switch off the `rest`, `graphqlSchema`, `jsResource`, and `fastifyRoutes` defaults your application relies on. See [applications](/reference/v5/components/applications). ::: -Once deployed, the route's contract is its status code, which is what lets a load -balancer consume it without parsing anything: +Once deployed, the route's contract is its status code, which is what lets a load balancer consume it without parsing anything: - `GET /status` returns `200` when the node is available, `404` when it is not - `POST /status` marks the node available (authenticated) - `DELETE /status` marks the node unavailable (authenticated) -It does also return a body, which is useful when you are checking by hand: a short -message on `200`, and an RFC 9457 problem-details document on `404`. +It does also return a body, which is useful when you are checking by hand: a short message on `200`, and an RFC 9457 problem-details document on `404`. @@ -128,48 +99,25 @@ await fetch(base, { method: 'POST', headers: auth }); // back in -Point your traffic layer's health check at `GET /status` on `9926`, not at the -Operations API and not at your application's root. A `404` is the node telling the -traffic layer to stop sending work, and it will keep saying so until something -sets it back. +Point your traffic layer's health check at `GET /status` on `9926`, not at the Operations API and not at your application's root. A `404` is the node telling the traffic layer to stop sending work, and it will keep saying so until something sets it back. :::note -The availability flag is node-local out of the box. The component stores it in a -table declared `replicate: false`, and the Operations API's own status values are -stored the same way, so neither propagates to peers. - -That is the property you want, and it is worth knowing why: a flag that replicated -would let one node's maintenance state reach its peers, turning a routine drain -into a cluster-wide outage. So the rule is to preserve it rather than to establish -it. If you fork the component or persist the flag some other way, keep -`replicate: false` on whatever holds it. See -[Operating Replication](./operating-replication.mdx). +The availability flag is node-local out of the box. The component stores it in a table declared `replicate: false`, and the Operations API's own status values are stored the same way, so neither propagates to peers. + +That is the property you want, and it is worth knowing why: a flag that replicated would let one node's maintenance state reach its peers, turning a routine drain into a cluster-wide outage. So the rule is to preserve it rather than to establish it. If you fork the component or persist the flag some other way, keep `replicate: false` on whatever holds it. See [Operating Replication](./operating-replication.mdx). ::: ### A note on `set_status` -The Operations API also offers -[`set_status`, `get_status`, and `clear_status`](/reference/v5/operations-api/operations#set_status--get_status--clear_status) -for application-defined status values, with types for primary, maintenance, and -availability. +The Operations API also offers [`set_status`, `get_status`, and `clear_status`](/reference/v5/operations-api/operations#set_status--get_status--clear_status) for application-defined status values, with types for primary, maintenance, and availability. -These are a coordination primitive for your own automation, not a health report, -and not a substitute for a real readiness check. Nothing in Harper acts on a value -you set through them. Prefer the `status-check` component for traffic admission, -because its contract is an HTTP status code that a load balancer can consume -directly, and reach for `set_status` when you need to coordinate something between -your own scripts. +These are a coordination primitive for your own automation, not a health report, and not a substitute for a real readiness check. Nothing in Harper acts on a value you set through them. Prefer the `status-check` component for traffic admission, because its contract is an HTTP status code that a load balancer can consume directly, and reach for `set_status` when you need to coordinate something between your own scripts. ## Write an application readiness route -Liveness and the availability flag both answer questions about the node. -Readiness answers a question about the journey: if traffic arrives for this route -right now, will it succeed? +Liveness and the availability flag both answer questions about the node. Readiness answers a question about the journey: if traffic arrives for this route right now, will it succeed? -The rule that keeps this useful is to check only the dependencies the routed -journey actually needs. A readiness route that checks everything will report a -node unready because of a subsystem that route never touches, and you will have -converted a partial degradation into a full outage yourself. +The rule that keeps this useful is to check only the dependencies the routed journey actually needs. A readiness route that checks everything will report a node unready because of a subsystem that route never touches, and you will have converted a partial degradation into a full outage yourself. Add a resource to your application's `resources.js`: @@ -211,28 +159,15 @@ export class Readiness extends Resource { } ``` -The `jsResource` plugin is enabled by default, so the route is served at -`/Readiness` on the application port as soon as the class is exported. See -[Harper Applications in Depth](../developers/harper-applications-in-depth.mdx) for -the resource and export mechanics. +The `jsResource` plugin is enabled by default, so the route is served at `/Readiness` on the application port as soon as the class is exported. See [Harper Applications in Depth](../developers/harper-applications-in-depth.mdx) for the resource and export mechanics. :::note -Because the handler is a `static` method, it replaces Harper's built-in dispatch -along with the authorization check that lives inside it, so `GET /Readiness` is -unauthenticated. That is what a load balancer probe needs, and it is why the -endpoint must not return anything you would not publish. Keep the response to -check names and outcomes, never connection strings, credentials, or internal -hostnames. - -If you rewrite this as an instance `get()` instead, authorization comes back and -defaults to `super_user` only, which will make your probe start failing with an -authorization error rather than a readiness one. In that form you need -`allowRead() { return true; }` to keep it reachable. +Because the handler is a `static` method, it replaces Harper's built-in dispatch along with the authorization check that lives inside it, so `GET /Readiness` is unauthenticated. That is what a load balancer probe needs, and it is why the endpoint must not return anything you would not publish. Keep the response to check names and outcomes, never connection strings, credentials, or internal hostnames. + +If you rewrite this as an instance `get()` instead, authorization comes back and defaults to `super_user` only, which will make your probe start failing with an authorization error rather than a readiness one. In that form you need `allowRead() { return true; }` to keep it reachable. ::: -The sentinel is a record you create once, on purpose, and leave alone. Seed it on -every node before you point anything at this route, or readiness will report -`failed` forever and you will have built a probe that never passes: +The sentinel is a record you create once, on purpose, and leave alone. Seed it on every node before you point anything at this route, or readiness will report `failed` forever and you will have built a probe that never passes: ```bash curl -s -X PUT https://my-node.example.com:9926/Product/readiness-probe-sentinel \ @@ -241,133 +176,76 @@ curl -s -X PUT https://my-node.example.com:9926/Product/readiness-probe-sentinel -d '{"name":"readiness probe sentinel","description":"do not delete"}' ``` -Give it a name that says what it is, because the next person to find it will be -deciding whether it is safe to delete. If your table replicates, seeding it once -is enough; if it does not, seed it per node. +Give it a name that says what it is, because the next person to find it will be deciding whether it is safe to delete. If your table replicates, seeding it once is enough; if it does not, seed it per node. -Two details in that example are the point of it. The timeout on the downstream -call means a slow dependency cannot make your readiness check hang, which would -make the node look dead to a probe rather than unready. And returning the version -means that when you are staring at a dashboard during a rollout, the readiness -response itself tells you which build answered. +Two details in that example are the point of it. The timeout on the downstream call means a slow dependency cannot make your readiness check hang, which would make the node look dead to a probe rather than unready. And returning the version means that when you are staring at a dashboard during a rollout, the readiness response itself tells you which build answered. ## Drain a node and bring it back -The order matters, and the verification steps between them are the parts people -skip. +The order matters, and the verification steps between them are the parts people skip. -1. **Confirm the peers can carry it, before you drain anything.** The remaining - nodes have to stay inside their capacity budget with this node gone. If they - cannot, stop here and do not drain: see - [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx). Checking this after - the traffic has already moved is how a maintenance window becomes an incident. +1. **Confirm the peers can carry it, before you drain anything.** The remaining nodes have to stay inside their capacity budget with this node gone. If they cannot, stop here and do not drain: see [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx). Checking this after the traffic has already moved is how a maintenance window becomes an incident. 2. **Declare it unavailable.** `DELETE /status` on the target node. -3. **Verify traffic actually stopped.** Watch request volume on the target fall to - zero and rise on its peers. Do not trust the configured weight; check the - measured request count from - [analytics](/reference/v5/analytics/overview) or your traffic layer's own - metrics. Health check intervals, DNS TTLs, and client-side connection reuse all - add delay here, and the delay is yours to measure. -4. **Watch the peers absorb it.** If they are going outside budget despite step 1, - abort: `POST /status` to put this node back in rotation, and re-plan the change - for a lower-traffic window or with added capacity. +3. **Verify traffic actually stopped.** Watch request volume on the target fall to zero and rise on its peers. Do not trust the configured weight; check the measured request count from [analytics](/reference/v5/analytics/overview) or your traffic layer's own metrics. Health check intervals, DNS TTLs, and client-side connection reuse all add delay here, and the delay is yours to measure. +4. **Watch the peers absorb it.** If they are going outside budget despite step 1, abort: `POST /status` to put this node back in rotation, and re-plan the change for a lower-traffic window or with added capacity. 5. **Do the work.** Restart, upgrade, reconfigure, or investigate. -1. **Confirm the databases are current.** `cluster_status` should show the expected - peer sockets connected for every database this node serves, and convergence - should be complete rather than in progress. -2. **Confirm readiness passes locally.** Call your `/Readiness` route directly - against the node, bypassing the traffic layer. -3. **Run the journey synthetic against the node directly.** A real read, or a safe - write, through the same path a user would take. +1. **Confirm the databases are current.** `cluster_status` should show the expected peer sockets connected for every database this node serves, and convergence should be complete rather than in progress. +2. **Confirm readiness passes locally.** Call your `/Readiness` route directly against the node, bypassing the traffic layer. +3. **Run the journey synthetic against the node directly.** A real read, or a safe write, through the same path a user would take. 4. **Declare it available.** `POST /status`. -5. **Hold before restoring full weight.** Give it a stability window at partial - traffic and watch error rate and latency against its peers before treating it as - fully back. +5. **Hold before restoring full weight.** Give it a stability window at partial traffic and watch error rate and latency against its peers before treating it as fully back. -Step 5 on the return side is the one worth defending in a review. Failback is a -change like any other, and a node that has just synchronized under load is the -most likely one to surprise you. Design failback, do not just design failover. +Step 5 on the return side is the one worth defending in a review. Failback is a change like any other, and a node that has just synchronized under load is the most likely one to surprise you. Design failback, do not just design failover. ## Readiness hygiene -- **Keep the response cheap and bounded.** A readiness check that performs a broad - scan or writes data will amplify an incident, because it runs at probe frequency - across every node at exactly the moment the system is already struggling. -- **Never make it the only routing signal.** Liveness plus availability plus - readiness, consumed at the right layers. -- **Probe from more than one location** where your traffic layer supports it. A - single probe point cannot distinguish a network path problem from a node problem. -- **Make it observable.** Record response code, latency, the reason for a failure, - and the node and component version. A readiness check whose failures you cannot - explain after the fact is a check you will end up ignoring. -- **Version the contract.** If you change what readiness means, that is a change - to the traffic admission policy, and it deserves the same care as a code - release. +- **Keep the response cheap and bounded.** A readiness check that performs a broad scan or writes data will amplify an incident, because it runs at probe frequency across every node at exactly the moment the system is already struggling. +- **Never make it the only routing signal.** Liveness plus availability plus readiness, consumed at the right layers. +- **Probe from more than one location** where your traffic layer supports it. A single probe point cannot distinguish a network path problem from a node problem. +- **Make it observable.** Record response code, latency, the reason for a failure, and the node and component version. A readiness check whose failures you cannot explain after the fact is a check you will end up ignoring. +- **Version the contract.** If you change what readiness means, that is a change to the traffic admission policy, and it deserves the same care as a code release. ### Prove it -On a non-production cluster, restart one node under representative read and write -load, using the full drain and return sequence above. Record three numbers: how -long from `DELETE /status` until measured traffic on that node reaches zero, -whether any user-visible errors occurred during the transition, and how long from -process start until the node legitimately passed all three return gates. +On a non-production cluster, restart one node under representative read and write load, using the full drain and return sequence above. Record three numbers: how long from `DELETE /status` until measured traffic on that node reaches zero, whether any user-visible errors occurred during the transition, and how long from process start until the node legitimately passed all three return gates. -That third number is your real node re-entry time, and it is almost always longer -than people assume, because it includes convergence rather than just startup. +That third number is your real node re-entry time, and it is almost always longer than people assume, because it includes convergence rather than just startup. ## Operational notes -- **A returning node is not the same as a new node.** A node whose databases have - never synchronized downloads them in full. A node that was offline and comes back - catches up on the transactions it missed. Both need to finish before traffic - arrives, but they take very different amounts of time, so do not budget for the - first when you are planning a routine restart. -- **Set unavailable before recovery work, not after.** Any operation that touches - data on a node, including a restore, should happen with the node out of rotation. -- **Configuration changes need a restart to take effect**, so a node that has been - reconfigured but not restarted is running the old configuration while reporting - the new one. Sequence the restart into the same maintenance window. -- **Fabric provides its own cluster-level health and routing.** These signals still - matter, because the availability flag and your readiness route are what Fabric's - routing has to consult. +- **A returning node is not the same as a new node.** A node whose databases have never synchronized downloads them in full. A node that was offline and comes back catches up on the transactions it missed. Both need to finish before traffic arrives, but they take very different amounts of time, so do not budget for the first when you are planning a routine restart. +- **Set unavailable before recovery work, not after.** Any operation that touches data on a node, including a restore, should happen with the node out of rotation. +- **Configuration changes need a restart to take effect**, so a node that has been reconfigured but not restarted is running the old configuration while reporting the new one. Sequence the restart into the same maintenance window. +- **Fabric provides its own cluster-level health and routing.** These signals still matter, because the availability flag and your readiness route are what Fabric's routing has to consult. ## Readiness checklist -- [ ] `@harperdb/status-check` deployed, declared in the root `harper-config.yaml` - rather than deployed by hand +- [ ] `@harperdb/status-check` deployed, declared in the root `harper-config.yaml` rather than deployed by hand - [ ] Traffic layer health check points at `GET /status` on `9926` -- [ ] Availability flag storage still declares `replicate: false`, if you forked the - component or persist it yourself +- [ ] Availability flag storage still declares `replicate: false`, if you forked the component or persist it yourself - [ ] An application readiness route exists, scoped to one journey's dependencies - [ ] Every downstream call in the readiness route has a timeout - [ ] Readiness response includes the component version -- [ ] A critical-journey synthetic runs against the public route, separately from - the liveness probe +- [ ] A critical-journey synthetic runs against the public route, separately from the liveness probe - [ ] Drain sequence documented, with measured time-to-zero-traffic - [ ] Return sequence documented, with a stability window before full weight - [ ] Measured node re-entry time recorded, including convergence ## Additional Resources -- [`@harperdb/status-check`](https://github.com/HarperFast/status-check) component - source and options -- [Operations API operations](/reference/v5/operations-api/operations) for - `cluster_status`, `system_information`, and the status operations -- [Components overview](/reference/v5/components/overview) for the full list of - Harper-maintained components, including the Prometheus exporter -- [Analytics overview](/reference/v5/analytics/overview) for per-node request and - latency data -- [Harper Applications in Depth](../developers/harper-applications-in-depth.mdx) - for custom resources and the `jsResource` plugin -- [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) for whether your peers - can absorb a drained node +- [`@harperdb/status-check`](https://github.com/HarperFast/status-check) component source and options +- [Operations API operations](/reference/v5/operations-api/operations) for `cluster_status`, `system_information`, and the status operations +- [Components overview](/reference/v5/components/overview) for the full list of Harper-maintained components, including the Prometheus exporter +- [Analytics overview](/reference/v5/analytics/overview) for per-node request and latency data +- [Harper Applications in Depth](../developers/harper-applications-in-depth.mdx) for custom resources and the `jsResource` plugin +- [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) for whether your peers can absorb a drained node diff --git a/learn/administration/how-harper-runs-in-production.mdx b/learn/administration/how-harper-runs-in-production.mdx index 81e6160df..f3c60cd12 100644 --- a/learn/administration/how-harper-runs-in-production.mdx +++ b/learn/administration/how-harper-runs-in-production.mdx @@ -6,40 +6,26 @@ sidebar_position: 1 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -You have built an application, deployed it from a pipeline, and now real users are -going to depend on it. Operating Harper is not the same job as operating an -application tier in front of a database tier, because Harper does not have those -two tiers. One process holds your component code, the HTTP stack, the local -database, and the replication client. That shape removes several problems you may -be used to solving, and it changes where the remaining ones live. +You have built and deployed an application, and now real users are going to depend on it. Operating Harper is not the same as operating separate application and database tiers. In Harper, one process holds your component code, the HTTP stack, the local database, and the replication client. That "collapsed stack" architecture reduces several problems you may be used to solving. -This guide is the map for the rest of the Administration track. It covers what a -single node actually contains, what that means for failure and scale, and how to -take an inventory of your own deployment before you design anything around it. +This guide is the map for the rest of the Administration track. It covers what a single node actually contains, what that means for failure and scale, and how to take an inventory of your own deployment before you design anything around it. ## What You Will Learn - What a Harper node contains, and which port carries which kind of traffic -- Why the node is your unit of failure and your unit of scale, and what that - removes from your operating burden as well as what it adds +- Why the node is your unit of failure and your unit of scale, and what that removes from your operating burden as well as what it adds - The three ways a Harper runbook differs from a two-tier runbook -- How to inventory your service boundary with the Operations API, so later guides - have something concrete to work from +- How to inventory your service boundary with the Operations API, so later guides have something concrete to work from ## Prerequisites -- A running Harper instance, either a [Harper Fabric](/fabric) cluster or a - [local installation](../getting-started/install-and-connect-harper.mdx) +- A running Harper instance, either a [Harper Fabric](/fabric) cluster or a [local installation](../getting-started/install-and-connect-harper.mdx) - A `super_user` credential for the Operations API -- An application deployed to it - ([Create your First Application](../getting-started/create-your-first-application.mdx)) +- An application deployed to it ([Create your First Application](../getting-started/create-your-first-application.mdx)) ## What one node contains -A Harper node is a single process running your components, an HTTP server, a local -storage engine, and peer replication. There is no network hop between your -application code and the data it reads, no separate cache tier to keep coherent, -and no connection pool to tune between tiers. +A Harper node is a single process running your components, an HTTP server, a local storage engine, and peer replication. There is no network hop between your application code and the data it reads, no separate cache tier to keep coherent, and no connection pool to tune between tiers. Three ports carry the traffic you will operate around: @@ -49,9 +35,7 @@ Three ports carry the traffic you will operate around: | `9925` | The [Operations API](/reference/v5/operations-api/overview) | Operators and your pipeline only | | `9933` | Secure peer [replication](/reference/v5/replication/overview) | Other nodes in the cluster only | -These are documented defaults, not guarantees about your cluster. Confirm the live -values rather than assuming them, because a replication port in particular can be -inherited from other configuration: +These are documented defaults, not guarantees about your cluster. Confirm the live values rather than assuming them, because a replication port in particular can be inherited from other configuration: ```json { @@ -59,68 +43,40 @@ inherited from other configuration: } ``` -Read back `http.port`, `operationsApi.network.port`, and the `replication` block. -Record what you find. Later guides in this track assume you know these numbers for -your own deployment. +Read back `http.port`, `operationsApi.network.port`, and the `replication` block. Record what you find. Later guides in this track assume you know these numbers for your own deployment. :::tip -Keep `9925` off any public route. The Operations API can deploy components, read -logs, and read configuration, so it is an administrative surface, not an -application one. See [security overview](/reference/v5/security/overview). +Keep `9925` off any public route. The Operations API can deploy components, read logs, and read configuration, so it is an administrative surface, not an application one. See [security overview](/reference/v5/security/overview). ::: ## The node is your unit of failure and your unit of scale -Because one process holds the runtime and the local data together, there is no -internal application-to-database seam that can fail over independently. The node -is the practical unit of service failure. +Because one process holds the runtime and the local data together, there is no internal application-to-database seam that can fail over independently. The node is the practical unit of service failure. Start with what that removes, because it is the larger half of the trade: -- No cross-tier network latency on data access, and no tail latency from a - saturated connection pool between tiers -- No cache invalidation problem between an application cache and a database of - record, because they are the same thing -- No partial-outage state where the application tier is healthy and the data tier - is not, which is the failure mode that produces the most confusing incidents +- No cross-tier network latency on data access, and no tail latency from a saturated connection pool between tiers +- No cache invalidation problem between an application cache and a database of record, because they are the same thing +- No partial-outage state where the application tier is healthy and the data tier is not, which is the failure mode that produces the most confusing incidents - One capacity number to measure and one thing to size -Then the consequence: when you lose a node, you lose a whole slice of your -service, not one layer of it. This is why the first real design decision in -[Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) is not your peak -throughput. It is how many nodes you are willing to lose at once, and whether the -survivors can carry the load. +Then the consequence: when you lose a node, you lose a whole slice of your service, not one layer of it. This is why the first real design decision in [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) is not your peak throughput. It is how many nodes you are willing to lose at once, and whether the survivors can carry the load. -Scale works on the same unit. Adding capacity means adding a node that carries -both request handling and data, so a scaling event is also a data movement event. -That is not a problem, but it is a thing with a duration, and traffic should not -arrive until it finishes. +Scale works on the same unit. Adding capacity means adding a node that carries both request handling and data, so a scaling event is also a data movement event. That is not a problem, but it is a thing with a duration, and traffic should not arrive until it finishes. ## Three differences that change your runbook ### A process that is up is not a node that should take traffic -A Harper process will answer a TCP connection and return an HTTP response before -it is a good place to send a user request. A new or replacement node has to -synchronize the databases it serves before its answers are correct. A returning -node has to catch up on transactions it missed. +A Harper process will answer a TCP connection and return an HTTP response before it is a good place to send a user request. A new or replacement node has to synchronize the databases it serves before its answers are correct. A returning node has to catch up on transactions it missed. -So liveness and admission are two different decisions, and a load balancer health -check that only proves liveness will route users to a node that is technically -running and functionally wrong. This is the whole subject of -[Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx), -and it is the single most common gap in a first production deployment. +So liveness and admission are two different decisions, and a load balancer health check that only proves liveness will route users to a node that is technically running and functionally wrong. This is the whole subject of [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx), and it is the single most common gap in a first production deployment. ### Replication is peer-to-peer and scoped, so verify your own scope -Harper peers exchange data over WebSockets with mTLS on the secure replication -port and discover each other through configured routes. There is no primary. Data -mutations and transactions replicate; some things do not, and the scope is -configurable per database and per table. +Harper peers exchange data over WebSockets with mTLS on the secure replication port and discover each other through configured routes. There is no primary. Data mutations and transactions replicate; some things do not, and the scope is configurable per database and per table. -The important operating habit is not memorizing a default. It is checking what -your cluster actually replicates, because the answer depends on your -configuration, your version, and whether anyone has scoped it since: +The important operating habit is not memorizing a default. It is checking what your cluster actually replicates, because the answer depends on your configuration, your version, and whether anyone has scoped it since: ```json { @@ -128,31 +84,17 @@ configuration, your version, and whether anyone has scoped it since: } ``` -The response lists each peer connection and, within it, one socket per database -per peer. That tells you which databases are actually flowing, which is the -question that matters during an incident. What is in scope, what is deliberately -out of it, and how to prove convergence rather than just connection are covered in -[Operating Replication](./operating-replication.mdx). +The response lists each peer connection and, within it, one socket per database per peer. That tells you which databases are actually flowing, which is the question that matters during an incident. What is in scope, what is deliberately out of it, and how to prove convergence rather than just connection are covered in [Operating Replication](./operating-replication.mdx). ### Reversal is a redeploy, not an infrastructure event -Your application ships as a component, deployed with -[`deploy_component`](/reference/v5/operations-api/operations#deploy_component) -from an immutable reference. Rolling back means deploying the previous immutable -reference. There is no image to rebuild, no instance to replace, and no cluster to -rebuild to undo a bad release. +Your application ships as a component, deployed with [`deploy_component`](/reference/v5/operations-api/operations#deploy_component) from an immutable reference. Rolling back means deploying the previous immutable reference. There is no image to rebuild, no instance to replace, and no cluster to rebuild to undo a bad release. -That makes reversal fast enough to be a real option under pressure, which in turn -makes it worth designing for deliberately rather than improvising. It also means -code rollback, configuration rollback, and data recovery are three separate -actions with three different blast radii, and conflating them during an incident -is how a bad release becomes a data loss event. See -[Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx). +That makes reversal fast enough to be a real option under pressure, which in turn makes it worth designing for deliberately rather than improvising. It also means code rollback, configuration rollback, and data recovery are three separate actions with three different blast radii, and conflating them during an incident is how a bad release becomes a data loss event. See [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx). ## Inventory your service boundary -Everything else in this track builds on knowing what you have. Run these four -operations against each node and write down the answers. +Everything else in this track builds on knowing what you have. Run these four operations against each node and write down the answers. @@ -185,22 +127,14 @@ await fetch('https://my-node.example.com:9925/', { :::warning -The `attributes` array silently drops names it does not recognize, so a typo -returns a smaller response rather than an error. The valid values are `system`, -`time`, `cpu`, `memory`, `disk`, `network`, `harperdb_processes`, `table_size`, -`metrics`, and `threads`. If a response is missing a section you asked for, check -the spelling before you check the node. +The `attributes` array silently drops names it does not recognize, so a typo returns a smaller response rather than an error. The valid values are `system`, `time`, `cpu`, `memory`, `disk`, `network`, `harperdb_processes`, `table_size`, `metrics`, and `threads`. If a response is missing a section you asked for, check the spelling before you check the node. ::: Then: -- [`get_components`](/reference/v5/components/applications#get_components) for - what is deployed, including which extensions are present -- [`list_deployments`](/reference/v5/operations-api/operations#list_deployments) - for what changed and when -- [`registration_info`](/reference/v5/operations-api/operations#registration_info) - for the Harper version, which `system_information` does not report (it returns - the node and npm versions, not Harper's) +- [`get_components`](/reference/v5/components/applications#get_components) for what is deployed, including which extensions are present +- [`list_deployments`](/reference/v5/operations-api/operations#list_deployments) for what changed and when +- [`registration_info`](/reference/v5/operations-api/operations#registration_info) for the Harper version, which `system_information` does not report (it returns the node and npm versions, not Harper's) - `cluster_status` for the peers and databases actually connected Fill in a boundary record you can keep: @@ -216,36 +150,18 @@ Fill in a boundary record you can keep: | Critical journeys | The user-facing paths that must work, named | | Downstream dependencies | Anything Harper calls that can fail independently | -The last two rows are the ones people skip and the ones that matter most. A -healthy Harper process cannot compensate for a failed downstream dependency or for -application logic returning wrong answers, so an operating model that only watches -Harper will miss the incidents your users actually notice. +The last two rows are the ones people skip and the ones that matter most. A healthy Harper process cannot compensate for a failed downstream dependency or for application logic returning wrong answers, so an operating model that only watches Harper will miss the incidents your users actually notice. ### Prove it -Before moving on, confirm the picture is real rather than assumed. On a -non-production cluster, stop one node and watch what happens to the others: -whether peers keep serving, how long the remaining nodes take to show the change -in `cluster_status`, and what your traffic layer does about it. You are not -measuring anything precisely yet. You are checking that the boundary you wrote -down matches the system you have. +Before moving on, confirm the picture is real rather than assumed. On a non-production cluster, stop one node and watch what happens to the others: whether peers keep serving, how long the remaining nodes take to show the change in `cluster_status`, and what your traffic layer does about it. You are not measuring anything precisely yet. You are checking that the boundary you wrote down matches the system you have. ## Operational notes -- **Fabric and self-managed differ in what you own, not in how Harper behaves.** - On [Fabric](/fabric), cluster creation, certificates, and the metrics pipeline - are managed for you. The failure unit, the replication model, and the admission - problem are identical. -- **Version parity across nodes is an operating requirement, not a nicety.** - Mixed versions in a cluster change replication and deployment behavior. Record - the version per node in your boundary inventory and alert on drift. -- **Configuration changes made through the API take effect on restart.** A - `set_configuration` call that has not been followed by a restart or - `restart_service` leaves a node running something other than its stated - configuration. Track pending changes in your change record. -- **`get_status` reports a `restartRequired` flag, but it tracks component and - code restarts rather than configuration changes.** Do not rely on it to tell you - a configuration change is still pending. +- **Fabric and self-managed differ in what you own, not in how Harper behaves.** On [Fabric](/fabric), cluster creation, certificates, and the metrics pipeline are managed for you. The failure unit, the replication model, and the admission problem are identical. +- **Version parity across nodes is an operating requirement, not a nicety.** Mixed versions in a cluster change replication and deployment behavior. Record the version per node in your boundary inventory and alert on drift. +- **Configuration changes made through the API take effect on restart.** A `set_configuration` call that has not been followed by a restart or `restart_service` leaves a node running something other than its stated configuration. Track pending changes in your change record. +- **`get_status` reports a `restartRequired` flag, but it tracks component and code restarts rather than configuration changes.** Do not rely on it to tell you a configuration change is still pending. ## Readiness checklist @@ -260,17 +176,10 @@ down matches the system you have. ## Additional Resources -- [HTTP server reference](/reference/v5/http/overview) for the application port and - server architecture -- [Operations API overview](/reference/v5/operations-api/overview) and the - [full operation list](/reference/v5/operations-api/operations) -- [Replication overview](/reference/v5/replication/overview) for the peer model, - mTLS, routes, and scope -- [Components overview](/reference/v5/components/overview) for the component model - and the available extensions -- [Database overview](/reference/v5/database/overview) for storage engines and - transaction boundaries -- [Configuration overview](/reference/v5/configuration/overview) for - `harper-config.yaml` and restart requirements -- [Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx) for how - application code reaches these nodes +- [HTTP server reference](/reference/v5/http/overview) for the application port and server architecture +- [Operations API overview](/reference/v5/operations-api/overview) and the [full operation list](/reference/v5/operations-api/operations) +- [Replication overview](/reference/v5/replication/overview) for the peer model, mTLS, routes, and scope +- [Components overview](/reference/v5/components/overview) for the component model and the available extensions +- [Database overview](/reference/v5/database/overview) for storage engines and transaction boundaries +- [Configuration overview](/reference/v5/configuration/overview) for `harper-config.yaml` and restart requirements +- [Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx) for how application code reaches these nodes diff --git a/learn/administration/monitoring-and-triage.mdx b/learn/administration/monitoring-and-triage.mdx index d51d8b89b..035e8356d 100644 --- a/learn/administration/monitoring-and-triage.mdx +++ b/learn/administration/monitoring-and-triage.mdx @@ -6,21 +6,14 @@ sidebar_position: 5 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -A dashboard is useful when it lets an operator place a symptom. Users are seeing -errors: is it the edge, the traffic layer, one node, the replication path, a -downstream dependency, or the change someone shipped twenty minutes ago? A -dashboard that shows CPU across the cluster cannot answer that, which is why it -gets ignored during incidents. +A dashboard is useful when it lets an operator place a symptom. Users are seeing errors: is it the edge, the traffic layer, one node, the replication path, a downstream dependency, or the change someone shipped twenty minutes ago? A dashboard that shows CPU across the cluster cannot answer that, which is why it gets ignored during incidents. -This guide builds monitoring around the boundaries you can actually act on, names -the specific Harper metrics worth alerting on, and gives you a triage sequence -short enough to run under pressure. +This guide builds monitoring around the boundaries you can actually act on, names the specific Harper metrics worth alerting on, and gives you a triage sequence short enough to run under pressure. ## What You Will Learn - The layers a Harper symptom can live in, and the minimum signal for each -- Which Harper metrics are worth an alert, by name, and which are only worth a - dashboard +- Which Harper metrics are worth an alert, by name, and which are only worth a dashboard - How to get metrics out of Harper, on Fabric and self-managed - How to preload an APM or tracing agent so it can instrument your application - How to write log entries that are still useful during an incident @@ -30,10 +23,8 @@ short enough to run under pressure. - A cluster with your application deployed and taking traffic - A `super_user` credential for the Operations API -- Somewhere to send metrics: [Grafana](/fabric/grafana-integration) on Fabric, or - a Prometheus-compatible system for self-managed -- [Operating Replication](./operating-replication.mdx), since replication signals - are half of what you will watch +- Somewhere to send metrics: [Grafana](/fabric/grafana-integration) on Fabric, or a Prometheus-compatible system for self-managed +- [Operating Replication](./operating-replication.mdx), since replication signals are half of what you will watch ## Monitor at the boundaries you operate @@ -48,15 +39,11 @@ short enough to run under pressure. | Change | Component and config version, cohort, operator, start and end, outcome | `list_deployments`, `get_deployment`, `get_components`, `get_configuration` read-back | | Recovery | Backup age, job state, verification result, last restore drill | `list_backups`, `verify_backup`, `get_job` | -The Change row is the one teams leave out and the one that resolves incidents -fastest. Most production symptoms correlate with something a human did, so being -able to overlay deployments onto a latency graph is worth more than another -resource metric. +The Change row is the one teams leave out and the one that resolves incidents fastest. Most production symptoms correlate with something a human did, so being able to overlay deployments onto a latency graph is worth more than another resource metric. ## The signals worth an alert -Harper records a large standard metric set automatically. Most of it belongs on a -dashboard. This much belongs on a pager: +Harper records a large standard metric set automatically. Most of it belongs on a dashboard. This much belongs on a pager: | Alert on | Metric or source | Why this one | | -------------------------------------- | ------------------------------------------------------------------ | ---------------------------------------------------------------------------------------- | @@ -71,32 +58,22 @@ dashboard. This much belongs on a pager: | Missing peer | `cluster_status` | Covered in [Operating Replication](./operating-replication.mdx) | | Backup age exceeding RPO | `list_backups` | The failure you will not notice until you need it | -Two notes on how to set these. Alert on the percentile that appears in your SLO, -because a mean latency graph will look fine through an incident that is failing -your slowest ten percent of users. And set saturation thresholds from the -per-node capacity work in -[Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) rather than from a -generic number, since the point where latency leaves your SLO is specific to your -application and your data. +Two notes on how to set these. Alert on the percentile that appears in your SLO, because a mean latency graph will look fine through an incident that is failing your slowest ten percent of users. And set saturation thresholds from the per-node capacity work in [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) rather than from a generic number, since the point where latency leaves your SLO is specific to your application and your data. ## Get the metrics out -Harper stores analytics locally in `hdb_raw_analytics` and aggregates them into -`hdb_analytics`. You can query those directly, but for a real monitoring setup +Harper stores analytics locally in `hdb_raw_analytics` and aggregates them into `hdb_analytics`. You can query those directly, but for a real monitoring setup export them. -Use the [Grafana integration](/fabric/grafana-integration). It ships dashboards -over Harper's analytics without you building a pipeline, which is the fastest path -to the alert list above. +Use the [Grafana integration](/fabric/grafana-integration). It ships dashboards over Harper's analytics without you building a pipeline, which is the fastest path to the alert list above. -Deploy [`@harperdb/prometheus-exporter`](https://github.com/HarperFast/prometheus-exporter) -and scrape it into whatever you already run: +Deploy [`@harperdb/prometheus-exporter`](https://github.com/HarperFast/prometheus-exporter) and scrape it into whatever you already run: ```json { @@ -107,31 +84,18 @@ and scrape it into whatever you already run: } ``` -Once deployed, the scrape endpoint is `//metrics` on the application -port, so the call above exposes it at `/prometheus-exporter/metrics`. The route is -authorized, so give your scraper a credential rather than expecting it to be open. +Once deployed, the scrape endpoint is `//metrics` on the application port, so the call above exposes it at `/prometheus-exporter/metrics`. The route is authorized, so give your scraper a credential rather than expecting it to be open. :::warning -**The exporter does not cover every metric in the alert table above.** It -translates the request and replication metrics, including `success`, `duration`, -and `replication-latency`, but it does not currently export -`main-thread-utilization` and its `taskQueueLatency` attribute, -`transaction-commit-time`, `write-transaction-queue-depth`, `database-size`, -`storage-volume`, or `table-size`. The `utilization` it does expose is a thread -utilization figure drawn from `system_information`, which is not the same as the -analytics `utilization` metric. - -For the saturation, commit-time, and storage rows you will need to query -`hdb_analytics` directly or collect `system_information` on your own schedule. -Check what your version actually exports before you build a dashboard on the -assumption that everything above arrives in Prometheus. +**The exporter does not cover every metric in the alert table above.** It translates the request and replication metrics, including `success`, `duration`, and `replication-latency`, but it does not currently export `main-thread-utilization` and its `taskQueueLatency` attribute, `transaction-commit-time`, `write-transaction-queue-depth`, `database-size`, `storage-volume`, or `table-size`. The `utilization` it does expose is a thread utilization figure drawn from `system_information`, which is not the same as the analytics `utilization` metric. + +For the saturation, commit-time, and storage rows you will need to query `hdb_analytics` directly or collect `system_information` on your own schedule. Check what your version actually exports before you build a dashboard on the assumption that everything above arrives in Prometheus. ::: -To find out what is actually available on your version rather than guessing from -documentation: +To find out what is actually available on your version rather than guessing from documentation: ```json { @@ -140,33 +104,19 @@ documentation: } ``` -Then [`describe_metric`](/reference/v5/analytics/operations#describe_metric) for -the shape of any one of them. +Then [`describe_metric`](/reference/v5/analytics/operations#describe_metric) for the shape of any one of them. -Your application can add its own metrics with -[`server.recordAnalytics()`](/reference/v5/http/api#serverrecordanalyticsvalue-metric-path-method-type), -which is how you get business-level signals such as checkout completion onto the -same dashboard as node saturation. That correlation is usually what tells you -whether a technical symptom matters. +Your application can add its own metrics with [`server.recordAnalytics()`](/reference/v5/http/api#serverrecordanalyticsvalue-metric-path-method-type), which is how you get business-level signals such as checkout completion onto the same dashboard as node saturation. That correlation is usually what tells you whether a technical symptom matters. :::tip -Check `analytics.replicate` in the -[analytics configuration](/reference/v5/analytics/overview#analytics-configuration) -before building dashboards. Whether metrics stay node-local or replicate changes -what a cluster-wide query means, and it is easier to decide that deliberately than -to discover it after building panels on the wrong assumption. +Check `analytics.replicate` in the [analytics configuration](/reference/v5/analytics/overview#analytics-configuration) before building dashboards. Whether metrics stay node-local or replicate changes what a cluster-wide query means, and it is easier to decide that deliberately than to discover it after building panels on the wrong assumption. ::: ## Run an APM or tracing agent -Harper's own metrics tell you how the node is behaving. They do not give you -distributed traces across your application code and its downstream calls, which is -what you want when a journey is slow and you need to know where the time went. -That comes from an instrumentation agent, and an agent has to load before the code -it instruments. +Harper's own metrics tell you how the node is behaving. They do not give you distributed traces across your application code and its downstream calls, which is what you want when a journey is slow and you need to know where the time went. That comes from an instrumentation agent, and an agent has to load before the code it instruments. -Two configuration keys put a module on each worker thread's startup, ahead of -Harper's own modules and yours: +Two configuration keys put a module on each worker thread's startup, ahead of Harper's own modules and yours: ```yaml threads: @@ -174,34 +124,17 @@ threads: preload: dd-trace/register.js # ESM loader hooks for automatic instrumentation ``` -`threads.preload` (Added in: v5.2.0) loads a module via Node's `--import`, which is -how an agent installs the loader hooks that let it instrument modules imported -later. `threads.preloadRequire` (Added in: v5.2.0) uses `--require`, which runs the -module body and is typically how an agent's initialization entry actually starts -it. Both are documented under -[threads configuration](/reference/v5/configuration/options#threads). - -Installing loader hooks and starting the agent are two different jobs, and which -of your agent's entry points does which is specific to that agent. Some ship one -entry that does both; others split them, in which case set both keys. The example -above is the split-entry case, and it is the one that produces the most confusing -failure: with only the hooks loaded, a tracer will hand out spans with plausible -trace ids that are no-ops and export nothing, so your instrumentation looks -installed and your collector stays empty. - -Two constraints worth knowing before you plan around this. Both keys apply to -worker threads only, and neither works under Bun. And bare specifiers resolve -against the `node_modules` of your installed components, so an agent can ship as a -dependency of a deployed component rather than as a host-level install. - -Whatever agent you use, follow its own worker-thread documentation, and finish by -confirming spans actually arrive at your collector rather than assuming the -configuration took. +`threads.preload` (Added in: v5.2.0) loads a module via Node's `--import`, which is how an agent installs the loader hooks that let it instrument modules imported later. `threads.preloadRequire` (Added in: v5.2.0) uses `--require`, which runs the module body and is typically how an agent's initialization entry actually starts it. Both are documented under [threads configuration](/reference/v5/configuration/options#threads). + +Installing loader hooks and starting the agent are two different jobs, and which of your agent's entry points does which is specific to that agent. Some ship one entry that does both; others split them, in which case set both keys. The example above is the split-entry case, and it is the one that produces the most confusing failure: with only the hooks loaded, a tracer will hand out spans with plausible trace ids that are no-ops and export nothing, so your instrumentation looks installed and your collector stays empty. + +Two constraints worth knowing before you plan around this. Both keys apply to worker threads only, and neither works under Bun. And bare specifiers resolve against the `node_modules` of your installed components, so an agent can ship as a dependency of a deployed component rather than as a host-level install. + +Whatever agent you use, follow its own worker-thread documentation, and finish by confirming spans actually arrive at your collector rather than assuming the configuration took. ## Logs you can use during an incident -Harper's `logger` global takes a message plus a context object from component -code: +Harper's `logger` global takes a message plus a context object from component code: ```javascript logger.info('order submitted', { @@ -215,49 +148,27 @@ logger.info('order submitted', { }); ``` -Include the component and its version, the node, a request or trace identifier, -the operation, its duration, the outcome, and a safe error class. That field set -is what lets you answer "was this only the new version" and "was this only one -node" without guessing, and those are the first two questions in almost every -incident. +Include the component and its version, the node, a request or trace identifier, the operation, its duration, the outcome, and a safe error class. That field set is what lets you answer "was this only the new version" and "was this only one node" without guessing, and those are the first two questions in almost every incident. Three details in that example need care: -- **The node name comes from `server.hostname`**, not an environment variable. - Harper does not set a node-name variable in the process environment, and - `server.hostname` is the same identity analytics uses, so it is what correlates - with your metrics. -- **`context.requestId` is only populated when `http.logging.id` is enabled**, and - HTTP request logging is off by default. Without it the field is `undefined` and - you silently lose your correlation id. Either enable it in - [logging configuration](/reference/v5/logging/configuration) or generate an id - in your own code. -- **`APP_VERSION` is yours to inject.** Harper does not provide it. Set it in your - deployment so the log line can name the build. +- **The node name comes from `server.hostname`**, not an environment variable. Harper does not set a node-name variable in the process environment, and `server.hostname` is the same identity analytics uses, so it is what correlates with your metrics. +- **`context.requestId` is only populated when `http.logging.id` is enabled**, and HTTP request logging is off by default. Without it the field is `undefined` and you silently lose your correlation id. Either enable it in [logging configuration](/reference/v5/logging/configuration) or generate an id in your own code. +- **`APP_VERSION` is yours to inject.** Harper does not provide it. Set it in your deployment so the log line can name the build. :::warning -Harper's logger is built on Node's `Console`, so this renders as a formatted text -line, not JSON. The context object is inspected into the message rather than -emitted as separate fields: +Harper's logger is built on Node's `Console`, so this renders as a formatted text line, not JSON. The context object is inspected into the message rather than emitted as separate fields: ```text [main/3] [info]: order submitted { component: 'orders-api', version: '2.4.1', ... } ``` -That is fine for reading during an incident, but a log pipeline cannot key on -`component` or `outcome` without parsing the line. If you need queryable fields, -serialize the context yourself and log a single JSON string. +That is fine for reading during an incident, but a log pipeline cannot key on `component` or `outcome` without parsing the line. If you need queryable fields, serialize the context yourself and log a single JSON string. ::: -Levels run `trace`, `debug`, `info`, `warn`, `error`, `fatal`, and `notify`. The -default is `warn`, and `notify` is always logged regardless of level. Choose -levels deliberately: a production log at `debug` is a log nobody can read, and one -at `error` only has the incidents in it and none of the context. +Levels run `trace`, `debug`, `info`, `warn`, `error`, `fatal`, and `notify`. The default is `warn`, and `notify` is always logged regardless of level. Choose levels deliberately: a production log at `debug` is a log nobody can read, and one at `error` only has the incidents in it and none of the context. -`console.log` output does not reach the log files unless `logging.console` is -enabled, so unstructured console output is not a production record. Centralize -logs off the node, because the node you most need logs from is the one you are -about to restart. +`console.log` output does not reach the log files unless `logging.console` is enabled, so unstructured console output is not a production record. Centralize logs off the node, because the node you most need logs from is the one you are about to restart. Read logs through the API when you need them from a specific node: @@ -273,21 +184,13 @@ Read logs through the API when you need them from a specific node: Run these in order. The goal is not diagnosis, it is narrowing. -1. **Confirm and bound the impact.** What journey, starting exactly when, in which - geography or cohort, and which component version is live. Without a start time - you cannot correlate anything. +1. **Confirm and bound the impact.** What journey, starting exactly when, in which geography or cohort, and which component version is live. Without a start time you cannot correlate anything. -2. **Compare the public route against a direct node call.** Request the same - journey through your traffic layer, then directly against each node on `9926`. - If direct calls succeed and the public route fails, you are looking at the - traffic layer or the availability flag, not at Harper. +2. **Compare the public route against a direct node call.** Request the same journey through your traffic layer, then directly against each node on `9926`. If direct calls succeed and the public route fails, you are looking at the traffic layer or the availability flag, not at Harper. -3. **Check admission state on every node.** `GET /status` on `9926` and your - readiness route. A node advertising unavailable is a node deliberately or - accidentally out of rotation, and finding that here saves a lot of time. +3. **Check admission state on every node.** `GET /status` on `9926` and your readiness route. A node advertising unavailable is a node deliberately or accidentally out of rotation, and finding that here saves a lot of time. -4. **Compare a suspect node against a healthy peer.** Same call, both nodes, then - diff: +4. **Compare a suspect node against a healthy peer.** Same call, both nodes, then diff: ```json { @@ -296,73 +199,41 @@ Run these in order. The goal is not diagnosis, it is narrowing. } ``` - Differences between peers are more informative than absolute values, because - they tell you whether this is one node or the whole cluster. + Differences between peers are more informative than absolute values, because they tell you whether this is one node or the whole cluster. -5. **Check the replication path.** `cluster_status` for the databases this journey - needs. Look for a missing peer, `connected: false`, or a widening gap between - `lastReceivedRemoteTime` and `lastReceivedLocalTime`, which means stale reads. +5. **Check the replication path.** `cluster_status` for the databases this journey needs. Look for a missing peer, `connected: false`, or a widening gap between `lastReceivedRemoteTime` and `lastReceivedLocalTime`, which means stale reads. -6. **Correlate with the last change.** `list_deployments` for what shipped and - when, and `get_configuration` read back against what you believe is configured. - Remember that a configuration change applied without a restart leaves a node - running something other than its stated configuration. +6. **Correlate with the last change.** `list_deployments` for what shipped and when, and `get_configuration` read back against what you believe is configured. Remember that a configuration change applied without a restart leaves a node running something other than its stated configuration. -7. **Contain with the smallest reversible action.** Stop a ramp, drain one node, - deactivate a feature, restore prior traffic weights, or isolate suspected data. - Record the decision, who made it, and the next decision deadline. +7. **Contain with the smallest reversible action.** Stop a ramp, drain one node, deactivate a feature, restore prior traffic weights, or isolate suspected data. Record the decision, who made it, and the next decision deadline. -Two rules make this sequence work. Silence is a failed gate: if telemetry is -missing for the thing you are checking, treat that as a negative signal rather -than skipping the step. And containment comes before root cause. You can diagnose -after users are being served again. +Two rules make this sequence work. Silence is a failed gate: if telemetry is missing for the thing you are checking, treat that as a negative signal rather than skipping the step. And containment comes before root cause. You can diagnose after users are being served again. ### Prove it -Pick a fault and inject it on a non-production cluster, then time yourself. Good -candidates: saturate one node's workers, block the replication port on one peer, -make a downstream dependency return errors, or deploy a component that fails on -one route. +Pick a fault and inject it on a non-production cluster, then time yourself. Good candidates: saturate one node's workers, block the replication port on one peer, make a downstream dependency return errors, or deploy a component that fails on one route. -Measure three things. How long until an alert fired. How long until an operator -following the sequence above could name the layer. And whether any step gave a -misleading answer, which is the most valuable output of the drill, because a -misleading signal during a real incident costs more than a missing one. +Measure three things. How long until an alert fired. How long until an operator following the sequence above could name the layer. And whether any step gave a misleading answer, which is the most valuable output of the drill, because a misleading signal during a real incident costs more than a missing one. ## Operational notes -- **Node-level dashboards, not cluster averages.** A cluster average conceals the - single node doing twice the work, which is the most common cause of a latency - complaint that looks like nothing on a dashboard. -- **Watch measured request distribution, not configured weights.** Per-node - request counts are the ground truth. Sticky sessions, DNS caching, and - connection reuse all skew actual distribution away from intent. -- **Annotate deployments onto your graphs.** If your monitoring supports - annotations, feed `list_deployments` into them. This single change resolves more - incidents faster than any additional metric. -- **`read_audit_log` needs transaction logging enabled** and is a heavier tool for - reconstructing what changed in a table. Know before an incident whether you have - it on, because turning it on afterwards does not help. -- **Protect your telemetry surfaces.** `read_log` and `get_components` can expose - configuration and source detail, so they are `super_user` operations for a - reason. Restrict and log their use. +- **Node-level dashboards, not cluster averages.** A cluster average conceals the single node doing twice the work, which is the most common cause of a latency complaint that looks like nothing on a dashboard. +- **Watch measured request distribution, not configured weights.** Per-node request counts are the ground truth. Sticky sessions, DNS caching, and connection reuse all skew actual distribution away from intent. +- **Annotate deployments onto your graphs.** If your monitoring supports annotations, feed `list_deployments` into them. This single change resolves more incidents faster than any additional metric. +- **`read_audit_log` needs transaction logging enabled** and is a heavier tool for reconstructing what changed in a table. Know before an incident whether you have it on, because turning it on afterwards does not help. +- **Protect your telemetry surfaces.** `read_log` and `get_components` can expose configuration and source detail, so they are `super_user` operations for a reason. Restrict and log their use. ## Readiness checklist -- [ ] A dashboard exists that can place a symptom at edge, traffic layer, node, - replication, data, or change +- [ ] A dashboard exists that can place a symptom at edge, traffic layer, node, replication, data, or change - [ ] Alerts on journey success and latency at the SLO percentile, not the mean -- [ ] Alerts on worker `utilization` and `taskQueueLatency`, thresholds derived - from measured per-node capacity +- [ ] Alerts on worker `utilization` and `taskQueueLatency`, thresholds derived from measured per-node capacity - [ ] Alerts on replication convergence lag and missing peers - [ ] Alerts on storage headroom and backup age -- [ ] Metrics exported off the node, via Grafana on Fabric or the Prometheus - exporter self-managed -- [ ] If you run an APM agent, both `threads.preload` and `threads.preloadRequire` - set as that agent requires, with spans confirmed at the collector +- [ ] Metrics exported off the node, via Grafana on Fabric or the Prometheus exporter self-managed +- [ ] If you run an APM agent, both `threads.preload` and `threads.preloadRequire` set as that agent requires, with spans confirmed at the collector - [ ] `analytics.replicate` setting known and deliberate -- [ ] Structured logging includes component, version, node, request id, operation, - duration, and outcome +- [ ] Structured logging includes component, version, node, request id, operation, duration, and outcome - [ ] Logs centralized off the node - [ ] Deployments annotated onto dashboards - [ ] Triage sequence written down where on-call can find it @@ -370,17 +241,11 @@ misleading signal during a real incident costs more than a missing one. ## Additional Resources -- [Analytics overview](/reference/v5/analytics/overview) for the full standard - metric catalog and configuration options -- [Analytics operations](/reference/v5/analytics/operations) for `list_metrics` - and `describe_metric` +- [Analytics overview](/reference/v5/analytics/overview) for the full standard metric catalog and configuration options +- [Analytics operations](/reference/v5/analytics/operations) for `list_metrics` and `describe_metric` - [Grafana integration](/fabric/grafana-integration) for Fabric dashboards -- [`@harperdb/prometheus-exporter`](https://github.com/HarperFast/prometheus-exporter) - for self-managed metric scraping -- [Logging overview](/reference/v5/logging/overview), - [configuration](/reference/v5/logging/configuration), and - [operations](/reference/v5/logging/operations) +- [`@harperdb/prometheus-exporter`](https://github.com/HarperFast/prometheus-exporter) for self-managed metric scraping +- [Logging overview](/reference/v5/logging/overview), [configuration](/reference/v5/logging/configuration), and [operations](/reference/v5/logging/operations) - [HTTP API reference](/reference/v5/http/api) for `server.recordAnalytics()` - [Fabric logging](/fabric/logging) for log access on Fabric -- [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx) for the - gates these signals feed +- [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx) for the gates these signals feed diff --git a/learn/administration/operating-replication.mdx b/learn/administration/operating-replication.mdx index 93baa86c7..c18964755 100644 --- a/learn/administration/operating-replication.mdx +++ b/learn/administration/operating-replication.mdx @@ -6,45 +6,28 @@ sidebar_position: 4 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -Replication is how a Harper cluster stays available: peers exchange data directly -over WebSockets, there is no primary, and a node that falls behind catches up on -its own. The [replication reference](/reference/v5/replication/overview) covers -how to configure that. This guide covers the part that only matters once real -users depend on it, which is how to tell whether replication is actually working. +Replication is how a Harper cluster stays available: peers exchange data directly over WebSockets, there is no primary, and a node that falls behind catches up on its own. The [replication reference](/reference/v5/replication/overview) covers how to configure that. This guide covers the part that only matters once real users depend on it, which is how to tell whether replication is actually working. -The distinction that runs through this guide is between a connection and -convergence. A connected socket proves two nodes can talk. It does not prove the -data a user is about to read is current. Those are different claims, and only one -of them is what your traffic admission decision depends on. +The distinction that runs through this guide is between a connection and convergence. A connected socket proves two nodes can talk. It does not prove the data a user is about to read is current. Those are different claims, and only one of them is what your traffic admission decision depends on. ## What You Will Learn -- How to determine what your cluster actually replicates, rather than assuming a - default -- What moves between peers automatically, what does not, and which of those will - surprise you -- How to read `cluster_status` as an operator, including which timing field - actually indicates a node is behind -- How to prove convergence with a sentinel rather than inferring it from - connection state -- Which application behaviors replication cannot make safe, and what to do about - them instead +- How to determine what your cluster actually replicates, rather than assuming a default +- What moves between peers automatically, what does not, and which of those will surprise you +- How to read `cluster_status` as an operator, including which timing field actually indicates a node is behind +- How to prove convergence with a sentinel rather than inferring it from connection state +- Which application behaviors replication cannot make safe, and what to do about them instead ## Prerequisites -- A cluster of at least three nodes, so you can interrupt one peer and still - observe the others +- A cluster of at least three nodes, so you can interrupt one peer and still observe the others - A `super_user` credential for the Operations API -- [How Harper Runs in Production](./how-harper-runs-in-production.mdx) and your - service boundary inventory -- Familiarity with how your cluster was joined, either through - `harper-config.yaml` routes or the - [clustering operations](/reference/v5/replication/clustering) +- [How Harper Runs in Production](./how-harper-runs-in-production.mdx) and your service boundary inventory +- Familiarity with how your cluster was joined, either through `harper-config.yaml` routes or the [clustering operations](/reference/v5/replication/clustering) ## Know your own scope -By default Harper replicates all data in all databases. Scope can be narrowed two -ways: per database in configuration, and per table in the schema. +By default Harper replicates all data in all databases. Scope can be narrowed two ways: per database in configuration, and per table in the schema. ```yaml replication: @@ -60,10 +43,7 @@ type LocalTableForNode @table(replicate: false) { } ``` -All tables in a replicated database replicate unless the table opts out. So the -scope you are operating is the product of a config list, a set of schema -directives, and any directional routes someone added later. Do not reconstruct it -from memory. Read it off the running cluster: +All tables in a replicated database replicate unless the table opts out. So the scope you are operating is the product of a config list, a set of schema directives, and any directional routes someone added later. Do not reconstruct it from memory. Read it off the running cluster: @@ -92,23 +72,10 @@ await fetch('https://my-node.example.com:9925/', { -There is one socket per database per peer, so the sockets present in the response -are the ground truth about what is flowing. Run this from every node, not one. -Replication direction can be constrained per route, so node A's view of the -cluster is not necessarily node B's view, and a one-sided picture is how -directional configuration mistakes survive into production. +There is one socket per database per peer, so the sockets present in the response are the ground truth about what is flowing. Run this from every node, not one. Replication direction can be constrained per route, so node A's view of the cluster is not necessarily node B's view, and a one-sided picture is how directional configuration mistakes survive into production. :::warning -Whether the `system` database is in your replication scope is the highest-stakes -scoping question in a Harper cluster, because `system` holds `hdb_user`, -`hdb_role`, and `hdb_nodes`. **It is in scope by default**, since the default -replication scope is every database, so unless someone has narrowed it your users -and roles already propagate and every node that receives `system` must be trusted -with its contents, including encrypted secret rows. That is usually what you want, -but it should be a decision rather than a surprise. Confirm your own answer from -`cluster_status` and your configuration, and read -[replicating the system database with controlled flow](/reference/v5/replication/overview#replicating-the-system-database-with-controlled-flow) -before changing it. +Whether the `system` database is in your replication scope is the highest-stakes scoping question in a Harper cluster, because `system` holds `hdb_user`, `hdb_role`, and `hdb_nodes`. **It is in scope by default**, since the default replication scope is every database, so unless someone has narrowed it your users and roles already propagate and every node that receives `system` must be trusted with its contents, including encrypted secret rows. That is usually what you want, but it should be a decision rather than a surprise. Confirm your own answer from `cluster_status` and your configuration, and read [replicating the system database with controlled flow](/reference/v5/replication/overview#replicating-the-system-database-with-controlled-flow) before changing it. ::: ## What moves automatically, and what does not @@ -125,43 +92,18 @@ before changing it. | Node registry (`hdb_nodes`) | Each node rewrites its own self-record from its `harper-config.yaml` routes on restart or component reload | Scoping applied through `add_node` alone does not survive a restart. Put durable constraints in config routes | | Destructive schema changes | `drop_database` and `drop_table` replicate by default; `drop_attribute` does not | A drop is a cluster-wide event unless you pass `"replicated": false`. Verify the result on every node | -The three rows most likely to bite you are the returning node, the new node, and -the node registry. - -On the returning node: it is common to see a routine restart budgeted as though it -were a full resynchronization, which makes teams avoid restarts they should be -comfortable with. A node whose databases have never synced does download them in -full. A node that was briefly offline catches up on what it missed. Measure both -on your own data volume once, and use the right number for the right situation. - -On the new node, the cost is larger than "one copy from one peer," because the -full-copy decision is made per peer and per database rather than once against a -bootstrap source. A joining node requests a full copy from every peer it has no -resume cursor for. And because replication is bidirectional, each established peer -independently decides it has no cursor for the newcomer and requests a full copy -_from_ it as well. Joining a cluster of N peers is therefore N inbound transfers -plus N outbound transfers of the new node's own (empty) databases, not a single -stream from one source. - -The practical consequences are worth planning around. Load lands on every existing -peer at once rather than on one, so a join during peak traffic is a capacity event -for the whole cluster. If you want a single designated source instead, `add_node` -accepts `isLeader: true`, which tells the joining node to request its full copy -from that peer alone. - -On the registry: a node's advertised record is derived from its configuration -file, and it replicates. That means a topology constraint you applied -imperatively is superseded the next time that node restarts or reloads -components, and the node quietly goes back to advertising itself more broadly -than you intended. +The three rows most likely to bite you are the returning node, the new node, and the node registry. + +On the returning node: it is common to see a routine restart budgeted as though it were a full resynchronization, which makes teams avoid restarts they should be comfortable with. A node whose databases have never synced does download them in full. A node that was briefly offline catches up on what it missed. Measure both on your own data volume once, and use the right number for the right situation. + +On the new node, the cost is larger than "one copy from one peer," because the full-copy decision is made per peer and per database rather than once against a bootstrap source. A joining node requests a full copy from every peer it has no resume cursor for. And because replication is bidirectional, each established peer independently decides it has no cursor for the newcomer and requests a full copy _from_ it as well. Joining a cluster of N peers is therefore N inbound transfers plus N outbound transfers of the new node's own (empty) databases, not a single stream from one source. + +The practical consequences are worth planning around. Load lands on every existing peer at once rather than on one, so a join during peak traffic is a capacity event for the whole cluster. If you want a single designated source instead, `add_node` accepts `isLeader: true`, which tells the joining node to request its full copy from that peer alone. + +On the registry: a node's advertised record is derived from its configuration file, and it replicates. That means a topology constraint you applied imperatively is superseded the next time that node restarts or reloads components, and the node quietly goes back to advertising itself more broadly than you intended. :::danger -When replicating configuration, only send cluster-appropriate parameters. -Replicating a node-local value such as a port, `node.hostname`, a file path, TLS -material, or `replication.hostname`, `url`, or `routes` overwrites every peer's -own local value. To apply a cluster-wide change safely, use -`set_configuration` with `"replicated": true` for the parameter, then -`restart_service` with `"replicated": true`, which restarts nodes one at a time. +When replicating configuration, only send cluster-appropriate parameters. Replicating a node-local value such as a port, `node.hostname`, a file path, TLS material, or `replication.hostname`, `url`, or `routes` overwrites every peer's own local value. To apply a cluster-wide change safely, use `set_configuration` with `"replicated": true` for the parameter, then `restart_service` with `"replicated": true`, which restarts nodes one at a time. ::: ## Read `cluster_status` like an operator @@ -194,61 +136,34 @@ A trimmed response, with the fields that matter: What each field is telling you: -- **`connected`** is the liveness of this one database's socket to this one peer. - A missing peer, or a peer present with `connected: false`, is actionable before - users notice. -- **`latency`** is the round trip to that peer in milliseconds. Alert on sustained - growth rather than on a single sample. -- **`lastCommitConfirmed`** is the last time this peer acknowledged receiving one - of your commits. If it stops advancing while you are still writing, your writes - are not landing on that peer. -- **`lastReceivedRemoteTime`** is the source node's timestamp on the newest - transaction you have received. +- **`connected`** is the liveness of this one database's socket to this one peer. A missing peer, or a peer present with `connected: false`, is actionable before users notice. +- **`latency`** is the round trip to that peer in milliseconds. Alert on sustained growth rather than on a single sample. +- **`lastCommitConfirmed`** is the last time this peer acknowledged receiving one of your commits. If it stops advancing while you are still writing, your writes are not landing on that peer. +- **`lastReceivedRemoteTime`** is the source node's timestamp on the newest transaction you have received. - **`lastReceivedLocalTime`** is your own clock when you received it. -The last two are the pair that matters. **A widening gap between -`lastReceivedRemoteTime` and `lastReceivedLocalTime` means this node is behind and -working through a backlog.** That is the signal to alert on for convergence, and -it is the one that tells you a returning node is not ready for traffic yet. -`sendingMessage` appears while a transaction is actively being sent and is absent -when the socket is idle, so its absence is not a fault. - -Two things will mislead you if you automate on that pair without knowing them. -The gap only means "behind" while transactions are actually arriving: on an idle -source both stamps freeze, and the gap sits at whatever constant it last reached -rather than signalling lag. And because the two stamps come from two different -clocks, skew between peers is added to the gap directly, so calibrate your -threshold against a healthy baseline rather than treating the raw number as -replication delay. +The last two are the pair that matters. **A widening gap between `lastReceivedRemoteTime` and `lastReceivedLocalTime` means this node is behind and working through a backlog.** That is the signal to alert on for convergence, and it is the one that tells you a returning node is not ready for traffic yet. `sendingMessage` appears while a transaction is actively being sent and is absent when the socket is idle, so its absence is not a fault. + +Two things will mislead you if you automate on that pair without knowing them. The gap only means "behind" while transactions are actually arriving: on an idle source both stamps freeze, and the gap sits at whatever constant it last reached rather than signalling lag. And because the two stamps come from two different clocks, skew between peers is added to the gap directly, so calibrate your threshold against a healthy baseline rather than treating the raw number as replication delay. :::tip -While a database is taking a full copy, these timing fields render as the literal -string `"Copying"` rather than a date. Anything that parses them as timestamps -will fail on a joining node, which is exactly the node you most want to be -watching. Handle that value explicitly. +While a database is taking a full copy, these timing fields render as the literal string `"Copying"` rather than a date. Anything that parses them as timestamps will fail on a joining node, which is exactly the node you most want to be watching. Handle that value explicitly. ::: -Inventory `system.hdb_nodes` alongside this and compare it to your intended -topology. Configuration intent and live peer state should agree, and the node's -own row is in there too, not just its peers. +Inventory `system.hdb_nodes` alongside this and compare it to your intended topology. Configuration intent and live peer state should agree, and the node's own row is in there too, not just its peers. ### What to alert on -- A peer missing entirely from `connections`, or `connected: false` on a database - that serves a critical journey +- A peer missing entirely from `connections`, or `connected: false` on a database that serves a critical journey - Sustained growth in `latency`, judged against your own baseline - `lastCommitConfirmed` not advancing on a peer while writes are occurring -- A `lastReceivedRemoteTime` to `lastReceivedLocalTime` gap exceeding your - admission budget, evaluated only while writes are flowing -- Repeated reconnects, which are visible in the logs even when a point-in-time - status check looks healthy +- A `lastReceivedRemoteTime` to `lastReceivedLocalTime` gap exceeding your admission budget, evaluated only while writes are flowing +- Repeated reconnects, which are visible in the logs even when a point-in-time status check looks healthy - Version or configuration drift between peers ## Prove convergence, not connection -Socket state cannot tell you that a specific business record is current. For -anything where the answer matters, write a sentinel and read it back from the -peer. +Socket state cannot tell you that a specific business record is current. For anything where the answer matters, write a sentinel and read it back from the peer. ```javascript // On the source node: write a sentinel with a known value @@ -262,120 +177,63 @@ curl -s https://server-2.example.net:9926/OpsProbe/convergence-probe \ -u 'admin:password' ``` -The interval between the write and the moment the peer returns the new value is -your measured convergence time for that database, under whatever load the cluster -is carrying at the time. Run it under load, not on an idle cluster, and record the -result. That number is what your node admission gate should be compared against, -and it is an input to the RPO work in -[Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx). - -The `replication-latency` metric from -[analytics](/reference/v5/analytics/overview#replication-metrics) gives you the -continuous version of the same measurement: the difference between the source -commit timestamp and local time, reported per node, database, and table. Use the -sentinel for a definitive answer during a change, and the metric for a dashboard. - -Note that `replication-latency` is not recorded for the `system` database. Since -`system` is in replication scope by default and carries your users and roles, a -dashboard built only on this metric will show nothing for the database that -propagates identity. Use the sentinel, or the `cluster_status` receive-time pair, -if you need to watch `system` convergence. +The interval between the write and the moment the peer returns the new value is your measured convergence time for that database, under whatever load the cluster is carrying at the time. Run it under load, not on an idle cluster, and record the result. That number is what your node admission gate should be compared against, and it is an input to the RPO work in [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx). + +The `replication-latency` metric from [analytics](/reference/v5/analytics/overview#replication-metrics) gives you the continuous version of the same measurement: the difference between the source commit timestamp and local time, reported per node, database, and table. Use the sentinel for a definitive answer during a change, and the metric for a dashboard. + +Note that `replication-latency` is not recorded for the `system` database. Since `system` is in replication scope by default and carries your users and roles, a dashboard built only on this metric will show nothing for the database that propagates identity. Use the sentinel, or the `cluster_status` receive-time pair, if you need to watch `system` convergence. ## Consistency belongs in the application design -Replication makes data available on every peer. It does not make every peer agree -at every instant, and a distributed decision made on a node that has not yet -converged can observe stale state and act on it. +Replication makes data available on every peer. It does not make every peer agree at every instant, and a distributed decision made on a node that has not yet converged can observe stale state and act on it. -This is not a Harper limitation to work around, it is the property that lets any -node serve any request without a coordinator. But it means a specific class of -operation is unsafe if you write it as a plain read followed by a write: +This is not a Harper limitation to work around, it is the property that lets any node serve any request without a coordinator. But it means a specific class of operation is unsafe if you write it as a plain read followed by a write: -- Claim-once actions: redeeming a code, assigning a unique handle, awarding a - one-per-customer offer -- Hard floors: inventory that must not go negative, a balance that must not - overdraw +- Claim-once actions: redeeming a code, assigning a unique handle, awarding a one-per-customer offer +- Hard floors: inventory that must not go negative, a balance that must not overdraw - Global limits: a rate limit or quota enforced across the whole cluster -- State machine transitions where two nodes could both believe they are making - the same transition +- State machine transitions where two nodes could both believe they are making the same transition -For each of these, choose one of three designs: route all decisions for a given -key to a single owner, use a serialization mechanism so the conflict is resolved -in one place, or delegate to an external coordinator. Then test your -read-after-write expectations through the actual public route rather than against -a single node, because a single node always looks consistent to itself. +For each of these, choose one of three designs: route all decisions for a given key to a single owner, use a serialization mechanism so the conflict is resolved in one place, or delegate to an external coordinator. Then test your read-after-write expectations through the actual public route rather than against a single node, because a single node always looks consistent to itself. ### Prove it On a non-production cluster under write load: 1. Record baseline `cluster_status` timing fields on every node. -2. Interrupt replication to one peer, by stopping the node or blocking the - replication port, while writes continue elsewhere. -3. Watch what your monitoring reports, and how long it takes to say anything. Note - whether `connected` flipped, whether latency alerted, and how long until a - human would have known. -4. Verify your application's actual behavior on the isolated node. Does the - critical journey fail, serve stale data, or serve correctly? All three are - possible and you should know which. -5. Restore the peer, then measure convergence with the sentinel until it is - current. -6. Compare the convergence time against the admission gate in - [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). - If your gate would have admitted the node before step 5 finished, the gate is - wrong. +2. Interrupt replication to one peer, by stopping the node or blocking the replication port, while writes continue elsewhere. +3. Watch what your monitoring reports, and how long it takes to say anything. Note whether `connected` flipped, whether latency alerted, and how long until a human would have known. +4. Verify your application's actual behavior on the isolated node. Does the critical journey fail, serve stale data, or serve correctly? All three are possible and you should know which. +5. Restore the peer, then measure convergence with the sentinel until it is current. +6. Compare the convergence time against the admission gate in [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). If your gate would have admitted the node before step 5 finished, the gate is wrong. ## Operational notes -- **mTLS is required on the secure replication port and cannot be disabled.** - Certificate expiry is therefore a replication outage, so treat certificate - lifetime as an operational deadline with its own alert. See - [certificate management](/reference/v5/security/certificate-management). -- **Gossip discovery means one route can join a whole cluster.** A node that - connects to one peer discovers the rest. That is convenient and it means an - accidental route can widen your topology more than you intended. -- **Sharding is a separate scope control.** If you use - [sharding](/reference/v5/replication/sharding), not every node holds every - record, so "this node is converged" and "this node can answer this query - locally" become different questions. -- **Analytics data has its own replication setting.** See `analytics.replicate` in - the [analytics configuration](/reference/v5/analytics/overview#analytics-configuration) - before you build dashboards that assume metrics are or are not cluster-wide. -- **Keep node-local operational state out of scope.** Availability flags, - maintenance markers, and anything else that describes one node rather than the - cluster should not replicate, or a routine drain propagates to peers. +- **mTLS is required on the secure replication port and cannot be disabled.** Certificate expiry is therefore a replication outage, so treat certificate lifetime as an operational deadline with its own alert. See [certificate management](/reference/v5/security/certificate-management). +- **Gossip discovery means one route can join a whole cluster.** A node that connects to one peer discovers the rest. That is convenient and it means an accidental route can widen your topology more than you intended. +- **Sharding is a separate scope control.** If you use [sharding](/reference/v5/replication/sharding), not every node holds every record, so "this node is converged" and "this node can answer this query locally" become different questions. +- **Analytics data has its own replication setting.** See `analytics.replicate` in the [analytics configuration](/reference/v5/analytics/overview#analytics-configuration) before you build dashboards that assume metrics are or are not cluster-wide. +- **Keep node-local operational state out of scope.** Availability flags, maintenance markers, and anything else that describes one node rather than the cluster should not replicate, or a routine drain propagates to peers. ## Readiness checklist -- [ ] Replication scope read off the running cluster, from every node, not - reconstructed from memory -- [ ] Whether the `system` database is in scope is a documented, deliberate - decision +- [ ] Replication scope read off the running cluster, from every node, not reconstructed from memory +- [ ] Whether the `system` database is in scope is a documented, deliberate decision - [ ] Identity provisioning procedure exists and matches that decision - [ ] Tables requiring one transaction boundary confirmed to be in one database -- [ ] Durable topology constraints live in `harper-config.yaml` routes, not only in - `add_node` calls +- [ ] Durable topology constraints live in `harper-config.yaml` routes, not only in `add_node` calls - [ ] Measured convergence time recorded under load, per critical database -- [ ] Alerts configured on missing peers, sustained latency growth, stalled - `lastCommitConfirmed`, and the remote-to-local receive gap -- [ ] Non-commutative operations identified and given an owner, a serialization - point, or an external coordinator +- [ ] Alerts configured on missing peers, sustained latency growth, stalled `lastCommitConfirmed`, and the remote-to-local receive gap +- [ ] Non-commutative operations identified and given an owner, a serialization point, or an external coordinator - [ ] Replication certificate expiry dates tracked with an alert - [ ] Replication interruption and recovery exercise completed and dated ## Additional Resources -- [Replication overview](/reference/v5/replication/overview) for routes, scope, - controlled flow, and securing connections -- [Clustering operations](/reference/v5/replication/clustering) for `add_node`, - `set_node`, and the `cluster_status` response in full -- [Sharding](/reference/v5/replication/sharding) for controlling how many nodes - hold a given record -- [Replication metrics](/reference/v5/analytics/overview#replication-metrics) for - `replication-latency` and byte counters -- [Certificate management](/reference/v5/security/certificate-management) and - [certificate verification](/reference/v5/security/certificate-verification) -- [Database schema](/reference/v5/database/schema) for the `replicate` table - directive -- [Monitoring and Triage](./monitoring-and-triage.mdx) for putting these signals - on a dashboard +- [Replication overview](/reference/v5/replication/overview) for routes, scope, controlled flow, and securing connections +- [Clustering operations](/reference/v5/replication/clustering) for `add_node`, `set_node`, and the `cluster_status` response in full +- [Sharding](/reference/v5/replication/sharding) for controlling how many nodes hold a given record +- [Replication metrics](/reference/v5/analytics/overview#replication-metrics) for `replication-latency` and byte counters +- [Certificate management](/reference/v5/security/certificate-management) and [certificate verification](/reference/v5/security/certificate-verification) +- [Database schema](/reference/v5/database/schema) for the `replicate` table directive +- [Monitoring and Triage](./monitoring-and-triage.mdx) for putting these signals on a dashboard diff --git a/learn/administration/production-readiness-checklist.mdx b/learn/administration/production-readiness-checklist.mdx index 45d1165ba..882c39202 100644 --- a/learn/administration/production-readiness-checklist.mdx +++ b/learn/administration/production-readiness-checklist.mdx @@ -3,33 +3,21 @@ title: Production Readiness Checklist sidebar_position: 9 --- -This is the aggregated launch gate for the whole Administration track. Every item -appears in one of the earlier guides, which is where the reasoning lives. This -page exists so you have one thing to work down before a launch, and one thing to -hand to a reviewer who asks how you operate Harper. +This is the aggregated launch gate for the whole Administration track. Every item appears in one of the earlier guides, which is where the reasoning lives. This page exists so you have one thing to work down before a launch, and one thing to hand to a reviewer who asks how you operate Harper. -Copy it into your own runbook and adapt it. It is a starting point for your gate, -not a substitute for having one. +Copy it into your own runbook and adapt it. It is a starting point for your gate, not a substitute for having one. ## How to use this -Each item is a claim about your deployment. A claim counts when there is evidence: -a measured number with a date, a written procedure, a configured alert, or a -completed drill. "We know about that" is not evidence, and neither is a passing -intention. +Each item is a claim about your deployment. A claim counts when there is evidence: a measured number with a date, a written procedure, a configured alert, or a completed drill. "We know about that" is not evidence, and neither is a passing intention. -Some items will not apply to you, and that is fine. Mark them not applicable with -a reason rather than deleting them, so a reviewer can see the decision was made -rather than missed. +Some items will not apply to you, and that is fine. Mark them not applicable with a reason rather than deleting them, so a reviewer can see the decision was made rather than missed. Three things are worth deciding before you start: - **Who signs off.** A gate with no named approver is a document, not a gate. -- **What "outstanding" means.** Some items can be accepted as gaps with an owner - and a date. Decide in advance which ones cannot. -- **When you run it again.** This is not a one-time launch artifact. Re-run it - after a Harper version upgrade, a topology change, or significant data growth, - since most of the measured numbers expire. +- **What "outstanding" means.** Some items can be accepted as gaps with an owner and a date. Decide in advance which ones cannot. +- **When you run it again.** This is not a one-time launch artifact. Re-run it after a Harper version upgrade, a topology change, or significant data growth, since most of the measured numbers expire. ## Service definition @@ -49,37 +37,28 @@ From [How Harper Runs in Production](./how-harper-runs-in-production.mdx). From [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx). -- [ ] `F`, the tolerated simultaneous node loss, declared explicitly and visible to - change operators -- [ ] Per-node throughput measured with your own application code, data shape, and - query mix +- [ ] `F`, the tolerated simultaneous node loss, declared explicitly and visible to change operators +- [ ] Per-node throughput measured with your own application code, data shape, and query mix - [ ] Measurement taken with a peer synchronizing, not on an idle cluster - [ ] Target utilization derived from latency at your SLO percentile - [ ] `(N - F) x per-node x utilization >= peak` verified with real numbers - [ ] Measurement conditions recorded alongside the number, and dated -- [ ] Topology's "what must be proven" satisfied, including independence of failure - domains +- [ ] Topology's "what must be proven" satisfied, including independence of failure domains - [ ] Maintenance policy states what happens when a drain would breach `F` - [ ] Storage growth thresholds set, separately from request-rate thresholds ## Health and traffic admission -From -[Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). +From [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). -- [ ] Liveness, availability flag, application readiness, and journey synthetic all - exist as separate signals -- [ ] `@harperdb/status-check` deployed, declared in the root `harper-config.yaml` - rather than by hand +- [ ] Liveness, availability flag, application readiness, and journey synthetic all exist as separate signals +- [ ] `@harperdb/status-check` deployed, declared in the root `harper-config.yaml` rather than by hand - [ ] Traffic layer health check points at `GET /status` on the application port -- [ ] Availability flag storage still declares `replicate: false`, if you forked the - component or persist it yourself -- [ ] Readiness route scoped to one journey's dependencies, with a timeout on every - downstream call +- [ ] Availability flag storage still declares `replicate: false`, if you forked the component or persist it yourself +- [ ] Readiness route scoped to one journey's dependencies, with a timeout on every downstream call - [ ] Readiness response includes the component version - [ ] Drain sequence documented, with measured time to zero traffic -- [ ] Return sequence documented, including convergence verification and a - stability window before full weight +- [ ] Return sequence documented, including convergence verification and a stability window before full weight - [ ] Measured node re-entry time recorded ## Replication @@ -87,17 +66,13 @@ From From [Operating Replication](./operating-replication.mdx). - [ ] Replication scope read off the running cluster, from every node -- [ ] Whether the `system` database is in scope is a deliberate, documented - decision +- [ ] Whether the `system` database is in scope is a deliberate, documented decision - [ ] Identity provisioning procedure matches that decision - [ ] Tables that must commit together confirmed to be in one database -- [ ] Durable topology constraints live in `harper-config.yaml` routes, not only in - `add_node` calls +- [ ] Durable topology constraints live in `harper-config.yaml` routes, not only in `add_node` calls - [ ] Measured convergence time recorded under load, per critical database -- [ ] Alerts on missing peers, sustained latency growth, stalled - `lastCommitConfirmed`, and the receive-time gap -- [ ] Non-commutative operations identified and given an owner, a serialization - point, or an external coordinator +- [ ] Alerts on missing peers, sustained latency growth, stalled `lastCommitConfirmed`, and the receive-time gap +- [ ] Non-commutative operations identified and given an owner, a serialization point, or an external coordinator - [ ] Replication certificate expiry tracked with an alert - [ ] Replication interruption and recovery exercise completed and dated @@ -116,8 +91,7 @@ From [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx). - [ ] Configuration changes go through the same change record as code - [ ] `get_configuration` read back after every configuration change - [ ] Node-local parameters never replicated -- [ ] Schema changes follow expand then contract, with destructive operations - documented and verified per node +- [ ] Schema changes follow expand then contract, with destructive operations documented and verified per node - [ ] Forward repair defined for changes that alter persisted meaning - [ ] Measured release reversal time recorded from a rehearsal @@ -125,17 +99,14 @@ From [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx). From [Monitoring and Triage](./monitoring-and-triage.mdx). -- [ ] A dashboard exists that can place a symptom at edge, traffic layer, node, - replication, data, or change +- [ ] A dashboard exists that can place a symptom at edge, traffic layer, node, replication, data, or change - [ ] Alerts on journey success and latency at the SLO percentile, not the mean -- [ ] Alerts on worker utilization and task queue latency, with thresholds derived - from measured capacity +- [ ] Alerts on worker utilization and task queue latency, with thresholds derived from measured capacity - [ ] Alerts on replication convergence lag - [ ] Alerts on storage headroom and backup age - [ ] Metrics exported off the node - [ ] `analytics.replicate` setting known and deliberate -- [ ] Structured logs include component, version, node, request id, operation, - duration, and outcome +- [ ] Structured logs include component, version, node, request id, operation, duration, and outcome - [ ] Logs centralized off the node - [ ] Deployments annotated onto dashboards - [ ] Triage sequence written where on-call can find it @@ -148,31 +119,24 @@ From [Backup and Recovery](./backup-and-recovery.mdx). - [ ] Failure classes enumerated, each with the mechanism that addresses it - [ ] Recovery point and recovery time stated per database - [ ] Mechanism chosen per database according to its storage engine -- [ ] No database in scope uses per-table storage paths, or the exclusion is known - and accepted +- [ ] No database in scope uses per-table storage paths, or the exclusion is known and accepted - [ ] Backup cadence matches the stated recovery point -- [ ] Backup volume sized against the full cost, including the non-incremental - transaction-log and blob snapshots +- [ ] Backup volume sized against the full cost, including the non-incremental transaction-log and blob snapshots - [ ] Off-host copy exists in a destination that does not share a failure domain -- [ ] Managed repository copies take the whole per-database directory, quiesced or - from an atomic snapshot +- [ ] Managed repository copies take the whole per-database directory, quiesced or from an atomic snapshot - [ ] `verify_backup` runs on a schedule - [ ] Blob integrity understood to be unverified by `verify_backup` -- [ ] Restore constraints documented for user databases, component-held databases, - and `system` +- [ ] Restore constraints documented for user databases, component-held databases, and `system` - [ ] Restore authority named, and restore execution logged -- [ ] Restore procedure isolates the node from replication, not just from user - traffic -- [ ] Restore drill completed and dated, with measured recovery time and actual - data loss +- [ ] Restore procedure isolates the node from replication, not just from user traffic +- [ ] Restore drill completed and dated, with measured recovery time and actual data loss - [ ] `system` database restore rehearsed offline ## Objectives From [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx). -- [ ] Recovery point, recovery time, and availability stated per journey with the - measurement boundary named +- [ ] Recovery point, recovery time, and availability stated per journey with the measurement boundary named - [ ] Exclusions documented - [ ] Every scenario in the scenario map has a named mechanism - [ ] Availability target converted to minutes per month @@ -188,18 +152,15 @@ From [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx). - [ ] Least privilege enforced for operators, applications, and automation - [ ] `super_user` credentials inventoried, with a rotation procedure - [ ] Operations API restricted to administrative networks -- [ ] TLS material inventoried, with expiry alerts for both application and - replication certificates +- [ ] TLS material inventoried, with expiry alerts for both application and replication certificates - [ ] Deploy credentials held in a secret store, not in pipeline configuration - [ ] Administrative access logged, including CLI access on the hosts -- [ ] Filesystem access on nodes understood to permit offline restore without an - API credential +- [ ] Filesystem access on nodes understood to permit offline restore without an API credential ## Ownership and runbooks - [ ] On-call rotation and escalation path defined -- [ ] Runbooks exist for drain, return, deploy, reverse, restore, and cluster - expansion +- [ ] Runbooks exist for drain, return, deploy, reverse, restore, and cluster expansion - [ ] Every runbook has been executed by someone other than its author - [ ] Change record location known, and used - [ ] Incident and postmortem process defined @@ -207,8 +168,7 @@ From [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx). ## Exercises completed -Record the date of the most recent run of each. An undated exercise is an -undocumented one. +Record the date of the most recent run of each. An undated exercise is an undocumented one. | Exercise | Guide | Last run | Result | | ----------------------------------------- | ------------------------------------------------------------------------ | -------- | ------ | @@ -224,10 +184,7 @@ undocumented one. ## Additional Resources -- [Reliability Plan Template](./reliability-plan-template.mdx) for the document - that holds the answers this checklist asks for +- [Reliability Plan Template](./reliability-plan-template.mdx) for the document that holds the answers this checklist asks for - Every guide in this section, linked per group above -- [Security overview](/reference/v5/security/overview) and - [certificate management](/reference/v5/security/certificate-management) -- [Configuration options](/reference/v5/configuration/options) for the settings - referenced throughout +- [Security overview](/reference/v5/security/overview) and [certificate management](/reference/v5/security/certificate-management) +- [Configuration options](/reference/v5/configuration/options) for the settings referenced throughout diff --git a/learn/administration/reliability-plan-template.mdx b/learn/administration/reliability-plan-template.mdx index f580ef251..a563f7723 100644 --- a/learn/administration/reliability-plan-template.mdx +++ b/learn/administration/reliability-plan-template.mdx @@ -3,16 +3,11 @@ title: Reliability Plan Template sidebar_position: 10 --- -The [Production Readiness Checklist](./production-readiness-checklist.mdx) asks -the questions. This is the document that holds the answers. +The [Production Readiness Checklist](./production-readiness-checklist.mdx) asks the questions. This is the document that holds the answers. -Copy it, keep one per service, and keep it where your on-call can reach it during -an incident rather than in a wiki nobody remembers. It is deliberately short. -A reliability plan that takes a day to read is a reliability plan nobody consults -at 3am. +Copy it, keep one per service, and keep it where your on-call can reach it during an incident rather than in a wiki nobody remembers. It is deliberately short. A reliability plan that takes a day to read is a reliability plan nobody consults at 3am. -Example values are included in italics to show the intended level of specificity. -Replace them. +Example values are included in italics to show the intended level of specificity. Replace them. --- @@ -32,8 +27,7 @@ Replace them. ## 2. Critical journeys -The user-facing paths that must work. Everything else in this plan is stated per -journey, so this table defines the scope of all of it. +The user-facing paths that must work. Everything else in this plan is stated per journey, so this table defines the scope of all of it. | Journey | Description | Databases and tables involved | Downstream dependencies | Owner | | ---------------- | ----------------------------- | ----------------------------- | ----------------------- | ----- | @@ -48,8 +42,7 @@ journey, so this table defines the scope of all of it. Monthly unavailability budget: _21 min 55 sec at 99.95%_ -Error budget policy: _what changes when the remaining budget falls below a -threshold, and who decides_ +Error budget policy: _what changes when the remaining budget falls below a threshold, and who decides_ ## 4. Architecture and topology @@ -69,8 +62,7 @@ threshold, and who decides_ | Target utilization | | _derived from latency at the SLO percentile_ | | | Surviving capacity at `F` | | _measured in the drain drill, not calculated_ | | -Maintenance policy when a drain would breach `F`: _pause, add capacity, or accept -with documented duration_ +Maintenance policy when a drain would breach `F`: _pause, add capacity, or accept with documented duration_ ## 6. Traffic admission @@ -158,8 +150,7 @@ with documented duration_ ## 14. Open risks -Accepted gaps, each with an owner and a date. This section existing and being -honest is worth more than it being empty. +Accepted gaps, each with an owner and a date. This section existing and being honest is worth more than it being empty. | Risk | Impact if realized | Why accepted | Owner | Review date | | ---- | ------------------ | ------------ | ----- | ----------- | @@ -175,25 +166,19 @@ honest is worth more than it being empty. ## Keeping this current -Most of the numbers in this plan are properties of a Harper version, a data -volume, a topology, and a set of components. All four change, so the numbers -expire. Re-measure and revise after: +Most of the numbers in this plan are properties of a Harper version, a data volume, a topology, and a set of components. All four change, so the numbers expire. Re-measure and revise after: - A Harper version upgrade - A topology change, including adding or removing nodes - Significant data growth - Any incident that produced a surprise -Date every number. An undated measurement in a reliability plan is worse than a -blank field, because a blank field prompts someone to go and measure. +Date every number. An undated measurement in a reliability plan is worse than a blank field, because a blank field prompts someone to go and measure. ## Additional Resources -- [Production Readiness Checklist](./production-readiness-checklist.mdx) for the - gate that populates this plan -- [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx) for how - to derive sections 3 and 11 +- [Production Readiness Checklist](./production-readiness-checklist.mdx) for the gate that populates this plan +- [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx) for how to derive sections 3 and 11 - [Backup and Recovery](./backup-and-recovery.mdx) for section 10 -- [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx) for - section 8 +- [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx) for section 8 - [Monitoring and Triage](./monitoring-and-triage.mdx) for section 9 diff --git a/learn/administration/safe-deployments-and-rollback.mdx b/learn/administration/safe-deployments-and-rollback.mdx index baf18bffc..9fdee4bb9 100644 --- a/learn/administration/safe-deployments-and-rollback.mdx +++ b/learn/administration/safe-deployments-and-rollback.mdx @@ -6,39 +6,24 @@ sidebar_position: 6 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -[Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx) gets an -immutable artifact onto your cluster from a pipeline. This guide is about what -happens around that call: how to limit who sees a change, what evidence justifies -expanding it, and how to reverse it when the evidence says stop. - -Harper makes reversal unusually cheap, because your application is a component -deployed from an immutable reference, so rolling back is deploying the previous -reference. There is no image to rebuild and no instance to replace. That only -helps if the previous reference is still addressable and someone has done it -before under calm conditions, which is what makes rollback a designed capability -rather than a hope. +[Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx) gets an immutable artifact onto your cluster from a pipeline. This guide is about what happens around that call: how to limit who sees a change, what evidence justifies expanding it, and how to reverse it when the evidence says stop. + +Harper makes reversal unusually cheap, because your application is a component deployed from an immutable reference, so rolling back is deploying the previous reference. There is no image to rebuild and no instance to replace. That only helps if the previous reference is still addressable and someone has done it before under calm conditions, which is what makes rollback a designed capability rather than a hope. ## What You Will Learn -- How to separate build, deploy, activate, and expose into four decisions with - four different control points +- How to separate build, deploy, activate, and expose into four decisions with four different control points - The change loop, and what counts as evidence at each step - Five rollout patterns and the Harper operations behind each -- What `"restart": true` and `"restart": "rolling"` actually do, which is not what - most people assume -- How to classify a change so you reverse the right thing, and why reversing the - wrong thing can cause data loss +- What `"restart": true` and `"restart": "rolling"` actually do, which is not what most people assume +- How to classify a change so you reverse the right thing, and why reversing the wrong thing can cause data loss ## Prerequisites -- A cluster of at least three nodes, so you can hold one out of rotation and still - meet capacity -- A working pipeline that deploys from an immutable reference - ([Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx)) -- [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx), - since every pattern here depends on being able to move traffic -- [Monitoring and Triage](./monitoring-and-triage.mdx), since every gate here - depends on being able to compare versions +- A cluster of at least three nodes, so you can hold one out of rotation and still meet capacity +- A working pipeline that deploys from an immutable reference ([Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx)) +- [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx), since every pattern here depends on being able to move traffic +- [Monitoring and Triage](./monitoring-and-triage.mdx), since every gate here depends on being able to compare versions ## Separate four decisions @@ -51,63 +36,31 @@ Most bad deployments come from collapsing these into one action. | **Activate** | Which code path is actually enabled? | `urlPath` and `host` mounting, component configuration, a feature flag, a header or tenant rule | | **Expose** | Which production traffic reaches it? | Traffic layer weights, the availability flag, cohort or geography targeting | -Once these are separate you get options that do not exist otherwise. You can -install an artifact on every node and activate it nowhere. You can activate a -route and expose it only to internal traffic. And when something is wrong you can -reverse exposure in seconds without touching what is installed, which is the -fastest containment action available to you. +Once these are separate you get options that do not exist otherwise. You can install an artifact on every node and activate it nowhere. You can activate a route and expose it only to internal traffic. And when something is wrong you can reverse exposure in seconds without touching what is installed, which is the fastest containment action available to you. :::danger -**`replicated` is opt-out, not opt-in.** A `deploy_component` call with no -`replicated` field replicates to every peer. So does `add_component`, -`drop_component`, `set_component_file`, `set_env_value`, `delete_env_value`, and -the destructive `drop_database` and `drop_table`. Only `"replicated": false` -changes anything; `"replicated": true` is the default spelled out. - -This is the opposite of what most operators assume, and it is why the single-node -validation pattern below has to say `false` explicitly. If you intend to touch one -node, you must say so. ([`set_configuration`](/reference/v5/configuration/operations) -is the deliberate exception: it replicates only when you ask.) +**`replicated` is opt-out, not opt-in.** A `deploy_component` call with no `replicated` field replicates to every peer. So does `add_component`, `drop_component`, `set_component_file`, `set_env_value`, `delete_env_value`, and the destructive `drop_database` and `drop_table`. Only `"replicated": false` changes anything; `"replicated": true` is the default spelled out. + +This is the opposite of what most operators assume, and it is why the single-node validation pattern below has to say `false` explicitly. If you intend to touch one node, you must say so. ([`set_configuration`](/reference/v5/configuration/operations) is the deliberate exception: it replicates only when you ask.) ::: -`urlPath` mounts a component at an HTTP path. -`host` (Added in: v5.2.0) serves it on a virtual hostname. Both are -persisted on the component's root config entry, so they are part of the deployed -state rather than a runtime toggle. See -[HTTP middleware routing](/reference/v5/http/overview#middleware-routing). +`urlPath` mounts a component at an HTTP path. `host` (Added in: v5.2.0) serves it on a virtual hostname. Both are persisted on the component's root config entry, so they are part of the deployed state rather than a runtime toggle. See [HTTP middleware routing](/reference/v5/http/overview#middleware-routing). ## The change loop -1. **Preflight.** Confirm the target version, the component inventory from - `get_components`, `cluster_status` convergence, peer capacity with one node - held out, backup posture if data is at risk, and that the previous known-good - artifact is still addressable. +1. **Preflight.** Confirm the target version, the component inventory from `get_components`, `cluster_status` convergence, peer capacity with one node held out, backup posture if data is at risk, and that the previous known-good artifact is still addressable. -2. **Limit.** Choose the smallest cohort that produces useful evidence. One - drained node, an internal cohort, a low-risk geography, a tenant set, or a - small weighted slice. Smaller is better right up until the sample is too small - to distinguish signal from noise. +2. **Limit.** Choose the smallest cohort that produces useful evidence. One drained node, an internal cohort, a low-risk geography, a tenant set, or a small weighted slice. Smaller is better right up until the sample is too small to distinguish signal from noise. -3. **Observe.** Compare the new and old versions on the same metrics: request - success, latency at your SLO percentile, worker saturation, logs and traces, - data correctness, replication behavior, and downstream errors. Comparison - against the other cohort is the point. Absolute numbers on the new version tell - you much less. +3. **Observe.** Compare the new and old versions on the same metrics: request success, latency at your SLO percentile, worker saturation, logs and traces, data correctness, replication behavior, and downstream errors. Comparison against the other cohort is the point. Absolute numbers on the new version tell you much less. -4. **Decide.** Advance, hold, stop, or roll back, against thresholds declared - before you started, with a named owner. Missing telemetry is a failed gate, not - a pass. +4. **Decide.** Advance, hold, stop, or roll back, against thresholds declared before you started, with a named owner. Missing telemetry is a failed gate, not a pass. -5. **Expand.** Increase exposure only after minimum sample and hold conditions - pass, and keep enough healthy capacity in the old version to reverse. +5. **Expand.** Increase exposure only after minimum sample and hold conditions pass, and keep enough healthy capacity in the old version to reverse. -6. **Close.** Verify uniform artifact and configuration across nodes, restore - intended traffic, record the actual outcome, and keep the evidence with the - change record. +6. **Close.** Verify uniform artifact and configuration across nodes, restore intended traffic, record the actual outcome, and keep the evidence with the change record. -Declaring thresholds in step 4 before step 2 is the part that gets skipped and the -part that matters. A threshold invented while looking at a live graph is not a -threshold, it is a negotiation, and it always resolves toward shipping. +Declaring thresholds in step 4 before step 2 is the part that gets skipped and the part that matters. A threshold invented while looking at a live graph is not a threshold, it is a negotiation, and it always resolves toward shipping. ## Five rollout patterns @@ -134,8 +87,7 @@ Isolated node validation, concretely: } ``` -Drain the node first, deploy, run your readiness route and journey synthetic -against it directly, then admit a bounded cohort. +Drain the node first, deploy, run your readiness route and journey synthetic against it directly, then admit a bounded cohort. @@ -150,103 +102,49 @@ against it directly, then admit a bounded cohort. } ``` -Same immutable reference, now cluster-wide. The deploy itself is finished when the -call returns, so the thing still outstanding is the restart: poll the returned -`restartJobId` with -[`get_job`](/reference/v5/operations-api/operations#get_job). Read -[`get_deployment`](/reference/v5/operations-api/operations#get_deployment) for the -per-peer detail the response summarizes away, including which peers received the -artifact and which failed. +Same immutable reference, now cluster-wide. The deploy itself is finished when the call returns, so the thing still outstanding is the restart: poll the returned `restartJobId` with [`get_job`](/reference/v5/operations-api/operations#get_job). Read [`get_deployment`](/reference/v5/operations-api/operations#get_deployment) for the per-peer detail the response summarizes away, including which peers received the artifact and which failed. -Prefer a pinned version or an immutable tarball over a moving branch reference. -A branch moves, so you can neither audit what was running yesterday nor redeploy -it. +Prefer a pinned version or an immutable tarball over a moving branch reference. A branch moves, so you can neither audit what was running yesterday nor redeploy it. :::tip -A pinned registry version such as `@my-org/orders-api@2.4.1` is the -best-travelled form. If you pin a tarball URL instead, note that a bare `https://` -URL pointing at `github.com`, `gitlab.com`, or `bitbucket.org` is treated as a git -clone rather than a download, so a GitHub release asset URL will not behave the -way the example above does. Host release tarballs somewhere neutral, or use a -registry version. +A pinned registry version such as `@my-org/orders-api@2.4.1` is the best-travelled form. If you pin a tarball URL instead, note that a bare `https://` URL pointing at `github.com`, `gitlab.com`, or `bitbucket.org` is treated as a git clone rather than a download, so a GitHub release asset URL will not behave the way the example above does. Host release tarballs somewhere neutral, or use a registry version. ::: ## What restart actually does This is worth reading carefully, because the naming invites a wrong assumption. -**`"restart": true`** starts a restart of the HTTP worker threads on the node -handling the call and returns immediately, without waiting for it. A `200` means -the deploy succeeded and a restart has been requested. It does not mean the new -code is serving yet. Until a worker has been replaced it is still running the -previous code, and on platforms where replacements share a listening port it keeps -accepting connections during the changeover. +**`"restart": true`** starts a restart of the HTTP worker threads on the node handling the call and returns immediately, without waiting for it. A `200` means the deploy succeeded and a restart has been requested. It does not mean the new code is serving yet. Until a worker has been replaced it is still running the previous code, and on platforms where replacements share a listening port it keeps accepting connections during the changeover. -**`"restart": "rolling"`** does not restart inline either, but it is observable. -It starts a replicated `restart_service` job and returns a `restartJobId` you can -poll with [`get_job`](/reference/v5/operations-api/operations#get_job). That job -walks the cluster **one node at a time**, waiting for each node to come back -before starting the next, so the cluster keeps serving throughout. +**`"restart": "rolling"`** does not restart inline either, but it is observable. It starts a replicated `restart_service` job and returns a `restartJobId` you can poll with [`get_job`](/reference/v5/operations-api/operations#get_job). That job walks the cluster **one node at a time**, waiting for each node to come back before starting the next, so the cluster keeps serving throughout. :::warning -**On a cluster, prefer `"rolling"`.** `"restart": true` is forwarded verbatim to -every peer, and peers are dispatched in parallel, so a replicated deploy with -`"restart": true` restarts every node in the cluster at roughly the same moment. -That is a cluster-wide availability event, not a node-local one. Reserve -`"restart": true` for a single-node or development instance, and use `"rolling"` -anywhere you care about staying up. See -[Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx). +**On a cluster, prefer `"rolling"`.** `"restart": true` is forwarded verbatim to every peer, and peers are dispatched in parallel, so a replicated deploy with `"restart": true` restarts every node in the cluster at roughly the same moment. That is a cluster-wide availability event, not a node-local one. Reserve `"restart": true` for a single-node or development instance, and use `"rolling"` anywhere you care about staying up. See [Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx). ::: -Two more consequences for your pipeline. Neither restart mode reports completion -in the deploy response, so if you need to know the new code is live, poll the -rolling restart job or probe the nodes themselves. And a failed restart does not -fail the deploy: the component is installed and replicated either way, so your -pipeline needs to check both outcomes separately rather than assuming one implies -the other. +Two more consequences for your pipeline. Neither restart mode reports completion in the deploy response, so if you need to know the new code is live, poll the rolling restart job or probe the nodes themselves. And a failed restart does not fail the deploy: the component is installed and replicated either way, so your pipeline needs to check both outcomes separately rather than assuming one implies the other. ### Waiting for the restart -From v5.3.0, `"restart": true` waits for the worker restart to finish before -responding, rather than returning as soon as it has been requested. The wait -follows the restart's own progress rather than a fixed timeout, so the call can -take tens of seconds on a slow install with many worker threads, with a hard -ceiling of ten minutes. +From v5.3.0, `"restart": true` waits for the worker restart to finish before responding, rather than returning as soon as it has been requested. The wait follows the restart's own progress rather than a fixed timeout, so the call can take tens of seconds on a slow install with many worker threads, with a hard ceiling of ten minutes. -Two things do not change, and both matter more than the wait itself. A caller that -gives up early does not stop the restart, it only loses the result. And the -response still does not carry the restart's outcome: a restart that stalls, times -out, or leaves workers on the old code is reported in the node's log, not to you. -So even on v5.3.0, treat a `200` as "the deploy landed and a restart ran," and -confirm the version actually serving through your readiness route or journey -synthetic rather than through the deploy response. +Two things do not change, and both matter more than the wait itself. A caller that gives up early does not stop the restart, it only loses the result. And the response still does not carry the restart's outcome: a restart that stalls, times out, or leaves workers on the old code is reported in the node's log, not to you. So even on v5.3.0, treat a `200` as "the deploy landed and a restart ran," and confirm the version actually serving through your readiness route or journey synthetic rather than through the deploy response. Two parameters worth setting deliberately on replicated deploys: -- `deployment_timeout` (Added in: v5.1.4) is how long a peer waits - for the replicated payload before failing, defaulting to 120000 ms. Raise it for - large components or slow links. -- `ignore_replication_errors` (Added in: v5.1.4) treats a peer that - fails to receive the deploy as non-fatal. By default a failed peer makes the - whole operation return a non-2xx status, while the component is still deployed - on the origin node. Decide which behavior you want before you need it, because - the default leaves you in a mixed-version state with a failed response, and that - is a confusing thing to reason about mid-incident. +- `deployment_timeout` (Added in: v5.1.4) is how long a peer waits for the replicated payload before failing, defaulting to 120000 ms. Raise it for large components or slow links. +- `ignore_replication_errors` (Added in: v5.1.4) treats a peer that fails to receive the deploy as non-fatal. By default a failed peer makes the whole operation return a non-2xx status, while the component is still deployed on the origin node. Decide which behavior you want before you need it, because the default leaves you in a mixed-version state with a failed response, and that is a confusing thing to reason about mid-incident. ## Configuration changes are deployments too -A configuration change carries the same risk as a code change and gets less -ceremony, which is backwards. +A configuration change carries the same risk as a code change and gets less ceremony, which is backwards. -`set_configuration` supports `"replicated": true` (Added in: v5.2.0) -to apply a change across the cluster in one call, with per-node outcomes in the -response. To finish the change cluster-wide, follow with `restart_service` using -`"replicated": true`, which restarts nodes one at a time. +`set_configuration` supports `"replicated": true` (Added in: v5.2.0) to apply a change across the cluster in one call, with per-node outcomes in the response. To finish the change cluster-wide, follow with `restart_service` using `"replicated": true`, which restarts nodes one at a time. ```json { @@ -257,25 +155,14 @@ response. To finish the change cluster-wide, follow with `restart_service` using ``` :::danger -Only replicate cluster-appropriate parameters. Node-local values such as ports, -`node.hostname`, file paths, TLS material, and `replication.hostname`, `url`, or -`routes` would overwrite every peer's own values. Replicating one of these is a -cluster-wide outage delivered in a single API call. +Only replicate cluster-appropriate parameters. Node-local values such as ports, `node.hostname`, file paths, TLS material, and `replication.hostname`, `url`, or `routes` would overwrite every peer's own values. Replicating one of these is a cluster-wide outage delivered in a single API call. ::: -Two more things to hold onto. A change takes effect only after a restart, so a -node that has been reconfigured and not restarted is running the old -configuration while reporting the new one. And `get_status` reports a -`restartRequired` flag, but it tracks component and code restarts rather than -configuration changes, so it will not tell you a configuration change is still -pending. Track pending configuration in your change record instead, and read back -`get_configuration` after the restart to confirm. +Two more things to hold onto. A change takes effect only after a restart, so a node that has been reconfigured and not restarted is running the old configuration while reporting the new one. And `get_status` reports a `restartRequired` flag, but it tracks component and code restarts rather than configuration changes, so it will not tell you a configuration change is still pending. Track pending configuration in your change record instead, and read back `get_configuration` after the restart to confirm. ## Rollback is a designed capability -Classify the change before choosing a reversal path, because these have different -compatibility requirements, different authorities, and very different blast -radii: +Classify the change before choosing a reversal path, because these have different compatibility requirements, different authorities, and very different blast radii: | Change type | Reversal | Watch out for | | ---------------------- | ------------------------------------------------- | ------------------------------------------------------------------- | @@ -288,97 +175,59 @@ radii: Practices that make each of these real: -- **Keep the previous package addressable and rehearse redeploying it** before - launch, not during an incident. -- **Prefer expand-then-contract schema evolution.** Add the new shape, migrate, - then remove the old shape in a separate change. Destructive schema behavior is - operation and version dependent, so each destructive change needs an exact - written procedure and verification on every node. -- **Use the same gates for rollback as for forward movement.** Availability, - journey synthetic, peer stability, data validation, traffic reconciliation. A - rollback is a deployment and can fail like one. +- **Keep the previous package addressable and rehearse redeploying it** before launch, not during an incident. +- **Prefer expand-then-contract schema evolution.** Add the new shape, migrate, then remove the old shape in a separate change. Destructive schema behavior is operation and version dependent, so each destructive change needs an exact written procedure and verification on every node. +- **Use the same gates for rollback as for forward movement.** Availability, journey synthetic, peer stability, data validation, traffic reconciliation. A rollback is a deployment and can fail like one. :::warning -If a release changed the meaning of persisted data, code rollback alone will not -fix it, and restoring a database to undo application code is usually the wrong -move. A restore rolls back every write in the window, including all the valid -ones, so it can violate your RPO in order to fix a code bug. Define forward repair -or replay for these cases instead. See -[Backup and Recovery](./backup-and-recovery.mdx). +If a release changed the meaning of persisted data, code rollback alone will not fix it, and restoring a database to undo application code is usually the wrong move. A restore rolls back every write in the window, including all the valid ones, so it can violate your RPO in order to fix a code bug. Define forward repair or replay for these cases instead. See [Backup and Recovery](./backup-and-recovery.mdx). ::: ### Prove it Rehearse a stopped rollout end to end on a non-production cluster: -1. Declare a threshold before you start, for example "stop if journey success on - the new cohort is more than 0.5 percent below the control cohort over five - minutes." +1. Declare a threshold before you start, for example "stop if journey success on the new cohort is more than 0.5 percent below the control cohort over five minutes." 2. Deploy a component that fails that threshold deliberately, to a bounded cohort. 3. Detect it through your dashboards rather than because you know what you did. -4. Reverse it, and time from decision to restored traffic. That number is your - release RTO, and it belongs in your reliability plan. -5. Verify uniform state afterwards: `get_components` on every node, and - `list_deployments` showing the reversal. +4. Reverse it, and time from decision to restored traffic. That number is your release RTO, and it belongs in your reliability plan. +5. Verify uniform state afterwards: `get_components` on every node, and `list_deployments` showing the reversal. -The step people fail is 5. A partially reversed cluster looks fine on a dashboard -because the healthy majority dominates the average. +The step people fail is 5. A partially reversed cluster looks fine on a dashboard because the healthy majority dominates the average. ## Operational notes -- **Version parity is an operating requirement.** Mixed Harper versions in a - cluster change replication and deployment behavior, so a rollout that stalls - halfway is a state you want to detect and exit, not sit in. -- **Keep deploy credentials in your delivery platform's secret store**, use TLS - and least privilege, and retain the operation result as change evidence. See - [secrets](/reference/v5/security/secrets). -- **Component deployment can replicate, so a deploy is a cluster event.** Keep - isolated single-node deployment available for validation, because if the only - deployment path you have is replicated then you have no way to test anything on - one node. -- **Flag cleanup is part of the release.** A feature flag with no owner and no - removal date becomes permanent configuration that nobody understands, and it - will eventually be the thing nobody can explain during an incident. -- **Record who decided, not only what happened.** Named decision authority is what - makes a stop gate function under pressure, and it costs nothing to write down in - advance. +- **Version parity is an operating requirement.** Mixed Harper versions in a cluster change replication and deployment behavior, so a rollout that stalls halfway is a state you want to detect and exit, not sit in. +- **Keep deploy credentials in your delivery platform's secret store**, use TLS and least privilege, and retain the operation result as change evidence. See [secrets](/reference/v5/security/secrets). +- **Component deployment can replicate, so a deploy is a cluster event.** Keep isolated single-node deployment available for validation, because if the only deployment path you have is replicated then you have no way to test anything on one node. +- **Flag cleanup is part of the release.** A feature flag with no owner and no removal date becomes permanent configuration that nobody understands, and it will eventually be the thing nobody can explain during an incident. +- **Record who decided, not only what happened.** Named decision authority is what makes a stop gate function under pressure, and it costs nothing to write down in advance. ## Readiness checklist - [ ] Artifacts are immutable and referenced by pinned version, never by branch - [ ] Deploy and expose are separate actions in your pipeline -- [ ] The previous known-good artifact is addressable, and redeploying it has been - rehearsed +- [ ] The previous known-good artifact is addressable, and redeploying it has been rehearsed - [ ] Cohort ladder defined, from smallest useful sample to full exposure - [ ] Stop thresholds and hold times declared before the rollout starts - [ ] A named decision owner for advance, hold, stop, and roll back -- [ ] Pipeline polls the restart job rather than treating the deploy response as - proof the new code is serving -- [ ] `"replicated": false` used deliberately wherever a change is meant to reach - one node only, since replication is the default +- [ ] Pipeline polls the restart job rather than treating the deploy response as proof the new code is serving +- [ ] `"replicated": false` used deliberately wherever a change is meant to reach one node only, since replication is the default - [ ] Cluster deploys use `"restart": "rolling"`, not `"restart": true` - [ ] `deployment_timeout` and `ignore_replication_errors` set deliberately - [ ] Configuration changes go through the same change record as code - [ ] `get_configuration` read back after every configuration change -- [ ] Schema changes follow expand-then-contract, with destructive operations - documented per node +- [ ] Schema changes follow expand-then-contract, with destructive operations documented per node - [ ] Forward repair defined for changes that alter persisted meaning - [ ] Measured release RTO recorded from a rehearsed reversal ## Additional Resources -- [Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx) for the - pipeline mechanics and deploy credentials -- [`deploy_component` reference](/reference/v5/operations-api/operations#deploy_component) - for every parameter including `urlPath`, `host`, and replication controls -- [Applications reference](/reference/v5/components/applications) for component - structure and the full component operation set -- [Deployment records](/reference/v5/operations-api/operations#deployment-operations) - for `list_deployments` and `get_deployment` -- [HTTP middleware routing](/reference/v5/http/overview#middleware-routing) for - `urlPath` and `host` activation -- [Configuration operations](/reference/v5/configuration/operations) for - `set_configuration` and restart requirements +- [Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx) for the pipeline mechanics and deploy credentials +- [`deploy_component` reference](/reference/v5/operations-api/operations#deploy_component) for every parameter including `urlPath`, `host`, and replication controls +- [Applications reference](/reference/v5/components/applications) for component structure and the full component operation set +- [Deployment records](/reference/v5/operations-api/operations#deployment-operations) for `list_deployments` and `get_deployment` +- [HTTP middleware routing](/reference/v5/http/overview#middleware-routing) for `urlPath` and `host` activation +- [Configuration operations](/reference/v5/configuration/operations) for `set_configuration` and restart requirements - [Database schema](/reference/v5/database/schema) for schema evolution -- [Multiple Applications on One Cluster](../developers/multiple-applications.mdx) - for running more than one component side by side +- [Multiple Applications on One Cluster](../developers/multiple-applications.mdx) for running more than one component side by side diff --git a/learn/administration/sizing-a-harper-cluster.mdx b/learn/administration/sizing-a-harper-cluster.mdx index 9a226224f..4e695fdb7 100644 --- a/learn/administration/sizing-a-harper-cluster.mdx +++ b/learn/administration/sizing-a-harper-cluster.mdx @@ -6,33 +6,22 @@ sidebar_position: 3 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -Most clusters get sized by dividing expected peak throughput by measured per-node -throughput and rounding up. That arithmetic produces a cluster that is exactly -large enough to handle a good day, which means the first node failure during peak -traffic becomes a user-visible outage. +Most clusters get sized by dividing expected peak throughput by measured per-node throughput and rounding up. That arithmetic produces a cluster that is exactly large enough to handle a good day, which means the first node failure during peak traffic becomes a user-visible outage. -Sizing Harper starts from a different question. Not "how much traffic do we -have," but "how many nodes are we willing to lose at once, and can the survivors -carry the load." This guide gives you the capacity rule that follows from that, -what to include when you measure a node, and how to turn an availability target -into an operating budget you can actually spend. +Sizing Harper starts from a different question. Not "how much traffic do we have," but "how many nodes are we willing to lose at once, and can the survivors carry the load." This guide gives you the capacity rule that follows from that, what to include when you measure a node, and how to turn an availability target into an operating budget you can actually spend. ## What You Will Learn - The capacity invariant that sizes for the failure state, with a worked example -- What to include in a per-node throughput measurement, and the three things - people leave out +- What to include in a per-node throughput measurement, and the three things people leave out - Why maintenance is a planned failure, and what that means for change windows - How to choose a topology from your objectives rather than from a diagram -- How to convert an availability SLO into minutes per month, and what each tier - demands of your automation +- How to convert an availability SLO into minutes per month, and what each tier demands of your automation ## Prerequisites -- [How Harper Runs in Production](./how-harper-runs-in-production.mdx), and your - service boundary inventory -- [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx), - because capacity headroom is meaningless if traffic cannot be moved off a node +- [How Harper Runs in Production](./how-harper-runs-in-production.mdx), and your service boundary inventory +- [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx), because capacity headroom is meaningless if traffic cannot be moved off a node - A load generation tool and a non-production cluster you can push to saturation - A stated peak throughput requirement for your critical journeys @@ -44,15 +33,9 @@ The rule: (N - F) x tested per-node throughput x target utilization >= required peak throughput ``` -Where `N` is your node count, `F` is the largest simultaneous node loss the design -tolerates, and target utilization is the fraction of measured capacity you are -willing to run at, which should leave room for latency to stay acceptable rather -than merely for requests to complete. +Where `N` is your node count, `F` is the largest simultaneous node loss the design tolerates, and target utilization is the fraction of measured capacity you are willing to run at, which should leave room for latency to stay acceptable rather than merely for requests to complete. -Worked example. Suppose your critical journey needs 12,000 requests per second at -peak, a node sustains 5,000 requests per second in your own test with acceptable -latency, you will run at 70 percent utilization, and you want to survive losing -one node: +Worked example. Suppose your critical journey needs 12,000 requests per second at peak, a node sustains 5,000 requests per second in your own test with acceptable latency, you will run at 70 percent utilization, and you want to survive losing one node: ```text Required surviving capacity = 12,000 / 0.7 = 17,143 req/s @@ -60,36 +43,20 @@ Surviving nodes needed = 17,143 / 5,000 = 3.43 -> 4 N = 4 + F (1) = 5 nodes ``` -Five nodes, not three. The naive calculation gives 12,000 / 5,000 = 2.4, rounded -up to 3, and that cluster degrades the moment anything goes wrong. +Five nodes, not three. The naive calculation gives 12,000 / 5,000 = 2.4, rounded up to 3, and that cluster degrades the moment anything goes wrong. -Two things to notice. Increasing `F` from 1 to 2 costs you one more node here, not -double the cluster, so tolerating a second simultaneous failure is often cheaper -than people expect at this size. And the utilization factor is doing as much work -as `F` is: sizing to 100 percent of measured capacity means your "surviving" -nodes are at saturation, where latency degrades long before throughput does. +Two things to notice. Increasing `F` from 1 to 2 costs you one more node here, not double the cluster, so tolerating a second simultaneous failure is often cheaper than people expect at this size. And the utilization factor is doing as much work as `F` is: sizing to 100 percent of measured capacity means your "surviving" nodes are at saturation, where latency degrades long before throughput does. ## Measure a node honestly -The per-node number in that formula is the one most likely to be wrong, because -benchmark conditions are kinder than production. Include all of this: - -- **Representative application code, data shape, and query mix.** A Harper node - runs your component logic in the same process as the data access, so your code - is part of the capacity measurement in a way it would not be for a standalone - database. A synthetic key-value benchmark tells you very little about the node's - capacity to serve your journey. -- **Realistic downstream latency.** If your resource calls an upstream pricing - service, its latency occupies worker capacity on the node. -- **Replication catch-up load.** A node that is feeding a recovering peer is doing - work that does not appear in its own request metrics. Measure with a peer - synchronizing, because that is precisely the state you will be in when you are - already down a node. -- **Traffic imbalance.** Configured weights express intent. Actual request - distribution is what consumes capacity, and it is rarely even. - -Measure each node separately rather than dividing a cluster total by node count. -An average conceals the one node that is about to tip. +The per-node number in that formula is the one most likely to be wrong, because benchmark conditions are kinder than production. Include all of this: + +- **Representative application code, data shape, and query mix.** A Harper node runs your component logic in the same process as the data access, so your code is part of the capacity measurement in a way it would not be for a standalone database. A synthetic key-value benchmark tells you very little about the node's capacity to serve your journey. +- **Realistic downstream latency.** If your resource calls an upstream pricing service, its latency occupies worker capacity on the node. +- **Replication catch-up load.** A node that is feeding a recovering peer is doing work that does not appear in its own request metrics. Measure with a peer synchronizing, because that is precisely the state you will be in when you are already down a node. +- **Traffic imbalance.** Configured weights express intent. Actual request distribution is what consumes capacity, and it is rarely even. + +Measure each node separately rather than dividing a cluster total by node count. An average conceals the one node that is about to tip. @@ -121,39 +88,23 @@ await fetch('https://my-node.example.com:9925/', { -Watch CPU, memory, event loop and worker pressure from the `threads` attribute, -disk, and network, and pair them with request rate and latency from -[analytics](/reference/v5/analytics/overview). The number you want is not the -point where requests start failing. It is the point where your journey's latency -leaves your SLO, which arrives earlier. +Watch CPU, memory, event loop and worker pressure from the `threads` attribute, disk, and network, and pair them with request rate and latency from [analytics](/reference/v5/analytics/overview). The number you want is not the point where requests start failing. It is the point where your journey's latency leaves your SLO, which arrives earlier. :::tip -Record the conditions alongside the number: Harper version, node size, data -volume, query mix, and whether a peer was synchronizing. A per-node capacity -figure without its conditions is not reusable, and six months later nobody will -remember whether the test included replication load. +Record the conditions alongside the number: Harper version, node size, data volume, query mix, and whether a peer was synchronizing. A per-node capacity figure without its conditions is not reusable, and six months later nobody will remember whether the test included replication load. ::: ## Maintenance is a planned failure -Draining a node for a deploy, an upgrade, or an investigation consumes exactly the -same headroom that a node failure does. So a cluster sized for `F = 1` is running -at `F = 0` for the duration of every maintenance window, with no tolerance left. +Draining a node for a deploy, an upgrade, or an investigation consumes exactly the same headroom that a node failure does. So a cluster sized for `F = 1` is running at `F = 0` for the duration of every maintenance window, with no tolerance left. -The operating rule that follows: if draining one node means the service can no -longer tolerate its declared `F`, pause the change. Either wait for lower traffic, -or add capacity for the window, or accept and document the reduced tolerance for a -bounded period. What you should not do is treat the maintenance window as free -because nothing is technically broken. +The operating rule that follows: if draining one node means the service can no longer tolerate its declared `F`, pause the change. Either wait for lower traffic, or add capacity for the window, or accept and document the reduced tolerance for a bounded period. What you should not do is treat the maintenance window as free because nothing is technically broken. -This is also the argument for sizing `F` at 2 in a cluster that deploys -frequently. It is not paranoia about correlated hardware failure. It is that you -want to be able to deploy during business hours and still survive an incident. +This is also the argument for sizing `F` at 2 in a cluster that deploys frequently. It is not paranoia about correlated hardware failure. It is that you want to be able to deploy during business hours and still survive an incident. ## Choose a topology from objectives -Pick the pattern that matches the failures you actually need to survive, and know -what you have to prove for it to count. +Pick the pattern that matches the failures you actually need to survive, and know what you have to prove for it to count. | Pattern | When it fits | What you must prove | | ------------------------------------ | ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | @@ -162,22 +113,13 @@ what you have to prove for it to count. | Multiple regions | Regional impairment or user locality is a real requirement | Data authority per region, route propagation time, convergence, degraded-mode behavior | | Edge or on-premises peers | Data locality or residency constrains where data can live | Replication scope per location, identity provisioning, backup path from each site | -The right-hand column is the useful one. A topology diagram is a claim, and the -claim is only true once you have exercised it. "Multiple failure domains" means -nothing if both domains draw from the same power feed, and you will not discover -that from your provider's documentation. +The right-hand column is the useful one. A topology diagram is a claim, and the claim is only true once you have exercised it. "Multiple failure domains" means nothing if both domains draw from the same power feed, and you will not discover that from your provider's documentation. -Note that adding a node is a data movement event as well as a capacity event, -since a Harper node carries data along with request handling. A node whose -databases have never synchronized downloads them in full before it is useful, so -scaling out is not instantaneous and cannot be your response to an unexpected -traffic spike. Size ahead of demand. +Note that adding a node is a data movement event as well as a capacity event, since a Harper node carries data along with request handling. A node whose databases have never synchronized downloads them in full before it is useful, so scaling out is not instantaneous and cannot be your response to an unexpected traffic spike. Size ahead of demand. ## Turn the SLO into a budget -An availability target is a quantity of unavailability you are permitted to spend -per month. Written that way it becomes an operating constraint rather than an -aspiration. +An availability target is a quantity of unavailability you are permitted to spend per month. Written that way it becomes an operating constraint rather than an aspiration. | Monthly SLO | Approximate maximum unavailability | What it demands | | ----------- | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | @@ -185,58 +127,34 @@ aspiration. | 99.95% | 21 min 55 sec | Automated node removal and a rehearsed release reversal become necessary | | 99.99% | 4 min 23 sec | Node failure must be close to transparent. Detection, traffic removal, and validation have to be automated, because no human response fits in the budget | -Figures assume a 30.44 day month. Your contractual definition, exclusions, and -measurement boundary may differ, and the boundary matters more than the number: -availability measured at your CDN edge and availability measured at the Harper -node are different quantities. +Figures assume a 30.44 day month. Your contractual definition, exclusions, and measurement boundary may differ, and the boundary matters more than the number: availability measured at your CDN edge and availability measured at the Harper node are different quantities. -The practical consequence is that the SLO tier determines how much automation you -need, not how many nodes. At three nines a person can be paged, look at a -dashboard, and drain a node. At four nines that same sequence has already spent -the entire month's budget. +The practical consequence is that the SLO tier determines how much automation you need, not how many nodes. At three nines a person can be paged, look at a dashboard, and drain a node. At four nines that same sequence has already spent the entire month's budget. ### Prove it -Run the failure state rather than calculating it. On a cluster carrying -representative load at your expected peak: +Run the failure state rather than calculating it. On a cluster carrying representative load at your expected peak: 1. Record per-node request rate, latency, and saturation as a baseline. -2. Drain one node using the sequence from - [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). -3. Measure what the survivors do: request rate, latency at your SLO percentile, - CPU and worker pressure, and whether any user-visible errors appeared during - the transition. -4. While still down a node, bring the drained node back and let it synchronize, so - the survivors are carrying peak traffic and feeding a recovering peer at the - same time. This is the real worst case and it is the one nobody tests. -5. Compare measured surviving capacity against your invariant. If step 4 pushed - latency out of SLO, your target utilization is too high or your `N` is too low. +2. Drain one node using the sequence from [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). +3. Measure what the survivors do: request rate, latency at your SLO percentile, CPU and worker pressure, and whether any user-visible errors appeared during the transition. +4. While still down a node, bring the drained node back and let it synchronize, so the survivors are carrying peak traffic and feeding a recovering peer at the same time. This is the real worst case and it is the one nobody tests. +5. Compare measured surviving capacity against your invariant. If step 4 pushed latency out of SLO, your target utilization is too high or your `N` is too low. ## Operational notes -- **Utilization targets belong per journey, not per cluster.** A write-heavy - journey and a cached read journey consume very different resources on the same - node, so a single cluster-wide utilization figure will be wrong for both. -- **Re-measure after a version upgrade.** Per-node capacity is a property of a - Harper version, your component code, and your data volume. All three change. -- **Watch measured distribution, not configured weights.** Per-node request counts - from analytics are the ground truth. A misconfigured weight, a sticky session - policy, or DNS caching can leave one node doing far more work than the topology - claims. -- **Storage growth is a capacity dimension too.** Disk headroom, compaction - behavior, and backup space all scale with data volume rather than with request - rate, so they need their own thresholds. See - [compaction](/reference/v5/database/compaction). -- **On Fabric, cluster shape is managed but the invariant is unchanged.** You are - still choosing `N` and living with `F`. +- **Utilization targets belong per journey, not per cluster.** A write-heavy journey and a cached read journey consume very different resources on the same node, so a single cluster-wide utilization figure will be wrong for both. +- **Re-measure after a version upgrade.** Per-node capacity is a property of a Harper version, your component code, and your data volume. All three change. +- **Watch measured distribution, not configured weights.** Per-node request counts from analytics are the ground truth. A misconfigured weight, a sticky session policy, or DNS caching can leave one node doing far more work than the topology claims. +- **Storage growth is a capacity dimension too.** Disk headroom, compaction behavior, and backup space all scale with data volume rather than with request rate, so they need their own thresholds. See [compaction](/reference/v5/database/compaction). +- **On Fabric, cluster shape is managed but the invariant is unchanged.** You are still choosing `N` and living with `F`. ## Readiness checklist - [ ] `F` declared explicitly, and written down where change operators will see it - [ ] Per-node throughput measured with your own application code and data shape - [ ] Measurement taken with a peer synchronizing, not on an idle cluster -- [ ] Target utilization set from latency at your SLO percentile, not from request - success +- [ ] Target utilization set from latency at your SLO percentile, not from request success - [ ] `(N - F) x per-node x utilization >= peak` verified with real numbers - [ ] Measurement conditions recorded alongside the capacity figure - [ ] Maintenance policy states what happens when a drain would breach `F` @@ -246,16 +164,10 @@ representative load at your expected peak: ## Additional Resources -- [Analytics overview](/reference/v5/analytics/overview) and - [analytics operations](/reference/v5/analytics/operations) for per-node request - and latency metrics +- [Analytics overview](/reference/v5/analytics/overview) and [analytics operations](/reference/v5/analytics/operations) for per-node request and latency metrics - [Grafana integration](/fabric/grafana-integration) for dashboards on Fabric -- [`@harperdb/prometheus-exporter`](https://github.com/HarperFast/prometheus-exporter) - for self-managed metrics scraping -- [Storage tuning](/reference/v5/database/storage-tuning) for the durability and - throughput trade-offs available per database +- [`@harperdb/prometheus-exporter`](https://github.com/HarperFast/prometheus-exporter) for self-managed metrics scraping +- [Storage tuning](/reference/v5/database/storage-tuning) for the durability and throughput trade-offs available per database - [Compaction](/reference/v5/database/compaction) for storage growth behavior -- [Replication overview](/reference/v5/replication/overview) for peer - synchronization behavior when adding or returning a node -- [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx) for - scenario-level targets built on this capacity model +- [Replication overview](/reference/v5/replication/overview) for peer synchronization behavior when adding or returning a node +- [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx) for scenario-level targets built on this capacity model From b46aadc321ba7e6747b72844e82d9c7876f09626 Mon Sep 17 00:00:00 2001 From: Ethan Arrowood Date: Mon, 28 Sep 2026 12:53:39 -0600 Subject: [PATCH 08/21] human edit wip --- .../how-harper-runs-in-production.mdx | 36 +++++++++---------- 1 file changed, 17 insertions(+), 19 deletions(-) diff --git a/learn/administration/how-harper-runs-in-production.mdx b/learn/administration/how-harper-runs-in-production.mdx index f3c60cd12..0d87e64fe 100644 --- a/learn/administration/how-harper-runs-in-production.mdx +++ b/learn/administration/how-harper-runs-in-production.mdx @@ -8,12 +8,12 @@ import TabItem from '@theme/TabItem'; You have built and deployed an application, and now real users are going to depend on it. Operating Harper is not the same as operating separate application and database tiers. In Harper, one process holds your component code, the HTTP stack, the local database, and the replication client. That "collapsed stack" architecture reduces several problems you may be used to solving. -This guide is the map for the rest of the Administration track. It covers what a single node actually contains, what that means for failure and scale, and how to take an inventory of your own deployment before you design anything around it. +This guide covers what a single Harper node actually contains, and how to understand your deployment before you design anything around it. ## What You Will Learn - What a Harper node contains, and which port carries which kind of traffic -- Why the node is your unit of failure and your unit of scale, and what that removes from your operating burden as well as what it adds +- Why the node is your unit of failure and your unit of scale, and what that adds/removes from your operating burden - The three ways a Harper runbook differs from a two-tier runbook - How to inventory your service boundary with the Operations API, so later guides have something concrete to work from @@ -27,7 +27,7 @@ This guide is the map for the rest of the Administration track. It covers what a A Harper node is a single process running your components, an HTTP server, a local storage engine, and peer replication. There is no network hop between your application code and the data it reads, no separate cache tier to keep coherent, and no connection pool to tune between tiers. -Three ports carry the traffic you will operate around: +There are three main default ports: | Port | Serves | Who should reach it | | ------ | --------------------------------------------------------------------------- | --------------------------------- | @@ -35,34 +35,32 @@ Three ports carry the traffic you will operate around: | `9925` | The [Operations API](/reference/v5/operations-api/overview) | Operators and your pipeline only | | `9933` | Secure peer [replication](/reference/v5/replication/overview) | Other nodes in the cluster only | -These are documented defaults, not guarantees about your cluster. Confirm the live values rather than assuming them, because a replication port in particular can be inherited from other configuration: - -```json -{ - "operation": "get_configuration" -} -``` +These are the default values. Confirm the live values using the [`get_configuration`](TODO) operation. The replication port in particular can be inherited from other configurations. Read back `http.port`, `operationsApi.network.port`, and the `replication` block. Record what you find. Later guides in this track assume you know these numbers for your own deployment. +{/* TODO: Insert Tabs example of the get_configuration operation request with JSON parsing. CURL piped to jq and fetch with basic json access + console.log */} + :::tip -Keep `9925` off any public route. The Operations API can deploy components, read logs, and read configuration, so it is an administrative surface, not an application one. See [security overview](/reference/v5/security/overview). +Keep `9925` off any public route as the Operations API is for administrative purpsoes such as deploying components, reading logs, and managing configurations. See [security overview](/reference/v5/security/overview). ::: -## The node is your unit of failure and your unit of scale +## A node is the unit of both failure and scale -Because one process holds the runtime and the local data together, there is no internal application-to-database seam that can fail over independently. The node is the practical unit of service failure. +Since the single node is simultaneously the runtime, the database, and the application, they all scale and fail together too. This goes beyond the literal asyncronous execution model of Harper (more information at [TODO](TODO)). If a node fails for any reason, that entire node goes down; not just one part. -Start with what that removes, because it is the larger half of the trade: +This model is quite different from what separated system architectures are used to. As a result, one must think about their Harper deployment differently. -- No cross-tier network latency on data access, and no tail latency from a saturated connection pool between tiers -- No cache invalidation problem between an application cache and a database of record, because they are the same thing -- No partial-outage state where the application tier is healthy and the data tier is not, which is the failure mode that produces the most confusing incidents +- No network latency on data access, and no tail latency from a saturated connection pool between tiers +- No cache invalidation problem between an application cache and a database of record (because they are the same thing) +- No partial-outage state where the application tier is healthy and the data tier is not - One capacity number to measure and one thing to size -Then the consequence: when you lose a node, you lose a whole slice of your service, not one layer of it. This is why the first real design decision in [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) is not your peak throughput. It is how many nodes you are willing to lose at once, and whether the survivors can carry the load. +When you lose a node, you lose a whole slice of your service, not one layer of it. This is why the first real design decision in [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) is not your peak throughput. Instead, it is how many nodes you are willing to lose at once, and whether the survivors can carry the load. + +Scale works similarly. Adding capacity means adding a node that habdles both requests and data, so a scaling event is also a data movement event (as replication needs to be setup appropriately). Because of this, scaling a Harper system up takes slightly more time overall compared to scaling separted service architectures. -Scale works on the same unit. Adding capacity means adding a node that carries both request handling and data, so a scaling event is also a data movement event. That is not a problem, but it is a thing with a duration, and traffic should not arrive until it finishes. +All in all, whenever you think about your Harper application, think about it as one holistic unit. You don't have to worry about individually scaling API layer versus database nodes. Harper executes, fails, and scales all together. ## Three differences that change your runbook From cc92816509f1fd47190535833cfa87d043a2cb86 Mon Sep 17 00:00:00 2001 From: Ethan Arrowood Date: Mon, 28 Sep 2026 15:07:47 -0600 Subject: [PATCH 09/21] docs: apply the correctness items from Jeff's review Takes the factual corrections from Jeff's review and leaves the prose, structure and diagram requests for the human editing pass. Each was re-verified at the v5.2.14 tag rather than origin/main, since the guides document released 5.2 and main has moved. - Replication port. `9933` is what a fresh Harper Pro install writes into `harper-config.yaml`, but it is configured rather than constant: with no `replication` block the secure port falls back to the Operations API secure port. An in-place upgrade never backfills that block, so an upgraded node and a fresh one can replicate on different ports. Says so, and the `cluster_status` sample no longer hard-codes the number. - `host` / `urlPath`. Routing by either has existed since v5.1.0; what v5.2.0 added is declaring it on the root-config entry and the `host` parameter on `deploy_component`. Those were collapsed into one badge. - `hdb_nodes`. The registry lives in `system` and propagates with it, which is what makes gossip discovery work. Keeping it per cluster is a deliberate per-route `excludeTables` setting, which the guide now covers. - Backup storage root. The rejection rule is a database whose tables resolve to more than one storage root. A per-table `path` landing on the same root is fine, so the blanket "per-table paths are unsupported" was wrong. - `restart: true`. Each node restarts independently as it finishes applying the deploy, unsequenced and free to overlap. The previous wording implied a coordinated simultaneous restart. - `HARPER_SAFE_MODE` skips your applications and components; Harper's built-in root component still loads. Also resolves the `get_configuration` placeholder link. The remaining `[TODO](TODO)` on the asynchronous-execution sentence is left in place because its target is a drafting decision, and it is the only thing keeping the build red. Co-Authored-By: Claude Opus 5 --- learn/administration/backup-and-recovery.mdx | 6 ++--- .../how-harper-runs-in-production.mdx | 4 ++- .../administration/operating-replication.mdx | 26 ++++++++++--------- .../safe-deployments-and-rollback.mdx | 4 +-- 4 files changed, 22 insertions(+), 18 deletions(-) diff --git a/learn/administration/backup-and-recovery.mdx b/learn/administration/backup-and-recovery.mdx index 164222592..765a786b6 100644 --- a/learn/administration/backup-and-recovery.mdx +++ b/learn/administration/backup-and-recovery.mdx @@ -49,7 +49,7 @@ A backup is a whole-database copy: all tables, the transaction log, and any file Two limitations to check against your own deployment now rather than later: - **Managed backups require RocksDB.** For LMDB databases your options are `get_backup` or volume snapshots. -- **One storage root per database.** A database whose tables use per-table `path` storage configs spans multiple root stores and cannot be backed up with these operations at all. Database-level custom storage paths are fine. If someone has configured per-table paths, your backup strategy for that database does not exist yet and you need to know that before launch. +- **One storage root per database.** Backup is rejected when a database's tables resolve to more than one storage root. A database-level custom `storage.path` is fine, and so is a per-table `path` that still lands on the same root. It is a database whose tables are split across roots that cannot be backed up with these operations at all, and if you have one, your backup strategy for it does not exist yet. ::: @@ -152,7 +152,7 @@ RocksDB is single-writer, so an in-place restore requires the database to be ful Because the restore runs as a background job, a component holding the database open does not fail your request. It fails inside the job, so you find out from `get_job` rather than from the response. Harper does not track which component uses which database, so it cannot selectively stop one. :::tip -That last constraint has an escape hatch worth knowing before you need it. Setting `HARPER_SAFE_MODE` to any value starts Harper without loading any applications or components, while the database, Operations API, and HTTP server come up normally: +That last constraint has an escape hatch worth knowing before you need it. Setting `HARPER_SAFE_MODE` to any value starts Harper without loading your applications and components, while the database, Operations API, and HTTP server come up normally. Harper's own built-in root component still loads; what is skipped is everything you deployed: ```bash HARPER_SAFE_MODE=1 harper run @@ -218,7 +218,7 @@ Both measured numbers feed [Engineering RPO, RTO, and Uptime](./engineering-rpo- - [ ] Failure classes enumerated, with the mechanism that addresses each - [ ] Recovery point and recovery time stated per database, not per cluster - [ ] Storage engine confirmed per database, and the mechanism chosen accordingly -- [ ] No database in scope uses per-table storage paths, or its exclusion is known and accepted +- [ ] No database in scope has tables spanning more than one storage root, or the exclusion is known and accepted - [ ] Backup cadence matches the stated recovery point per database - [ ] Backup volume sized against the non-incremental transaction-log snapshot, and against full blob copies if the backup and blob paths are on different filesystems - [ ] An off-host copy exists in a destination that does not share a failure domain diff --git a/learn/administration/how-harper-runs-in-production.mdx b/learn/administration/how-harper-runs-in-production.mdx index 0d87e64fe..f4f36af82 100644 --- a/learn/administration/how-harper-runs-in-production.mdx +++ b/learn/administration/how-harper-runs-in-production.mdx @@ -35,7 +35,9 @@ There are three main default ports: | `9925` | The [Operations API](/reference/v5/operations-api/overview) | Operators and your pipeline only | | `9933` | Secure peer [replication](/reference/v5/replication/overview) | Other nodes in the cluster only | -These are the default values. Confirm the live values using the [`get_configuration`](TODO) operation. The replication port in particular can be inherited from other configurations. +These are the default values. Confirm the live values using the [`get_configuration`](/reference/v5/operations-api/operations#get_configuration) operation. + +Resolve the replication port rather than assuming it. `9933` is the default a fresh Harper Pro install writes into `harper-config.yaml`, but it is a configured value, not a constant. When no `replication` block is present at all, the secure replication port falls back to the Operations API secure port instead, so it lands on `9925`. That is not a hypothetical: an in-place upgrade carries the previous configuration forward and does not backfill a `replication` block, so an upgraded node with no such block replicates on a different port than a freshly installed one. Read back `http.port`, `operationsApi.network.port`, and the `replication` block. Record what you find. Later guides in this track assume you know these numbers for your own deployment. diff --git a/learn/administration/operating-replication.mdx b/learn/administration/operating-replication.mdx index c18964755..f4607e25d 100644 --- a/learn/administration/operating-replication.mdx +++ b/learn/administration/operating-replication.mdx @@ -80,17 +80,17 @@ Whether the `system` database is in your replication scope is the highest-stakes ## What moves automatically, and what does not -| Object or change | Behavior | What it means for you | -| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | -| Data mutations | Insert, update, upsert, delete, and bulk load replicate for tables in scope | Monitor connections, latency, and convergence for the databases that serve critical journeys | -| Transactions | Replicated atomically, and a transaction may span multiple tables | Tables that need to commit together must live in the same database | -| A brand new node | Requests a full copy of each in-scope database from every peer it has no resume cursor for, then enters incremental replication | Budget N transfers, not one. See the note below before sizing a join | -| A returning node | Resynchronizes automatically to catch up on the transactions it missed | This is not a full copy. A routine restart is much cheaper than a node replacement, so budget them separately | -| Component deployment | `deploy_component` replicates by default; pass `"replicated": false` to hold it to one node | Cluster-wide is the default, so single-node validation is the case you must ask for | -| Configuration | `set_configuration` supports `"replicated": true` | Never replicate node-local parameters. See the warning below | -| Users and roles | Propagate when the `system` database is in replication scope, which is the default | If you narrow scope to exclude `system`, identity must be provisioned on every node by your own automation | -| Node registry (`hdb_nodes`) | Each node rewrites its own self-record from its `harper-config.yaml` routes on restart or component reload | Scoping applied through `add_node` alone does not survive a restart. Put durable constraints in config routes | -| Destructive schema changes | `drop_database` and `drop_table` replicate by default; `drop_attribute` does not | A drop is a cluster-wide event unless you pass `"replicated": false`. Verify the result on every node | +| Object or change | Behavior | What it means for you | +| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Data mutations | Insert, update, upsert, delete, and bulk load replicate for tables in scope | Monitor connections, latency, and convergence for the databases that serve critical journeys | +| Transactions | Replicated atomically, and a transaction may span multiple tables | Tables that need to commit together must live in the same database | +| A brand new node | Requests a full copy of each in-scope database from every peer it has no resume cursor for, then enters incremental replication | Budget N transfers, not one. See the note below before sizing a join | +| A returning node | Resynchronizes automatically to catch up on the transactions it missed | This is not a full copy. A routine restart is much cheaper than a node replacement, so budget them separately | +| Component deployment | `deploy_component` replicates by default; pass `"replicated": false` to hold it to one node | Cluster-wide is the default, so single-node validation is the case you must ask for | +| Configuration | `set_configuration` supports `"replicated": true` | Never replicate node-local parameters. See the warning below | +| Users and roles | Propagate when the `system` database is in replication scope, which is the default | If you narrow scope to exclude `system`, identity must be provisioned on every node by your own automation | +| Node registry (`hdb_nodes`) | Lives in `system`, so it propagates with it. Each node rewrites its own self-record from its `harper-config.yaml` routes on restart or component reload | Scoping applied through `add_node` alone does not survive a restart. Put durable constraints in config routes, and see below if you need the registry kept per cluster | +| Destructive schema changes | `drop_database` and `drop_table` replicate by default; `drop_attribute` does not | A drop is a cluster-wide event unless you pass `"replicated": false`. Verify the result on every node | The three rows most likely to bite you are the returning node, the new node, and the node registry. @@ -102,6 +102,8 @@ The practical consequences are worth planning around. Load lands on every existi On the registry: a node's advertised record is derived from its configuration file, and it replicates. That means a topology constraint you applied imperatively is superseded the next time that node restarts or reloads components, and the node quietly goes back to advertising itself more broadly than you intended. +That propagation is what makes gossip discovery work, and since v5.2 the record each node advertises is directional, derived from its own routes, so replicating `system` no longer collapses a constrained topology into a full mesh. If you need the registry itself kept per cluster rather than shared, that is a deliberate per-route setting: `excludeTables: ['hdb_nodes']` on a route's `sendsTo` or `receivesFrom` entry stops those rows crossing that link while users, roles, and schema still propagate. Its main use is a migration bridge between clusters that should share identity but keep separate topology. + :::danger When replicating configuration, only send cluster-appropriate parameters. Replicating a node-local value such as a port, `node.hostname`, a file path, TLS material, or `replication.hostname`, `url`, or `routes` overwrites every peer's own local value. To apply a cluster-wide change safely, use `set_configuration` with `"replicated": true` for the parameter, then `restart_service` with `"replicated": true`, which restarts nodes one at a time. ::: @@ -117,7 +119,7 @@ A trimmed response, with the fields that matter: "is_enabled": true, "connections": [ { - "url": "wss://server-2.example.net:9933", + "url": "wss://server-2.example.net:", "name": "server-2.example.net", "database_sockets": [ { diff --git a/learn/administration/safe-deployments-and-rollback.mdx b/learn/administration/safe-deployments-and-rollback.mdx index 9fdee4bb9..f5b38080e 100644 --- a/learn/administration/safe-deployments-and-rollback.mdx +++ b/learn/administration/safe-deployments-and-rollback.mdx @@ -44,7 +44,7 @@ Once these are separate you get options that do not exist otherwise. You can ins This is the opposite of what most operators assume, and it is why the single-node validation pattern below has to say `false` explicitly. If you intend to touch one node, you must say so. ([`set_configuration`](/reference/v5/configuration/operations) is the deliberate exception: it replicates only when you ask.) ::: -`urlPath` mounts a component at an HTTP path. `host` (Added in: v5.2.0) serves it on a virtual hostname. Both are persisted on the component's root config entry, so they are part of the deployed state rather than a runtime toggle. See [HTTP middleware routing](/reference/v5/http/overview#middleware-routing). +`urlPath` mounts a component at an HTTP path and `host` serves it on a virtual hostname. Routing by either has been available since v5.1.0. What arrived in v5.2.0 is declaring that routing on the application's entry in the root `harper-config.yaml`, and the matching `host` parameter on `deploy_component`. Declared that way, the routing is persisted deployment state rather than a runtime toggle. See [HTTP middleware routing](/reference/v5/http/overview#middleware-routing). ## The change loop @@ -122,7 +122,7 @@ This is worth reading carefully, because the naming invites a wrong assumption. **`"restart": "rolling"`** does not restart inline either, but it is observable. It starts a replicated `restart_service` job and returns a `restartJobId` you can poll with [`get_job`](/reference/v5/operations-api/operations#get_job). That job walks the cluster **one node at a time**, waiting for each node to come back before starting the next, so the cluster keeps serving throughout. :::warning -**On a cluster, prefer `"rolling"`.** `"restart": true` is forwarded verbatim to every peer, and peers are dispatched in parallel, so a replicated deploy with `"restart": true` restarts every node in the cluster at roughly the same moment. That is a cluster-wide availability event, not a node-local one. Reserve `"restart": true` for a single-node or development instance, and use `"rolling"` anywhere you care about staying up. See [Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx). +**On a cluster, prefer `"rolling"`.** `"restart": true` is forwarded verbatim to every peer, so every node restarts independently as it finishes applying the deploy. Nothing sequences them and nothing waits, so the restarts are free to overlap. You get no rolling coordination and no guarantee that any node is still serving, which makes this a cluster-wide availability risk rather than a node-local one. Reserve `"restart": true` for a single-node or development instance, and use `"rolling"` anywhere you care about staying up. See [Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx). ::: Two more consequences for your pipeline. Neither restart mode reports completion in the deploy response, so if you need to know the new code is live, poll the rolling restart job or probe the nodes themselves. And a failed restart does not fail the deploy: the component is installed and replicated either way, so your pipeline needs to check both outcomes separately rather than assuming one implies the other. From 0f403ab2745b1506ff1eb9053521f615bda54cf6 Mon Sep 17 00:00:00 2001 From: Dawson Toth Date: Tue, 13 Oct 2026 21:09:42 -0400 Subject: [PATCH 10/21] Document Fabric load balancing, status-check, and taking an unhealthy node out of rotation Add a Learn guide for applications on Fabric that feed their own health into the node's availability record, which Fabric's status-check component serves to the load balancer at GET /status. It covers when to do it, six safety rules (hysteresis; never undoing Fabric's or an operator's write, tracked by the record's __updatedtime__ in a non-replicated marker table and re-read before every write; never taking the last node out; a recheck right after going out; surviving restarts; warn logging and an off switch), a generic sample component, a local drill against a throwaway Harper with the real status-check component and the CLI's harper login, and caveats, including the narrow windows a set_status without compare-and-swap leaves open. Add a Fabric page for status-check: what GET /status answers, that POST and DELETE need a super user, that the record is per node, persisted and not replicated, and how to take a node out with harper login and harper set_status, locally or from GitHub Actions with OIDC, as a role whose operations list names only set_status and get_status (v5.3.0). It warns not to deploy the @harperdb/status-check npm package on Fabric. In the set_status reference, say that set_status stores only status, that Harper core does not act on status values, and that on Fabric the availability record decides routing. Co-Authored-By: Claude Opus 5.5 --- fabric/load-balancing.md | 79 +++ .../taking-a-node-out-of-rotation.mdx | 627 ++++++++++++++++++ reference/operations-api/operations.md | 6 +- sidebarsFabric.ts | 5 + sidebarsLearn.ts | 5 + 5 files changed, 720 insertions(+), 2 deletions(-) create mode 100644 fabric/load-balancing.md create mode 100644 learn/developers/taking-a-node-out-of-rotation.mdx diff --git a/fabric/load-balancing.md b/fabric/load-balancing.md new file mode 100644 index 000000000..c6da0455c --- /dev/null +++ b/fabric/load-balancing.md @@ -0,0 +1,79 @@ +--- +title: Load Balancing and Node Availability +--- + +# Load Balancing and Node Availability + +A Harper Fabric cluster serves your applications from one hostname, the cluster's Fabric hostname (for example `my-cluster.my-org.harperfabric.com`) or a [custom domain](/fabric/custom-domains) pointed at it. Fabric's load balancer picks a node of the cluster for each new connection to that hostname. This page covers how it decides which nodes may receive traffic, and how to read or change that decision for one node. + +## How Fabric decides where traffic goes + +The load balancer checks each node with a `GET /status` request to that node. A node that answers `200` may receive new connections. A node that answers anything else, or does not answer, receives none until it answers `200` again. + +`/status` is served by a component that Fabric deploys to every node of the cluster, named `status-check`. Its source is public: [HarperFast/status-check-fabric](https://github.com/HarperFast/status-check-fabric). It answers from a single record on the node, the `availability` status, which Harper's [`set_status` and `get_status`](/reference/v5/operations-api/operations#set_status--get_status--clear_status) operations write and read: + +| `availability` record | `GET /status` answers | +| --------------------- | --------------------- | +| `Available` | `200` | +| `Unavailable` | `404` | +| No record | `404` | + +Fabric writes this record itself. For example, it takes a node out while it upgrades that node, and puts it back afterwards. + +### Each node has its own record + +Harper keeps status records in the `system.hdb_status` table, which it creates with replication turned off. Every node therefore has its own `availability` record: writing it on one node changes only that node. The record is stored on disk, so it keeps its value when Harper restarts. + +Because each node answers for itself, a request meant for one node has to reach that node. Through the cluster hostname, the load balancer decides which node receives it. Use the node's own hostname, which only that node answers. + +## The `status-check` component + +### `GET /status` + +Unauthenticated. + +- `200` with the body `"Status endpoint is reporting for duty."` when the node's `availability` record is `Available`. +- `404` with an error titled `Status endpoint is reporting downtime.` otherwise, including when the node has no `availability` record. + +```bash +curl -i https:///status +``` + +### `POST /status` + +Sets the node's `availability` record to `Available`. Requires a `super_user`: the component answers `204` on success, `401` to a request without credentials, and `403` to any other user. + +### `DELETE /status` + +Sets the node's `availability` record to `Unavailable`. Authorized the same way as `POST`. + +To change the record yourself, use the Harper CLI rather than these routes, as [Taking a node out of rotation](#taking-a-node-out-of-rotation) shows. It needs no password on the command line, and from v5.3.0 the user it runs as does not have to be a `super_user`. + +:::warning +Fabric routes traffic by this component. Do not remove it, and do not deploy another resource that serves `/status`. +::: + +:::note +This is not the [`@harperdb/status-check`](https://github.com/HarperFast/status-check) npm package, which keeps its own `hdb_status` table in the `data` database. Fabric does not read that table, and the package also serves `/status`, so do not deploy it on a Fabric cluster. On a self-managed cluster, [Health Checks and Traffic Admission](/learn/administration/health-checks-and-traffic-admission) covers it. +::: + +## Taking a node out of rotation + +To stop new traffic reaching a node, set its `availability` record to `Unavailable` with the [`set_status`](/reference/v5/operations-api/operations#set_status--get_status--clear_status) operation. To let traffic back, set it to `Available`. Send each command to the node's own hostname. From your own machine, [log in](/reference/v5/cli/authentication#method-1-persistent-login-recommended-for-local-development) once and the CLI keeps the token: + +```bash +harper login https:// +harper set_status id=availability status=Unavailable target=https:// +harper get_status id=availability target=https:// +harper set_status id=availability status=Available target=https:// +``` + +`get_status` returns the record, including `__updatedtime__`, the time of its last write. + +From a GitHub Actions workflow, run the same commands with [workload identity (OIDC)](/reference/v5/cli/authentication#workload-identity-oidc), so the workflow holds no Harper credential at all. [Deploying from a CI/CD Pipeline](/learn/developers/deploying-from-ci) shows how to set up the trust policy and the user it maps to. + +That user does not need to be a `super_user`. From v5.3.0, a role whose [`operations`](/reference/v5/users-and-roles/overview#operation-permissions) list names only `set_status` and `get_status` can run those two operations and nothing else. + +Connections already open to the node stay open: the load balancer only decides where new connections go. Fabric writes the same record, so a value you set can be replaced by Fabric's own operations on the node. + +An application can also feed its own health into this record, so that a node taking errors leaves rotation while its peers keep serving. See [Taking an Unhealthy Node Out of Rotation](/learn/developers/taking-a-node-out-of-rotation). diff --git a/learn/developers/taking-a-node-out-of-rotation.mdx b/learn/developers/taking-a-node-out-of-rotation.mdx new file mode 100644 index 000000000..b60dc3522 --- /dev/null +++ b/learn/developers/taking-a-node-out-of-rotation.mdx @@ -0,0 +1,627 @@ +--- +title: Taking an Unhealthy Node Out of Rotation +--- + +Harper Fabric's load balancer sends traffic to every node of a cluster whose `GET /status` answers `200`. That answer comes from a per-node record that Fabric maintains, so a node keeps receiving traffic while it is running, even when your application on that node cannot serve it. This guide shows how an application can write that record too, from its own health: a node that keeps failing takes itself out of rotation while another node serves, and puts itself back once it recovers, leaving alone what Fabric or an operator has set, outside the narrow windows described under [Caveats](#caveats). + +## What You Will Learn + +- How Fabric's load balancing uses the `status-check` component and the node's `availability` record +- When an application should take its node out of rotation, and when it should not +- The rules that keep an automated loop from fighting Fabric, flapping, or taking the last node out +- A small, generic component that implements those rules +- How to test it locally against a throwaway Harper with the real `status-check` component loaded + +## Prerequisites + +- A Harper Fabric cluster with at least two nodes, and a [Harper application](./harper-applications-in-depth.mdx) deployed to it +- Each node's own hostname, which only that node answers, as opposed to the cluster hostname +- For local testing: Node.js, and npm to install a throwaway copy of Harper + +## How Fabric decides where traffic goes + +Fabric deploys a component named `status-check` to every node. It serves `GET /status`, unauthenticated, and answers `200` when the node's `availability` record is `Available`, and `404` when the record is `Unavailable` or missing. The load balancer only sends new connections to nodes that answer `200`. See [Load Balancing and Node Availability](/fabric/load-balancing) for the component's routes. + +The `availability` record is read and written with the [`get_status` and `set_status`](/reference/v5/operations-api/operations#set_status--get_status--clear_status) operations. Three properties of it shape everything below: + +- **Each node has its own.** Harper stores it in `system.hdb_status`, which does not replicate, so a write on one node changes only that node. +- **It holds nothing but a status.** `set_status` stores the `status` value and drops any other field, so the record cannot also record who wrote it. `get_status` returns `__updatedtime__`, the time of the last write, which every write moves. +- **Fabric writes it too.** Fabric takes a node out while it upgrades it and puts it back afterwards. Operators can do the same with the Harper CLI, from their own machine or from a CI workflow; see [Taking a node out of rotation](/fabric/load-balancing#taking-a-node-out-of-rotation). + +## When to do this + +Take a node out when it fails in a way that is local to that node and that the load balancer cannot see, because `/status` keeps answering `200`. For example: + +- A dependency only this node reaches, such as a regional service, is unreachable from it. +- Something the application loads per node, such as a cache, a model, or a data file, failed to load here. +- This node has fallen behind on data that its requests need to be current. + +Do not use it for: + +- **Failures every node shares**, such as a central origin being down. Taking nodes out does not help, and the rule against taking the last node out stops it anyway. Leave shared dependencies out of the health check, or give them their own handling. +- **A node that is down altogether.** If Harper is not running, the loop is not either, but the node's `/status` does not answer, so the load balancer already leaves it out. +- **Planned maintenance.** Set the record yourself, or let Fabric do it. + +## The rules + +A loop that writes the `availability` record shares it with Fabric and with operators, and its mistakes take capacity out of the cluster. These rules keep it safe. Each one is in the [sample code](#build-it). + +### Wait before acting, longer before coming back + +The loop looks every 5 seconds. It takes the node out only after the health check has failed on every look for 30 seconds, and puts it back only after the check has passed on every look for 60 seconds. One look the other way restarts the count. + +**Why:** a node fails its health check for a few seconds while it starts, for example when a deploy restarts it. Waiting longer than that means a deploy never takes a node out. Requiring longer to come back than to go out keeps a node whose health check is flaky from flapping in and out of rotation. + +### Never undo someone else's write + +The loop only puts back a node it took out itself, and only while the record is still the one it wrote. Before it writes `Unavailable`, it stores a marker in a small table of its own. After the write, it adds the record's `__updatedtime__` to the marker. While the node is out, the record still having that `__updatedtime__` proves nobody has written it since. If anybody has (Fabric, or an operator putting the node back by hand), the loop leaves the record as they set it. It replaces its marker with a note that the record is theirs, and does not take the node out again until its health check has passed for the full 60 seconds. The note persists, so a restart, such as the one an upgrade brings, does not clear it. `set_status` has no compare-and-swap, so these checks leave small windows; [Caveats](#caveats) lists them. + +It never touches a node that is `Unavailable` without its marker, or a node that has no `availability` record. Asking the other nodes can take a few seconds, so the loop reads the record again just before each write, and leaves the write to its next look if anyone has written the record in between. + +**Why:** Fabric and operators have reasons the application cannot see. A loop that put back a node Fabric had taken out for an upgrade would send traffic to a node in the middle of one. The marker has to live outside the record, because `set_status` keeps only the status. + +When someone puts a node in while it is failing, the 30 seconds count from that write, not from when the failing began. Otherwise a node Fabric has just put back after an upgrade would be taken straight out again. + +### Never take the last node out + +Before it goes out, the loop asks every other node whether it is serving: its health check passes, and its `/status` answers `200`. It asks each one at that node's own hostname. If no other node is serving, it stays in and logs a warning. + +While it is out, it keeps asking. If no other node has served for 30 seconds, it comes back even though it is failing. + +**Why:** a node that fails some requests is better than no node at all. The question has to go to each node's own hostname: through the cluster hostname, the load balancer might send it to the node that is asking. + +### Look again straight after going out + +The loop looks again 1.5 seconds after it goes out. If no other node is serving at that look, or at any look in its first 10 seconds out, it comes straight back, and does not go out again until it restarts. + +**Why:** two nodes that start failing together can each see the other serving, and go out at the same moment. This catches that, and anything else that takes the other nodes out together with this one, such as a status record that replicates when it should not. + +### Survive a restart + +The marker table persists, so a loop that restarts while its node is out still knows the node is its to put back, and one that restarts while it leaves someone else's write in place still knows to leave it. The loop writes the marker before the record, and deletes it if the write fails. If the process stops between the two writes, the next look finds a marker without an `__updatedtime__`, and claims the record only if it went `Unavailable` within 5 seconds of the marker. + +**Why:** the `availability` record also persists. A node the loop took out stays out across a restart, and without the marker nothing would ever put it back. + +### Say what it does, and keep a switch + +Every write the loop makes is logged at `warn` with the reason, and so is every time it holds back for one of the rules above: staying in because no other node is serving, or leaving someone else's write in place. One environment variable turns the loop off. + +**Why:** a node leaving rotation changes where users' traffic goes. Whoever investigates later needs to find out why from the log, and needs a way to stop the loop without a code change. + +## Build it + +The sample is an application component with four parts: + +- `config.yaml` +- a marker table in `schema.graphql` +- a health check and a `/Health` endpoint in `resources.js` +- the loop in `availability.js` + +Replace the health check with your own. The rest is generic. + +### Configuration + +```yaml +# config.yaml +loadEnv: + files: '.env' +rest: true +graphqlSchema: + files: 'schema.graphql' +jsResource: + files: 'resources.js' +``` + +The loop reads its settings from the environment. [`loadEnv`](/reference/v5/environment-variables/overview) has to come first, so the variables are set before `resources.js` loads. + +```bash +# .env +# Every node's own base URL, this node's included, separated by commas. +AVAILABILITY_NODES=https://node-1.example.com,https://node-2.example.com +# Set to "off" to stop the loop. +AVAILABILITY_LOOP=on +# Used by this sample's health check; replace it with your own. +HEALTH_DEPENDENCY_URL=https://dependency.example.com/health +``` + +### The marker table + +```graphql +# schema.graphql +# What this application has done to this node's availability record, so that a +# restart doesn't forget it. Never replicated: like the record, it describes one node. +type RotationMark @table(replicate: false) { + id: ID @primaryKey + outAt: Float # when the application took the node out (ms since epoch) + statusAt: Float # the availability record's __updatedtime__ after that write + reason: String + asideAt: Float # when it found someone else had written the record since +} +``` + +[`replicate: false`](/reference/v5/replication/overview) keeps each node's marker on that node, so every node uses the same row id, `this-node`. A row with `outAt` means the loop has the node out; a row with `asideAt` means it is leaving someone else's write in place. The table has no `@export`, so it is not reachable over REST. + +### The health check and `/Health` + +```javascript +// resources.js +import { Resource, server } from 'harper'; +import { startAvailabilityLoop } from './availability.js'; + +// What "this node can serve" means for your application. Check what can fail +// on this node alone; this example asks a dependency the node reaches. +const DEPENDENCY_URL = process.env.HEALTH_DEPENDENCY_URL; + +async function checkDependency() { + if (!DEPENDENCY_URL) return { ok: true }; + try { + const response = await fetch(DEPENDENCY_URL, { + signal: AbortSignal.timeout(2000), + }); + return response.ok ? { ok: true } : { ok: false, reason: `dependency answered ${response.status}` }; + } catch (error) { + return { ok: false, reason: `dependency unreachable: ${error.message}` }; + } +} + +// At most one check every HEALTH_CACHE_MS, its answer shared, so traffic to the +// unauthenticated /Health can't multiply the requests this node makes. +const HEALTH_CACHE_MS = 2000; +let latest; +function checkHealth() { + if (!latest || Date.now() - latest.at >= HEALTH_CACHE_MS) { + latest = { at: Date.now(), result: checkDependency() }; + } + return latest.result; +} + +// GET /Health: unauthenticated, so the other nodes can ask it. `node` lets the +// availability loop recognize this node's own address in its list of nodes. The +// failure reason stays in the loop's log: anyone can read this response. +export class Health extends Resource { + allowRead() { + return true; + } + async get() { + const { ok } = await checkHealth(); + return { + status: ok ? 200 : 503, + headers: { 'Cache-Control': 'no-store' }, + data: { ok, node: server.hostname }, + }; + } +} + +startAvailabilityLoop(checkHealth); +``` + +Keep the check fast and bounded by a timeout: the loop runs it every 5 seconds, and the other nodes call `/Health` before they go out. The sample shares one answer for 2 seconds, so requests to `/Health` can't multiply the checks. `/Health` answers anyone, so it returns only whether the node is healthy and its name; the reason stays in the loop's log. + +### The loop + +```javascript +// availability.js +import { server, tables } from 'harper'; + +const RECORD = 'availability'; +const ROW = 'this-node'; // RotationMark doesn't replicate, so each node has this one row +const TICK_MS = 5_000; // how often to look +const RECHECK_MS = 1_500; // how soon to look again after going out +const OUT_AFTER_MS = 30_000; // failing on every look this long before going out +const BACK_AFTER_MS = 60_000; // passing on every look this long before coming back +const ALONE_MS = 30_000; // out, with no other node serving this long: come back anyway +const JUST_OUT_MS = 10_000; // no other node serving this soon after going out: come back now +const WRITE_MS = 5_000; // the loop's own Unavailable lands this soon after its row +const HEALTH_MS = 5_000; // how long the health check has to answer +const ASK_MS = 3_000; // how long another node has to answer + +const log = logger.withTag('availability'); +const { RotationMark } = tables; + +// With authorization off (the default), a component can call these without a user. +const getStatus = () => server.operation({ operation: 'get_status', id: RECORD }); +const setStatus = (status) => server.operation({ operation: 'set_status', id: RECORD, status }); +const writtenAt = (record) => Number(record?.__updatedtime__) || 0; +const secs = (ms) => `${Math.round(ms / 1000)} s`; +const messageOf = (error) => error?.message ?? String(error); + +// The health check, bounded: one that throws or doesn't answer counts as failing. +function healthNow(checkHealth) { + let timer; + const timeout = new Promise((resolve) => { + timer = setTimeout(resolve, HEALTH_MS, { + ok: false, + reason: 'health check timed out', + }); + }); + return Promise.race([Promise.resolve().then(checkHealth), timeout]) + .catch((error) => ({ + ok: false, + reason: `health check threw: ${messageOf(error)}`, + })) + .finally(() => clearTimeout(timer)); +} + +async function ask(base, path) { + try { + const response = await fetch(new URL(path, base), { + signal: AbortSignal.timeout(ASK_MS), + }); + return { + status: response.status, + body: await response.json().catch(() => null), + }; + } catch (error) { + return { status: 0, error: messageOf(error) }; + } +} + +// Which other nodes are serving: their /Health passes, and their /status says +// the load balancer may send them traffic. Each is asked at its own address. +async function otherNodesServing(nodes, me) { + const answers = await Promise.all( + nodes.map(async (base) => { + if (URL.canParse(base) && new URL(base).hostname === me) return null; // this node + const health = await ask(base, '/Health'); + if (health.body?.node === me) return null; // this node, under another name + if (health.status !== 200) return { base, why: `/Health ${health.status || health.error}` }; + const status = await ask(base, '/status'); + if (status.status !== 200) return { base, why: `/status ${status.status || status.error}` }; + return { base, serving: true }; + }) + ); + const others = answers.filter(Boolean); + return { + serving: others.filter((node) => node.serving).map((node) => node.base), + why: + others + .filter((node) => !node.serving) + .map((node) => `${node.base} ${node.why}`) + .join('; ') || 'no other node listed', + }; +} + +export function startAvailabilityLoop(checkHealth) { + if (server.workerIndex !== 0) return; + // Every node's own base URL, this one's included. Never the cluster's shared + // hostname: the load balancer could route the question back to this node. + const nodes = (process.env.AVAILABILITY_NODES ?? '') + .split(',') + .map((url) => url.trim()) + .filter(Boolean); + if (process.env.AVAILABILITY_LOOP === 'off' || nodes.length < 2) { + log.notify?.('loop off'); + return; + } + const me = server.hostname; + let failingSince = null; + let passingSince = null; + let aloneSince = null; + let stopped = false; // the others stopped serving as this node went out + let warnedAlone = false; + + async function goBack(record, why) { + // Leave it to the next look if someone has written the record since this one read it. + if (writtenAt(await getStatus()) !== writtenAt(record)) return; + await setStatus('Available'); + log.warn?.(`Available, back in rotation: ${why}`); + await RotationMark.delete(ROW); + aloneSince = null; + warnedAlone = false; + } + + async function tick() { + const now = Date.now(); + const health = await healthNow(checkHealth); + if (health.ok) { + failingSince = null; + passingSince ??= now; + warnedAlone = false; + } else { + passingSince = null; + failingSince ??= now; + } + const passedFor = passingSince === null ? 0 : now - passingSince; + + const record = await getStatus(); + const row = await RotationMark.get(ROW); + + if (row?.asideAt) { + // Someone else wrote the record after this loop took the node out. It is + // theirs until the health check has passed for BACK_AFTER_MS. + if (passedFor >= BACK_AFTER_MS) await RotationMark.delete(ROW); + return; + } + + if (row?.outAt) { + // Out by this loop's write, unless someone has written the record since. + // statusAt is 0 if the process stopped between the loop's two writes; the + // record is then the loop's only if it went Unavailable right after the row + // was written (Harper's clock can trail Date.now() slightly). + const unavailable = record?.status === 'Unavailable'; + const at = writtenAt(record); + const ours = + unavailable && (row.statusAt ? at === row.statusAt : at >= row.outAt - 1_000 && at <= row.outAt + WRITE_MS); + if (!ours) { + if (!unavailable && !row.statusAt && writtenAt(record) < row.outAt) { + await RotationMark.delete(ROW); // the loop's write never happened + return; + } + await RotationMark.put({ id: ROW, asideAt: now }); + log.warn?.( + `${record?.status ?? 'no record'}, written by someone else after this node took itself out: leaving it` + ); + return; + } + if (!row.statusAt) { + await RotationMark.put({ + id: ROW, + outAt: row.outAt, + statusAt: writtenAt(record), + reason: row.reason, + }); + } + if (passedFor >= BACK_AFTER_MS) return goBack(record, `health has passed for ${secs(passedFor)}`); + const others = await otherNodesServing(nodes, me); + if (others.serving.length) { + aloneSince = null; + return; + } + if (now - row.outAt < JUST_OUT_MS) { + stopped = true; // as if they went out with it: stay in until this node restarts + return goBack( + record, + `no other node is serving just after this one went out (${others.why}); not going out again until Harper restarts` + ); + } + aloneSince ??= now; + if (now - aloneSince >= ALONE_MS) { + return goBack(record, `no other node has served for ${secs(now - aloneSince)} (${others.why})`); + } + return; + } + + if (health.ok || stopped) return; + if (record?.status !== 'Available') return; // out by someone else, or never put in + // Count from when failing began or from the record's last write, whichever + // is later: a node someone has just put in gets the full time again. + if (now - Math.max(failingSince, writtenAt(record)) < OUT_AFTER_MS) return; + const others = await otherNodesServing(nodes, me); + if (!others.serving.length) { + if (!warnedAlone) + log.warn?.(`health failing (${health.reason}), but no other node is serving (${others.why}): staying in`); + warnedAlone = true; + return; + } + // Leave it to the next look if someone wrote the record while the others were asked. + if (writtenAt(await getStatus()) !== writtenAt(record)) return; + // The row first: if the process stops between the two writes, the next + // look still knows whether the Unavailable record is this loop's. + const outAt = Date.now(); + await RotationMark.put({ + id: ROW, + outAt, + statusAt: 0, + reason: health.reason, + }); + try { + await setStatus('Unavailable'); + } catch (error) { + await RotationMark.delete(ROW); + throw error; + } + log.warn?.( + `Unavailable, out of rotation: health has failed for ${secs(now - failingSince)} (${health.reason}); serving: ${others.serving.join(', ')}` + ); + await RotationMark.put({ + id: ROW, + outAt, + statusAt: writtenAt(await getStatus()), + reason: health.reason, + }); + return 'out'; + } + + let errorLoggedAt = 0; + async function loop() { + let next = TICK_MS; + try { + if ((await tick()) === 'out') next = RECHECK_MS; + } catch (error) { + if (Date.now() - errorLoggedAt > 60_000) { + errorLoggedAt = Date.now(); + log.warn?.(`could not check: ${messageOf(error)}`); + } + } finally { + setTimeout(loop, next).unref(); + } + } + setTimeout(loop, TICK_MS).unref(); + log.notify?.(`loop on; asks ${nodes.join(' ')}`); +} +``` + +A few details: + +- **One loop per node.** Every worker thread loads the component, so the loop starts only where `server.workerIndex` is `0`. An [isolated application](/reference/v5/components/applications#isolated-applications) runs in one worker, whose index is `0`. +- **A health check that throws, or does not answer within 5 seconds, counts as failing.** Either way the loop goes on looking. +- **No user needed.** [`server.operation()`](/reference/v5/http/api#serveroperationoperation-context-authorize) applies no authorization unless you pass `authorize: true`, so the loop can call `set_status` without a user or a role. The `status-check` component calls it the same way. +- **Recognizing itself.** `AVAILABILITY_NODES` lists every node, so the same `.env` works on all of them. The loop skips the address whose `/Health` reports this node's [`server.hostname`](/reference/v5/http/api#serverhostname). +- **The numbers are a starting point.** 30 seconds out and 60 seconds back suit a node that restarts in a few seconds. If your nodes take longer to start, raise `OUT_AFTER_MS` above that time. + +## Test it locally + +Run the loop against a throwaway Harper with Fabric's real `status-check` component, a stand-in for a second node, and a stand-in for the dependency. Use scratch values for the home directory, root path, credentials, and ports, so the test touches nothing of an existing Harper install. + +### Start a throwaway Harper + +In an empty directory: + +```bash +npm install harper@5.3.2 +export HOME="$PWD/home" ROOTPATH="$PWD/root" TC_AGREEMENT=yes \ + HDB_ADMIN_USERNAME=admin HDB_ADMIN_PASSWORD=scratch-password \ + OPERATIONSAPI_NETWORK_PORT=9965 HTTP_PORT=9966 +mkdir -p "$HOME" +npx harper install +``` + +`npx harper` runs the copy you just installed. The scratch `HOME` keeps `harper install` from finding an existing installation in your own home directory, and `harper login` from touching your own saved credentials. + +Point the application at the stand-ins. Variables already set in Harper's environment take precedence over the application's `.env`, because `loadEnv` does not override them by default. Then start Harper: + +```bash +export AVAILABILITY_NODES=http://127.0.0.1:9966,http://127.0.0.1:9967 \ + HEALTH_DEPENDENCY_URL=http://127.0.0.1:9968/ +npx harper start +``` + +[Log in](/reference/v5/cli/authentication#method-1-persistent-login-recommended-for-local-development) with the admin username and password you chose. The CLI keeps the token, and the commands below use it: + +```bash +npx harper login http://127.0.0.1:9965 +``` + +### Start the stand-ins + +The application's health check and its list of nodes point at two stand-ins. Save this as `stand-ins.mjs` and run it with `node stand-ins.mjs` in a second terminal. It answers as another node on port 9967 and as the dependency on port 9968, and you change their answers with `POST` requests: + +```javascript +// Stand-ins for a local test: another node on 9967, and the dependency on 9968. +// Flip them with POSTs: /Health/503 or /status/404 on the node, /down or /up on the dependency. +import http from 'node:http'; + +const node = { Health: 200, status: 200 }; +let dependencyUp = true; + +http + .createServer((request, response) => { + const [, path, code] = request.url.split('/'); + if (request.method === 'POST' && path in node) node[path] = Number(code); + const body = path === 'Health' ? { ok: node.Health === 200, node: 'other-node.localhost' } : null; + response.writeHead(node[path] ?? 404, { 'content-type': 'application/json' }).end(JSON.stringify(body)); + }) + .listen(9967, '127.0.0.1'); + +http + .createServer((request, response) => { + if (request.method === 'POST') dependencyUp = request.url === '/up'; + response.writeHead(dependencyUp ? 200 : 503).end(); + }) + .listen(9968, '127.0.0.1'); +``` + +### Deploy `status-check` and the application + +Deploy Fabric's `status-check` component from its public repository, then the application from its directory: + +```bash +npx harper deploy project=status-check package=HarperFast/status-check-fabric restart=true +cd my-app && npx harper deploy restart=true && cd .. +``` + +A fresh install has no `availability` record, so `/status` answers `404`, and the loop leaves the node alone. Put the node in, as Fabric would: + +```bash +npx harper set_status id=availability status=Available +curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:9966/status +``` + +That prints `200`. Harper's log, `root/log/hdb.log`, shows that the loop is on. The log lines in this guide omit each line's timestamp and thread: + +``` +[notify] [availability]: loop on; asks http://127.0.0.1:9966 http://127.0.0.1:9967 +``` + +### Run the drill + +Follow the loop in the log while you work through the steps below: + +```bash +tail -f root/log/hdb.log | grep availability +``` + +1. **Out.** Make the health check fail: + + ```bash + curl -X POST http://127.0.0.1:9968/down + ``` + + About 30 seconds later, `/status` answers `404`, `npx harper get_status id=availability` reports `Unavailable`, and the log says why: + + ``` + [warn] [availability]: Unavailable, out of rotation: health has failed for 30 s (dependency answered 503); serving: http://127.0.0.1:9967 + ``` + +2. **Back.** Let it recover with `curl -X POST http://127.0.0.1:9968/up`. About 60 seconds later, `/status` answers `200` again: + + ``` + [warn] [availability]: Available, back in rotation: health has passed for 60 s + ``` + +3. **Never the last node.** Take the other node out, then fail the health check: + + ```bash + curl -X POST http://127.0.0.1:9967/status/404 + curl -X POST http://127.0.0.1:9968/down + ``` + + The node stays in, and warns once: + + ``` + [warn] [availability]: health failing (dependency answered 503), but no other node is serving (http://127.0.0.1:9967 /status 404): staying in + ``` + + Put the other node back with `curl -X POST http://127.0.0.1:9967/status/200`, and this one goes out at its next look. + +4. **Someone else's write.** While the loop has the node out, put it back by hand, as an operator would: + + ```bash + npx harper set_status id=availability status=Available + ``` + + The loop leaves it in, even though the health check still fails and even if Harper restarts, until the check has passed for 60 seconds: + + ``` + [warn] [availability]: Available, written by someone else after this node took itself out: leaving it + ``` + + The same happens when Fabric takes the node out itself. Write `status=Unavailable` while the loop has the node out, then let the dependency recover: the node stays out. + +5. **A restart.** With the node out and the dependency back up, restart Harper, in the terminal where you exported the variables: + + ```bash + npx harper stop + npx harper start + ``` + + The record is still `Unavailable` after the restart. The loop finds its marker, and puts the node back about 60 seconds after Harper starts. + +6. **Straight back.** Fail the health check and wait for the node to go out, then within five seconds take the other node out with `curl -X POST http://127.0.0.1:9967/status/404`. At its next look, the loop comes back, and it does not go out again until Harper restarts: + + ``` + [warn] [availability]: Available, back in rotation: no other node is serving just after this one went out (http://127.0.0.1:9967 /status 404); not going out again until Harper restarts + ``` + +When you are done, stop Harper with `npx harper stop`, stop the stand-ins, and delete the directory. + +## Caveats + +- **Open connections stay.** The load balancer only decides where new connections go. Clients already connected to a node that goes out, including WebSocket and MQTT clients, stay on it until they disconnect. +- **Turning the loop off leaves its node as it is.** The `availability` record persists, so a node the loop had taken out stays out after you set `AVAILABILITY_LOOP=off` or remove the application. Before you do either, check each node's `/status`, and put back, with `harper set_status`, any node whose `RotationMark` row has an `outAt`. +- **The health check decides.** A check that fails when the node could serve takes capacity out for nothing. A check that depends on something every node shares makes every node fail together, and then the loop keeps them all in. +- **A node that cannot be asked counts as not serving.** If one node cannot reach another at its own hostname, or that node's `/Health` needs credentials, the loop treats it as not serving and keeps its own node in. The loop errs toward keeping nodes in, never toward taking them out. +- **Someone else's write pauses the loop.** After Fabric or an operator puts back a node the loop took out, the loop leaves that node alone until its health check has passed for 60 seconds, however long the node keeps failing. +- **"Serving" is what this node sees.** A node that answers at its own hostname counts as serving even if the load balancer cannot reach it, and so does a node that is still running but no longer in the cluster. Keep `AVAILABILITY_NODES` current. +- **The ownership check has small gaps.** The loop reads the record again just before each write, and after a failed or interrupted write it only claims an `Unavailable` written within 5 seconds of its marker. So someone else's write can be taken for the loop's if it lands a few milliseconds after the loop's own write, or within those 5 seconds when a restart interrupted the loop. Coming back has the same kind of gap: a write landing between the loop's last read and its own `Available` is overwritten. The reverse can happen too: if the loop's own write took longer than 5 seconds and a restart interrupted it, the loop leaves the node out and logs the write as someone else's. After a restart that hit a node mid-write, check its `/status`. +- **Operation logging.** At log level `info` or more verbose, Harper logs every operation call, including the loop's `get_status` every 5 seconds. + +## Additional Resources + +- [Load Balancing and Node Availability](/fabric/load-balancing) - the `status-check` component's routes and authorization +- [`set_status`, `get_status`, and `clear_status`](/reference/v5/operations-api/operations#set_status--get_status--clear_status) - the operations behind the `availability` record +- [`server.operation()`](/reference/v5/http/api#serveroperationoperation-context-authorize) - calling operations from a component +- [Replication: excluding a table](/reference/v5/replication/overview) - `@table(replicate: false)` +- [Environment Variables](/reference/v5/environment-variables/overview) - `loadEnv` +- [Logging API](/reference/v5/logging/api) - `logger` and tagged loggers +- [HarperFast/status-check-fabric](https://github.com/HarperFast/status-check-fabric) - the `status-check` component's source diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 4dd8ee857..b5b6796a6 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -1779,11 +1779,13 @@ Valid attribute names are `system`, `time`, `cpu`, `memory`, `disk`, `network`, ### `set_status` / `get_status` / `clear_status` -Manage application status values. The status types are fixed — `primary`, `maintenance`, `availability` — while the value you store against each is yours to define (availability only accepts `'Available'` or `'Unavailable'`). +Manage application status values. The status types are fixed — `primary`, `maintenance`, `availability` — while the value you store against each is yours to define (availability only accepts `'Available'` or `'Unavailable'`). `set_status` stores only `status`; any other field in the request is ignored. Values are **persisted**, stored in the `hdb_status` table in the `system` database, so they survive a restart. That table is declared `replicate: false`, so a status set on one node stays on that node and does not propagate to peers — which is what makes these values usable as node-local coordination markers. -These are a coordination primitive for your own automation: Harper stores and returns them but does not change its own behavior based on them, so they are not a substitute for a real health check. A `get_status` call with no `id` additionally returns Harper-derived state, including a `restartRequired` flag that tracks pending component and code restarts (not configuration changes). +These are a coordination primitive for your own automation: Harper core stores and returns them but does not change its own behavior based on them, so they are not a substitute for a real health check. A `get_status` call with no `id` additionally returns Harper-derived state, including a `restartRequired` flag that tracks pending component and code restarts (not configuration changes). + +On Harper Fabric, the `status-check` component answers the load balancer from the `availability` record, so that record decides whether a node receives traffic. See [Load Balancing and Node Availability](/fabric/load-balancing). ```json { "operation": "set_status", "id": "primary", "status": "active" } diff --git a/sidebarsFabric.ts b/sidebarsFabric.ts index 529e87780..61413a3f8 100644 --- a/sidebarsFabric.ts +++ b/sidebarsFabric.ts @@ -32,6 +32,11 @@ const sidebarsFabric: SidebarsConfig = { id: 'custom-domains', label: 'Custom Domains', }, + { + type: 'doc', + id: 'load-balancing', + label: 'Load Balancing & Node Availability', + }, { type: 'doc', id: 'managing-applications', diff --git a/sidebarsLearn.ts b/sidebarsLearn.ts index 721b1690c..6282dbd6f 100644 --- a/sidebarsLearn.ts +++ b/sidebarsLearn.ts @@ -71,6 +71,11 @@ const sidebarsLearn: SidebarsConfig = { id: 'developers/deploying-from-ci', label: 'Deploying from a CI/CD Pipeline', }, + { + type: 'doc', + id: 'developers/taking-a-node-out-of-rotation', + label: 'Taking an Unhealthy Node Out of Rotation', + }, ], }, { From 35b88235cdaeae078da004872cc364c9e9e989c9 Mon Sep 17 00:00:00 2001 From: Ethan Arrowood Date: Wed, 14 Oct 2026 13:33:32 -0600 Subject: [PATCH 11/21] Make the production reliability guides easier to read Co-Authored-By: OpenAI Codex --- learn/administration/backup-and-recovery.mdx | 124 +++++++++++------- .../engineering-rpo-rto-and-uptime.mdx | 30 ++--- .../health-checks-and-traffic-admission.mdx | 91 +++++++------ .../how-harper-runs-in-production.mdx | 58 ++++---- .../administration/monitoring-and-triage.mdx | 53 ++++---- .../administration/operating-replication.mdx | 39 +++--- .../production-readiness-checklist.mdx | 35 +++-- .../reliability-plan-template.mdx | 19 ++- .../safe-deployments-and-rollback.mdx | 79 +++++++---- .../sizing-a-harper-cluster.mdx | 26 ++-- reference/backups/operations.md | 10 +- reference/backups/overview.md | 6 +- 12 files changed, 333 insertions(+), 237 deletions(-) diff --git a/learn/administration/backup-and-recovery.mdx b/learn/administration/backup-and-recovery.mdx index 765a786b6..e85d63ae5 100644 --- a/learn/administration/backup-and-recovery.mdx +++ b/learn/administration/backup-and-recovery.mdx @@ -6,15 +6,14 @@ sidebar_position: 7 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -Replication gives you availability. It does not give you recovery, and the reason is worth stating plainly: replication faithfully propagates whatever you write, including the bad write. A destructive operation, a schema mistake, or a release that corrupted data reaches every peer at the speed of your convergence time. +Replication supports availability, but also propagates destructive writes, schema errors, and application corruption to peers. You need backups to recover an earlier state. -Backups cover the failure classes replication cannot. This guide is about deciding what you need before you pick an operation, getting a copy somewhere the node's failure cannot reach, and knowing your restore constraints before you are in an incident rather than during one. +This guide covers choosing a backup mechanism, keeping an independent copy, and testing that you can restore service within your recovery targets. ## What You Will Learn -- The five questions that determine your backup design, answered before any command -- Which backup mechanism you actually have, since it depends on your storage engine -- Why a copy on the node that created it is not a backup, and the specific way copying one wrong breaks it +- How to choose a backup mechanism for your storage engine and recovery targets +- How to copy and retain backups independently of the source node - What `verify_backup` does and does not check - Which databases can be restored with the server running, and which cannot - Why restoring a database to undo a code bug is usually the wrong move @@ -28,13 +27,13 @@ Backups cover the failure classes replication cannot. This guide is about decidi ## Decide before you choose a command -Answer these first. Every operation below is easy, and every one of them is the wrong choice for some of these answers. +Record these decisions in your reliability plan: -1. **Which failure classes must you survive?** Node loss, storage loss, a bad deployment, an accidental destructive operation, a corrupting application bug, a site or region event, and a control-plane or credential loss are seven different problems. Replication addresses the first one well and the sixth one partially. It addresses none of the rest. -2. **What is the recovery point and recovery time per database?** Not per cluster. A catalog table and an order ledger rarely deserve the same answer, and paying ledger-grade backup cadence for catalog data is how backup cost becomes a reason to reduce frequency. -3. **What granularity do you need?** Harper backs up and restores whole databases. There is no per-table restore. If your recovery story requires restoring one table, that requirement has to be met by database layout or by forward repair, and it is far cheaper to learn that now. -4. **Where must copies live?** How many independent locations, under what retention, and with what access control. -5. **Who is authorized to execute a restore, and how is that logged?** A restore destroys current data by design. It deserves a named authority and an audit trail. +1. **Which failures must you survive?** Include node or storage loss, bad deployments, destructive operations, application corruption, regional outages, and compromised credentials. Choose a recovery mechanism for each. +2. **What are the recovery targets?** State acceptable data loss per database and time to restore each affected journey. Use those targets to set backup frequency and test the restore procedure. +3. **What granularity do you need?** Managed backups restore whole databases. To recover selected tables or records, restore into an isolated environment and plan a separate data repair. LMDB snapshot options differ; see [backup operations](/reference/v5/backups/operations#get_backup). +4. **Where must copies live?** Choose independent storage locations, retention periods, and access controls. +5. **Who may authorize and execute a restore?** Name the approver and operator, and record their actions. An in-place restore replaces current data. ## Know which mechanism you have @@ -49,7 +48,7 @@ A backup is a whole-database copy: all tables, the transaction log, and any file Two limitations to check against your own deployment now rather than later: - **Managed backups require RocksDB.** For LMDB databases your options are `get_backup` or volume snapshots. -- **One storage root per database.** Backup is rejected when a database's tables resolve to more than one storage root. A database-level custom `storage.path` is fine, and so is a per-table `path` that still lands on the same root. It is a database whose tables are split across roots that cannot be backed up with these operations at all, and if you have one, your backup strategy for it does not exist yet. +- **One storage root per database.** Backup is rejected when a database's tables resolve to multiple storage roots. Custom database or table paths are valid if all tables still resolve to one root. Use a separate backup strategy for databases split across roots. ::: @@ -78,7 +77,7 @@ harper purge_backups database=data keep_count=7 Every backup operation runs from the CLI under the same name. With the server running the CLI forwards to the server; with it stopped the command operates directly on the files. -`get_backup` is the exception. It streams from a running server and has no offline form, so with Harper stopped it fails as an unknown command. If you need a copy off a node that is down, start it, or take the backup directory or a volume snapshot instead. Worth knowing before an incident, because a stopped node is exactly when you reach for it. +`get_backup` requires a running server. Document how to copy the managed backup repository or take a volume snapshot if the server cannot start. @@ -86,41 +85,42 @@ Every backup operation runs from the CLI under the same name. With the server ru Backups of the same database share unchanged RocksDB data files, so the first one copies everything and later ones copy only what changed. Shared files are reference-counted, so deleting a backup removes only files no remaining backup references. :::warning -The incremental behavior applies to the RocksDB data files only. **The transaction-log snapshot is copied in full on every backup.** With a large audit-retention window that is real, recurring disk cost that the data-only view hides, so size your backup volume against it rather than against the incremental figure. +Each backup is a full logical snapshot, but its disk cost depends on file sharing: -Blobs behave differently and are easy to over-budget. Each backup captures a full set of blobs, but the files are hard-linked rather than copied when the backup directory and the blob storage share a filesystem, which is the default layout. In that case the extra space is near zero. They become genuine full copies only when the two live on different filesystems, so putting backups on a separate volume is a real cost decision rather than a free one. Pass `exclude_blobs: true` to skip blobs when that is appropriate. +- RocksDB backups share unchanged data files within the same repository. +- The transaction-log snapshot is copied for each backup. Include it when measuring repository growth. +- Blob snapshots use hard links when the source and destination filesystem support them. Otherwise, they copy the files. Use `exclude_blobs: true` only if your recovery procedure can recover those blobs separately. +- An export that does not preserve sharing materializes those shared or linked files as separate copies. Budget for the full exported size. -One consequence for disaster recovery: because the blob snapshot is hard-linked, copying a backup repository with an ordinary recursive copy materializes every blob as a full, separate file at the destination. +Measure physical repository growth over several backups with your retention policy and filesystem. Do not assume that a full logical snapshot consumes its full logical size, or that all snapshot files are hard-linked. ::: Do not use `list_backups` sizes for capacity planning. The `size` and `file_count` fields come from the RocksDB backup engine and exclude the transaction-log and blob snapshots, so each entry undercounts, while the shared files between entries mean summing them overcounts. Measure the repository directory instead. ## Get a copy off the node -This is the step most likely to be missing, and the one where doing it slightly wrong produces a copy that cannot be restored. +Keep a recoverable copy outside the source node's failure and administrative boundaries. Separate credentials and retention controls should prevent a compromised node or backup writer from deleting that copy or weakening its retention policy. **Managed backups live on the node that created them.** The repository is a local directory, and RocksDB shares files across backup IDs, so **a backup ID is not a self-contained folder.** Two consequences: - A disaster-recovery copy has to take the entire per-database repository, `/`, not an individual backup. - It has to do that while no backup operation is running, or from an atomic filesystem snapshot. A live recursive copy can race `create_backup`, `delete_backup`, or `purge_backups` and produce an unrestorable copy. -An unrestorable copy is worse than no copy, because it will pass a "backups exist" check and fail during an incident. - -The simpler off-host path, and the one to prefer unless you specifically need retained managed backups off the node: +To take a fresh snapshot directly to another machine, run: ```bash # Pull a snapshot of the current state from a running node harper get_backup database=data out=./data-$(date +%Y%m%dT%H%M%S).tar.gz # Or pull a snapshot of another node, writing it to a file here -harper get_backup database=data target=https://node-2.example.com:9925 out=./data.tar.gz +harper get_backup database=data target=https://operations.node-2.example.com out=./data.tar.gz ``` `target=` only changes which node the snapshot is read from. It writes a file and nothing else: it does not restore, import, or clone that node's database onto the local one. Turning the file into a live database is a separate, deliberate restore step, performed with Harper stopped. Note also that `get_backup` always streams the current state. It cannot download a historical managed backup, so it is a way to take a fresh off-host copy, not a way to export your retention history. -Whichever path you choose, the destination must not share a failure domain with the source. A second directory on the same volume survives a deleted file and nothing else. +Give each exported recovery artifact a unique, immutable identity. Include a manifest with its database, Harper version, backup timestamp, included files, blob roots, and checksums. After upload, read back the destination metadata and confirm the expected identity, size, encryption, versioning, and retention settings where supported. Checksum verification must cover the copy at the destination. ## Verification is not optional @@ -135,13 +135,13 @@ Whichever path you choose, the destination must not share a failure domain with `verify_backup` checks the RocksDB file sizes, their checksums when `verify_checksum` is `true`, which is slower, and the framing of the transaction-log snapshot, which is always checked. -**Blob contents are not verified.** Verification confirms that a backup which recorded blobs still has its blob snapshot present, and fails the backup as corrupt if that snapshot is missing. It does not check the individual blob files for size, checksum, or readability. So verification can pass on a backup whose blobs are damaged, and only a real restore proves they are intact. +**Blob contents are not verified.** Verification checks that a required blob snapshot is present, but does not check individual blob files for size, checksum, or readability. Validate those separately during restore testing. -More generally, a verified backup is a well-formed backup, not a proven recovery. The only evidence that your recovery works is a restore you have actually performed, which is why the drill at the end of this guide is the point of it. +Verification checks the backup's structure. A restore test checks whether it can recover your application. ## Know your restore constraints before the incident -RocksDB is single-writer, so an in-place restore requires the database to be fully closed first. That produces hard constraints you cannot negotiate during an outage: +An in-place RocksDB restore requires the database to be closed: | Database | Online restore, server running | Offline restore, server stopped | | ---------------------------------------------- | ------------------------------------ | ------------------------------- | @@ -152,37 +152,49 @@ RocksDB is single-writer, so an in-place restore requires the database to be ful Because the restore runs as a background job, a component holding the database open does not fail your request. It fails inside the job, so you find out from `get_job` rather than from the response. Harper does not track which component uses which database, so it cannot selectively stop one. :::tip -That last constraint has an escape hatch worth knowing before you need it. Setting `HARPER_SAFE_MODE` to any value starts Harper without loading your applications and components, while the database, Operations API, and HTTP server come up normally. Harper's own built-in root component still loads; what is skipped is everything you deployed: +Safe mode skips user applications and components while keeping the built-in root component, database, Operations API, and HTTP server available: ```bash HARPER_SAFE_MODE=1 harper run ``` -Because no component loads, nothing holds a user database open, so a restore that would fail inside its job can run online. It is also how you get back in when a broken component is what stopped Harper in the first place, since the Operations API is available to inspect, repair, or remove it. Traffic should stay off the node either way. See [safe mode](/reference/v5/configuration/overview#harper_safe_mode). +This can release user database handles held by application components so an online restore can proceed. It also lets you inspect or remove a component that prevents normal startup. Keep user traffic and replication isolated during recovery. Safe mode does not remove the offline requirement for restoring `system`. See [safe mode](/reference/v5/configuration/overview#harper_safe_mode). ::: -Two more constraints worth writing into your procedure: +Also include these constraints in the restore procedure: - **`target_database` requires the server stopped.** Restoring into a separate database rather than overwriting the source is CLI-only with Harper down. The target must not already exist or must be an empty directory. - **An interrupted restore leaves the database unloadable.** On a crash or power loss mid-restore, Harper marks the database as incompletely restored and refuses to load it on the next start. Recover by rerunning `restore_backup` for the same database and `backup_id`. Do not try to load or hand-repair the directory. ## Restoring in a replicated cluster -A restore is a point-in-time rollback of one node's data. In a cluster, that node then has to rejoin peers that never rolled back, so sequence it deliberately: +A restored node may be behind peers that have continued accepting writes. Choose which data should be authoritative before reconnecting it. + +```mermaid +flowchart TD + Live[Live cluster] --> Artifact[Create immutable recovery artifact] + Artifact --> Retained[Independent retained destination] + Retained --> Test[Restore in disposable isolated environment] + Test --> Validate[Validate storage and application] + Validate --> Policy[Choose data authority and replay policy] + Policy --> Drain[Drain user traffic from recovery node] + Drain --> Isolate[Separately isolate replication] + Isolate --> Restore[Restore and validate recovery node] + Restore --> Peers[Controlled replication re-entry] + Peers --> Traffic[Progressive traffic admission] +``` + +For an in-place recovery: 1. Take the node out of rotation with the availability flag, per [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). Never restore a node that is serving traffic. -2. **Isolate it from its peers.** This is a separate action from step 1, and the step most likely to be skipped. See the warning below. -3. Decide what should happen to the restored data: whether it should propagate to peers, or be overwritten by them. Those are two very different outcomes and it is not a decision to improvise once the node is back on the mesh. +2. **Isolate replication separately.** Stop peer exchange using your tested topology or network controls. Keep that isolation in place across restarts. +3. Confirm the data authority and replay policy: which writes to preserve, repair, or reapply, and how peers will converge. 4. Perform the restore, online or offline according to the table above. 5. Verify data expectations locally before any peer sees the node. -6. Re-admit the node through the full return sequence, including convergence verification. +6. Reconnect replication under that policy and verify convergence. Then complete the traffic-admission checks before restoring traffic. :::danger -**The availability flag does not stop replication.** It tells your traffic layer to stop sending user requests. Replication is a separate path: peer connections stay up, and the node keeps exchanging transactions the whole time it is "out of rotation." - -So a node that is drained but still connected is not isolated. Restore it in place and the rolled-back database is live on the mesh immediately, before you have validated anything. Two things can then happen, both bad: peers overwrite your restored data with newer transactions, or your rolled-back state propagates outward to peers that were never affected. - -Isolation is its own step. Stopping Harper and restoring offline is the unambiguous way to get it, which is a further argument for the offline path in a cluster even where the table above permits an online restore. If you restore online, you must have removed the node from the replication topology first, and put it back deliberately in step 6. +**Draining traffic does not stop replication.** A drained node continues exchanging transactions with peers. An offline restore stops that exchange while Harper is down, but configured routes can reconnect as soon as it starts. Keep replication isolated until validation is complete and you are ready to execute the re-entry policy. ::: ## What backups cannot fix @@ -191,26 +203,29 @@ If a release changed the meaning of persisted data, restoring the database is us For that failure class, define forward repair instead: a targeted correction, or a replay from the transaction log, that fixes the affected records and leaves the rest alone. Decide which of your failure classes get restore and which get forward repair while you are calm, and record the decision. See [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx) for how this interacts with release reversal. -### Prove it +## Test the backup program + +Test the copy you would actually use after losing a node, including its destination controls: + +1. Create a backup and verify it with `verify_checksum: true`. For other mechanisms, use the corresponding integrity checks. +2. Export it using the procedure above. Record the immutable artifact ID, backup timestamp, manifest, and checksums. Read back destination metadata and check that the backup writer cannot weaken retention. +3. Restore the destination copy into a disposable environment with a compatible Harper version. Isolate it from production traffic, replication, and application side effects. +4. Validate the expected databases, tables, record counts, blob contents, and business invariants. Run the critical user journeys against the restored application. +5. Rehearse the failure-to-recovery procedure on a non-production cluster. Measure elapsed time from the simulated failure, including detection, transfer, restore, validation, and traffic admission. Compare recovered writes against the failure time to measure actual data loss. +6. Record the artifact ID, backup time, validation results, operator, approver, and achieved RPO and RTO. Rehearse an offline `system` restore separately. -A backup you have never restored is a hypothesis. Run this on a non-production cluster and record the numbers: +Use these results in [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx). If recovery misses a target, improve the procedure or revise the commitment. -1. Create a managed backup, then verify it with `verify_checksum: true`. -2. Take an off-host copy using the correct procedure for your mechanism, quiesced or from an atomic snapshot if you are copying a managed repository. -3. Destroy the source database. -4. Restore it, taking the node out of rotation first, and time from decision to restored service. That is your measured recovery time. -5. Determine how much data was actually lost against the backup timestamp. That is your measured recovery point. -6. Validate correctness through the application, not just at the storage layer. Row counts agreeing is not the same as the journey working. -7. Repeat for the `system` database specifically, offline, since it has different constraints and it is the one people never rehearse. +Monitor the age of the newest recovery point that has passed the required destination and recovery checks. A successful backup job alone does not establish a usable recovery point. -Both measured numbers feed [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx). If they do not meet your stated targets, one of the two has to change, and it is better that it is the target than a promise you cannot keep. +For records requiring long-term preservation, keep a separate export in a documented format that can be read independently of Harper. Operational backups serve disaster recovery; preservation exports also need to remain usable after the original software is retired. Choose cadence, storage provider, and any physical-media process for your own environment. ## Operational notes - **Retention is a policy, not a side effect of disk space.** Use `purge_backups` with an explicit `keep_count` that matches a written retention decision. - **Backup operations are `super_user` through the server, and filesystem permissions offline.** An operator with shell access on a node can restore without an API credential, so protect the host accordingly. - **Schedule backups per database, matched to that database's recovery point.** A single cluster-wide cadence overspends on some databases and underspends on the ones that matter. -- **Alert on backup age, and on job failure.** A `create_backup` job that fails silently produces a gap you will find at the worst possible time. Backup age exceeding your recovery point objective is a pageable condition. +- **Alert on job failure and recovery-point age.** Track the newest fully verified recovery point and alert before its age exceeds your RPO. - **Record the storage engine per database in your inventory** and re-check after migrations, since your entire mechanism choice depends on it. ## Readiness checklist @@ -220,8 +235,11 @@ Both measured numbers feed [Engineering RPO, RTO, and Uptime](./engineering-rpo- - [ ] Storage engine confirmed per database, and the mechanism chosen accordingly - [ ] No database in scope has tables spanning more than one storage root, or the exclusion is known and accepted - [ ] Backup cadence matches the stated recovery point per database -- [ ] Backup volume sized against the non-incremental transaction-log snapshot, and against full blob copies if the backup and blob paths are on different filesystems -- [ ] An off-host copy exists in a destination that does not share a failure domain +- [ ] Repository growth measured across successive backups, including transaction-log copies and blob hard-link behavior +- [ ] Export capacity accounts for full copies where file sharing is not preserved +- [ ] An off-host copy exists across separate failure and administrative boundaries +- [ ] Artifact identities, manifests, checksums, and destination metadata verified +- [ ] Backup writer cannot weaken retention - [ ] Managed repository copies take the whole `/` directory, quiesced or from an atomic snapshot - [ ] `verify_backup` runs on a schedule, with `verify_checksum` at least periodically - [ ] Blob contents understood to be unchecked by `verify_backup`, which confirms only that the blob snapshot is present @@ -229,9 +247,13 @@ Both measured numbers feed [Engineering RPO, RTO, and Uptime](./engineering-rpo- - [ ] Restore authority named, and restore execution logged - [ ] Restore procedure isolates the node from replication, not just from user traffic, since the availability flag does not stop peer exchange - [ ] Safe mode known as the way to restore a database a component holds open, and to reach a node a broken component stopped -- [ ] Restore drill completed and dated, with measured recovery time and recovery point +- [ ] Destination copy restored in a disposable, isolated environment with a compatible Harper version +- [ ] Inventory, counts, blobs, business invariants, and critical journeys validated +- [ ] Artifact ID, timestamp, validation results, operator, approver, achieved RPO, and achieved RTO recorded +- [ ] Alerts use the newest fully verified recovery point - [ ] `system` database restore rehearsed offline - [ ] Forward repair defined for corruption caused by application code +- [ ] Separate preservation export defined where long-term access is required ## Additional Resources diff --git a/learn/administration/engineering-rpo-rto-and-uptime.mdx b/learn/administration/engineering-rpo-rto-and-uptime.mdx index 6b1cc11d0..d92e56e6e 100644 --- a/learn/administration/engineering-rpo-rto-and-uptime.mdx +++ b/learn/administration/engineering-rpo-rto-and-uptime.mdx @@ -3,9 +3,9 @@ title: Engineering RPO, RTO, and Uptime sidebar_position: 8 --- -Recovery point, recovery time, and availability are usually written down once, in a document nobody consults, as three numbers that were never checked against a mechanism. This guide is about making them real: stating them per journey, mapping each failure scenario to the mechanism that addresses it, and reconciling the targets against numbers you have actually measured. +Define recovery point objective (RPO), recovery time objective (RTO), and availability for each critical user journey and failure scenario. This guide helps you choose a recovery mechanism for each target and check it against measured results. -If a target and a measurement disagree, one of them has to change. The point of doing this deliberately is that you get to choose which, rather than finding out during an incident that the answer was the target all along. +When measured results miss a target, improve the recovery mechanism or revise the target before committing to it. ## What You Will Learn @@ -31,11 +31,11 @@ If a target and a measurement disagree, one of them has to change. The point of | Recovery time objective | How long until service is restored | Per journey, measured from the moment the journey starts failing, not from the moment someone acts | | Availability target | How much unavailability is permitted | Per journey, as minutes per month, with the measurement boundary named | -Three details make the difference between a testable statement and a slogan. +For each target, record the affected journey, when measurement starts, and where you measure it. **Per journey, not per cluster.** A catalog browse and an order submission fail differently, matter differently, and recover differently. One cluster-wide number overcommits on the cheap journey and undercommits on the expensive one. -**Recovery time starts when the journey breaks, not when someone acts on it.** Detection is inside the clock, not before it, and the gap between something breaking and someone knowing is often the largest part of an outage. Starting the count at the moment a human decided to act, or even at the moment an alert fired, produces a number that flatters your automation and misleads your planning. Your users were already down for all of it. +**Include detection in recovery time.** Measure from the start of the failure to verified service restoration. Starting at the alert or the operator's decision leaves out time when users were already affected. **Name the measurement boundary.** Availability measured at your CDN edge and availability measured at a Harper node are different quantities, and the difference is exactly the part of the stack you may not control. Whichever you choose, say so in the same sentence as the number. @@ -45,13 +45,13 @@ Your recovery point objective is a statement about how much acknowledged data yo [`storage.writeAsync`](/reference/v5/database/storage-tuning#storagewriteasync) defaults to `false`. Set it to `true` and Harper stops calling `fsync` on commit, so a write returns as soon as it is queued to the operating system's page cache. Throughput on write-heavy workloads improves substantially. The cost is that a power loss or OS crash between the commit and the flush to disk loses the most recent transactions, even though your application was told they succeeded. The database stays structurally consistent; only the newest writes are gone. -The trap is that this is invisible to everything else in this guide. Backup timestamps look right, replication looks healthy, and no alert fires. You discover it the first time a node loses power, which is also the first time the gap between your stated recovery point and your real one matters. +Healthy replication and recent backups do not prove that every acknowledged write has reached durable storage. Include the possible loss of unflushed writes in your RPO. So treat it as an RPO decision rather than a performance one. Enabling it is defensible when the data is reproducible from an upstream source, when the workload is bulk ingest that can simply be re-run, or when peers acknowledge writes before the loss window opens. Outside those cases, leave it off. If it is on anywhere, record which databases and say so next to the recovery point in your plan, because the number no longer means what it appears to. ## Map scenarios to mechanisms -For each scenario, know the mechanism, and know what must be proven for the mechanism to count. The right-hand column is the whole exercise. +For each scenario, choose a recovery mechanism and record the evidence that it meets your target. | Scenario | Primary mechanism | What must be proven | | ------------------------------------- | --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | @@ -68,7 +68,7 @@ For each scenario, know the mechanism, and know what must be proven for the mech | Replication certificate expiry | Certificate lifecycle management | Expiry dates tracked, with an alert far enough ahead to act | | Downstream dependency outage | Declared degraded mode | The journey's behavior when the dependency is down, verified rather than assumed | -Two rows in that table are the ones most often missing. Node replacement is budgeted as though it were a restart, when it is a full synchronization and can be much slower. And the downstream dependency row usually has no answer at all, which means the answer is whatever the code happens to do. +Measure node replacement separately from a restart: full synchronization can take much longer than incremental catch-up. Also test how each journey behaves when a downstream dependency is unavailable. ## Convert availability into a budget @@ -82,13 +82,13 @@ An availability target is a quantity of unavailability you may spend per month. Figures assume a 30.44 day month. Your contractual definition, exclusions, and measurement boundary may differ. -Read down that right-hand column rather than the middle one. The tier you choose determines how much automation you are committing to build, not how many nodes you buy. Adding nodes without automating detection and traffic removal moves you across the table's rows without moving you down its column. +Use the availability budget to decide how quickly detection, traffic removal, and recovery must happen. Additional nodes provide capacity, but those procedures still need to fit within the budget. -Treat the budget as spendable. If the month's remaining budget is small, that is an argument for pausing discretionary change, and if it is large, that is permission to ship. An error budget that never changes anyone's behavior is just a number in a document. +Define when a depleted error budget should pause discretionary changes, who makes that decision, and what must improve before changes resume. ## Reconcile targets against measurements -This is the step that turns the exercise into engineering. Fill in both columns and look for rows where they disagree. +Fill in both columns and investigate any measurement that misses its target. | Commitment | Measured from | Your target | Your measurement | | --------------------------- | ---------------------------------------------------------------------------------------- | ----------- | ---------------- | @@ -106,17 +106,17 @@ Three reconciliations catch most problems: - **Backup cadence against recovery point.** If you back up every six hours, your recovery point cannot be one hour, no matter what the document says. - **Restore time plus detection time against recovery time.** Recovery time includes noticing. A twenty minute restore behind a forty minute detection gap is a one hour recovery. -- **Convergence time against your admission gate.** If your gate admits a node faster than it converges, you are serving stale data on purpose and calling it availability. +- **Convergence time against your admission gate.** Keep a returning node out of traffic until its data meets the journey's freshness requirements. -Where a target and a measurement disagree, the resolution is one of three things: invest in the mechanism, relax the target, or accept the gap explicitly with an owner and a date. All three are legitimate. Leaving it unreconciled is not. +For each missed target, improve the mechanism, revise the target, or record an accepted gap with an owner and a review date. ## Declare your degraded modes -Between fully working and fully down there is a range of states, and if you have not decided what they should be, the code has decided for you. +Define which reduced-service behaviors are acceptable during each failure. For each critical journey, write down what happens when a dependency is unavailable, when a node is behind on replication, and when the cluster is below its capacity budget. Reads served from slightly stale data may be entirely acceptable for a catalog and entirely unacceptable for a balance check, and the right answer differs by journey rather than by system. -Then decide who can declare a degraded mode, and whether the declaration is manual or automatic. A degraded mode nobody is authorized to invoke is not a degraded mode. +Name who can invoke each mode and whether the transition is manual or automatic. ### Prove it @@ -126,7 +126,7 @@ Two passes, and both are needed. **Then run one real drill per scenario class**, spread over a quarter rather than attempted in a day. The classes are: node loss, node return, node replacement, release reversal, restore, and dependency failure. Record the date, the measured numbers, what surprised you, and what you changed as a result. -The surprises are the deliverable. A drill that goes exactly as expected has confirmed your documentation. A drill that does not has found the thing that would have hurt you. +Use unexpected results to update the runbook, monitoring, or recovery mechanism before repeating the drill. ## Operational notes diff --git a/learn/administration/health-checks-and-traffic-admission.mdx b/learn/administration/health-checks-and-traffic-admission.mdx index a4d48196e..db7580ef8 100644 --- a/learn/administration/health-checks-and-traffic-admission.mdx +++ b/learn/administration/health-checks-and-traffic-admission.mdx @@ -6,14 +6,14 @@ sidebar_position: 2 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -The default health check on most load balancers asks one question: did something answer on this port? A Harper node answers that question correctly while still being the wrong place to send a user request, because a node can be running before its databases have synchronized, while it is catching up on missed transactions, or while a dependency it needs for one particular journey is down. +An HTTP response proves that a Harper node is running. Before sending user traffic to it, also check that its data is current and the application can serve the requested journey. -This guide builds four separate signals so your traffic layer can tell the difference between "the process is alive" and "this node will serve this journey correctly," and then uses them to drain and return a node safely. +This guide explains the checks to use and the order for draining a node, changing it, and returning it to service. ## What You Will Learn - Why liveness, availability, readiness, and journey verification are four signals rather than one, and which layer consumes each -- How to install and drive the `@harperdb/status-check` component to control whether a node advertises itself as available +- How to control node availability on Fabric or with `@harperdb/status-check` on a self-managed cluster - How to write an application readiness route that checks only what the routed journey needs - The order of operations for draining a node and returning it to service - What makes a readiness endpoint safe under load, and what makes one dangerous @@ -27,29 +27,32 @@ This guide builds four separate signals so your traffic layer can tell the diffe ## Four signals, not one -| Signal | Where it lives | What it proves | Who consumes it | -| -------------------------- | ---------------------------------------------------------- | ----------------------------------------------------- | ---------------------------------- | -| Process liveness | Any trivial response on `9925` or `9926` | Harper is running | Process supervisor, restart policy | -| Node availability flag | `GET /status` on `9926`, from the `status-check` component | An operator or automation says this node should serve | Traffic layer health check | -| Application readiness | A component route you write on `9926` | This node can serve this journey right now | Traffic layer, or your own gating | -| Critical-journey synthetic | A real request through the public route | Users are actually being served | Your SLO and alerting | +| Signal | Where it lives | What it proves | Who consumes it | +| -------------------------- | ---------------------------------------------- | ----------------------------------------------------- | ---------------------------------- | +| Process liveness | A trivial response on the node's HTTP endpoint | Harper is running | Process supervisor, restart policy | +| Node availability flag | `GET /status` on the application URL | An operator or automation says this node should serve | Traffic layer health check | +| Application readiness | A component route on the application URL | This node can serve this journey right now | Traffic layer, or your own gating | +| Critical-journey synthetic | A real request through the public route | Users are actually being served | Your SLO and alerting | -The distinction that does the most work is the second one. The availability flag is not a measurement, it is a declaration. It exists so that you, or your automation, can take a node out of rotation deliberately, before doing something to it, and put it back afterwards. Nothing else in this list can be set by an operator, and nothing else is safe to use as the primary routing signal. +The availability flag lets an operator or automation control whether a node receives new traffic. Use it as the routing signal, and use readiness, replication checks, and journey tests to decide when to set it. ## Install the availability flag -[`@harperdb/status-check`](https://github.com/HarperFast/status-check) is a Harper component that adds a `/status` route on the application port. Deploy it like any other component: +On **Fabric**, the status component is already installed. It reads the node's `availability` record in `system.hdb_status`. Use the CLI to change that record, as described in [Load Balancing and Node Availability](/fabric/load-balancing#taking-a-node-out-of-rotation). Fabric also writes it during maintenance. Do not install the npm package below on Fabric: it serves the same `/status` path but uses a different table. + +On a **self-managed cluster**, [`@harperdb/status-check`](https://github.com/HarperFast/status-check) adds a `/status` route on the application port. Deploy it with: ```json { "operation": "deploy_component", "project": "status-check", "package": "@harperdb/status-check", + "replicated": false, "restart": true } ``` -Or declare it in the root `harper-config.yaml`, so the component is part of the node's configuration rather than something an operator has to remember to deploy: +Repeat on each node, or declare it in each node's root `harper-config.yaml`: ```yaml status-check: @@ -57,36 +60,36 @@ status-check: ``` :::note -That entry belongs in the root `harper-config.yaml` (in the Harper `rootPath`, typically `~/hdb`), not in an application's own `config.yaml`. The two files look alike but behave differently: in the root config the entry name is free-form, while in a component's `config.yaml` it must match a `package.json` dependency. A component `config.yaml` also **replaces** Harper's default component configuration outright instead of merging with it, so a file containing only this entry would switch off the `rest`, `graphqlSchema`, `jsResource`, and `fastifyRoutes` defaults your application relies on. See [applications](/reference/v5/components/applications). +This entry belongs in the root `harper-config.yaml` under the Harper `rootPath`, typically `~/hdb`. An application's `config.yaml` configures its plugins and replaces the default plugin configuration. See [applications](/reference/v5/components/applications). ::: -Once deployed, the route's contract is its status code, which is what lets a load balancer consume it without parsing anything: +The self-managed component provides these routes: - `GET /status` returns `200` when the node is available, `404` when it is not - `POST /status` marks the node available (authenticated) - `DELETE /status` marks the node unavailable (authenticated) -It does also return a body, which is useful when you are checking by hand: a short message on `200`, and an RFC 9457 problem-details document on `404`. +Use the target node's application URL. The examples assume HTTPS is configured on that URL; a default local installation uses `http://localhost:9926`. ```bash # Is this node advertising itself as available? -curl -s -o /dev/null -w '%{http_code}\n' https://my-node.example.com:9926/status +curl -s -o /dev/null -w '%{http_code}\n' https://my-node.example.com/status # Take it out of rotation -curl -s -X DELETE https://my-node.example.com:9926/status -u 'admin:password' +curl -s -X DELETE https://my-node.example.com/status -u 'admin:password' # Put it back -curl -s -X POST https://my-node.example.com:9926/status -u 'admin:password' +curl -s -X POST https://my-node.example.com/status -u 'admin:password' ``` ```javascript -const base = 'https://my-node.example.com:9926/status'; +const base = 'https://my-node.example.com/status'; const auth = { Authorization: 'Basic ' + btoa('admin:password') }; const res = await fetch(base); @@ -99,25 +102,23 @@ await fetch(base, { method: 'POST', headers: auth }); // back in -Point your traffic layer's health check at `GET /status` on `9926`, not at the Operations API and not at your application's root. A `404` is the node telling the traffic layer to stop sending work, and it will keep saying so until something sets it back. +Point the traffic layer's health check at `GET /status` on each node's application URL. A `404` should stop new traffic reaching that node. Existing connections may remain open, so measure the drain before restarting it. :::note -The availability flag is node-local out of the box. The component stores it in a table declared `replicate: false`, and the Operations API's own status values are stored the same way, so neither propagates to peers. - -That is the property you want, and it is worth knowing why: a flag that replicated would let one node's maintenance state reach its peers, turning a routine drain into a cluster-wide outage. So the rule is to preserve it rather than to establish it. If you fork the component or persist the flag some other way, keep `replicate: false` on whatever holds it. See [Operating Replication](./operating-replication.mdx). +Both implementations keep the flag node-local with `replicate: false`. Preserve that setting if you fork a component or store the flag elsewhere. A replicated flag could take every peer out of rotation when you drain one node. ::: ### A note on `set_status` The Operations API also offers [`set_status`, `get_status`, and `clear_status`](/reference/v5/operations-api/operations#set_status--get_status--clear_status) for application-defined status values, with types for primary, maintenance, and availability. -These are a coordination primitive for your own automation, not a health report, and not a substitute for a real readiness check. Nothing in Harper acts on a value you set through them. Prefer the `status-check` component for traffic admission, because its contract is an HTTP status code that a load balancer can consume directly, and reach for `set_status` when you need to coordinate something between your own scripts. +On Fabric, `set_status` with `id=availability` controls the record its status component reads. The self-managed npm package uses its own `data.hdb_status` table, so `set_status` does not control that package. Use its `POST /status` and `DELETE /status` routes instead. Neither flag measures application health by itself. ## Write an application readiness route Liveness and the availability flag both answer questions about the node. Readiness answers a question about the journey: if traffic arrives for this route right now, will it succeed? -The rule that keeps this useful is to check only the dependencies the routed journey actually needs. A readiness route that checks everything will report a node unready because of a subsystem that route never touches, and you will have converted a partial degradation into a full outage yourself. +Check only the dependencies needed by the journey routed to this node. A failure in an unrelated subsystem should not remove capacity from a working journey. Add a resource to your application's `resources.js`: @@ -159,7 +160,7 @@ export class Readiness extends Resource { } ``` -The `jsResource` plugin is enabled by default, so the route is served at `/Readiness` on the application port as soon as the class is exported. See [Harper Applications in Depth](../developers/harper-applications-in-depth.mdx) for the resource and export mechanics. +The `jsResource` plugin loads the exported class. Keep it enabled if your application has an explicit `config.yaml`. At the default application mount, the route is `/Readiness`; adjust the probe URL if you configured `urlPath` or `host` routing. See [resource routing](/reference/v5/resources/overview#exporting-resources-as-endpoints) for the path rules. :::note Because the handler is a `static` method, it replaces Harper's built-in dispatch along with the authorization check that lives inside it, so `GET /Readiness` is unauthenticated. That is what a load balancer probe needs, and it is why the endpoint must not return anything you would not publish. Keep the response to check names and outcomes, never connection strings, credentials, or internal hostnames. @@ -167,27 +168,43 @@ Because the handler is a `static` method, it replaces Harper's built-in dispatch If you rewrite this as an instance `get()` instead, authorization comes back and defaults to `super_user` only, which will make your probe start failing with an authorization error rather than a readiness one. In that form you need `allowRead() { return true; }` to keep it reachable. ::: -The sentinel is a record you create once, on purpose, and leave alone. Seed it on every node before you point anything at this route, or readiness will report `failed` forever and you will have built a probe that never passes: +Create the sentinel record before enabling the probe. The example assumes `Product` is exported at the server root: ```bash -curl -s -X PUT https://my-node.example.com:9926/Product/readiness-probe-sentinel \ +curl -s -X PUT https://my-node.example.com/Product/readiness-probe-sentinel \ -H 'Content-Type: application/json' \ -u 'admin:password' \ -d '{"name":"readiness probe sentinel","description":"do not delete"}' ``` -Give it a name that says what it is, because the next person to find it will be deciding whether it is safe to delete. If your table replicates, seeding it once is enough; if it does not, seed it per node. +If the table replicates, seed once and confirm the record reaches every node. Otherwise, seed it on each node. Keep the record in place while the probe is in use. -Two details in that example are the point of it. The timeout on the downstream call means a slow dependency cannot make your readiness check hang, which would make the node look dead to a probe rather than unready. And returning the version means that when you are staring at a dashboard during a rollout, the readiness response itself tells you which build answered. +Bound downstream checks with a timeout. Set `APP_VERSION` in your deployment so the response identifies the component build answering during a rollout. ## Drain a node and bring it back -The order matters, and the verification steps between them are the parts people skip. +Keep the node unavailable until all return checks pass. On Fabric, set `availability` through the CLI as linked above; the `DELETE /status` and `POST /status` calls below are the self-managed package's controls. + +```mermaid +flowchart TD + Serving --> Unavailable[Declare unavailable] + Unavailable --> Drained[Measured user traffic is zero] + Drained --> Change[Change or restart] + Change --> Live{Process live?} + Live -->|Yes| Synced{Replication converged?} + Synced -->|Yes| Ready{Readiness and journey test pass?} + Ready -->|Yes| Available[Declare available] + Available --> Restored[Restore traffic progressively] + Live -->|No| Hold[Hold unavailable and investigate] + Synced -->|No| Hold + Ready -->|No| Hold + Hold --> Live +``` -1. **Confirm the peers can carry it, before you drain anything.** The remaining nodes have to stay inside their capacity budget with this node gone. If they cannot, stop here and do not drain: see [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx). Checking this after the traffic has already moved is how a maintenance window becomes an incident. +1. **Check peer capacity.** Confirm that the remaining nodes can carry the expected traffic within their capacity budget. If they cannot, postpone the change or add capacity. See [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx). 2. **Declare it unavailable.** `DELETE /status` on the target node. 3. **Verify traffic actually stopped.** Watch request volume on the target fall to zero and rise on its peers. Do not trust the configured weight; check the measured request count from [analytics](/reference/v5/analytics/overview) or your traffic layer's own metrics. Health check intervals, DNS TTLs, and client-side connection reuse all add delay here, and the delay is yours to measure. 4. **Watch the peers absorb it.** If they are going outside budget despite step 1, abort: `POST /status` to put this node back in rotation, and re-plan the change for a lower-traffic window or with added capacity. @@ -205,7 +222,7 @@ The order matters, and the verification steps between them are the parts people -Step 5 on the return side is the one worth defending in a review. Failback is a change like any other, and a node that has just synchronized under load is the most likely one to surprise you. Design failback, do not just design failover. +Set the initial traffic weight before declaring the node available, where your traffic layer supports it. Increase it only while error rate and latency remain acceptable. If a return check fails, leave the node unavailable and investigate. ## Readiness hygiene @@ -219,19 +236,19 @@ Step 5 on the return side is the one worth defending in a review. Failback is a On a non-production cluster, restart one node under representative read and write load, using the full drain and return sequence above. Record three numbers: how long from `DELETE /status` until measured traffic on that node reaches zero, whether any user-visible errors occurred during the transition, and how long from process start until the node legitimately passed all three return gates. -That third number is your real node re-entry time, and it is almost always longer than people assume, because it includes convergence rather than just startup. +Record the third number as node re-entry time, including convergence and application validation. ## Operational notes - **A returning node is not the same as a new node.** A node whose databases have never synchronized downloads them in full. A node that was offline and comes back catches up on the transactions it missed. Both need to finish before traffic arrives, but they take very different amounts of time, so do not budget for the first when you are planning a routine restart. - **Set unavailable before recovery work, not after.** Any operation that touches data on a node, including a restore, should happen with the node out of rotation. - **Configuration changes need a restart to take effect**, so a node that has been reconfigured but not restarted is running the old configuration while reporting the new one. Sequence the restart into the same maintenance window. -- **Fabric provides its own cluster-level health and routing.** These signals still matter, because the availability flag and your readiness route are what Fabric's routing has to consult. +- **Fabric routes by its status component.** It does not automatically consume your readiness route. For application-driven availability, see [Taking an Unhealthy Node Out of Rotation](../developers/taking-a-node-out-of-rotation.mdx). ## Readiness checklist -- [ ] `@harperdb/status-check` deployed, declared in the root `harper-config.yaml` rather than deployed by hand -- [ ] Traffic layer health check points at `GET /status` on `9926` +- [ ] Fabric's existing status component retained, or `@harperdb/status-check` configured on each self-managed node +- [ ] Traffic layer health check points at `GET /status` on each node's application URL - [ ] Availability flag storage still declares `replicate: false`, if you forked the component or persist it yourself - [ ] An application readiness route exists, scoped to one journey's dependencies - [ ] Every downstream call in the readiness route has a timeout diff --git a/learn/administration/how-harper-runs-in-production.mdx b/learn/administration/how-harper-runs-in-production.mdx index f4f36af82..c89f2bab1 100644 --- a/learn/administration/how-harper-runs-in-production.mdx +++ b/learn/administration/how-harper-runs-in-production.mdx @@ -6,7 +6,7 @@ sidebar_position: 1 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -You have built and deployed an application, and now real users are going to depend on it. Operating Harper is not the same as operating separate application and database tiers. In Harper, one process holds your component code, the HTTP stack, the local database, and the replication client. That "collapsed stack" architecture reduces several problems you may be used to solving. +Harper runs your application code, HTTP server, local database, and replication client on the same node. To operate it in production, you need to know how much traffic each node can handle, which data it serves, and how to move traffic when it needs maintenance. This guide covers what a single Harper node actually contains, and how to understand your deployment before you design anything around it. @@ -27,42 +27,42 @@ This guide covers what a single Harper node actually contains, and how to unders A Harper node is a single process running your components, an HTTP server, a local storage engine, and peer replication. There is no network hop between your application code and the data it reads, no separate cache tier to keep coherent, and no connection pool to tune between tiers. -There are three main default ports: +Record the ports used by your deployment: -| Port | Serves | Who should reach it | -| ------ | --------------------------------------------------------------------------- | --------------------------------- | -| `9926` | Application traffic: REST, WebSocket, MQTT-over-WebSocket, component routes | Your traffic layer and your users | -| `9925` | The [Operations API](/reference/v5/operations-api/overview) | Operators and your pipeline only | -| `9933` | Secure peer [replication](/reference/v5/replication/overview) | Other nodes in the cluster only | +| Port | Serves | Who should reach it | +| ---------------------------------- | --------------------------------------------------------------------------- | --------------------------------- | +| `9926` | Application traffic: REST, WebSocket, MQTT-over-WebSocket, component routes | Your traffic layer and your users | +| `9925` | The [Operations API](/reference/v5/operations-api/overview) | Operators and your pipeline only | +| Configured secure replication port | Secure peer [replication](/reference/v5/replication/overview) | Other nodes in the cluster only | -These are the default values. Confirm the live values using the [`get_configuration`](/reference/v5/operations-api/operations#get_configuration) operation. +`9925` and `9926` are the default HTTP ports. HTTPS uses `operationsApi.network.securePort` and `http.securePort`, or a TLS-terminating proxy. Use the node's configured URL in the examples throughout this track. -Resolve the replication port rather than assuming it. `9933` is the default a fresh Harper Pro install writes into `harper-config.yaml`, but it is a configured value, not a constant. When no `replication` block is present at all, the secure replication port falls back to the Operations API secure port instead, so it lands on `9925`. That is not a hypothetical: an in-place upgrade carries the previous configuration forward and does not backfill a `replication` block, so an upgraded node with no such block replicates on a different port than a freshly installed one. +Resolve the effective replication port from `replication.securePort`, falling back to `operationsApi.network.securePort` when it is unset. An upgrade retains the existing configuration, so do not assume that a newly installed node and an upgraded one use the same port. -Read back `http.port`, `operationsApi.network.port`, and the `replication` block. Record what you find. Later guides in this track assume you know these numbers for your own deployment. +Read the configuration from each node and record its HTTP, HTTPS, and replication ports: -{/* TODO: Insert Tabs example of the get_configuration operation request with JSON parsing. CURL piped to jq and fetch with basic json access + console.log */} +```json +{ "operation": "get_configuration" } +``` :::tip -Keep `9925` off any public route as the Operations API is for administrative purpsoes such as deploying components, reading logs, and managing configurations. See [security overview](/reference/v5/security/overview). +Restrict the Operations API to operators and deployment automation. It can deploy components, read logs, and manage configuration. See [security overview](/reference/v5/security/overview). ::: ## A node is the unit of both failure and scale -Since the single node is simultaneously the runtime, the database, and the application, they all scale and fail together too. This goes beyond the literal asyncronous execution model of Harper (more information at [TODO](TODO)). If a node fails for any reason, that entire node goes down; not just one part. +Losing a node removes both its request-handling capacity and access to its local data. Plan for the remaining nodes to serve the application while it is unavailable. -This model is quite different from what separated system architectures are used to. As a result, one must think about their Harper deployment differently. +Keeping application code and data together removes some of the work associated with separate tiers: - No network latency on data access, and no tail latency from a saturated connection pool between tiers - No cache invalidation problem between an application cache and a database of record (because they are the same thing) -- No partial-outage state where the application tier is healthy and the data tier is not -- One capacity number to measure and one thing to size +- No separate application-to-database connection to monitor +- Application and data access measured together in your capacity tests -When you lose a node, you lose a whole slice of your service, not one layer of it. This is why the first real design decision in [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) is not your peak throughput. Instead, it is how many nodes you are willing to lose at once, and whether the survivors can carry the load. +Choose how many simultaneous node failures your service must tolerate, then check whether the survivors can carry peak traffic. [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) walks through that calculation. -Scale works similarly. Adding capacity means adding a node that habdles both requests and data, so a scaling event is also a data movement event (as replication needs to be setup appropriately). Because of this, scaling a Harper system up takes slightly more time overall compared to scaling separted service architectures. - -All in all, whenever you think about your Harper application, think about it as one holistic unit. You don't have to worry about individually scaling API layer versus database nodes. Harper executes, fails, and scales all together. +Adding a node also moves data. Allow time for the databases it serves to synchronize before sending it user traffic. ## Three differences that change your runbook @@ -70,13 +70,13 @@ All in all, whenever you think about your Harper application, think about it as A Harper process will answer a TCP connection and return an HTTP response before it is a good place to send a user request. A new or replacement node has to synchronize the databases it serves before its answers are correct. A returning node has to catch up on transactions it missed. -So liveness and admission are two different decisions, and a load balancer health check that only proves liveness will route users to a node that is technically running and functionally wrong. This is the whole subject of [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx), and it is the single most common gap in a first production deployment. +Keep a starting node out of traffic until its data and application checks pass. [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx) describes the checks and the return sequence. ### Replication is peer-to-peer and scoped, so verify your own scope Harper peers exchange data over WebSockets with mTLS on the secure replication port and discover each other through configured routes. There is no primary. Data mutations and transactions replicate; some things do not, and the scope is configurable per database and per table. -The important operating habit is not memorizing a default. It is checking what your cluster actually replicates, because the answer depends on your configuration, your version, and whether anyone has scoped it since: +Read the live replication state on each node: ```json { @@ -90,17 +90,19 @@ The response lists each peer connection and, within it, one socket per database Your application ships as a component, deployed with [`deploy_component`](/reference/v5/operations-api/operations#deploy_component) from an immutable reference. Rolling back means deploying the previous immutable reference. There is no image to rebuild, no instance to replace, and no cluster to rebuild to undo a bad release. -That makes reversal fast enough to be a real option under pressure, which in turn makes it worth designing for deliberately rather than improvising. It also means code rollback, configuration rollback, and data recovery are three separate actions with three different blast radii, and conflating them during an incident is how a bad release becomes a data loss event. See [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx). +Immutable component references make code rollback repeatable. Configuration rollback and data recovery need separate procedures because they change different parts of the service. See [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx). ## Inventory your service boundary -Everything else in this track builds on knowing what you have. Run these four operations against each node and write down the answers. +Collect the following information from each node and keep it with your runbooks. + +The examples use `https://operations.my-node.example.com/` for a configured Operations API endpoint. Replace it with the node's own Operations API URL; a default local installation uses `http://localhost:9925/`. ```bash -curl -s -X POST https://my-node.example.com:9925/ \ +curl -s -X POST https://operations.my-node.example.com/ \ -H 'Content-Type: application/json' \ -u 'admin:password' \ -d '{"operation":"system_information","attributes":["system","cpu","memory","disk","threads"]}' @@ -110,7 +112,7 @@ curl -s -X POST https://my-node.example.com:9925/ \ ```javascript -await fetch('https://my-node.example.com:9925/', { +await fetch('https://operations.my-node.example.com/', { method: 'POST', headers: { 'Content-Type': 'application/json', @@ -150,11 +152,11 @@ Fill in a boundary record you can keep: | Critical journeys | The user-facing paths that must work, named | | Downstream dependencies | Anything Harper calls that can fail independently | -The last two rows are the ones people skip and the ones that matter most. A healthy Harper process cannot compensate for a failed downstream dependency or for application logic returning wrong answers, so an operating model that only watches Harper will miss the incidents your users actually notice. +Include application journeys and downstream dependencies in your monitoring. Node health alone will not reveal incorrect responses or failures in services the application calls. ### Prove it -Before moving on, confirm the picture is real rather than assumed. On a non-production cluster, stop one node and watch what happens to the others: whether peers keep serving, how long the remaining nodes take to show the change in `cluster_status`, and what your traffic layer does about it. You are not measuring anything precisely yet. You are checking that the boundary you wrote down matches the system you have. +On a non-production cluster, stop one node. Confirm that peers keep serving, note when `cluster_status` shows the change, and check whether the traffic layer stops routing to the node. Compare the result with your inventory and topology plan. ## Operational notes diff --git a/learn/administration/monitoring-and-triage.mdx b/learn/administration/monitoring-and-triage.mdx index 035e8356d..0e35bfe02 100644 --- a/learn/administration/monitoring-and-triage.mdx +++ b/learn/administration/monitoring-and-triage.mdx @@ -6,9 +6,9 @@ sidebar_position: 5 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -A dashboard is useful when it lets an operator place a symptom. Users are seeing errors: is it the edge, the traffic layer, one node, the replication path, a downstream dependency, or the change someone shipped twenty minutes ago? A dashboard that shows CPU across the cluster cannot answer that, which is why it gets ignored during incidents. +Monitor the parts of the service you can investigate or change: the public route, individual Harper nodes, replication, downstream dependencies, and recent deployments. -This guide builds monitoring around the boundaries you can actually act on, names the specific Harper metrics worth alerting on, and gives you a triage sequence short enough to run under pressure. +This guide identifies useful signals at each boundary and gives you a procedure for narrowing an incident. ## What You Will Learn @@ -39,31 +39,30 @@ This guide builds monitoring around the boundaries you can actually act on, name | Change | Component and config version, cohort, operator, start and end, outcome | `list_deployments`, `get_deployment`, `get_components`, `get_configuration` read-back | | Recovery | Backup age, job state, verification result, last restore drill | `list_backups`, `verify_backup`, `get_job` | -The Change row is the one teams leave out and the one that resolves incidents fastest. Most production symptoms correlate with something a human did, so being able to overlay deployments onto a latency graph is worth more than another resource metric. +Show deployments and configuration changes alongside request errors and latency. Their timing helps you identify whether a release affected one node, one version, or the whole cluster. ## The signals worth an alert Harper records a large standard metric set automatically. Most of it belongs on a dashboard. This much belongs on a pager: -| Alert on | Metric or source | Why this one | -| -------------------------------------- | ------------------------------------------------------------------ | ---------------------------------------------------------------------------------------- | -| Journey success rate below SLO | `success`, by resource path and method | The only signal that maps directly to user impact | -| Journey latency at your SLO percentile | `duration`, by resource path and method | Alert on the percentile in your SLO, never on the mean | -| Worker saturation | `utilization`, per worker thread | Percentage of time the worker was processing requests. This is your real headroom signal | -| Event loop backpressure | `main-thread-utilization`, the `taskQueueLatency` attribute | Rises before throughput drops, so it is an early warning rather than a postmortem input | -| Replication convergence lag | `replication-latency`, or the receive-time gap in `cluster_status` | A node serving stale data looks healthy on every other signal | -| Write commit time | `transaction-commit-time` | Storage-level degradation shows here before it shows in request latency | -| Write queue depth growing | `write-transaction-queue-depth` | A growing queue means the node is accepting work faster than it can commit it | -| Storage headroom | `storage-volume` free, and `database-size` | Disk exhaustion is an outage with no graceful degradation | -| Missing peer | `cluster_status` | Covered in [Operating Replication](./operating-replication.mdx) | -| Backup age exceeding RPO | `list_backups` | The failure you will not notice until you need it | - -Two notes on how to set these. Alert on the percentile that appears in your SLO, because a mean latency graph will look fine through an incident that is failing your slowest ten percent of users. And set saturation thresholds from the per-node capacity work in [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) rather than from a generic number, since the point where latency leaves your SLO is specific to your application and your data. +| Alert on | Metric or source | Why this one | +| -------------------------------------- | ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | +| Journey success rate below SLO | `success`, by resource path and method | The only signal that maps directly to user impact | +| Journey latency at your SLO percentile | `duration`, by resource path and method | Alert on the percentile in your SLO, never on the mean | +| Worker saturation | `utilization`, per worker thread | Percentage of time the worker was processing requests. This is your real headroom signal | +| Event loop backpressure | `main-thread-utilization`, the `taskQueueLatency` attribute | Rises before throughput drops, so it is an early warning rather than a postmortem input | +| Replication convergence lag | `replication-latency`, source timeline, and sentinel checks | Connection health alone does not establish that a node's data is current | +| Write commit time | `transaction-commit-time` | Storage-level degradation shows here before it shows in request latency | +| Write queue depth growing | `write-transaction-queue-depth` | A growing queue means the node is accepting work faster than it can commit it | +| Storage headroom | `storage-volume` free, and `database-size` | Disk exhaustion is an outage with no graceful degradation | +| Missing peer | `cluster_status` | Covered in [Operating Replication](./operating-replication.mdx) | +| Verified recovery point older than RPO | Backup manifest and destination/restore validation records | A successful job alone does not establish that the copy can restore service | + +Alert on the latency percentile in your SLO. Set saturation thresholds from your measurements in [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx), where you determine the load at which latency becomes unacceptable. ## Get the metrics out -Harper stores analytics locally in `hdb_raw_analytics` and aggregates them into `hdb_analytics`. You can query those directly, but for a real monitoring setup -export them. +Harper stores raw analytics in `hdb_raw_analytics` and aggregates them into `hdb_analytics`. Query these tables or export metrics to your monitoring system. @@ -126,9 +125,9 @@ threads: `threads.preload` (Added in: v5.2.0) loads a module via Node's `--import`, which is how an agent installs the loader hooks that let it instrument modules imported later. `threads.preloadRequire` (Added in: v5.2.0) uses `--require`, which runs the module body and is typically how an agent's initialization entry actually starts it. Both are documented under [threads configuration](/reference/v5/configuration/options#threads). -Installing loader hooks and starting the agent are two different jobs, and which of your agent's entry points does which is specific to that agent. Some ship one entry that does both; others split them, in which case set both keys. The example above is the split-entry case, and it is the one that produces the most confusing failure: with only the hooks loaded, a tracer will hand out spans with plausible trace ids that are no-ops and export nothing, so your instrumentation looks installed and your collector stays empty. +Agent entry points vary: some install loader hooks and start the agent together; others need separate `preload` and `preloadRequire` entries. Loading hooks without starting the agent can produce no-op spans and no exports. -Two constraints worth knowing before you plan around this. Both keys apply to worker threads only, and neither works under Bun. And bare specifiers resolve against the `node_modules` of your installed components, so an agent can ship as a dependency of a deployed component rather than as a host-level install. +Both keys apply only to worker threads and require Node.js. Bare specifiers resolve against installed components' `node_modules`, so the agent can be a component dependency. Whatever agent you use, follow its own worker-thread documentation, and finish by confirming spans actually arrive at your collector rather than assuming the configuration took. @@ -166,7 +165,7 @@ Harper's logger is built on Node's `Console`, so this renders as a formatted tex That is fine for reading during an incident, but a log pipeline cannot key on `component` or `outcome` without parsing the line. If you need queryable fields, serialize the context yourself and log a single JSON string. ::: -Levels run `trace`, `debug`, `info`, `warn`, `error`, `fatal`, and `notify`. The default is `warn`, and `notify` is always logged regardless of level. Choose levels deliberately: a production log at `debug` is a log nobody can read, and one at `error` only has the incidents in it and none of the context. +Levels run `trace`, `debug`, `info`, `warn`, `error`, `fatal`, and `notify`. The default is `warn`, and `notify` is always logged. Choose a level that retains useful incident context without producing excessive volume. `console.log` output does not reach the log files unless `logging.console` is enabled, so unstructured console output is not a production record. Centralize logs off the node, because the node you most need logs from is the one you are about to restart. @@ -182,7 +181,7 @@ Read logs through the API when you need them from a specific node: ## The five-minute triage sequence -Run these in order. The goal is not diagnosis, it is narrowing. +Use these checks to locate the affected layer, then contain the failure before investigating its cause. 1. **Confirm and bound the impact.** What journey, starting exactly when, in which geography or cohort, and which component version is live. Without a start time you cannot correlate anything. @@ -201,19 +200,19 @@ Run these in order. The goal is not diagnosis, it is narrowing. Differences between peers are more informative than absolute values, because they tell you whether this is one node or the whole cluster. -5. **Check the replication path.** `cluster_status` for the databases this journey needs. Look for a missing peer, `connected: false`, or a widening gap between `lastReceivedRemoteTime` and `lastReceivedLocalTime`, which means stale reads. +5. **Check replication.** Inspect the expected peer/database sockets in `cluster_status`, sustained latency, and `lastCommitConfirmed` progress while writes continue. Compare `lastReceivedRemoteTime` with the expected source timeline and verify freshness with a sentinel or journey test. Timestamp differences alone do not prove convergence. 6. **Correlate with the last change.** `list_deployments` for what shipped and when, and `get_configuration` read back against what you believe is configured. Remember that a configuration change applied without a restart leaves a node running something other than its stated configuration. 7. **Contain with the smallest reversible action.** Stop a ramp, drain one node, deactivate a feature, restore prior traffic weights, or isolate suspected data. Record the decision, who made it, and the next decision deadline. -Two rules make this sequence work. Silence is a failed gate: if telemetry is missing for the thing you are checking, treat that as a negative signal rather than skipping the step. And containment comes before root cause. You can diagnose after users are being served again. +If required telemetry is missing, investigate the gap before declaring the service healthy. Record containment actions so the next operator can see what changed. ### Prove it Pick a fault and inject it on a non-production cluster, then time yourself. Good candidates: saturate one node's workers, block the replication port on one peer, make a downstream dependency return errors, or deploy a component that fails on one route. -Measure three things. How long until an alert fired. How long until an operator following the sequence above could name the layer. And whether any step gave a misleading answer, which is the most valuable output of the drill, because a misleading signal during a real incident costs more than a missing one. +Record time to alert, time to identify the affected layer, and any misleading or missing signals. Update the procedure before repeating the drill. ## Operational notes @@ -229,7 +228,7 @@ Measure three things. How long until an alert fired. How long until an operator - [ ] Alerts on journey success and latency at the SLO percentile, not the mean - [ ] Alerts on worker `utilization` and `taskQueueLatency`, thresholds derived from measured per-node capacity - [ ] Alerts on replication convergence lag and missing peers -- [ ] Alerts on storage headroom and backup age +- [ ] Alerts on storage headroom and the newest fully verified recovery point's age - [ ] Metrics exported off the node, via Grafana on Fabric or the Prometheus exporter self-managed - [ ] If you run an APM agent, both `threads.preload` and `threads.preloadRequire` set as that agent requires, with spans confirmed at the collector - [ ] `analytics.replicate` setting known and deliberate diff --git a/learn/administration/operating-replication.mdx b/learn/administration/operating-replication.mdx index f4607e25d..0b9df2f35 100644 --- a/learn/administration/operating-replication.mdx +++ b/learn/administration/operating-replication.mdx @@ -6,9 +6,9 @@ sidebar_position: 4 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -Replication is how a Harper cluster stays available: peers exchange data directly over WebSockets, there is no primary, and a node that falls behind catches up on its own. The [replication reference](/reference/v5/replication/overview) covers how to configure that. This guide covers the part that only matters once real users depend on it, which is how to tell whether replication is actually working. +Harper peers exchange data over WebSockets and catch up after interruptions. This guide explains how to monitor replication and verify convergence in production. For configuration, see the [replication reference](/reference/v5/replication/overview). -The distinction that runs through this guide is between a connection and convergence. A connected socket proves two nodes can talk. It does not prove the data a user is about to read is current. Those are different claims, and only one of them is what your traffic admission decision depends on. +A connected socket proves that peers can communicate. Before admitting user traffic, also verify that the data needed by the application has arrived. ## What You Will Learn @@ -49,7 +49,7 @@ All tables in a replicated database replicate unless the table opts out. So the ```bash -curl -s -X POST https://my-node.example.com:9925/ \ +curl -s -X POST https://operations.my-node.example.com/ \ -H 'Content-Type: application/json' \ -u 'admin:password' \ -d '{"operation":"cluster_status"}' @@ -59,7 +59,7 @@ curl -s -X POST https://my-node.example.com:9925/ \ ```javascript -await fetch('https://my-node.example.com:9925/', { +await fetch('https://operations.my-node.example.com/', { method: 'POST', headers: { 'Content-Type': 'application/json', @@ -75,7 +75,7 @@ await fetch('https://my-node.example.com:9925/', { There is one socket per database per peer, so the sockets present in the response are the ground truth about what is flowing. Run this from every node, not one. Replication direction can be constrained per route, so node A's view of the cluster is not necessarily node B's view, and a one-sided picture is how directional configuration mistakes survive into production. :::warning -Whether the `system` database is in your replication scope is the highest-stakes scoping question in a Harper cluster, because `system` holds `hdb_user`, `hdb_role`, and `hdb_nodes`. **It is in scope by default**, since the default replication scope is every database, so unless someone has narrowed it your users and roles already propagate and every node that receives `system` must be trusted with its contents, including encrypted secret rows. That is usually what you want, but it should be a decision rather than a surprise. Confirm your own answer from `cluster_status` and your configuration, and read [replicating the system database with controlled flow](/reference/v5/replication/overview#replicating-the-system-database-with-controlled-flow) before changing it. +**`system` replicates by default.** It contains users, roles, and encrypted secret rows, so trust every node that receives it with those contents. Confirm the scope in your configuration and live connections before changing it. See [controlled system replication](/reference/v5/replication/overview#replicating-the-system-database-with-controlled-flow) for directional routes and table exclusions. ::: ## What moves automatically, and what does not @@ -92,17 +92,15 @@ Whether the `system` database is in your replication scope is the highest-stakes | Node registry (`hdb_nodes`) | Lives in `system`, so it propagates with it. Each node rewrites its own self-record from its `harper-config.yaml` routes on restart or component reload | Scoping applied through `add_node` alone does not survive a restart. Put durable constraints in config routes, and see below if you need the registry kept per cluster | | Destructive schema changes | `drop_database` and `drop_table` replicate by default; `drop_attribute` does not | A drop is a cluster-wide event unless you pass `"replicated": false`. Verify the result on every node | -The three rows most likely to bite you are the returning node, the new node, and the node registry. - -On the returning node: it is common to see a routine restart budgeted as though it were a full resynchronization, which makes teams avoid restarts they should be comfortable with. A node whose databases have never synced does download them in full. A node that was briefly offline catches up on what it missed. Measure both on your own data volume once, and use the right number for the right situation. +Measure incremental catch-up and full synchronization separately. A briefly offline node can catch up from retained transaction history. If its required history has been pruned, a full copy may be needed even though the node has synchronized before. On the new node, the cost is larger than "one copy from one peer," because the full-copy decision is made per peer and per database rather than once against a bootstrap source. A joining node requests a full copy from every peer it has no resume cursor for. And because replication is bidirectional, each established peer independently decides it has no cursor for the newcomer and requests a full copy _from_ it as well. Joining a cluster of N peers is therefore N inbound transfers plus N outbound transfers of the new node's own (empty) databases, not a single stream from one source. -The practical consequences are worth planning around. Load lands on every existing peer at once rather than on one, so a join during peak traffic is a capacity event for the whole cluster. If you want a single designated source instead, `add_node` accepts `isLeader: true`, which tells the joining node to request its full copy from that peer alone. +A node join can load every existing peer, so include it in cluster capacity planning. To use one full-copy source, pass `isLeader: true` to `add_node`. -On the registry: a node's advertised record is derived from its configuration file, and it replicates. That means a topology constraint you applied imperatively is superseded the next time that node restarts or reloads components, and the node quietly goes back to advertising itself more broadly than you intended. +Each node derives its advertised `hdb_nodes` record from its configuration. A restart or component reload can replace constraints applied only through `add_node`; keep durable constraints in config routes. -That propagation is what makes gossip discovery work, and since v5.2 the record each node advertises is directional, derived from its own routes, so replicating `system` no longer collapses a constrained topology into a full mesh. If you need the registry itself kept per cluster rather than shared, that is a deliberate per-route setting: `excludeTables: ['hdb_nodes']` on a route's `sendsTo` or `receivesFrom` entry stops those rows crossing that link while users, roles, and schema still propagate. Its main use is a migration bridge between clusters that should share identity but keep separate topology. +Registry rows can replicate with `system`. Since v5.2, advertised records describe directional routes, so that exchange can preserve a constrained topology. To keep registries separate across a migration bridge, explicitly exclude `hdb_nodes` on the relevant routes while allowing users and roles through. See [controlled system replication](/reference/v5/replication/overview#replicating-the-system-database-with-controlled-flow). :::danger When replicating configuration, only send cluster-appropriate parameters. Replicating a node-local value such as a port, `node.hostname`, a file path, TLS material, or `replication.hostname`, `url`, or `routes` overwrites every peer's own local value. To apply a cluster-wide change safely, use `set_configuration` with `"replicated": true` for the parameter, then `restart_service` with `"replicated": true`, which restarts nodes one at a time. @@ -144,9 +142,14 @@ What each field is telling you: - **`lastReceivedRemoteTime`** is the source node's timestamp on the newest transaction you have received. - **`lastReceivedLocalTime`** is your own clock when you received it. -The last two are the pair that matters. **A widening gap between `lastReceivedRemoteTime` and `lastReceivedLocalTime` means this node is behind and working through a backlog.** That is the signal to alert on for convergence, and it is the one that tells you a returning node is not ready for traffic yet. `sendingMessage` appears while a transaction is actively being sent and is absent when the socket is idle, so its absence is not a fault. +Check connection state and sustained latency alongside commit progress during writes. Compare `lastReceivedRemoteTime` with the source's expected write timeline, then use a sentinel or journey test to verify freshness. `sendingMessage` appears only while a transaction is being sent; its absence on an idle socket is normal. + +The receive-time gap can help diagnose delay, with two cautions: + +- **It describes received transactions.** On an idle or stalled connection, the timestamps stop moving. Their difference does not measure how far the receiver is behind the latest source write. +- **It includes clock skew.** The timestamps come from different nodes. Compare against a healthy baseline with synchronized clocks. -Two things will mislead you if you automate on that pair without knowing them. The gap only means "behind" while transactions are actually arriving: on an idle source both stamps freeze, and the gap sits at whatever constant it last reached rather than signalling lag. And because the two stamps come from two different clocks, skew between peers is added to the gap directly, so calibrate your threshold against a healthy baseline rather than treating the raw number as replication delay. +Use these fields as supporting evidence, rather than as the sole admission check. :::tip While a database is taking a full copy, these timing fields render as the literal string `"Copying"` rather than a date. Anything that parses them as timestamps will fail on a joining node, which is exactly the node you most want to be watching. Handle that value explicitly. @@ -159,7 +162,7 @@ Inventory `system.hdb_nodes` alongside this and compare it to your intended topo - A peer missing entirely from `connections`, or `connected: false` on a database that serves a critical journey - Sustained growth in `latency`, judged against your own baseline - `lastCommitConfirmed` not advancing on a peer while writes are occurring -- A `lastReceivedRemoteTime` to `lastReceivedLocalTime` gap exceeding your admission budget, evaluated only while writes are flowing +- Received transaction progress falling behind the expected source timeline, confirmed by a stale sentinel or journey result - Repeated reconnects, which are visible in the logs even when a point-in-time status check looks healthy - Version or configuration drift between peers @@ -175,7 +178,7 @@ await tables.OpsProbe.put(marker.id, marker); ```bash # On the target node: read it back and compare writtenAt -curl -s https://server-2.example.net:9926/OpsProbe/convergence-probe \ +curl -s https://server-2.example.net/OpsProbe/convergence-probe \ -u 'admin:password' ``` @@ -183,13 +186,13 @@ The interval between the write and the moment the peer returns the new value is The `replication-latency` metric from [analytics](/reference/v5/analytics/overview#replication-metrics) gives you the continuous version of the same measurement: the difference between the source commit timestamp and local time, reported per node, database, and table. Use the sentinel for a definitive answer during a change, and the metric for a dashboard. -Note that `replication-latency` is not recorded for the `system` database. Since `system` is in replication scope by default and carries your users and roles, a dashboard built only on this metric will show nothing for the database that propagates identity. Use the sentinel, or the `cluster_status` receive-time pair, if you need to watch `system` convergence. +`replication-latency` is not recorded for `system`. Check its peer sockets and commit progress, then verify that expected identity changes have reached each node. Use a dedicated sentinel if your runbook needs a repeatable test of `system` convergence. ## Consistency belongs in the application design Replication makes data available on every peer. It does not make every peer agree at every instant, and a distributed decision made on a node that has not yet converged can observe stale state and act on it. -This is not a Harper limitation to work around, it is the property that lets any node serve any request without a coordinator. But it means a specific class of operation is unsafe if you write it as a plain read followed by a write: +Concurrent read-then-write operations on different nodes need additional coordination when correctness requires a single decision: - Claim-once actions: redeeming a code, assigning a unique handle, awarding a one-per-customer offer - Hard floors: inventory that must not go negative, a balance that must not overdraw @@ -225,7 +228,7 @@ On a non-production cluster under write load: - [ ] Tables requiring one transaction boundary confirmed to be in one database - [ ] Durable topology constraints live in `harper-config.yaml` routes, not only in `add_node` calls - [ ] Measured convergence time recorded under load, per critical database -- [ ] Alerts configured on missing peers, sustained latency growth, stalled `lastCommitConfirmed`, and the remote-to-local receive gap +- [ ] Alerts configured on missing peers, sustained latency growth, stalled commit progress during writes, and stale sentinel values - [ ] Non-commutative operations identified and given an owner, a serialization point, or an external coordinator - [ ] Replication certificate expiry dates tracked with an alert - [ ] Replication interruption and recovery exercise completed and dated diff --git a/learn/administration/production-readiness-checklist.mdx b/learn/administration/production-readiness-checklist.mdx index 882c39202..03c68a999 100644 --- a/learn/administration/production-readiness-checklist.mdx +++ b/learn/administration/production-readiness-checklist.mdx @@ -3,19 +3,19 @@ title: Production Readiness Checklist sidebar_position: 9 --- -This is the aggregated launch gate for the whole Administration track. Every item appears in one of the earlier guides, which is where the reasoning lives. This page exists so you have one thing to work down before a launch, and one thing to hand to a reviewer who asks how you operate Harper. +Use this checklist to review a service before launch. It brings together the checks from the Administration guides, with links to the procedures and explanations. -Copy it into your own runbook and adapt it. It is a starting point for your gate, not a substitute for having one. +Copy it into your runbook and adapt it to your deployment. ## How to use this -Each item is a claim about your deployment. A claim counts when there is evidence: a measured number with a date, a written procedure, a configured alert, or a completed drill. "We know about that" is not evidence, and neither is a passing intention. +Complete each item with evidence: a dated measurement, a written procedure, a configured alert, or a completed drill. -Some items will not apply to you, and that is fine. Mark them not applicable with a reason rather than deleting them, so a reviewer can see the decision was made rather than missed. +Mark items that do not apply with a reason, so the reviewer can see which decisions were deliberate. -Three things are worth deciding before you start: +Before you start, agree on: -- **Who signs off.** A gate with no named approver is a document, not a gate. +- **Who signs off.** Name the person responsible for accepting the evidence. - **What "outstanding" means.** Some items can be accepted as gaps with an owner and a date. Decide in advance which ones cannot. - **When you run it again.** This is not a one-time launch artifact. Re-run it after a Harper version upgrade, a topology change, or significant data growth, since most of the measured numbers expire. @@ -52,7 +52,7 @@ From [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx). From [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). - [ ] Liveness, availability flag, application readiness, and journey synthetic all exist as separate signals -- [ ] `@harperdb/status-check` deployed, declared in the root `harper-config.yaml` rather than by hand +- [ ] Fabric's existing status component retained, or `@harperdb/status-check` configured for a self-managed cluster - [ ] Traffic layer health check points at `GET /status` on the application port - [ ] Availability flag storage still declares `replicate: false`, if you forked the component or persist it yourself - [ ] Readiness route scoped to one journey's dependencies, with a timeout on every downstream call @@ -71,7 +71,7 @@ From [Operating Replication](./operating-replication.mdx). - [ ] Tables that must commit together confirmed to be in one database - [ ] Durable topology constraints live in `harper-config.yaml` routes, not only in `add_node` calls - [ ] Measured convergence time recorded under load, per critical database -- [ ] Alerts on missing peers, sustained latency growth, stalled `lastCommitConfirmed`, and the receive-time gap +- [ ] Alerts on missing peers, sustained latency growth, stalled commit progress during writes, and stale sentinel values - [ ] Non-commutative operations identified and given an owner, a serialization point, or an external coordinator - [ ] Replication certificate expiry tracked with an alert - [ ] Replication interruption and recovery exercise completed and dated @@ -86,7 +86,7 @@ From [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx). - [ ] Cohort ladder defined, from smallest useful sample to full exposure - [ ] Stop thresholds and hold times declared before a rollout begins - [ ] Named decision owner for advance, hold, stop, and roll back -- [ ] Pipeline polls the deployment record and the restart job separately +- [ ] Pipeline checks deployment details and waits for the restart job, then verifies the serving version on every node - [ ] `deployment_timeout` and `ignore_replication_errors` set deliberately - [ ] Configuration changes go through the same change record as code - [ ] `get_configuration` read back after every configuration change @@ -103,7 +103,7 @@ From [Monitoring and Triage](./monitoring-and-triage.mdx). - [ ] Alerts on journey success and latency at the SLO percentile, not the mean - [ ] Alerts on worker utilization and task queue latency, with thresholds derived from measured capacity - [ ] Alerts on replication convergence lag -- [ ] Alerts on storage headroom and backup age +- [ ] Alerts on storage headroom and the age of the newest fully verified recovery point - [ ] Metrics exported off the node - [ ] `analytics.replicate` setting known and deliberate - [ ] Structured logs include component, version, node, request id, operation, duration, and outcome @@ -119,18 +119,25 @@ From [Backup and Recovery](./backup-and-recovery.mdx). - [ ] Failure classes enumerated, each with the mechanism that addresses it - [ ] Recovery point and recovery time stated per database - [ ] Mechanism chosen per database according to its storage engine -- [ ] No database in scope uses per-table storage paths, or the exclusion is known and accepted +- [ ] No database in scope has tables spanning multiple storage roots, or the exclusion is known and accepted - [ ] Backup cadence matches the stated recovery point -- [ ] Backup volume sized against the full cost, including the non-incremental transaction-log and blob snapshots -- [ ] Off-host copy exists in a destination that does not share a failure domain +- [ ] Repository capacity measured across successive backups, including transaction-log copies and blob hard links +- [ ] Export capacity accounts for full copies where the destination does not preserve file sharing +- [ ] Off-host copy exists in a separate failure and administrative boundary +- [ ] Backup artifacts have immutable identities, manifests, and checksums +- [ ] Destination encryption, versioning, and retention checked after upload where applicable +- [ ] Backup writer cannot weaken the destination's retention policy - [ ] Managed repository copies take the whole per-database directory, quiesced or from an atomic snapshot - [ ] `verify_backup` runs on a schedule - [ ] Blob integrity understood to be unverified by `verify_backup` - [ ] Restore constraints documented for user databases, component-held databases, and `system` - [ ] Restore authority named, and restore execution logged - [ ] Restore procedure isolates the node from replication, not just from user traffic -- [ ] Restore drill completed and dated, with measured recovery time and actual data loss +- [ ] Restore tested in a disposable, isolated environment using a compatible Harper version +- [ ] Restored inventory, counts, business invariants, and critical journeys validated +- [ ] Artifact ID, backup time, validation results, operator, approver, achieved RPO, and achieved RTO recorded - [ ] `system` database restore rehearsed offline +- [ ] Preservation export planned separately where records require long-term, software-independent access ## Objectives diff --git a/learn/administration/reliability-plan-template.mdx b/learn/administration/reliability-plan-template.mdx index a563f7723..269c97d53 100644 --- a/learn/administration/reliability-plan-template.mdx +++ b/learn/administration/reliability-plan-template.mdx @@ -5,7 +5,7 @@ sidebar_position: 10 The [Production Readiness Checklist](./production-readiness-checklist.mdx) asks the questions. This is the document that holds the answers. -Copy it, keep one per service, and keep it where your on-call can reach it during an incident rather than in a wiki nobody remembers. It is deliberately short. A reliability plan that takes a day to read is a reliability plan nobody consults at 3am. +Keep one plan per service, somewhere your on-call team can reach during an incident. Replace the example values with your deployment's measurements and procedures. Example values are included in italics to show the intended level of specificity. Replace them. @@ -36,9 +36,9 @@ The user-facing paths that must work. Everything else in this plan is stated per ## 3. Objectives -| Journey | Availability target | Measurement boundary | Recovery time objective | Recovery point objective | Exclusions | -| ------- | ------------------- | -------------------- | ----------------------- | ------------------------ | ------------------------------------------ | -| | _99.95%_ | _at the CDN edge_ | _15 min from detection_ | _5 min_ | _planned maintenance, third-party outages_ | +| Journey | Availability target | Measurement boundary | Recovery time objective | Recovery point objective | Exclusions | +| ------- | ------------------- | -------------------- | ------------------------------ | ------------------------ | ------------------------------------------ | +| | _99.95%_ | _at the CDN edge_ | _15 min from start of failure_ | _5 min_ | _planned maintenance, third-party outages_ | Monthly unavailability budget: _21 min 55 sec at 99.95%_ @@ -114,6 +114,13 @@ Maintenance policy when a drain would breach `F`: _pause, add capacity, or accep - Restore constraints per database: _online, or offline only because a component holds it open_ - `system` database restore procedure: _offline only, link to runbook_ - Failure classes assigned to restore versus forward repair: _table_ +- Artifact record: _immutable ID, backup timestamp, manifest and checksum locations_ +- Destination controls: _failure and administrative boundaries, encryption, versioning, retention, read-back evidence_ +- Backup writer permissions: _how retention changes are prevented_ +- Newest fully verified recovery point: _artifact ID, timestamp, validation results, operator, approver_ +- Restore validation: _database/table inventory, counts, business invariants, critical journeys, compatible Harper version_ +- Achieved RPO and RTO: _measured data loss and time from failure to verified service_ +- Preservation export, if required: _format, reader, and retention independent of Harper_ ## 11. Scenario responses @@ -150,7 +157,7 @@ Maintenance policy when a drain would breach `F`: _pause, add capacity, or accep ## 14. Open risks -Accepted gaps, each with an owner and a date. This section existing and being honest is worth more than it being empty. +Record accepted gaps with their impact, owner, and review date. | Risk | Impact if realized | Why accepted | Owner | Review date | | ---- | ------------------ | ------------ | ----- | ----------- | @@ -173,7 +180,7 @@ Most of the numbers in this plan are properties of a Harper version, a data volu - Significant data growth - Any incident that produced a surprise -Date every number. An undated measurement in a reliability plan is worse than a blank field, because a blank field prompts someone to go and measure. +Date each measurement so operators can judge whether it still describes the current service. ## Additional Resources diff --git a/learn/administration/safe-deployments-and-rollback.mdx b/learn/administration/safe-deployments-and-rollback.mdx index f5b38080e..db8584d1c 100644 --- a/learn/administration/safe-deployments-and-rollback.mdx +++ b/learn/administration/safe-deployments-and-rollback.mdx @@ -6,16 +6,16 @@ sidebar_position: 6 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -[Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx) gets an immutable artifact onto your cluster from a pipeline. This guide is about what happens around that call: how to limit who sees a change, what evidence justifies expanding it, and how to reverse it when the evidence says stop. +[Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx) covers installing a release from a pipeline. This guide covers limiting its initial traffic, deciding when to expand the rollout, and reversing it if checks fail. -Harper makes reversal unusually cheap, because your application is a component deployed from an immutable reference, so rolling back is deploying the previous reference. There is no image to rebuild and no instance to replace. That only helps if the previous reference is still addressable and someone has done it before under calm conditions, which is what makes rollback a designed capability rather than a hope. +To roll back component code, redeploy the previous immutable reference. Keep that reference addressable and rehearse the procedure before launch. ## What You Will Learn - How to separate build, deploy, activate, and expose into four decisions with four different control points - The change loop, and what counts as evidence at each step - Five rollout patterns and the Harper operations behind each -- What `"restart": true` and `"restart": "rolling"` actually do, which is not what most people assume +- How `"restart": true` and `"restart": "rolling"` differ, and how to verify completion - How to classify a change so you reverse the right thing, and why reversing the wrong thing can cause data loss ## Prerequisites @@ -27,7 +27,7 @@ Harper makes reversal unusually cheap, because your application is a component d ## Separate four decisions -Most bad deployments come from collapsing these into one action. +Decide separately what to build, where to install it, which behavior to enable, and who receives it. | Decision | The question | Harper control point | | ------------ | ----------------------------------------- | ----------------------------------------------------------------------------------------------- | @@ -36,31 +36,44 @@ Most bad deployments come from collapsing these into one action. | **Activate** | Which code path is actually enabled? | `urlPath` and `host` mounting, component configuration, a feature flag, a header or tenant rule | | **Expose** | Which production traffic reaches it? | Traffic layer weights, the availability flag, cohort or geography targeting | -Once these are separate you get options that do not exist otherwise. You can install an artifact on every node and activate it nowhere. You can activate a route and expose it only to internal traffic. And when something is wrong you can reverse exposure in seconds without touching what is installed, which is the fastest containment action available to you. +For example, validate a route with internal traffic before exposing it publicly. If errors rise, reduce traffic to the release or disable its feature flag while you investigate. :::danger -**`replicated` is opt-out, not opt-in.** A `deploy_component` call with no `replicated` field replicates to every peer. So does `add_component`, `drop_component`, `set_component_file`, `set_env_value`, `delete_env_value`, and the destructive `drop_database` and `drop_table`. Only `"replicated": false` changes anything; `"replicated": true` is the default spelled out. - -This is the opposite of what most operators assume, and it is why the single-node validation pattern below has to say `false` explicitly. If you intend to touch one node, you must say so. ([`set_configuration`](/reference/v5/configuration/operations) is the deliberate exception: it replicates only when you ask.) +**Component deployment replicates by default.** Pass `"replicated": false` when validating on one node. Other management operations also replicate by default; check the operation's reference before using it during an isolated rollout. [`set_configuration`](/reference/v5/configuration/operations) requires an explicit `"replicated": true`. ::: `urlPath` mounts a component at an HTTP path and `host` serves it on a virtual hostname. Routing by either has been available since v5.1.0. What arrived in v5.2.0 is declaring that routing on the application's entry in the root `harper-config.yaml`, and the matching `host` parameter on `deploy_component`. Declared that way, the routing is persisted deployment state rather than a runtime toggle. See [HTTP middleware routing](/reference/v5/http/overview#middleware-routing). ## The change loop -1. **Preflight.** Confirm the target version, the component inventory from `get_components`, `cluster_status` convergence, peer capacity with one node held out, backup posture if data is at risk, and that the previous known-good artifact is still addressable. +```mermaid +flowchart TD + Preflight --> Limit[Limit initial traffic] + Limit --> Observe + Observe --> Decide{Decide} + Decide -->|Advance| Expand[Expand traffic] + Expand -->|More stages| Observe + Expand -->|Rollout complete| Close[Verify and close] + Decide -->|Hold| Hold[Keep current exposure] + Hold --> Observe + Decide -->|Reverse| Reverse[Restore prior release or traffic] + Reverse --> Verify[Verify recovery] + Verify --> Close +``` + +1. **Preflight.** Confirm the target version, component inventory, replication convergence, peer capacity with one node held out, and backups if data is at risk. Check that the prior artifact is still addressable. Name the decision owner and set stop thresholds before starting. -2. **Limit.** Choose the smallest cohort that produces useful evidence. One drained node, an internal cohort, a low-risk geography, a tenant set, or a small weighted slice. Smaller is better right up until the sample is too small to distinguish signal from noise. +2. **Limit.** Start with one drained node, an internal group, selected tenants, a region, or a small traffic percentage. Choose enough requests to make the comparison meaningful. -3. **Observe.** Compare the new and old versions on the same metrics: request success, latency at your SLO percentile, worker saturation, logs and traces, data correctness, replication behavior, and downstream errors. Comparison against the other cohort is the point. Absolute numbers on the new version tell you much less. +3. **Observe.** Compare old and new versions on request success, SLO latency, worker saturation, data correctness, replication, and downstream errors. Use logs and traces to investigate differences. -4. **Decide.** Advance, hold, stop, or roll back, against thresholds declared before you started, with a named owner. Missing telemetry is a failed gate, not a pass. +4. **Decide.** Advance, hold, stop, or roll back against the agreed thresholds. Hold if required telemetry is missing. 5. **Expand.** Increase exposure only after minimum sample and hold conditions pass, and keep enough healthy capacity in the old version to reverse. 6. **Close.** Verify uniform artifact and configuration across nodes, restore intended traffic, record the actual outcome, and keep the evidence with the change record. -Declaring thresholds in step 4 before step 2 is the part that gets skipped and the part that matters. A threshold invented while looking at a live graph is not a threshold, it is a negotiation, and it always resolves toward shipping. +Record each decision and the measurements that supported it. ## Five rollout patterns @@ -102,7 +115,27 @@ Drain the node first, deploy, run your readiness route and journey synthetic aga } ``` -Same immutable reference, now cluster-wide. The deploy itself is finished when the call returns, so the thing still outstanding is the restart: poll the returned `restartJobId` with [`get_job`](/reference/v5/operations-api/operations#get_job). Read [`get_deployment`](/reference/v5/operations-api/operations#get_deployment) for the per-peer detail the response summarizes away, including which peers received the artifact and which failed. +On Harper Pro 5.2.15, this returns a `deployment_id` for the deployment record and a separate `restartJobId` for the rolling restart. Read each from the actual response: + +```json +{ + "operation": "get_job", + "id": "" +} +``` + +Poll the node that received the deployment. `get_job` returns an array; wait for its job's `status` to be `COMPLETE`, and stop on `ERROR`. An empty array means the job was not found on that node. Retry a failed request if the node is restarting. + +Use the other identifier to read the deployment details: + +```json +{ + "operation": "get_deployment", + "deployment_id": "" +} +``` + +On 5.2, check its `status` and `peer_results` for install and replication failures, then confirm the serving version directly on every node. Later releases change how the rolling job applies the release; use the version-specific [CI wait procedure](../developers/deploying-from-ci.mdx#path-a-deploy-a-tagged-release-from-your-repository) with your installed version. @@ -115,17 +148,17 @@ A pinned registry version such as `@my-org/orders-api@2.4.1` is the best-travell ## What restart actually does -This is worth reading carefully, because the naming invites a wrong assumption. +On Harper 5.2, the two modes behave as follows: **`"restart": true`** starts a restart of the HTTP worker threads on the node handling the call and returns immediately, without waiting for it. A `200` means the deploy succeeded and a restart has been requested. It does not mean the new code is serving yet. Until a worker has been replaced it is still running the previous code, and on platforms where replacements share a listening port it keeps accepting connections during the changeover. -**`"restart": "rolling"`** does not restart inline either, but it is observable. It starts a replicated `restart_service` job and returns a `restartJobId` you can poll with [`get_job`](/reference/v5/operations-api/operations#get_job). That job walks the cluster **one node at a time**, waiting for each node to come back before starting the next, so the cluster keeps serving throughout. +**`"restart": "rolling"`** starts a replicated `restart_service` job. Poll its `restartJobId` with [`get_job`](/reference/v5/operations-api/operations#get_job). It sequences node restarts, but your peers still need enough capacity and your traffic-admission checks must pass. :::warning -**On a cluster, prefer `"rolling"`.** `"restart": true` is forwarded verbatim to every peer, so every node restarts independently as it finishes applying the deploy. Nothing sequences them and nothing waits, so the restarts are free to overlap. You get no rolling coordination and no guarantee that any node is still serving, which makes this a cluster-wide availability risk rather than a node-local one. Reserve `"restart": true` for a single-node or development instance, and use `"rolling"` anywhere you care about staying up. See [Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx). +**Use `"rolling"` for cluster rollouts.** With `"restart": true`, each node restarts independently as it applies the replicated deployment. Those restarts can overlap and reduce serving capacity. Use `"restart": true` for the drained single-node validation above, or for a development instance. ::: -Two more consequences for your pipeline. Neither restart mode reports completion in the deploy response, so if you need to know the new code is live, poll the rolling restart job or probe the nodes themselves. And a failed restart does not fail the deploy: the component is installed and replicated either way, so your pipeline needs to check both outcomes separately rather than assuming one implies the other. +Wait for the rolling job and verify the serving version on each node. A successful deployment record alone does not establish that every worker is running the release. ### Waiting for the restart @@ -133,16 +166,16 @@ Two more consequences for your pipeline. Neither restart mode reports completion From v5.3.0, `"restart": true` waits for the worker restart to finish before responding, rather than returning as soon as it has been requested. The wait follows the restart's own progress rather than a fixed timeout, so the call can take tens of seconds on a slow install with many worker threads, with a hard ceiling of ten minutes. -Two things do not change, and both matter more than the wait itself. A caller that gives up early does not stop the restart, it only loses the result. And the response still does not carry the restart's outcome: a restart that stalls, times out, or leaves workers on the old code is reported in the node's log, not to you. So even on v5.3.0, treat a `200` as "the deploy landed and a restart ran," and confirm the version actually serving through your readiness route or journey synthetic rather than through the deploy response. +A caller timeout does not stop the restart. On v5.3, the deploy response does not report restart success; check the logs for restart failures and verify the serving version through readiness or a journey test. For later versions, check the [restart and certification behavior](/reference/v5/operations-api/operations#restarting-restart) before choosing pipeline checks. -Two parameters worth setting deliberately on replicated deploys: +Set these parameters to match your payload size, network, and failure policy: - `deployment_timeout` (Added in: v5.1.4) is how long a peer waits for the replicated payload before failing, defaulting to 120000 ms. Raise it for large components or slow links. - `ignore_replication_errors` (Added in: v5.1.4) treats a peer that fails to receive the deploy as non-fatal. By default a failed peer makes the whole operation return a non-2xx status, while the component is still deployed on the origin node. Decide which behavior you want before you need it, because the default leaves you in a mixed-version state with a failed response, and that is a confusing thing to reason about mid-incident. ## Configuration changes are deployments too -A configuration change carries the same risk as a code change and gets less ceremony, which is backwards. +Apply the same review, rollout, and rollback checks to configuration changes. `set_configuration` supports `"replicated": true` (Added in: v5.2.0) to apply a change across the cluster in one call, with per-node outcomes in the response. To finish the change cluster-wide, follow with `restart_service` using `"replicated": true`, which restarts nodes one at a time. @@ -158,7 +191,7 @@ A configuration change carries the same risk as a code change and gets less cere Only replicate cluster-appropriate parameters. Node-local values such as ports, `node.hostname`, file paths, TLS material, and `replication.hostname`, `url`, or `routes` would overwrite every peer's own values. Replicating one of these is a cluster-wide outage delivered in a single API call. ::: -Two more things to hold onto. A change takes effect only after a restart, so a node that has been reconfigured and not restarted is running the old configuration while reporting the new one. And `get_status` reports a `restartRequired` flag, but it tracks component and code restarts rather than configuration changes, so it will not tell you a configuration change is still pending. Track pending configuration in your change record instead, and read back `get_configuration` after the restart to confirm. +Configuration changes take effect after restart. `get_status.restartRequired` tracks component and code restarts, so it does not identify pending `set_configuration` changes. Record those changes yourself, restart the affected nodes, and read back `get_configuration` to confirm the intended values. ## Rollback is a designed capability @@ -193,7 +226,7 @@ Rehearse a stopped rollout end to end on a non-production cluster: 4. Reverse it, and time from decision to restored traffic. That number is your release RTO, and it belongs in your reliability plan. 5. Verify uniform state afterwards: `get_components` on every node, and `list_deployments` showing the reversal. -The step people fail is 5. A partially reversed cluster looks fine on a dashboard because the healthy majority dominates the average. +Check every node after rollback; a cluster average can hide nodes still running the failed release. ## Operational notes diff --git a/learn/administration/sizing-a-harper-cluster.mdx b/learn/administration/sizing-a-harper-cluster.mdx index 4e695fdb7..d4998acc8 100644 --- a/learn/administration/sizing-a-harper-cluster.mdx +++ b/learn/administration/sizing-a-harper-cluster.mdx @@ -6,14 +6,14 @@ sidebar_position: 3 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -Most clusters get sized by dividing expected peak throughput by measured per-node throughput and rounding up. That arithmetic produces a cluster that is exactly large enough to handle a good day, which means the first node failure during peak traffic becomes a user-visible outage. +Size a Harper cluster so that, after losing `F` nodes, the survivors can carry peak demand within your utilization limit. -Sizing Harper starts from a different question. Not "how much traffic do we have," but "how many nodes are we willing to lose at once, and can the survivors carry the load." This guide gives you the capacity rule that follows from that, what to include when you measure a node, and how to turn an availability target into an operating budget you can actually spend. +This guide works through the capacity calculation, the load tests it depends on, and the headroom needed for maintenance and recovery. ## What You Will Learn - The capacity invariant that sizes for the failure state, with a worked example -- What to include in a per-node throughput measurement, and the three things people leave out +- What workload and recovery conditions to include in a per-node throughput measurement - Why maintenance is a planned failure, and what that means for change windows - How to choose a topology from your objectives rather than from a diagram - How to convert an availability SLO into minutes per month, and what each tier demands of your automation @@ -43,13 +43,13 @@ Surviving nodes needed = 17,143 / 5,000 = 3.43 -> 4 N = 4 + F (1) = 5 nodes ``` -Five nodes, not three. The naive calculation gives 12,000 / 5,000 = 2.4, rounded up to 3, and that cluster degrades the moment anything goes wrong. +This requires five nodes. Three would handle the peak only while all remain available, without the chosen utilization headroom. -Two things to notice. Increasing `F` from 1 to 2 costs you one more node here, not double the cluster, so tolerating a second simultaneous failure is often cheaper than people expect at this size. And the utilization factor is doing as much work as `F` is: sizing to 100 percent of measured capacity means your "surviving" nodes are at saturation, where latency degrades long before throughput does. +In this example, tolerating two simultaneous failures requires six nodes. The 70 percent utilization limit also matters: it leaves the surviving nodes room to handle variable load while keeping latency acceptable. -## Measure a node honestly +## Measure per-node capacity -The per-node number in that formula is the one most likely to be wrong, because benchmark conditions are kinder than production. Include all of this: +Use production-like conditions when measuring throughput: - **Representative application code, data shape, and query mix.** A Harper node runs your component logic in the same process as the data access, so your code is part of the capacity measurement in a way it would not be for a standalone database. A synthetic key-value benchmark tells you very little about the node's capacity to serve your journey. - **Realistic downstream latency.** If your resource calls an upstream pricing service, its latency occupies worker capacity on the node. @@ -62,7 +62,7 @@ Measure each node separately rather than dividing a cluster total by node count. ```bash -curl -s -X POST https://my-node.example.com:9925/ \ +curl -s -X POST https://operations.my-node.example.com/ \ -H 'Content-Type: application/json' \ -u 'admin:password' \ -d '{"operation":"system_information","attributes":["cpu","memory","disk","network","threads","harperdb_processes"]}' @@ -72,7 +72,7 @@ curl -s -X POST https://my-node.example.com:9925/ \ ```javascript -await fetch('https://my-node.example.com:9925/', { +await fetch('https://operations.my-node.example.com/', { method: 'POST', headers: { 'Content-Type': 'application/json', @@ -100,7 +100,7 @@ Draining a node for a deploy, an upgrade, or an investigation consumes exactly t The operating rule that follows: if draining one node means the service can no longer tolerate its declared `F`, pause the change. Either wait for lower traffic, or add capacity for the window, or accept and document the reduced tolerance for a bounded period. What you should not do is treat the maintenance window as free because nothing is technically broken. -This is also the argument for sizing `F` at 2 in a cluster that deploys frequently. It is not paranoia about correlated hardware failure. It is that you want to be able to deploy during business hours and still survive an incident. +Sizing for `F = 2` can let you drain one node for maintenance and still tolerate another node's failure at peak load. ## Choose a topology from objectives @@ -113,7 +113,7 @@ Pick the pattern that matches the failures you actually need to survive, and kno | Multiple regions | Regional impairment or user locality is a real requirement | Data authority per region, route propagation time, convergence, degraded-mode behavior | | Edge or on-premises peers | Data locality or residency constrains where data can live | Replication scope per location, identity provisioning, backup path from each site | -The right-hand column is the useful one. A topology diagram is a claim, and the claim is only true once you have exercised it. "Multiple failure domains" means nothing if both domains draw from the same power feed, and you will not discover that from your provider's documentation. +Test the failures your topology is meant to survive. Confirm that supposedly independent domains do not share a power supply, network path, or other dependency that could take them down together. Note that adding a node is a data movement event as well as a capacity event, since a Harper node carries data along with request handling. A node whose databases have never synchronized downloads them in full before it is useful, so scaling out is not instantaneous and cannot be your response to an unexpected traffic spike. Size ahead of demand. @@ -129,7 +129,7 @@ An availability target is a quantity of unavailability you are permitted to spen Figures assume a 30.44 day month. Your contractual definition, exclusions, and measurement boundary may differ, and the boundary matters more than the number: availability measured at your CDN edge and availability measured at the Harper node are different quantities. -The practical consequence is that the SLO tier determines how much automation you need, not how many nodes. At three nines a person can be paged, look at a dashboard, and drain a node. At four nines that same sequence has already spent the entire month's budget. +Compare your measured detection and recovery times with the budget. A manual response that fits a 99.9 percent target may consume the entire budget at 99.99 percent. ### Prove it @@ -138,7 +138,7 @@ Run the failure state rather than calculating it. On a cluster carrying represen 1. Record per-node request rate, latency, and saturation as a baseline. 2. Drain one node using the sequence from [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). 3. Measure what the survivors do: request rate, latency at your SLO percentile, CPU and worker pressure, and whether any user-visible errors appeared during the transition. -4. While still down a node, bring the drained node back and let it synchronize, so the survivors are carrying peak traffic and feeding a recovering peer at the same time. This is the real worst case and it is the one nobody tests. +4. Keep the drained node out of user traffic while it rejoins replication. Measure the survivors while they carry peak traffic and feed the recovering peer. 5. Compare measured surviving capacity against your invariant. If step 4 pushed latency out of SLO, your target utilization is too high or your `N` is too low. ## Operational notes diff --git a/reference/backups/operations.md b/reference/backups/operations.md index d736a460c..80f71bc6a 100644 --- a/reference/backups/operations.md +++ b/reference/backups/operations.md @@ -33,7 +33,7 @@ Offline invocation matters most for restore: an in-place `restore_backup` of the Creates an incremental directory backup of the database under the configured backup root ([`storage.backupPath`](../configuration/options.md#storage), default `/backup`). Through a running server this runs as a background [job](../operations-api/operations.md#jobs): the operation returns a `job_id` immediately, and [`get_job`](../operations-api/operations.md#get_job) reports the outcome including the new `backup_id`, `size`, and `timestamp`. -Backups of the same database share unchanged RocksDB data files, so the second and subsequent backups only copy the data that changed. The transaction-log snapshot — and the blob snapshot, if the database has file-backed blobs — is copied in full on every backup, so those portions are not incremental; with a large audit-retention window, frequent backups are not free for them. Pass `exclude_blobs: true` to skip the blob snapshot. Use [`purge_backups`](#purge_backups) to manage retention. +Backups of the same database share unchanged RocksDB data files within the repository. Each backup also copies the transaction-log snapshot. Blob snapshots use hard links when the source and destination filesystem support them, and copy the files otherwise. Pass `exclude_blobs: true` to skip the blob snapshot. Measure physical repository growth to plan capacity, and use [`purge_backups`](#purge_backups) to manage retention. ```json { "operation": "create_backup", "database": "data" } @@ -68,7 +68,13 @@ harper list_backups database=data -Verifies a managed backup's RocksDB file sizes — and their checksums when `verify_checksum` is `true` (slower) — together with the framing of its transaction-log snapshot (always checked). For a backup that recorded blobs, verification confirms the blob snapshot is still present and fails the backup as corrupt if it is missing, but it does not check individual blob files for size, checksum, or readability — so verification can pass on a backup whose blob contents are damaged. Through a running server this runs as a background [job](../operations-api/operations.md#jobs). `backup_id` is required. +Verifies a managed backup. `backup_id` is required. + +- **RocksDB files:** checks file sizes and, when `verify_checksum` is `true`, checksums. Checksum verification takes longer. +- **Transaction log:** always checks the snapshot's framing. +- **Blob snapshot:** checks that a snapshot is present if the backup recorded blobs. A missing snapshot fails verification. Individual blob sizes, checksums, and readability are not checked, so damaged blob contents can pass verification. + +Through a running server, verification runs as a background [job](../operations-api/operations.md#jobs). Poll `get_job` for its result. ```json { "operation": "verify_backup", "database": "data", "backup_id": 1, "verify_checksum": true } diff --git a/reference/backups/overview.md b/reference/backups/overview.md index 4db448213..815a730ff 100644 --- a/reference/backups/overview.md +++ b/reference/backups/overview.md @@ -20,7 +20,7 @@ Managed backups use the RocksDB backup engine. Each backup is a directory under Every managed backup includes the database's transaction log — so a restored database keeps its `read_audit_log` history as of the backup point — and, for a database with file-backed blobs, a snapshot of those blobs (skip it with `exclude_blobs`). -Only the RocksDB data files are shared and incremental. The transaction log and the blob snapshot are each copied in full on every backup, so with a large audit-retention window or many blobs, frequent backups cost more than the data-only view suggests. +Each backup copies the transaction-log snapshot. Blob snapshots use hard links when supported on the same filesystem, and copy the files otherwise. Measure physical repository growth over successive backups, including the log copies and blob snapshots. Exports that do not preserve file sharing require space for the full materialized copies. The long-running operations (`create_backup`, `verify_backup`, `restore_backup`) run as background [jobs](../operations-api/operations.md#jobs) when invoked through a running server: the operation returns a `job_id` immediately, and [`get_job`](../operations-api/operations.md#get_job) reports the outcome. @@ -34,7 +34,7 @@ The offline path matters for restore. RocksDB is single-writer, so an in-place r - **Managed backups require the RocksDB storage engine.** For LMDB databases, use [`get_backup`](./operations.md#get_backup) or [volume snapshots](../cli/commands.md#how-backups-work). - **Whole-database granularity.** There is no per-table backup or restore. (The one exception: `get_backup` on an LMDB database can stream individual tables, and includes the audit store only with `include_audit`.) -- **One storage root per database.** A database whose tables use per-table `path` storage configs spans multiple root stores and cannot be backed up with these operations. +- **One storage root per database.** Backup is rejected if the database's tables resolve to multiple storage roots. A custom database-level `storage.path`, or per-table paths that still resolve to a single root, are supported. - **Backups live on the node that created them.** The backup repository is a local directory. RocksDB shares files across backup IDs (a backup ID is not a self-contained folder), so disaster-recovery copies must take the entire per-database repository — `/` — not an individual backup, and must do so while no backup operation is running, or from an atomic filesystem snapshot. A live recursive copy can race `create_backup`/`delete_backup`/`purge_backups` and produce an unrestorable copy. Alternatively, use `get_backup` to pull a snapshot from a running server. - **`get_backup` always streams the current state.** It cannot download a historical managed backup; to move a retained backup off-host, copy its whole per-database repository as above. - **A restore is a point-in-time rollback.** In a replicated cluster, coordinate a restore with replication before bringing the node back. @@ -63,7 +63,7 @@ Create a managed backup of the `data` database (the default): harper create_backup database=data ``` -The first backup copies the entire database; each one after that only copies the RocksDB data files that changed, so re-backing-up the data is cheap. (The transaction-log snapshot, and the blob snapshot for databases with file-backed blobs, are copied in full each time — see [how managed backups work](#how-managed-backups-work).) Schedule `create_backup` as often as your recovery point requires, and manage retention with `purge_backups`: +Later backups share unchanged RocksDB data files, while transaction-log copies and blob snapshots have their own storage cost; see [how managed backups work](#how-managed-backups-work). Schedule `create_backup` as often as your recovery point requires, and manage retention with `purge_backups`: ```bash harper purge_backups database=data keep_count=7 From 1c9665c0a46756ec43dfd44bf4551e307f2a1547 Mon Sep 17 00:00:00 2001 From: Ethan Arrowood Date: Wed, 14 Oct 2026 13:57:06 -0600 Subject: [PATCH 12/21] Tighten readiness and recovery guidance after independent review Co-Authored-By: OpenAI Codex --- learn/administration/backup-and-recovery.mdx | 10 +- .../health-checks-and-traffic-admission.mdx | 93 ++++++++++++------- .../administration/monitoring-and-triage.mdx | 2 +- .../production-readiness-checklist.mdx | 1 + .../reliability-plan-template.mdx | 1 + .../safe-deployments-and-rollback.mdx | 8 +- reference/backups/operations.md | 2 + reference/components/applications.md | 6 +- reference/configuration/options.md | 2 +- reference/replication/clustering.md | 1 + reference/replication/overview.md | 6 +- release-notes/v5-lincoln/5.2.md | 2 +- 12 files changed, 86 insertions(+), 48 deletions(-) diff --git a/learn/administration/backup-and-recovery.mdx b/learn/administration/backup-and-recovery.mdx index e85d63ae5..8b2f356c7 100644 --- a/learn/administration/backup-and-recovery.mdx +++ b/learn/administration/backup-and-recovery.mdx @@ -90,6 +90,10 @@ Each backup is a full logical snapshot, but its disk cost depends on file sharin - RocksDB backups share unchanged data files within the same repository. - The transaction-log snapshot is copied for each backup. Include it when measuring repository growth. - Blob snapshots use hard links when the source and destination filesystem support them. Otherwise, they copy the files. Use `exclude_blobs: true` only if your recovery procedure can recover those blobs separately. +- Restoring a backup created with `exclude_blobs: true` requires `allow_engine_only: + true`, even into a new database. This explicitly accepts the risk of records pointing at missing or unrelated blob + files. Include separate blob recovery and validation in the drill; do not add the flag just to bypass the safety + check. - An export that does not preserve sharing materializes those shared or linked files as separate copies. Budget for the full exported size. Measure physical repository growth over several backups with your retention policy and filesystem. Do not assume that a full logical snapshot consumes its full logical size, or that all snapshot files are hard-linked. @@ -168,7 +172,7 @@ Also include these constraints in the restore procedure: ## Restoring in a replicated cluster -A restored node may be behind peers that have continued accepting writes. Choose which data should be authoritative before reconnecting it. +A restored node may be behind peers that have continued accepting writes. Peers can replay newer writes, including the destructive mutations you intended to undo. Restoring one node does not make its backup authoritative; choose which data to preserve or repair and rehearse the re-entry policy before reconnecting it. ```mermaid flowchart TD @@ -187,11 +191,11 @@ flowchart TD For an in-place recovery: 1. Take the node out of rotation with the availability flag, per [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). Never restore a node that is serving traffic. -2. **Isolate replication separately.** Stop peer exchange using your tested topology or network controls. Keep that isolation in place across restarts. +2. **Isolate replication separately.** For example, use a network policy that blocks peer replication in both directions and closes existing peer connections. Keep it effective across restarts, and verify that transactions cannot cross the boundary. Removing a node from the load balancer or setting `"replicated": false` on one operation does not isolate replication. 3. Confirm the data authority and replay policy: which writes to preserve, repair, or reapply, and how peers will converge. 4. Perform the restore, online or offline according to the table above. 5. Verify data expectations locally before any peer sees the node. -6. Reconnect replication under that policy and verify convergence. Then complete the traffic-admission checks before restoring traffic. +6. Reconnect replication under the policy you proved in an isolated multi-node drill. If that policy would reintroduce corruption from a peer, keep the affected peers isolated and reconcile their data first. Verify convergence, then complete the traffic-admission checks before restoring traffic. :::danger **Draining traffic does not stop replication.** A drained node continues exchanging transactions with peers. An offline restore stops that exchange while Harper is down, but configured routes can reconnect as soon as it starts. Keep replication isolated until validation is complete and you are ready to execute the re-entry policy. diff --git a/learn/administration/health-checks-and-traffic-admission.mdx b/learn/administration/health-checks-and-traffic-admission.mdx index db7580ef8..28d483207 100644 --- a/learn/administration/health-checks-and-traffic-admission.mdx +++ b/learn/administration/health-checks-and-traffic-admission.mdx @@ -31,7 +31,7 @@ This guide explains the checks to use and the order for draining a node, changin | -------------------------- | ---------------------------------------------- | ----------------------------------------------------- | ---------------------------------- | | Process liveness | A trivial response on the node's HTTP endpoint | Harper is running | Process supervisor, restart policy | | Node availability flag | `GET /status` on the application URL | An operator or automation says this node should serve | Traffic layer health check | -| Application readiness | A component route on the application URL | This node can serve this journey right now | Traffic layer, or your own gating | +| Application readiness | A component route on the application URL | This node passes its admission checks | Operator or admission controller | | Critical-journey synthetic | A real request through the public route | Users are actually being served | Your SLO and alerting | The availability flag lets an operator or automation control whether a node receives new traffic. Use it as the routing signal, and use readiness, replication checks, and journey tests to decide when to set it. @@ -118,40 +118,67 @@ On Fabric, `set_status` with `id=availability` controls the record its status co Liveness and the availability flag both answer questions about the node. Readiness answers a question about the journey: if traffic arrives for this route right now, will it succeed? -Check only the dependencies needed by the journey routed to this node. A failure in an unrelated subsystem should not remove capacity from a working journey. +Use readiness to gate admission, not as the load balancer's direct routing signal. Keep node-local failures separate from shared dependency failures: taking every node out during a shared outage removes capacity without fixing the dependency. Report shared failures as degradation and use journey tests to measure their user impact. + +Add a node-local probe table to your application's schema. Keep the `graphqlSchema` and `rest` plugins enabled so the authenticated seeding request below can reach the table: + +```graphql +type ReadinessProbe @table(replicate: false) @export { + id: ID @primaryKey + description: String +} +``` Add a resource to your application's `resources.js`: ```javascript import { Resource, tables } from 'harper'; +let cachedChecks; +let cacheUntil = 0; +let inFlight; + +async function checkReadiness() { + const checks = {}; + + // Keep the node-local storage check to one primary-key lookup. + // Use a fresh server-side context for this fixed, non-sensitive read. + try { + const sentinel = await tables.ReadinessProbe.get('readiness-probe-sentinel', {}); + checks.data = sentinel ? 'ok' : 'failed'; + } catch (error) { + checks.data = 'failed'; + } + + // A shared dependency outage is degradation, not a reason to drain every node. + try { + const res = await fetch('https://pricing.internal.example.com/health', { + signal: AbortSignal.timeout(500), + }); + checks.pricing = res.ok ? 'ok' : 'degraded'; + } catch (error) { + checks.pricing = 'degraded'; + } + return checks; +} + +function getChecks() { + if (Date.now() < cacheUntil) return Promise.resolve(cachedChecks); + return (inFlight ??= checkReadiness() + .then((checks) => { + cachedChecks = checks; + cacheUntil = Date.now() + 2000; + return checks; + }) + .finally(() => { + inFlight = undefined; + })); +} + export class Readiness extends Resource { static async get() { - const checks = {}; - - // A bounded read against the table this journey serves. - // Keep it to a single primary-key lookup, never a scan. - // A missing record resolves to undefined rather than throwing, - // so check the result as well as catching a storage fault. - try { - const sentinel = await tables.Product.get('readiness-probe-sentinel'); - checks.data = sentinel ? 'ok' : 'failed'; - } catch (error) { - checks.data = 'failed'; - } - - // Only the downstream dependencies this route needs. - try { - const res = await fetch('https://pricing.internal.example.com/health', { - signal: AbortSignal.timeout(500), - }); - checks.pricing = res.ok ? 'ok' : 'failed'; - } catch (error) { - checks.pricing = 'failed'; - } - - const ready = Object.values(checks).every((v) => v === 'ok'); - + const checks = await getChecks(); + const ready = checks.data === 'ok'; return new Response(JSON.stringify({ ready, checks, version: process.env.APP_VERSION }), { status: ready ? 200 : 503, headers: { 'Content-Type': 'application/json' }, @@ -163,21 +190,21 @@ export class Readiness extends Resource { The `jsResource` plugin loads the exported class. Keep it enabled if your application has an explicit `config.yaml`. At the default application mount, the route is `/Readiness`; adjust the probe URL if you configured `urlPath` or `host` routing. See [resource routing](/reference/v5/resources/overview#exporting-resources-as-endpoints) for the path rules. :::note -Because the handler is a `static` method, it replaces Harper's built-in dispatch along with the authorization check that lives inside it, so `GET /Readiness` is unauthenticated. That is what a load balancer probe needs, and it is why the endpoint must not return anything you would not publish. Keep the response to check names and outcomes, never connection strings, credentials, or internal hostnames. +Because the handler is a `static` method, it replaces Harper's built-in dispatch along with its authorization check, so `GET /Readiness` is unauthenticated. Keep the response to check names and outcomes, never connection strings, credentials, or internal hostnames. The shared in-flight check and two-second cache bound repeated reads and outbound requests per worker; also restrict probe access or rate-limit it at your traffic boundary. -If you rewrite this as an instance `get()` instead, authorization comes back and defaults to `super_user` only, which will make your probe start failing with an authorization error rather than a readiness one. In that form you need `allowRead() { return true; }` to keep it reachable. +If you rewrite this as an instance `get()` instead, authorization comes back and defaults to `super_user` only. Configure explicit authorization for your probe caller rather than assuming that the route remains public. ::: -Create the sentinel record before enabling the probe. The example assumes `Product` is exported at the server root: +Create the sentinel record on each node before enabling the probe. The example assumes `ReadinessProbe` is exported at the server root: ```bash -curl -s -X PUT https://my-node.example.com/Product/readiness-probe-sentinel \ +curl -s -X PUT https://my-node.example.com/ReadinessProbe/readiness-probe-sentinel \ -H 'Content-Type: application/json' \ -u 'admin:password' \ - -d '{"name":"readiness probe sentinel","description":"do not delete"}' + -d '{"description":"readiness probe sentinel - do not delete"}' ``` -If the table replicates, seed once and confirm the record reaches every node. Otherwise, seed it on each node. Keep the record in place while the probe is in use. +Keep the probe table node-local and exclude it from application cleanup jobs. A shared pricing failure reports `degraded` but does not make every node unavailable. A `200` therefore establishes node-local readiness, not that every journey dependency is healthy; check the response and the journey synthetic as well. Bound downstream checks with a timeout. Set `APP_VERSION` in your deployment so the response identifies the component build answering during a rollout. diff --git a/learn/administration/monitoring-and-triage.mdx b/learn/administration/monitoring-and-triage.mdx index 0e35bfe02..c829a909a 100644 --- a/learn/administration/monitoring-and-triage.mdx +++ b/learn/administration/monitoring-and-triage.mdx @@ -123,7 +123,7 @@ threads: preload: dd-trace/register.js # ESM loader hooks for automatic instrumentation ``` -`threads.preload` (Added in: v5.2.0) loads a module via Node's `--import`, which is how an agent installs the loader hooks that let it instrument modules imported later. `threads.preloadRequire` (Added in: v5.2.0) uses `--require`, which runs the module body and is typically how an agent's initialization entry actually starts it. Both are documented under [threads configuration](/reference/v5/configuration/options#threads). +`threads.preload` loads a module via Node's `--import`, which is how an agent installs the loader hooks that let it instrument modules imported later. `threads.preloadRequire` uses `--require`, which runs the module body and is typically how an agent's initialization entry actually starts it. Both are documented under [threads configuration](/reference/v5/configuration/options#threads). Agent entry points vary: some install loader hooks and start the agent together; others need separate `preload` and `preloadRequire` entries. Loading hooks without starting the agent can produce no-op spans and no exports. diff --git a/learn/administration/production-readiness-checklist.mdx b/learn/administration/production-readiness-checklist.mdx index 03c68a999..8935c1b7a 100644 --- a/learn/administration/production-readiness-checklist.mdx +++ b/learn/administration/production-readiness-checklist.mdx @@ -83,6 +83,7 @@ From [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx). - [ ] Artifacts immutable, referenced by pinned version rather than by branch - [ ] Deploy and expose are separate actions in the pipeline - [ ] Previous known-good artifact addressable, and redeploying it rehearsed +- [ ] On v5.3.0 and later, the previous `deployment_id` still retained on every target node and activation rehearsed - [ ] Cohort ladder defined, from smallest useful sample to full exposure - [ ] Stop thresholds and hold times declared before a rollout begins - [ ] Named decision owner for advance, hold, stop, and roll back diff --git a/learn/administration/reliability-plan-template.mdx b/learn/administration/reliability-plan-template.mdx index 269c97d53..768c88f4a 100644 --- a/learn/administration/reliability-plan-template.mdx +++ b/learn/administration/reliability-plan-template.mdx @@ -90,6 +90,7 @@ Maintenance policy when a drain would breach `F`: _pause, add capacity, or accep - Stop thresholds and hold times: _stated numerically_ - Decision authority for advance, hold, stop, roll back: _name or role_ - Restart mode used, and how completion is confirmed: _`rolling`, polling the job_ +- Rollback reference and retention check: _pinned package, or retained `deployment_id` on every node for v5.3.0 and later_ - Measured release reversal time: _value, date_ - Forward repair procedures for data-affecting releases: _link_ - Change record location: _link_ diff --git a/learn/administration/safe-deployments-and-rollback.mdx b/learn/administration/safe-deployments-and-rollback.mdx index db8584d1c..e385200e8 100644 --- a/learn/administration/safe-deployments-and-rollback.mdx +++ b/learn/administration/safe-deployments-and-rollback.mdx @@ -170,14 +170,14 @@ A caller timeout does not stop the restart. On v5.3, the deploy response does no Set these parameters to match your payload size, network, and failure policy: -- `deployment_timeout` (Added in: v5.1.4) is how long a peer waits for the replicated payload before failing, defaulting to 120000 ms. Raise it for large components or slow links. -- `ignore_replication_errors` (Added in: v5.1.4) treats a peer that fails to receive the deploy as non-fatal. By default a failed peer makes the whole operation return a non-2xx status, while the component is still deployed on the origin node. Decide which behavior you want before you need it, because the default leaves you in a mixed-version state with a failed response, and that is a confusing thing to reason about mid-incident. +- `deployment_timeout` is how long a peer waits for the replicated payload before failing, defaulting to 120000 ms. Raise it for large components or slow links. +- `ignore_replication_errors` treats a peer that fails to receive the deploy as non-fatal. By default a failed peer makes the whole operation return a non-2xx status, while the component is still deployed on the origin node. Decide which behavior you want before you need it. ## Configuration changes are deployments too Apply the same review, rollout, and rollback checks to configuration changes. -`set_configuration` supports `"replicated": true` (Added in: v5.2.0) to apply a change across the cluster in one call, with per-node outcomes in the response. To finish the change cluster-wide, follow with `restart_service` using `"replicated": true`, which restarts nodes one at a time. +`set_configuration` supports `"replicated": true` to apply a change across the cluster in one call, with per-node outcomes in the response. To finish the change cluster-wide, follow with `restart_service` using `"replicated": true`, which restarts nodes one at a time. ```json { @@ -209,6 +209,7 @@ Classify the change before choosing a reversal path, because these have differen Practices that make each of these real: - **Keep the previous package addressable and rehearse redeploying it** before launch, not during an incident. +- **Prefer activating a retained release on v5.3.0 and later.** Use `deploy_component` with the previous `deployment_id` and the appropriate restart mode to restore its installed bytes without resolving or installing again. Verify that every target node still retains it: `deployment.stagingRetention.maxCount` can prune replaced releases. See [going back to a previous release](/reference/v5/operations-api/operations#going-back-to-a-previous-release). - **Prefer expand-then-contract schema evolution.** Add the new shape, migrate, then remove the old shape in a separate change. Destructive schema behavior is operation and version dependent, so each destructive change needs an exact written procedure and verification on every node. - **Use the same gates for rollback as for forward movement.** Availability, journey synthetic, peer stability, data validation, traffic reconciliation. A rollback is a deployment and can fail like one. @@ -241,6 +242,7 @@ Check every node after rollback; a cluster average can hide nodes still running - [ ] Artifacts are immutable and referenced by pinned version, never by branch - [ ] Deploy and expose are separate actions in your pipeline - [ ] The previous known-good artifact is addressable, and redeploying it has been rehearsed +- [ ] On v5.3.0 and later, the previous `deployment_id` is still retained on every target node and activation has been rehearsed - [ ] Cohort ladder defined, from smallest useful sample to full exposure - [ ] Stop thresholds and hold times declared before the rollout starts - [ ] A named decision owner for advance, hold, stop, and roll back diff --git a/reference/backups/operations.md b/reference/backups/operations.md index 80f71bc6a..b2efa9ba7 100644 --- a/reference/backups/operations.md +++ b/reference/backups/operations.md @@ -118,6 +118,8 @@ harper purge_backups database=data keep_count=3 Restores a database in place from a managed backup. `backup_id` defaults to the latest backup. The audit/transaction log is restored alongside the data, and — for a database with file-backed blobs — the blob roots are purged and rewritten from the backup's blob snapshot. + `allow_engine_only` (optional, default `false`) permits restoring a backup created with `exclude_blobs: true`. Without it, that restore is refused, including when using `target_database`. Enabling it accepts the risk that restored records reference missing or unrelated blob files; recover and validate those blobs separately. + Through a running server this runs as a background [job](../operations-api/operations.md#jobs): Harper closes the database across all worker threads, restores it, and reloads it. This works only when no loaded component is holding the database open — if one does, the job ends in `ERROR` (surfaced by [`get_job`](../operations-api/operations.md#get_job)) telling you to restore offline. Restoring the `system` database is rejected up front, before a job is created, as is `target_database` while the server is running. These cases require running the command from the CLI with the server stopped. See [when can a database be restored?](./overview.md#when-can-a-database-be-restored) As of v5.3.1, subscriptions opened before a restore do not carry over to the restored data. When Harper reloads the database, each one ends, and its last message is a `DatabaseGenerationChangedError` (status code 409, code `DATABASE_GENERATION_CHANGED`). Resubscribe to resynchronize against the restored state. A restore that fails before it changes anything reloads the database as it was; those subscriptions end instead with a retryable `DatabaseClosingError` (status code 503, code `DATABASE_CLOSING`); resubscribe to continue. diff --git a/reference/components/applications.md b/reference/components/applications.md index 1f6c84d10..88bc0da81 100644 --- a/reference/components/applications.md +++ b/reference/components/applications.md @@ -466,7 +466,7 @@ Creates a new component project in the component root directory using a template - `install_command` _(optional)_ — Install command. Defaults to `npm install` - `install_timeout` _(optional)_ — Install timeout in milliseconds. Defaults to `300000` (5 minutes) - `install_allow_scripts` _(optional)_ — Allow install scripts to run. Defaults to `false`, which causes `--ignore-scripts` to be passed to the install command (this is ignored with `install_command`). -- `replicated` _(optional)_ — Replicate to all cluster nodes. Defaults to `true`; pass `false` to apply on this node only +- `replicated` _(optional)_ — On Harper Pro and Fabric, replicate to all cluster nodes by default; pass `false` to apply on this node only. Harper core on its own does not replicate ```json { @@ -570,7 +570,7 @@ On a `.env` file it replaces the whole file with the payload, so saving back the - `file` _(required)_ — Path relative to project folder - `payload` _(required)_ — File content to write - `encoding` _(optional)_ — File encoding. Defaults to `utf8` -- `replicated` _(optional)_ — Replicate the update to all cluster nodes. Defaults to `true`; pass `false` to apply on this node only +- `replicated` _(optional)_ — On Harper Pro and Fabric, replicate the update to all cluster nodes by default; pass `false` to apply on this node only. Harper core on its own does not replicate ```json { @@ -593,7 +593,7 @@ For deploying from private repositories, SSH keys must be registered on the Harp - `host` _(required)_ — Host alias for SSH config (used in `package` URL): a single alias, not a pattern - `hostname` _(required)_ — Actual domain (e.g., `github.com`) - `known_hosts` _(optional)_ — Public SSH keys of the host. Auto-retrieved for `github.com` -- `replicated` _(optional)_ — Replicate to all cluster nodes. Defaults to `true`; pass `false` to apply on this node only +- `replicated` _(optional)_ — On Harper Pro and Fabric, replicate to all cluster nodes by default; pass `false` to apply on this node only. Harper core on its own does not replicate A `key`, `host` or `hostname` that ssh couldn't use is refused with a `400` naming the problem; see [what `key`, `host` and `hostname` must be](../operations-api/operations.md#what-key-host-and-hostname-must-be). Each key's settings live in a block of `/ssh/config` that Harper owns; you can edit the file outside those blocks (see [the key's block in the ssh config](../operations-api/operations.md#the-keys-block-in-the-ssh-config)). diff --git a/reference/configuration/options.md b/reference/configuration/options.md index 65b2601e8..e1d4217a5 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -244,7 +244,7 @@ logging: - `path` — Explicit log file path (overrides `root`) - `stdStreams` — Write to stdout/stderr; _Default_: `false` - `console` — Include `console.*` output in the log file; _Default_: `false` -- `auditLog` — Enable table transaction audit logging; _Default_: `false` +- `auditLog` — Enable table transaction audit logging; _Default_: `true` - `auditRetention` — Audit log retention duration; _Default_: `3d` - `external` — Logging for components using the logger API; sub-options: `level`, `path` - `rotation.enabled` / `rotation.compress` / `rotation.interval` / `rotation.maxSize` / `rotation.path` — Log file rotation (activates when `interval` or `maxSize` is set) diff --git a/reference/replication/clustering.md b/reference/replication/clustering.md index 3d4760670..134db14eb 100644 --- a/reference/replication/clustering.md +++ b/reference/replication/clustering.md @@ -27,6 +27,7 @@ Adds a new Harper instance to the cluster. If `subscriptions` are provided, it c - `revoked_certificates` _(optional)_ — array of revoked certificate serial numbers that will not be accepted for any connections - `shard` _(optional)_ — shard number for this node. Only needed when using sharding - `start_time` _(optional)_ — ISO 8601 UTC datetime. If set, only data after this time is downloaded during initial synchronization instead of the entire database +- `isLeader` _(optional)_ — select the added node as the source for this node's initial full copy. This controls bootstrap synchronization, not primary ownership or write authority; subsequent replication still follows the configured routes - `subscriptions` _(optional)_ — explicit table-level replication relationships. This is optional (and discouraged). Each subscription is an object with: - `database` — database name - `table` — table name diff --git a/reference/replication/overview.md b/reference/replication/overview.md index c79bbf6aa..ac537614c 100644 --- a/reference/replication/overview.md +++ b/reference/replication/overview.md @@ -360,7 +360,7 @@ See [Clustering Operations](./clustering.md#cluster-status) for the full respons ## Initial Synchronization and Resynchronization -When a new node is added and its database has not been previously synced, Harper downloads the full database from the first node it connects to. After the initial sync completes, the node enters replication mode and receives incremental updates. +Initial synchronization is decided per peer and per database. A new node can request a full copy from each in-scope peer for which it has no resume cursor, so a join can load several peers at once. To select one full-copy source, use [`add_node` with `isLeader: true`](./clustering.md#add-node). After synchronization, replication continues incrementally. If a node goes offline and comes back, it resynchronizes automatically to catch up on missed transactions. @@ -389,11 +389,11 @@ The following data operations are replicated across the cluster: - `drop_database` and `drop_table` **replicate to the whole cluster by default**, like the other operations that accept a `replicated` flag. Pass `"replicated": false` to drop on one node only. - `drop_attribute` is not replicated and must be run on each node independently. -Note that `replicated` is opt-out rather than opt-in across the operations that support it: omitting the flag replicates. [`set_configuration`](../configuration/operations.md#set-configuration) is the deliberate exception, replicating only when you pass `"replicated": true`, because configuration bodies routinely carry node-local values. +Data mutations, `drop_database`, `drop_table`, and component deployment replicate by default on Pro and Fabric; pass `"replicated": false` to limit them to one node. [`set_configuration`](../configuration/operations.md#set-configuration) and [`restart_service`](../operations-api/operations.md#restart_service) are opt-in: pass `"replicated": true` for a cluster-wide configuration update or rolling restart. Users and roles **do** propagate by default. They live in the `system` database, and the default replication scope is every database, so `system` is in scope unless you have narrowed it. As of v5.2 replicating `system` no longer forces a full mesh — see [Replicating the `system` database with controlled flow](#replicating-the-system-database-with-controlled-flow). If you narrow the scope to exclude `system`, users and roles stop propagating and must be provisioned on every node by your own automation. -Certain management operations — including component deployment and rolling restarts — are replicated across the cluster as well, following the same opt-out rule described above. +Check each management operation's replication default before using it in a single-node procedure. ## Inspecting Cluster Configuration diff --git a/release-notes/v5-lincoln/5.2.md b/release-notes/v5-lincoln/5.2.md index a189ac0df..1bb1b0e27 100644 --- a/release-notes/v5-lincoln/5.2.md +++ b/release-notes/v5-lincoln/5.2.md @@ -10,7 +10,7 @@ All patch release notes for 5.2.x are available on the [releases page](https://g ## Backup & Restore -New managed-backup operations for RocksDB databases: `create_backup`, `list_backups`, `verify_backup`, `delete_backup`, `purge_backups`, and `restore_backup`. Managed backups are a server-side repository under the configured backup root (`storage.backupPath`, default `/backup`); the RocksDB data files are shared and incremental across backups, while the audit/transaction log — and, for databases with file-backed blobs, the blobs (skip with `exclude_blobs`) — are captured in full on every backup. `verify_backup` checks the RocksDB files (checksums with `verify_checksum`) and the transaction-log snapshot; it does not verify blobs. `restore_backup` restores a user database in place without stopping Harper, as long as no loaded component is holding that database open; a database held open by a component — and always the `system` database — is restored offline by running `restore_backup` from the CLI with the server stopped. Each of these operations can be run from the CLI under its operation name (e.g. `harper create_backup database=data`): while Harper is running the operation is forwarded to the server, and while it is stopped the command operates directly on the files. See [Backups](/reference/v5/backups/overview) and [Backup Operations](/reference/v5/backups/operations). +New managed-backup operations for RocksDB databases: `create_backup`, `list_backups`, `verify_backup`, `delete_backup`, `purge_backups`, and `restore_backup`. Managed backups are a server-side repository under the configured backup root (`storage.backupPath`, default `/backup`); the RocksDB data files are shared and incremental across backups, while the audit/transaction-log snapshot is copied on every backup. Blob snapshots use hard links when supported on the same filesystem and copies otherwise (skip with `exclude_blobs`). `verify_backup` checks the RocksDB files (checksums with `verify_checksum`) and the transaction-log snapshot; it requires a blob snapshot directory when the backup recorded blobs, but does not validate individual blob contents. `restore_backup` restores a user database in place without stopping Harper, as long as no loaded component is holding that database open; a database held open by a component — and always the `system` database — is restored offline by running `restore_backup` from the CLI with the server stopped. Each of these operations can be run from the CLI under its operation name (e.g. `harper create_backup database=data`): while Harper is running the operation is forwarded to the server, and while it is stopped the command operates directly on the files. See [Backups](/reference/v5/backups/overview) and [Backup Operations](/reference/v5/backups/operations). `get_backup` now understands RocksDB: it streams a full-snapshot `tar` of the database — including any file-backed blobs (pass `exclude_blobs: true` to omit them), gzipped by default (pass `gzip: false` for a plain `tar`) — instead of failing. The LMDB behavior (streaming the `.mdb`, with `table`/`tables`/`include_audit`) is unchanged. From 79680a167fc270737c43668742269f8a5b642f31 Mon Sep 17 00:00:00 2001 From: Ethan Arrowood Date: Thu, 15 Oct 2026 09:35:02 -0600 Subject: [PATCH 13/21] Clarify synchronization and recovery contracts in the guides Co-Authored-By: OpenAI Codex --- learn/administration/operating-replication.mdx | 2 +- learn/administration/production-readiness-checklist.mdx | 2 +- learn/administration/safe-deployments-and-rollback.mdx | 4 ++-- learn/developers/taking-a-node-out-of-rotation.mdx | 5 +++-- reference/http/overview.md | 2 +- reference/replication/clustering.md | 2 +- reference/replication/overview.md | 2 +- release-notes/v5-lincoln/5.2.md | 4 ++-- 8 files changed, 12 insertions(+), 11 deletions(-) diff --git a/learn/administration/operating-replication.mdx b/learn/administration/operating-replication.mdx index 0b9df2f35..4c75b3a09 100644 --- a/learn/administration/operating-replication.mdx +++ b/learn/administration/operating-replication.mdx @@ -96,7 +96,7 @@ Measure incremental catch-up and full synchronization separately. A briefly offl On the new node, the cost is larger than "one copy from one peer," because the full-copy decision is made per peer and per database rather than once against a bootstrap source. A joining node requests a full copy from every peer it has no resume cursor for. And because replication is bidirectional, each established peer independently decides it has no cursor for the newcomer and requests a full copy _from_ it as well. Joining a cluster of N peers is therefore N inbound transfers plus N outbound transfers of the new node's own (empty) databases, not a single stream from one source. -A node join can load every existing peer, so include it in cluster capacity planning. To use one full-copy source, pass `isLeader: true` to `add_node`. +A node join can load every existing peer, so include it in cluster capacity planning. Passing `isLeader: true` to `add_node` identifies a bootstrap source, but does not prevent full-copy requests to other peers that lack a resume cursor. Do not use it to budget for a single transfer. Each node derives its advertised `hdb_nodes` record from its configuration. A restart or component reload can replace constraints applied only through `add_node`; keep durable constraints in config routes. diff --git a/learn/administration/production-readiness-checklist.mdx b/learn/administration/production-readiness-checklist.mdx index 8935c1b7a..cebc03515 100644 --- a/learn/administration/production-readiness-checklist.mdx +++ b/learn/administration/production-readiness-checklist.mdx @@ -118,7 +118,7 @@ From [Monitoring and Triage](./monitoring-and-triage.mdx). From [Backup and Recovery](./backup-and-recovery.mdx). - [ ] Failure classes enumerated, each with the mechanism that addresses it -- [ ] Recovery point and recovery time stated per database +- [ ] Recovery point stated per database and recovery time stated per critical journey - [ ] Mechanism chosen per database according to its storage engine - [ ] No database in scope has tables spanning multiple storage roots, or the exclusion is known and accepted - [ ] Backup cadence matches the stated recovery point diff --git a/learn/administration/safe-deployments-and-rollback.mdx b/learn/administration/safe-deployments-and-rollback.mdx index e385200e8..c181c2fe2 100644 --- a/learn/administration/safe-deployments-and-rollback.mdx +++ b/learn/administration/safe-deployments-and-rollback.mdx @@ -224,7 +224,7 @@ Rehearse a stopped rollout end to end on a non-production cluster: 1. Declare a threshold before you start, for example "stop if journey success on the new cohort is more than 0.5 percent below the control cohort over five minutes." 2. Deploy a component that fails that threshold deliberately, to a bounded cohort. 3. Detect it through your dashboards rather than because you know what you did. -4. Reverse it, and time from decision to restored traffic. That number is your release RTO, and it belongs in your reliability plan. +4. Reverse it, and record both the time from the reversal decision to verified service and the total time from failure to verified service. Compare the total, including detection and decision time, with your RTO. 5. Verify uniform state afterwards: `get_components` on every node, and `list_deployments` showing the reversal. Check every node after rollback; a cluster average can hide nodes still running the failed release. @@ -254,7 +254,7 @@ Check every node after rollback; a cluster average can hide nodes still running - [ ] `get_configuration` read back after every configuration change - [ ] Schema changes follow expand-then-contract, with destructive operations documented per node - [ ] Forward repair defined for changes that alter persisted meaning -- [ ] Measured release RTO recorded from a rehearsed reversal +- [ ] Reversal duration and total recovery time recorded from a rehearsal, with the total compared against RTO ## Additional Resources diff --git a/learn/developers/taking-a-node-out-of-rotation.mdx b/learn/developers/taking-a-node-out-of-rotation.mdx index b60dc3522..e7b81320a 100644 --- a/learn/developers/taking-a-node-out-of-rotation.mdx +++ b/learn/developers/taking-a-node-out-of-rotation.mdx @@ -159,8 +159,9 @@ async function checkDependency() { signal: AbortSignal.timeout(2000), }); return response.ok ? { ok: true } : { ok: false, reason: `dependency answered ${response.status}` }; - } catch (error) { - return { ok: false, reason: `dependency unreachable: ${error.message}` }; + } catch { + // Fetch errors can include credentials from the URL. Keep them out of logs and markers. + return { ok: false, reason: 'dependency unreachable' }; } } diff --git a/reference/http/overview.md b/reference/http/overview.md index a33be64a4..3c7c09a3e 100644 --- a/reference/http/overview.md +++ b/reference/http/overview.md @@ -85,7 +85,7 @@ A mount is a routing prefix, not an isolation boundary: Host matching ignores the port and is case-insensitive. IPv6 hosts are configured as a bare literal (`::1`), not bracketed. -Custom components can configure the same behavior programmatically with `server.http(listener, { host, urlPath })`. See [`HttpOptions`](./api#httpoptions) for matching priority and middleware ordering options. +Custom components can configure handler routing programmatically with `server.http(listener, { host, urlPath })`. These handler options have been available since v5.1.0; the root-configuration and deployment options above were added in v5.2.0. See [`HttpOptions`](./api#httpoptions) for matching priority and middleware ordering options. ## Protocols Served diff --git a/reference/replication/clustering.md b/reference/replication/clustering.md index 134db14eb..29c8955bd 100644 --- a/reference/replication/clustering.md +++ b/reference/replication/clustering.md @@ -27,7 +27,7 @@ Adds a new Harper instance to the cluster. If `subscriptions` are provided, it c - `revoked_certificates` _(optional)_ — array of revoked certificate serial numbers that will not be accepted for any connections - `shard` _(optional)_ — shard number for this node. Only needed when using sharding - `start_time` _(optional)_ — ISO 8601 UTC datetime. If set, only data after this time is downloaded during initial synchronization instead of the entire database -- `isLeader` _(optional)_ — select the added node as the source for this node's initial full copy. This controls bootstrap synchronization, not primary ownership or write authority; subsequent replication still follows the configured routes +- `isLeader` _(optional)_ — identify the added node as a bootstrap source for this node. This does not grant primary ownership or write authority, or prevent full-copy requests to other peers for which this node has no resume cursor. Subsequent replication still follows the configured routes - `subscriptions` _(optional)_ — explicit table-level replication relationships. This is optional (and discouraged). Each subscription is an object with: - `database` — database name - `table` — table name diff --git a/reference/replication/overview.md b/reference/replication/overview.md index ac537614c..ff9ff6e89 100644 --- a/reference/replication/overview.md +++ b/reference/replication/overview.md @@ -360,7 +360,7 @@ See [Clustering Operations](./clustering.md#cluster-status) for the full respons ## Initial Synchronization and Resynchronization -Initial synchronization is decided per peer and per database. A new node can request a full copy from each in-scope peer for which it has no resume cursor, so a join can load several peers at once. To select one full-copy source, use [`add_node` with `isLeader: true`](./clustering.md#add-node). After synchronization, replication continues incrementally. +Initial synchronization is decided per peer and per database. A new node can request a full copy from each in-scope peer for which it has no resume cursor, so a join can load several peers at once. [`add_node` with `isLeader: true`](./clustering.md#add-node) identifies a bootstrap source; it does not prevent full-copy requests to other cursorless peers. After synchronization, replication continues incrementally. If a node goes offline and comes back, it resynchronizes automatically to catch up on missed transactions. diff --git a/release-notes/v5-lincoln/5.2.md b/release-notes/v5-lincoln/5.2.md index 1bb1b0e27..8649d62ef 100644 --- a/release-notes/v5-lincoln/5.2.md +++ b/release-notes/v5-lincoln/5.2.md @@ -86,9 +86,9 @@ Because Harper keeps one refresh-token hash per user, issuing a token revokes th ### Middleware routing and ordering -Applications can now be routed by virtual hostname, URL prefix, or both, with no dispatch code. Declare `host` and `urlPath` on the application's entry in the root `harper-config.yaml` — or pass them to `deploy_component` — and every handler the application registers is served under that hostname and path. Because the routing lives in the root config, the same application can be mounted differently per environment without editing it. A plugin's own `urlPath` still positions it within the application, and the application's mount is prefixed onto it. +Applications can now declare virtual-hostname and URL-prefix routing in the root configuration or at deployment time. Declare `host` and `urlPath` on the application's entry in the root `harper-config.yaml` — or pass them to `deploy_component` — and every handler the application registers is served under that hostname and path. Because the routing lives in the root config, the same application can be mounted differently per environment without editing it. A plugin's own `urlPath` still positions it within the application, and the application's mount is prefixed onto it. -Components can also pass `host` and `urlPath` directly to `server.http()`, `server.ws()`, and `server.upgrade()`. The new `name`, `before`, and `after` options provide explicit middleware ordering. See [HTTP middleware routing](/reference/v5/http/overview#middleware-routing) and [`HttpOptions`](/reference/v5/http/api#httpoptions). +Components can still pass `host` and `urlPath` directly to `server.http()`, `server.ws()`, and `server.upgrade()`, as in v5.1. The new `name`, `before`, and `after` options provide explicit middleware ordering. See [HTTP middleware routing](/reference/v5/http/overview#middleware-routing) and [`HttpOptions`](/reference/v5/http/api#httpoptions). ## Security From e104cfa15fa30fa20d7803a5f3fd3d5f660d98e7 Mon Sep 17 00:00:00 2001 From: Ethan Arrowood Date: Thu, 15 Oct 2026 10:01:08 -0600 Subject: [PATCH 14/21] Align reliability objectives and document engine-specific recovery limits Co-Authored-By: OpenAI Codex --- learn/administration/backup-and-recovery.mdx | 6 +++-- .../engineering-rpo-rto-and-uptime.mdx | 8 +++---- .../administration/monitoring-and-triage.mdx | 24 +++++++++++-------- .../administration/operating-replication.mdx | 2 +- .../production-readiness-checklist.mdx | 4 ++-- .../reliability-plan-template.mdx | 10 ++++---- .../safe-deployments-and-rollback.mdx | 6 +++-- .../taking-a-node-out-of-rotation.mdx | 13 ++++------ reference/backups/operations.md | 2 +- reference/backups/overview.md | 10 +++++--- reference/http/api.md | 4 +++- reference/replication/overview.md | 2 +- 12 files changed, 52 insertions(+), 39 deletions(-) diff --git a/learn/administration/backup-and-recovery.mdx b/learn/administration/backup-and-recovery.mdx index 8b2f356c7..261322710 100644 --- a/learn/administration/backup-and-recovery.mdx +++ b/learn/administration/backup-and-recovery.mdx @@ -42,7 +42,9 @@ Record these decisions in your reliability plan: | **Managed backups** | | Incremental, verifiable backups in a server-side repository under `storage.backupPath` | | **Snapshot download** | | [`get_backup`](/reference/v5/backups/operations#get_backup) streams a snapshot over HTTP, with no server-side artifact | -A backup is a whole-database copy: all tables, the transaction log, and any file-backed blobs. So a restored database keeps its `read_audit_log` history as of the backup point. +RocksDB managed backups and snapshot downloads include all tables, the transaction log, and file-backed blobs unless you request `exclude_blobs`. Restoring that complete backup preserves its `read_audit_log` history. + +LMDB `get_backup` streams the `.mdb` file and does not include external blob files. Table-filtered downloads include the audit store only with `include_audit: true`. If an LMDB database uses file-backed blobs, use a coordinated filesystem or volume backup of the database and every blob root with Harper stopped. An `.mdb` download alone cannot recover those blobs after the source node is lost. :::warning Two limitations to check against your own deployment now rather than later: @@ -235,7 +237,7 @@ For records requiring long-term preservation, keep a separate export in a docume ## Readiness checklist - [ ] Failure classes enumerated, with the mechanism that addresses each -- [ ] Recovery point and recovery time stated per database, not per cluster +- [ ] Database recovery points and journey recovery times recorded using the [objective definitions](./engineering-rpo-rto-and-uptime.mdx#state-objectives-for-each-journey-and-its-data) - [ ] Storage engine confirmed per database, and the mechanism chosen accordingly - [ ] No database in scope has tables spanning more than one storage root, or the exclusion is known and accepted - [ ] Backup cadence matches the stated recovery point per database diff --git a/learn/administration/engineering-rpo-rto-and-uptime.mdx b/learn/administration/engineering-rpo-rto-and-uptime.mdx index d92e56e6e..b63add4ea 100644 --- a/learn/administration/engineering-rpo-rto-and-uptime.mdx +++ b/learn/administration/engineering-rpo-rto-and-uptime.mdx @@ -3,7 +3,7 @@ title: Engineering RPO, RTO, and Uptime sidebar_position: 8 --- -Define recovery point objective (RPO), recovery time objective (RTO), and availability for each critical user journey and failure scenario. This guide helps you choose a recovery mechanism for each target and check it against measured results. +Define recovery time objective (RTO) and availability per critical journey, and recovery point objective (RPO) per database. Map failures to recovery mechanisms and validate the targets with measured results. When measured results miss a target, improve the recovery mechanism or revise the target before committing to it. @@ -23,7 +23,7 @@ When measured results miss a target, improve the recovery mechanism or revise th - [Backup and Recovery](./backup-and-recovery.mdx) for the recovery mechanisms - A named owner who can approve or reject a target, because this exercise produces decisions rather than data -## State the three per journey +## State objectives for each journey and its data | Term | The question it answers | How to state it so it is testable | | ------------------------ | ------------------------------------ | -------------------------------------------------------------------------------------------------- | @@ -33,7 +33,7 @@ When measured results miss a target, improve the recovery mechanism or revise th For each target, record the affected journey, when measurement starts, and where you measure it. -**Per journey, not per cluster.** A catalog browse and an order submission fail differently, matter differently, and recover differently. One cluster-wide number overcommits on the cheap journey and undercommits on the expensive one. +Scope availability and recovery time to each journey, and record the recovery point for every database it depends on. Catalog browsing and order submission can have different targets even when they share a cluster. **Include detection in recovery time.** Measure from the start of the failure to verified service restoration. Starting at the alert or the operator's decision leaves out time when users were already affected. @@ -47,7 +47,7 @@ Your recovery point objective is a statement about how much acknowledged data yo Healthy replication and recent backups do not prove that every acknowledged write has reached durable storage. Include the possible loss of unflushed writes in your RPO. -So treat it as an RPO decision rather than a performance one. Enabling it is defensible when the data is reproducible from an upstream source, when the workload is bulk ingest that can simply be re-run, or when peers acknowledge writes before the loss window opens. Outside those cases, leave it off. If it is on anywhere, record which databases and say so next to the recovery point in your plan, because the number no longer means what it appears to. +Leave `writeAsync` off unless you can accept the loss of unflushed writes, for example because the data can be reproduced from an upstream source or a bulk ingest can be rerun. Asynchronous replication does not guarantee a durable peer copy before the client receives success. If you enable it, record the affected databases and the accepted loss window beside their recovery objectives. ## Map scenarios to mechanisms diff --git a/learn/administration/monitoring-and-triage.mdx b/learn/administration/monitoring-and-triage.mdx index c829a909a..36f8fd0f6 100644 --- a/learn/administration/monitoring-and-triage.mdx +++ b/learn/administration/monitoring-and-triage.mdx @@ -133,18 +133,22 @@ Whatever agent you use, follow its own worker-thread documentation, and finish b ## Logs you can use during an incident -Harper's `logger` global takes a message plus a context object from component code: +Harper's `logger` global takes a message plus a context object from component code. Call this helper after a successful submission, passing the request context and measured duration in milliseconds: ```javascript -logger.info('order submitted', { - component: 'orders-api', - version: process.env.APP_VERSION, // whatever your build injects - node: server.hostname, - requestId: context.requestId, // requires http.logging.id, see below - operation: 'submit', - durationMs: elapsed, - outcome: 'ok', -}); +import { server } from 'harper'; + +function logOrderSubmitted(context, elapsed) { + logger.info('order submitted', { + component: 'orders-api', + version: process.env.APP_VERSION, + node: server.hostname, + requestId: context.requestId, + operation: 'submit', + durationMs: elapsed, + outcome: 'ok', + }); +} ``` Include the component and its version, the node, a request or trace identifier, the operation, its duration, the outcome, and a safe error class. That field set is what lets you answer "was this only the new version" and "was this only one node" without guessing, and those are the first two questions in almost every incident. diff --git a/learn/administration/operating-replication.mdx b/learn/administration/operating-replication.mdx index 4c75b3a09..1d152faba 100644 --- a/learn/administration/operating-replication.mdx +++ b/learn/administration/operating-replication.mdx @@ -100,7 +100,7 @@ A node join can load every existing peer, so include it in cluster capacity plan Each node derives its advertised `hdb_nodes` record from its configuration. A restart or component reload can replace constraints applied only through `add_node`; keep durable constraints in config routes. -Registry rows can replicate with `system`. Since v5.2, advertised records describe directional routes, so that exchange can preserve a constrained topology. To keep registries separate across a migration bridge, explicitly exclude `hdb_nodes` on the relevant routes while allowing users and roles through. See [controlled system replication](/reference/v5/replication/overview#replicating-the-system-database-with-controlled-flow). +Registry rows can replicate with `system`. Advertised records describe directional routes, so that exchange can preserve a constrained topology. To keep registries separate across a migration bridge, explicitly exclude `hdb_nodes` on the relevant routes while allowing users and roles through. See [controlled system replication](/reference/v5/replication/overview#replicating-the-system-database-with-controlled-flow). :::danger When replicating configuration, only send cluster-appropriate parameters. Replicating a node-local value such as a port, `node.hostname`, a file path, TLS material, or `replication.hostname`, `url`, or `routes` overwrites every peer's own local value. To apply a cluster-wide change safely, use `set_configuration` with `"replicated": true` for the parameter, then `restart_service` with `"replicated": true`, which restarts nodes one at a time. diff --git a/learn/administration/production-readiness-checklist.mdx b/learn/administration/production-readiness-checklist.mdx index cebc03515..d6cc18153 100644 --- a/learn/administration/production-readiness-checklist.mdx +++ b/learn/administration/production-readiness-checklist.mdx @@ -118,7 +118,7 @@ From [Monitoring and Triage](./monitoring-and-triage.mdx). From [Backup and Recovery](./backup-and-recovery.mdx). - [ ] Failure classes enumerated, each with the mechanism that addresses it -- [ ] Recovery point stated per database and recovery time stated per critical journey +- [ ] Database recovery points and journey recovery times recorded using the [objective definitions](./engineering-rpo-rto-and-uptime.mdx#state-objectives-for-each-journey-and-its-data) - [ ] Mechanism chosen per database according to its storage engine - [ ] No database in scope has tables spanning multiple storage roots, or the exclusion is known and accepted - [ ] Backup cadence matches the stated recovery point @@ -144,7 +144,7 @@ From [Backup and Recovery](./backup-and-recovery.mdx). From [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx). -- [ ] Recovery point, recovery time, and availability stated per journey with the measurement boundary named +- [ ] Objectives recorded using the [definitions](./engineering-rpo-rto-and-uptime.mdx#state-objectives-for-each-journey-and-its-data), with each journey's measurement boundary and dependent databases named - [ ] Exclusions documented - [ ] Every scenario in the scenario map has a named mechanism - [ ] Availability target converted to minutes per month diff --git a/learn/administration/reliability-plan-template.mdx b/learn/administration/reliability-plan-template.mdx index 768c88f4a..4c59030fd 100644 --- a/learn/administration/reliability-plan-template.mdx +++ b/learn/administration/reliability-plan-template.mdx @@ -27,7 +27,7 @@ Example values are included in italics to show the intended level of specificity ## 2. Critical journeys -The user-facing paths that must work. Everything else in this plan is stated per journey, so this table defines the scope of all of it. +List the user-facing paths that must work, including the databases and dependencies each one needs. | Journey | Description | Databases and tables involved | Downstream dependencies | Owner | | ---------------- | ----------------------------- | ----------------------------- | ----------------------- | ----- | @@ -36,9 +36,11 @@ The user-facing paths that must work. Everything else in this plan is stated per ## 3. Objectives -| Journey | Availability target | Measurement boundary | Recovery time objective | Recovery point objective | Exclusions | -| ------- | ------------------- | -------------------- | ------------------------------ | ------------------------ | ------------------------------------------ | -| | _99.95%_ | _at the CDN edge_ | _15 min from start of failure_ | _5 min_ | _planned maintenance, third-party outages_ | +Use the [objective definitions](./engineering-rpo-rto-and-uptime.mdx#state-objectives-for-each-journey-and-its-data): availability and recovery time per journey, recovery point per database. + +| Journey | Availability target | Measurement boundary | Recovery time objective | Recovery point objective by database | Exclusions | +| ------- | ------------------- | -------------------- | ------------------------------ | ------------------------------------ | ------------------------------------------ | +| | _99.95%_ | _at the CDN edge_ | _15 min from start of failure_ | _orders: 5 min; catalog: 1 hour_ | _planned maintenance, third-party outages_ | Monthly unavailability budget: _21 min 55 sec at 99.95%_ diff --git a/learn/administration/safe-deployments-and-rollback.mdx b/learn/administration/safe-deployments-and-rollback.mdx index c181c2fe2..a4c61c225 100644 --- a/learn/administration/safe-deployments-and-rollback.mdx +++ b/learn/administration/safe-deployments-and-rollback.mdx @@ -177,7 +177,9 @@ Set these parameters to match your payload size, network, and failure policy: Apply the same review, rollout, and rollback checks to configuration changes. -`set_configuration` supports `"replicated": true` to apply a change across the cluster in one call, with per-node outcomes in the response. To finish the change cluster-wide, follow with `restart_service` using `"replicated": true`, which restarts nodes one at a time. +`set_configuration` supports `"replicated": true` to apply a change across the cluster in one call, with per-node outcomes in the response. For settings that apply on worker restart, follow with `restart_service` using `"replicated": true`, which restarts workers on each node in turn. + +Check the setting's restart requirements first. Process-level settings such as [`threads.v8Flags`](/reference/v5/configuration/options#applying-v8-flags) require a full process restart; `restart_service` does not apply them. Drain and restart each node in turn, using `restart` or your process/container manager, and verify it before advancing. The `restart` operation does not replicate. ```json { @@ -191,7 +193,7 @@ Apply the same review, rollout, and rollback checks to configuration changes. Only replicate cluster-appropriate parameters. Node-local values such as ports, `node.hostname`, file paths, TLS material, and `replication.hostname`, `url`, or `routes` would overwrite every peer's own values. Replicating one of these is a cluster-wide outage delivered in a single API call. ::: -Configuration changes take effect after restart. `get_status.restartRequired` tracks component and code restarts, so it does not identify pending `set_configuration` changes. Record those changes yourself, restart the affected nodes, and read back `get_configuration` to confirm the intended values. +`get_status.restartRequired` tracks component and code restarts, so it does not identify pending `set_configuration` changes. Record those changes yourself, use the required restart type, and read back `get_configuration`. Read-back confirms the configured value; verify the running behavior too. ## Rollback is a designed capability diff --git a/learn/developers/taking-a-node-out-of-rotation.mdx b/learn/developers/taking-a-node-out-of-rotation.mdx index e7b81320a..024ef4ae8 100644 --- a/learn/developers/taking-a-node-out-of-rotation.mdx +++ b/learn/developers/taking-a-node-out-of-rotation.mdx @@ -180,10 +180,7 @@ function checkHealth() { // availability loop recognize this node's own address in its list of nodes. The // failure reason stays in the loop's log: anyone can read this response. export class Health extends Resource { - allowRead() { - return true; - } - async get() { + static async get() { const { ok } = await checkHealth(); return { status: ok ? 200 : 503, @@ -206,15 +203,15 @@ import { server, tables } from 'harper'; const RECORD = 'availability'; const ROW = 'this-node'; // RotationMark doesn't replicate, so each node has this one row -const TICK_MS = 5_000; // how often to look -const RECHECK_MS = 1_500; // how soon to look again after going out +const TICK_MS = 5_000; +const RECHECK_MS = 1_500; const OUT_AFTER_MS = 30_000; // failing on every look this long before going out const BACK_AFTER_MS = 60_000; // passing on every look this long before coming back const ALONE_MS = 30_000; // out, with no other node serving this long: come back anyway const JUST_OUT_MS = 10_000; // no other node serving this soon after going out: come back now const WRITE_MS = 5_000; // the loop's own Unavailable lands this soon after its row -const HEALTH_MS = 5_000; // how long the health check has to answer -const ASK_MS = 3_000; // how long another node has to answer +const HEALTH_MS = 5_000; +const ASK_MS = 3_000; const log = logger.withTag('availability'); const { RotationMark } = tables; diff --git a/reference/backups/operations.md b/reference/backups/operations.md index b2efa9ba7..c59dd65f4 100644 --- a/reference/backups/operations.md +++ b/reference/backups/operations.md @@ -145,7 +145,7 @@ harper restore_backup database=data backup_id=1 target_database=data_restored Streams a full snapshot of the specified database in the HTTP response for download — there is no server-side artifact and nothing to clean up. The server must be running; this is the one backup operation with no offline form. Behavior depends on the storage engine: - **RocksDB** — streams a `tar` archive of the current database state — all tables, the transaction log, and any file-backed blobs — gzipped by default (`gzip` is a RocksDB-only option and compresses the snapshot substantially). Pass `"gzip": false` for a plain `tar`, or `"exclude_blobs": true` to leave blobs out. The snapshot is always the current state; downloading a specific historical `backup_id` is not supported — to move a retained managed backup off-host, copy its whole per-database backup repository (see [Limitations](./overview.md#limitations)). Restoring a RocksDB snapshot by hand takes an extra step when the database has blobs — see [download a snapshot and restore it manually](./overview.md#example-download-a-snapshot-and-restore-it-manually). -- **LMDB** — streams the `.mdb` file. Specify `"table"` for a single table or `"tables"` for a set, and `"include_audit": true` to include the audit store. These options are LMDB-only. +- **LMDB** — streams the `.mdb` file, without external blob files. Specify `"table"` for a single table or `"tables"` for a set; a table-filtered download includes the audit store only with `"include_audit": true`. These options are LMDB-only. For databases with file-backed blobs, use a coordinated filesystem or volume backup of the database and all blob roots with Harper stopped; see [LMDB recovery](./overview.md#lmdb-). ```json { "operation": "get_backup", "database": "data" } diff --git a/reference/backups/overview.md b/reference/backups/overview.md index 815a730ff..c97754f86 100644 --- a/reference/backups/overview.md +++ b/reference/backups/overview.md @@ -5,12 +5,12 @@ title: Backups -Harper can back up and restore its databases natively. A backup is a whole-database copy — all tables, the audit/transaction log, and any file-backed blobs — so a restored database is a consistent point-in-time image of everything in it. (`get_backup` on an LMDB database is the exception: it can stream a subset of tables, and includes the audit store only when requested.) +Harper can back up and restore its databases natively. RocksDB managed backups and snapshot downloads include all tables, the audit/transaction log, and file-backed blobs unless `exclude_blobs` is requested. LMDB `get_backup` streams only the `.mdb` file, optionally filtered by table; external blob files need a separate, coordinated backup. Two complementary mechanisms are available: - **Managed backups** — incremental, verifiable backups kept in a server-side repository. Harper creates, lists, verifies, deletes, purges, and restores them through the [backup operations](./operations.md). -- **Snapshot download** — [`get_backup`](./operations.md#get_backup) streams a full snapshot of a database over HTTP for storage off-host, with no server-side artifact. +- **Snapshot download** — [`get_backup`](./operations.md#get_backup) streams a RocksDB archive or an LMDB `.mdb` file over HTTP for storage off-host, with no server-side artifact. Every backup operation is available through the [Operations API](../operations-api/overview.md) and as a [CLI command](./operations.md#running-backup-operations-from-the-cli) under the same operation name. @@ -33,7 +33,7 @@ The offline path matters for restore. RocksDB is single-writer, so an in-place r ## Limitations - **Managed backups require the RocksDB storage engine.** For LMDB databases, use [`get_backup`](./operations.md#get_backup) or [volume snapshots](../cli/commands.md#how-backups-work). -- **Whole-database granularity.** There is no per-table backup or restore. (The one exception: `get_backup` on an LMDB database can stream individual tables, and includes the audit store only with `include_audit`.) +- **Whole-database granularity.** There is no per-table backup or restore. The exception is `get_backup` on LMDB, which can stream selected tables; a table-filtered download includes the audit store only with `include_audit`. - **One storage root per database.** Backup is rejected if the database's tables resolve to multiple storage roots. A custom database-level `storage.path`, or per-table paths that still resolve to a single root, are supported. - **Backups live on the node that created them.** The backup repository is a local directory. RocksDB shares files across backup IDs (a backup ID is not a self-contained folder), so disaster-recovery copies must take the entire per-database repository — `/` — not an individual backup, and must do so while no backup operation is running, or from an atomic filesystem snapshot. A live recursive copy can race `create_backup`/`delete_backup`/`purge_backups` and produce an unrestorable copy. Alternatively, use `get_backup` to pull a snapshot from a running server. - **`get_backup` always streams the current state.** It cannot download a historical managed backup; to move a retained backup off-host, copy its whole per-database repository as above. @@ -125,6 +125,10 @@ The target node must have at least as many blob roots configured as the snapshot ### LMDB +:::warning +This stream does not include external blob files. For LMDB databases with file-backed blobs, use a coordinated filesystem or volume backup of the `.mdb` file and every configured blob root with Harper stopped. Restore that matching set and validate blob reads before serving traffic; restoring the `.mdb` file alone leaves its external blob references unresolved if the source files are lost. +::: + For an LMDB database the stream is the database's single `.mdb` file: ```bash diff --git a/reference/http/api.md b/reference/http/api.md index 4d0646210..5c4843a49 100644 --- a/reference/http/api.md +++ b/reference/http/api.md @@ -40,7 +40,9 @@ To continue the middleware chain, call `next(request)`. To short-circuit, return ### `HttpOptions` - (`name`, `before`, `after`, `urlPath`, and `host`) + (`urlPath` and `host`) + + (`name`, `before`, and `after`) | Property | Type | Default | Description | | ------------ | ------- | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | diff --git a/reference/replication/overview.md b/reference/replication/overview.md index ff9ff6e89..8f19fc68b 100644 --- a/reference/replication/overview.md +++ b/reference/replication/overview.md @@ -386,7 +386,7 @@ The following data operations are replicated across the cluster: **Destructive schema operations differ from each other, so check before you run one:** -- `drop_database` and `drop_table` **replicate to the whole cluster by default**, like the other operations that accept a `replicated` flag. Pass `"replicated": false` to drop on one node only. +- `drop_database` and `drop_table` **replicate to the whole cluster by default** on Pro and Fabric. Pass `"replicated": false` to drop on one node only. - `drop_attribute` is not replicated and must be run on each node independently. Data mutations, `drop_database`, `drop_table`, and component deployment replicate by default on Pro and Fabric; pass `"replicated": false` to limit them to one node. [`set_configuration`](../configuration/operations.md#set-configuration) and [`restart_service`](../operations-api/operations.md#restart_service) are opt-in: pass `"replicated": true` for a cluster-wide configuration update or rolling restart. From e7a5cd7500b78d5d1137f6e6c7545610eee85655 Mon Sep 17 00:00:00 2001 From: Ethan Arrowood Date: Thu, 15 Oct 2026 10:23:43 -0600 Subject: [PATCH 15/21] Align restart, routing, and logging guidance across pages Co-Authored-By: OpenAI Codex --- .../health-checks-and-traffic-admission.mdx | 6 +++--- learn/administration/how-harper-runs-in-production.mdx | 2 +- learn/administration/monitoring-and-triage.mdx | 2 +- learn/administration/operating-replication.mdx | 2 +- learn/administration/safe-deployments-and-rollback.mdx | 2 +- learn/developers/taking-a-node-out-of-rotation.mdx | 8 ++++---- reference/configuration/options.md | 2 +- reference/http/api.md | 4 +--- reference/logging/configuration.md | 4 ++-- release-notes/v5-lincoln/5.2.md | 2 +- 10 files changed, 16 insertions(+), 18 deletions(-) diff --git a/learn/administration/health-checks-and-traffic-admission.mdx b/learn/administration/health-checks-and-traffic-admission.mdx index 28d483207..90ac7f1eb 100644 --- a/learn/administration/health-checks-and-traffic-admission.mdx +++ b/learn/administration/health-checks-and-traffic-admission.mdx @@ -93,10 +93,10 @@ const base = 'https://my-node.example.com/status'; const auth = { Authorization: 'Basic ' + btoa('admin:password') }; const res = await fetch(base); -console.log(res.status); // 200 available, 404 unavailable +console.log(res.status); -await fetch(base, { method: 'DELETE', headers: auth }); // out of rotation -await fetch(base, { method: 'POST', headers: auth }); // back in +await fetch(base, { method: 'DELETE', headers: auth }); +await fetch(base, { method: 'POST', headers: auth }); ``` diff --git a/learn/administration/how-harper-runs-in-production.mdx b/learn/administration/how-harper-runs-in-production.mdx index c89f2bab1..38554ae8b 100644 --- a/learn/administration/how-harper-runs-in-production.mdx +++ b/learn/administration/how-harper-runs-in-production.mdx @@ -162,7 +162,7 @@ On a non-production cluster, stop one node. Confirm that peers keep serving, not - **Fabric and self-managed differ in what you own, not in how Harper behaves.** On [Fabric](/fabric), cluster creation, certificates, and the metrics pipeline are managed for you. The failure unit, the replication model, and the admission problem are identical. - **Version parity across nodes is an operating requirement, not a nicety.** Mixed versions in a cluster change replication and deployment behavior. Record the version per node in your boundary inventory and alert on drift. -- **Configuration changes made through the API take effect on restart.** A `set_configuration` call that has not been followed by a restart or `restart_service` leaves a node running something other than its stated configuration. Track pending changes in your change record. +- **Configuration changes need the restart type required by the setting.** Some apply on worker restart; process-level settings need a full process restart. Track pending changes and follow the [configuration rollout procedure](./safe-deployments-and-rollback.mdx#configuration-changes-are-deployments-too). - **`get_status` reports a `restartRequired` flag, but it tracks component and code restarts rather than configuration changes.** Do not rely on it to tell you a configuration change is still pending. ## Readiness checklist diff --git a/learn/administration/monitoring-and-triage.mdx b/learn/administration/monitoring-and-triage.mdx index 36f8fd0f6..464881984 100644 --- a/learn/administration/monitoring-and-triage.mdx +++ b/learn/administration/monitoring-and-triage.mdx @@ -169,7 +169,7 @@ Harper's logger is built on Node's `Console`, so this renders as a formatted tex That is fine for reading during an incident, but a log pipeline cannot key on `component` or `outcome` without parsing the line. If you need queryable fields, serialize the context yourself and log a single JSON string. ::: -Levels run `trace`, `debug`, `info`, `warn`, `error`, `fatal`, and `notify`. The default is `warn`, and `notify` is always logged. Choose a level that retains useful incident context without producing excessive volume. +Levels run `trace`, `debug`, `info`, `warn`, `error`, `fatal`, and `notify`. The default is `warn`, which drops the example's `info` message. Set [`logging.external.level`](/reference/v5/logging/configuration#loggingexternal) to `info` to retain it, and check the resulting volume. `notify` is always logged. `console.log` output does not reach the log files unless `logging.console` is enabled, so unstructured console output is not a production record. Centralize logs off the node, because the node you most need logs from is the one you are about to restart. diff --git a/learn/administration/operating-replication.mdx b/learn/administration/operating-replication.mdx index 1d152faba..27203ffdb 100644 --- a/learn/administration/operating-replication.mdx +++ b/learn/administration/operating-replication.mdx @@ -103,7 +103,7 @@ Each node derives its advertised `hdb_nodes` record from its configuration. A re Registry rows can replicate with `system`. Advertised records describe directional routes, so that exchange can preserve a constrained topology. To keep registries separate across a migration bridge, explicitly exclude `hdb_nodes` on the relevant routes while allowing users and roles through. See [controlled system replication](/reference/v5/replication/overview#replicating-the-system-database-with-controlled-flow). :::danger -When replicating configuration, only send cluster-appropriate parameters. Replicating a node-local value such as a port, `node.hostname`, a file path, TLS material, or `replication.hostname`, `url`, or `routes` overwrites every peer's own local value. To apply a cluster-wide change safely, use `set_configuration` with `"replicated": true` for the parameter, then `restart_service` with `"replicated": true`, which restarts nodes one at a time. +When replicating configuration, only send cluster-appropriate parameters. Replicating a node-local value such as a port, `node.hostname`, a file path, TLS material, or `replication.hostname`, `url`, or `routes` overwrites every peer's own local value. Use `set_configuration` with `"replicated": true` for cluster-wide parameters. If the setting applies on worker restart, follow with `restart_service` using `"replicated": true`. For settings that require a full process restart, drain and restart each node in turn; see [configuration changes](./safe-deployments-and-rollback.mdx#configuration-changes-are-deployments-too). ::: ## Read `cluster_status` like an operator diff --git a/learn/administration/safe-deployments-and-rollback.mdx b/learn/administration/safe-deployments-and-rollback.mdx index a4c61c225..4cd7c3f45 100644 --- a/learn/administration/safe-deployments-and-rollback.mdx +++ b/learn/administration/safe-deployments-and-rollback.mdx @@ -39,7 +39,7 @@ Decide separately what to build, where to install it, which behavior to enable, For example, validate a route with internal traffic before exposing it publicly. If errors rise, reduce traffic to the release or disable its feature flag while you investigate. :::danger -**Component deployment replicates by default.** Pass `"replicated": false` when validating on one node. Other management operations also replicate by default; check the operation's reference before using it during an isolated rollout. [`set_configuration`](/reference/v5/configuration/operations) requires an explicit `"replicated": true`. +**Component deployment replicates by default.** Pass `"replicated": false` when validating on one node. Other operations have different [replication defaults](/reference/v5/replication/overview#replicated-transactions); check the operation's reference before an isolated rollout. ::: `urlPath` mounts a component at an HTTP path and `host` serves it on a virtual hostname. Routing by either has been available since v5.1.0. What arrived in v5.2.0 is declaring that routing on the application's entry in the root `harper-config.yaml`, and the matching `host` parameter on `deploy_component`. Declared that way, the routing is persisted deployment state rather than a runtime toggle. See [HTTP middleware routing](/reference/v5/http/overview#middleware-routing). diff --git a/learn/developers/taking-a-node-out-of-rotation.mdx b/learn/developers/taking-a-node-out-of-rotation.mdx index 024ef4ae8..b31f4545b 100644 --- a/learn/developers/taking-a-node-out-of-rotation.mdx +++ b/learn/developers/taking-a-node-out-of-rotation.mdx @@ -205,10 +205,10 @@ const RECORD = 'availability'; const ROW = 'this-node'; // RotationMark doesn't replicate, so each node has this one row const TICK_MS = 5_000; const RECHECK_MS = 1_500; -const OUT_AFTER_MS = 30_000; // failing on every look this long before going out -const BACK_AFTER_MS = 60_000; // passing on every look this long before coming back -const ALONE_MS = 30_000; // out, with no other node serving this long: come back anyway -const JUST_OUT_MS = 10_000; // no other node serving this soon after going out: come back now +const OUT_AFTER_MS = 30_000; +const BACK_AFTER_MS = 60_000; +const ALONE_MS = 30_000; +const JUST_OUT_MS = 10_000; const WRITE_MS = 5_000; // the loop's own Unavailable lands this soon after its row const HEALTH_MS = 5_000; const ASK_MS = 3_000; diff --git a/reference/configuration/options.md b/reference/configuration/options.md index e1d4217a5..47841fa85 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -232,7 +232,7 @@ logging: level: warn root: ~/hdb/log stdStreams: false - auditLog: false + auditLog: true rotation: interval: 1D maxSize: 100M diff --git a/reference/http/api.md b/reference/http/api.md index 5c4843a49..ec7b5186a 100644 --- a/reference/http/api.md +++ b/reference/http/api.md @@ -40,9 +40,7 @@ To continue the middleware chain, call `next(request)`. To short-circuit, return ### `HttpOptions` - (`urlPath` and `host`) - - (`name`, `before`, and `after`) + (`name`, `before`, `after`, `urlPath`, and `host`) | Property | Type | Default | Description | | ------------ | ------- | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | diff --git a/reference/logging/configuration.md b/reference/logging/configuration.md index f204b9325..a607d10c7 100644 --- a/reference/logging/configuration.md +++ b/reference/logging/configuration.md @@ -106,7 +106,7 @@ See [Database / Transaction Logging](../database/transaction.md) for details on ```yaml logging: - auditLog: false + auditLog: true ``` ### `logging.auditRetention` @@ -339,7 +339,7 @@ logging: file: true stdStreams: false console: false - auditLog: false + auditLog: true auditRetention: 3d rotation: enabled: true diff --git a/release-notes/v5-lincoln/5.2.md b/release-notes/v5-lincoln/5.2.md index 8649d62ef..ef1a16c0d 100644 --- a/release-notes/v5-lincoln/5.2.md +++ b/release-notes/v5-lincoln/5.2.md @@ -88,7 +88,7 @@ Because Harper keeps one refresh-token hash per user, issuing a token revokes th Applications can now declare virtual-hostname and URL-prefix routing in the root configuration or at deployment time. Declare `host` and `urlPath` on the application's entry in the root `harper-config.yaml` — or pass them to `deploy_component` — and every handler the application registers is served under that hostname and path. Because the routing lives in the root config, the same application can be mounted differently per environment without editing it. A plugin's own `urlPath` still positions it within the application, and the application's mount is prefixed onto it. -Components can still pass `host` and `urlPath` directly to `server.http()`, `server.ws()`, and `server.upgrade()`, as in v5.1. The new `name`, `before`, and `after` options provide explicit middleware ordering. See [HTTP middleware routing](/reference/v5/http/overview#middleware-routing) and [`HttpOptions`](/reference/v5/http/api#httpoptions). +Components can still pass `host` and `urlPath` directly to `server.http()`, `server.ws()`, and `server.upgrade()`, as in v5.1. The `name`, `before`, and `after` ordering options are also available since v5.1. See [HTTP middleware routing](/reference/v5/http/overview#middleware-routing) and [`HttpOptions`](/reference/v5/http/api#httpoptions). ## Security From 212c3b849f3e1b2edf9c110c668211d9b82e41ad Mon Sep 17 00:00:00 2001 From: Ethan Arrowood Date: Thu, 15 Oct 2026 10:32:35 -0600 Subject: [PATCH 16/21] Finish version annotations and remove repeated sample narration Co-Authored-By: OpenAI Codex --- learn/administration/backup-and-recovery.mdx | 8 +++++--- .../health-checks-and-traffic-admission.mdx | 4 ---- learn/administration/operating-replication.mdx | 6 +++--- learn/administration/safe-deployments-and-rollback.mdx | 2 +- learn/developers/taking-a-node-out-of-rotation.mdx | 2 -- 5 files changed, 9 insertions(+), 13 deletions(-) diff --git a/learn/administration/backup-and-recovery.mdx b/learn/administration/backup-and-recovery.mdx index 261322710..926252ca3 100644 --- a/learn/administration/backup-and-recovery.mdx +++ b/learn/administration/backup-and-recovery.mdx @@ -112,13 +112,15 @@ Keep a recoverable copy outside the source node's failure and administrative bou - A disaster-recovery copy has to take the entire per-database repository, `/`, not an individual backup. - It has to do that while no backup operation is running, or from an atomic filesystem snapshot. A live recursive copy can race `create_backup`, `delete_backup`, or `purge_backups` and produce an unrestorable copy. -To take a fresh snapshot directly to another machine, run: +Download a fresh snapshot from the configured node: ```bash -# Pull a snapshot of the current state from a running node harper get_backup database=data out=./data-$(date +%Y%m%dT%H%M%S).tar.gz +``` + +Or specify a remote target and write the snapshot to a local file: -# Or pull a snapshot of another node, writing it to a file here +```bash harper get_backup database=data target=https://operations.node-2.example.com out=./data.tar.gz ``` diff --git a/learn/administration/health-checks-and-traffic-admission.mdx b/learn/administration/health-checks-and-traffic-admission.mdx index 90ac7f1eb..a2905ce2d 100644 --- a/learn/administration/health-checks-and-traffic-admission.mdx +++ b/learn/administration/health-checks-and-traffic-admission.mdx @@ -75,13 +75,10 @@ Use the target node's application URL. The examples assume HTTPS is configured o ```bash -# Is this node advertising itself as available? curl -s -o /dev/null -w '%{http_code}\n' https://my-node.example.com/status -# Take it out of rotation curl -s -X DELETE https://my-node.example.com/status -u 'admin:password' -# Put it back curl -s -X POST https://my-node.example.com/status -u 'admin:password' ``` @@ -141,7 +138,6 @@ let inFlight; async function checkReadiness() { const checks = {}; - // Keep the node-local storage check to one primary-key lookup. // Use a fresh server-side context for this fixed, non-sensitive read. try { const sentinel = await tables.ReadinessProbe.get('readiness-probe-sentinel', {}); diff --git a/learn/administration/operating-replication.mdx b/learn/administration/operating-replication.mdx index 27203ffdb..81437b750 100644 --- a/learn/administration/operating-replication.mdx +++ b/learn/administration/operating-replication.mdx @@ -168,16 +168,16 @@ Inventory `system.hdb_nodes` alongside this and compare it to your intended topo ## Prove convergence, not connection -Socket state cannot tell you that a specific business record is current. For anything where the answer matters, write a sentinel and read it back from the peer. +Socket state cannot tell you that a specific business record is current. Write a sentinel on the source node: ```javascript -// On the source node: write a sentinel with a known value const marker = { id: 'convergence-probe', writtenAt: Date.now(), from: 'server-1' }; await tables.OpsProbe.put(marker.id, marker); ``` +Read it from the target node and compare `writtenAt`: + ```bash -# On the target node: read it back and compare writtenAt curl -s https://server-2.example.net/OpsProbe/convergence-probe \ -u 'admin:password' ``` diff --git a/learn/administration/safe-deployments-and-rollback.mdx b/learn/administration/safe-deployments-and-rollback.mdx index 4cd7c3f45..377be7594 100644 --- a/learn/administration/safe-deployments-and-rollback.mdx +++ b/learn/administration/safe-deployments-and-rollback.mdx @@ -179,7 +179,7 @@ Apply the same review, rollout, and rollback checks to configuration changes. `set_configuration` supports `"replicated": true` to apply a change across the cluster in one call, with per-node outcomes in the response. For settings that apply on worker restart, follow with `restart_service` using `"replicated": true`, which restarts workers on each node in turn. -Check the setting's restart requirements first. Process-level settings such as [`threads.v8Flags`](/reference/v5/configuration/options#applying-v8-flags) require a full process restart; `restart_service` does not apply them. Drain and restart each node in turn, using `restart` or your process/container manager, and verify it before advancing. The `restart` operation does not replicate. +Check the setting's restart requirements first. Process-level settings such as [`threads.v8Flags`](/reference/v5/configuration/options#applying-v8-flags) require a full process restart; `restart_service` does not apply them. Drain and restart each node in turn, using `restart` or your process/container manager, and verify it before advancing. The `restart` operation does not replicate. ```json { diff --git a/learn/developers/taking-a-node-out-of-rotation.mdx b/learn/developers/taking-a-node-out-of-rotation.mdx index b31f4545b..d642b4f44 100644 --- a/learn/developers/taking-a-node-out-of-rotation.mdx +++ b/learn/developers/taking-a-node-out-of-rotation.mdx @@ -254,8 +254,6 @@ async function ask(base, path) { } } -// Which other nodes are serving: their /Health passes, and their /status says -// the load balancer may send them traffic. Each is asked at its own address. async function otherNodesServing(nodes, me) { const answers = await Promise.all( nodes.map(async (base) => { From d6cd674a14592c6cd424c3d743b69abab5880698 Mon Sep 17 00:00:00 2001 From: Ethan Arrowood Date: Fri, 16 Oct 2026 08:34:15 -0600 Subject: [PATCH 17/21] Lead the deployments guide with copy-paste deploy and rollback steps Rework Safe Deployments and Rollback around commands operators can run: log in and deploy a create-harper backend, build a Vite frontend once and deploy only its output, deploy from GitHub Actions with OIDC, wait for the rolling job, and roll back by deployment_id from the CLI or Studio. State what restart=rolling actually does: it restarts nodes one at a time but leaves each node in rotation, runs no application checks between nodes, and (without canary certification) skips a node it cannot reach. Correct the "zero-downtime" and "keeps serving throughout" claims in the CLI reference and CI guide, and add a drain gate to the checklist. Co-Authored-By: Claude Opus 5.5 Dispatch-Task: fix-Ethan-Arrowood_documentation_671-1627592f --- .../production-readiness-checklist.mdx | 1 + .../safe-deployments-and-rollback.mdx | 436 +++++++++++------- learn/developers/deploying-from-ci.mdx | 2 +- reference/cli/commands.md | 2 +- 4 files changed, 276 insertions(+), 165 deletions(-) diff --git a/learn/administration/production-readiness-checklist.mdx b/learn/administration/production-readiness-checklist.mdx index d6cc18153..af12d1adf 100644 --- a/learn/administration/production-readiness-checklist.mdx +++ b/learn/administration/production-readiness-checklist.mdx @@ -88,6 +88,7 @@ From [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx). - [ ] Stop thresholds and hold times declared before a rollout begins - [ ] Named decision owner for advance, hold, stop, and roll back - [ ] Pipeline checks deployment details and waits for the restart job, then verifies the serving version on every node +- [ ] Nodes that must not serve while they change are drained first, since `restart=rolling` keeps each node in rotation - [ ] `deployment_timeout` and `ignore_replication_errors` set deliberately - [ ] Configuration changes go through the same change record as code - [ ] `get_configuration` read back after every configuration change diff --git a/learn/administration/safe-deployments-and-rollback.mdx b/learn/administration/safe-deployments-and-rollback.mdx index 377be7594..908db64c5 100644 --- a/learn/administration/safe-deployments-and-rollback.mdx +++ b/learn/administration/safe-deployments-and-rollback.mdx @@ -6,214 +6,328 @@ sidebar_position: 6 import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -[Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx) covers installing a release from a pipeline. This guide covers limiting its initial traffic, deciding when to expand the rollout, and reversing it if checks fail. - -To roll back component code, redeploy the previous immutable reference. Keep that reference addressable and rehearse the procedure before launch. +This guide deploys a Harper application to a cluster, waits until every node runs it, and rolls it back. It then covers limiting a release's first traffic and deciding when to expand or reverse it. The commands use the Harper CLI and match projects created with `npm create harper@latest`. [Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx) covers pipeline authentication, private sources, and registry artifacts in more depth. ## What You Will Learn -- How to separate build, deploy, activate, and expose into four decisions with four different control points -- The change loop, and what counts as evidence at each step -- Five rollout patterns and the Harper operations behind each -- How `"restart": true` and `"restart": "rolling"` differ, and how to verify completion +- How to deploy a Harper backend, or a frontend built once, from your machine or from GitHub Actions +- What `restart=rolling` does and does not do, and how to wait for it +- How to roll back from the CLI or from Studio +- How to limit a release's first traffic, and decide when to advance, hold, or reverse - How to classify a change so you reverse the right thing, and why reversing the wrong thing can cause data loss ## Prerequisites -- A cluster of at least three nodes, so you can hold one out of rotation and still meet capacity -- A working pipeline that deploys from an immutable reference ([Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx)) -- [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx), since every pattern here depends on being able to move traffic -- [Monitoring and Triage](./monitoring-and-triage.mdx), since every gate here depends on being able to compare versions +- A Harper Fabric cluster, or a self-managed Harper Pro cluster. Rolling back to a kept release needs v5.3.0 or later. +- The Harper CLI on your machine: `npm install -g harper` +- An application, for example one created with `npm create harper@latest` +- [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx), for taking a node out of rotation +- [Monitoring and Triage](./monitoring-and-triage.mdx), for comparing versions during a rollout -## Separate four decisions +## Deploy a release -Decide separately what to build, where to install it, which behavior to enable, and who receives it. +### Log in -| Decision | The question | Harper control point | -| ------------ | ----------------------------------------- | ----------------------------------------------------------------------------------------------- | -| **Build** | What immutable artifact exists? | A pinned package reference, a versioned tarball, a checksum, a dependency lock | -| **Deploy** | Which nodes have that artifact installed? | `deploy_component`, with `"replicated": false` to hold it to one node | -| **Activate** | Which code path is actually enabled? | `urlPath` and `host` mounting, component configuration, a feature flag, a header or tenant rule | -| **Expose** | Which production traffic reaches it? | Traffic layer weights, the availability flag, cohort or geography targeting | +Set the cluster's Operations API URL once per shell, then log in with the cluster's admin username and password: -For example, validate a route with internal traffic before exposing it publicly. If errors rise, reduce traffic to the release or disable its feature flag while you investigate. +```bash +export HARPER_CLI_TARGET=https://my-cluster.example.com:9925 +harper login "$HARPER_CLI_TARGET" +``` -:::danger -**Component deployment replicates by default.** Pass `"replicated": false` when validating on one node. Other operations have different [replication defaults](/reference/v5/replication/overview#replicated-transactions); check the operation's reference before an isolated rollout. -::: +The CLI saves a token for that URL. Later `harper` commands in the shell use `HARPER_CLI_TARGET` as their target. -`urlPath` mounts a component at an HTTP path and `host` serves it on a virtual hostname. Routing by either has been available since v5.1.0. What arrived in v5.2.0 is declaring that routing on the application's entry in the root `harper-config.yaml`, and the matching `host` parameter on `deploy_component`. Declared that way, the routing is persisted deployment state rather than a runtime toggle. See [HTTP middleware routing](/reference/v5/http/overview#middleware-routing). +### A Harper backend -## The change loop +An application that runs from source, such as a `create-harper` project without a frontend build, deploys straight from its directory: -```mermaid -flowchart TD - Preflight --> Limit[Limit initial traffic] - Limit --> Observe - Observe --> Decide{Decide} - Decide -->|Advance| Expand[Expand traffic] - Expand -->|More stages| Observe - Expand -->|Rollout complete| Close[Verify and close] - Decide -->|Hold| Hold[Keep current exposure] - Hold --> Observe - Decide -->|Reverse| Reverse[Restore prior release or traffic] - Reverse --> Verify[Verify recovery] - Verify --> Close +```bash +cd my-app +harper deploy project=my-app restart=rolling json=true > deploy.json ``` -1. **Preflight.** Confirm the target version, component inventory, replication convergence, peer capacity with one node held out, and backups if data is at risk. Check that the prior artifact is still addressable. Name the decision owner and set stop thresholds before starting. +`create-harper` projects run the same deploy, without `json=true`, as `npm run deploy`. The CLI packages the directory and uploads it, and Harper installs its dependencies. On Fabric and Harper Pro, a deploy reaches every node unless you pass `replicated=false`. -2. **Limit.** Start with one drained node, an internal group, selected tenants, a region, or a small traffic percentage. Choose enough requests to make the comparison meaningful. +The upload includes everything in the directory except `node_modules`, including `.git` and any `.env` file. Deploy from a clean checkout, or stage only the files you want to ship, as the next section does. + +### A frontend built once + +The React and Vue templates load the `@harperfast/vite` plugin, which builds the frontend on each node when the build output is missing, so every node also carries the build toolchain. To build once and ship only the output, keep a second configuration file for the release. It serves the build output with the built-in [`static`](/reference/v5/static-files/overview) plugin and leaves out the Vite plugin: -3. **Observe.** Compare old and new versions on request success, SLO latency, worker saturation, data correctness, replication, and downstream errors. Use logs and traces to investigate differences. +```yaml +# deploy.config.yaml +rest: true +graphqlSchema: + files: 'schemas/*.graphql' +jsResource: + files: 'resources/*.js' +static: + files: 'dist/**' +``` -4. **Decide.** Advance, hold, stop, or roll back against the agreed thresholds. Hold if required telemetry is missing. +Build, copy the release into its own directory, and deploy that directory: -5. **Expand.** Increase exposure only after minimum sample and hold conditions pass, and keep enough healthy capacity in the old version to reverse. +```bash +npm ci +npm run build +rm -rf deploy && mkdir deploy +cp -R dist schemas resources deploy/ +cp deploy.config.yaml deploy/config.yaml +(cd deploy && harper deploy project=my-app restart=rolling json=true) > deploy.json +``` -6. **Close.** Verify uniform artifact and configuration across nodes, restore intended traffic, record the actual outcome, and keep the evidence with the change record. +The staged directory has no `package.json`, so the nodes skip the install step. If your resources import npm packages, add a `package.json` to `deploy/` that lists only those runtime dependencies. Harper Studio [deploys itself this way](https://github.com/HarperFast/studio/blob/stage/.github/actions/studio-deploy/action.yaml). -Record each decision and the measurements that supported it. +### Wait for every node -## Five rollout patterns +With `restart=rolling`, the deploy returns once the node you called has the release. The other nodes take it in a job that keeps running afterwards. Poll it until it prints `COMPLETE`: -| Pattern | How Harper is used | Fits when, and watch out for | -| ----------------------------- | ------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------- | -| Isolated node validation | Drain one node, deploy with `"replicated": false`, validate, re-admit a small cohort | Strongest infrastructure and component validation. Requires N+1 capacity and node targeting | -| Replicated rolling deployment | `deploy_component` with `"replicated": true` and `"restart": "rolling"` | Efficient once you trust the artifact. Rolling alone is not progressive delivery, add traffic and telemetry gates | -| Feature-targeted release | Deploy compatible code broadly, activate by flag, header, tenant, or geography | Best when behavior can separate from code placement. Guard flag ownership and cleanup | -| Parallel environment | Deploy to a separate node pool, shift traffic after validation | Strong isolation and fast reversal. Write ownership and replication between pools need real design | -| Regional progression | Promote the same artifact region by region, exposing each after local gates | Good locality and blast radius control. One region does not predict every traffic shape | +```bash +harper get_job id="$(jq -r '.restartJobId' deploy.json)" json=true | jq -r '.[0].status' +``` -Isolated node validation, concretely: +`ERROR` means the rollout failed, and `harper get_job` without the `jq` filter shows the job's `message`. `null` means the node that answered does not have the job: Harper does not replicate jobs, so poll the node that received the deploy. See [the job lives on one node](../developers/deploying-from-ci.mdx#path-a-deploy-a-tagged-release-from-your-repository). When the job completes, check the serving version on every node through your readiness route or a journey test. - - +### Deploy from GitHub Actions -```json -{ - "operation": "deploy_component", - "project": "orders-api", - "package": "https://artifacts.example.net/orders-api-2.4.1.tgz", - "replicated": false, - "restart": true -} +`create-harper` projects include `.github/workflows/deploy.yaml`. It tests every pull request and, when a change merges to `main`, deploys with `restart=rolling` and waits for the job. It authenticates with GitHub's OIDC token, so the repository stores no Harper credential. Set up the cluster side once from your machine, logged in as a `super_user`: + +```bash +npm run deploy:setup-ci ``` -Drain the node first, deploy, run your readiness route and journey synthetic against it directly, then admit a bounded cohort. + + +The script runs [`harper deploy setup=true provider=github-actions`](/reference/v5/cli/commands#github-actions-deploys-oidc). It creates a deploy-only user and a trust policy pinned to `main` and the `production` environment, and sets the repository's `HARPER_CLI_TARGET` variable. With an earlier CLI, create the user and policy by hand, as [Let the workflow authenticate with OIDC](../developers/deploying-from-ci.mdx#let-the-workflow-authenticate-with-oidc) describes, pinning `workflow_ref` to `main`. Trust policies need a v5.3.0 or later cluster. + +For a frontend built once, replace the template's workflow with one that builds in one job and deploys in another: + +```yaml +# .github/workflows/deploy.yaml +name: Deploy to Harper +on: + push: + branches: [main] + +permissions: + contents: read + +concurrency: + group: deploy-${{ github.ref }} + +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + - uses: actions/setup-node@v4 + with: + node-version: 22 + - run: npm ci + - run: npm test + - run: npm run build + - name: Stage the release + run: | + mkdir deploy + cp -R dist schemas resources deploy/ + cp deploy.config.yaml deploy/config.yaml + - uses: actions/upload-artifact@v4 + with: + name: release + path: deploy/ + + deploy: + needs: build + runs-on: ubuntu-latest + environment: production + permissions: + id-token: write + env: + HARPER_CLI_TARGET: ${{ vars.HARPER_CLI_TARGET }} + steps: + - uses: actions/download-artifact@v4 + with: + name: release + path: deploy + - uses: actions/setup-node@v4 + with: + node-version: 22 + - run: npm install --global harper@^5.3 + - name: Deploy the release + working-directory: deploy + run: harper deploy project=my-app restart=rolling json=true > "$RUNNER_TEMP/deploy.json" + - name: Wait for every node to take the release + run: | + JOB_ID=$(jq -r '.restartJobId // empty' "$RUNNER_TEMP/deploy.json") + [ -n "$JOB_ID" ] || { echo "The rolling deploy returned no restartJobId"; exit 1; } + LAST="could not be read" + for attempt in $(seq 60); do + sleep 10 + harper get_job id="$JOB_ID" json=true > "$RUNNER_TEMP/job.json" || continue + case $(jq -r '.[0].status // "MISSING"' "$RUNNER_TEMP/job.json") in + COMPLETE) exit 0 ;; + ERROR) jq -r '.[0].message' "$RUNNER_TEMP/job.json"; exit 1 ;; + MISSING) LAST="was not on the node that answered; its outcome is unknown" ;; + *) LAST="was still running" ;; + esac + done + echo "After 60 polls, job $JOB_ID $LAST" + exit 1 +``` - - - -```json -{ - "operation": "deploy_component", - "project": "orders-api", - "package": "https://artifacts.example.net/orders-api-2.4.1.tgz", - "replicated": true, - "restart": "rolling" -} +The `build` job runs your dependencies' install scripts and build tools, and cannot request an identity token. The `deploy` job can, and it runs none of your repository's code: it installs only the Harper CLI and uploads the built release. [Path B](../developers/deploying-from-ci.mdx#path-b-build-in-ci-publish-to-github-packages-deploy-the-artifact) explains this split, and publishes the build to a registry instead, which gives the release a version you can redeploy from any machine. + +## What `restart=rolling` does + +`restart=rolling` restarts the node you called, then each other node in turn. It waits for one node's restart to finish before it starts the next, and returns a `restartJobId` to poll. It does not: + +- **Take a node out of rotation.** Each node keeps taking traffic while its workers are replaced a few at a time. Harper does not mark the node unavailable or drain its connections. [harper#2975](https://github.com/HarperFast/harper/issues/2975) tracks load-balancer-aware rolling deploys. +- **Run your checks between nodes.** It moves on when a node's restart finishes, not when your readiness route or journey tests pass. From v5.4.0, [canary certification](/reference/v5/operations-api/operations#certifying-a-release-in-a-canary-worker) rejects a release that fails to load, but it does not run your checks. +- **Prove that every node restarted.** Without canary certification (before v5.4.0, or when the deploy's `certification` is `unavailable`), the job records an error for a node it cannot reach and continues to the next one. A `COMPLETE` job can still leave a node on the old release. + +To keep traffic off a node while it changes, drain it first, as in [isolated node validation](#limit-a-releases-first-traffic) or the [drain and return sequence](./health-checks-and-traffic-admission.mdx#drain-a-node-and-bring-it-back). + +`restart=true` restarts each node as it applies the replicated deploy, so restarts can overlap and reduce serving capacity. Use it for one drained node, a single-node instance, or development. On v5.2, it returns as soon as the restart is requested. From v5.3.0, it waits for the restart on the node you called, up to ten minutes. A caller timeout does not stop the restart, and the response does not report restart success, so check the serving version. For later versions, see [restart and certification behavior](/reference/v5/operations-api/operations#restarting-restart). + +Two deploy parameters control replication failures: + +- `deployment_timeout` is how long a peer waits for the replicated payload before failing, defaulting to 120000 ms. Raise it for large components or slow links. +- `ignore_replication_errors` treats a peer that fails to receive the deploy as non-fatal. By default a failed peer makes the whole operation return a non-2xx status, while the component is still deployed on the origin node. Decide which behavior you want before you need it. + +## Roll back + +### From the CLI + + + +When a deploy replaces a release, each node keeps the replaced one under its `deployment_id`. List recent releases, newest first: + +```bash +harper list_deployments project=my-app status=success limit=5 json=true \ + | jq -r '.deployments[] | "\(.deployment_id) \(.started_at / 1000 | floor | todate)"' ``` -On Harper Pro 5.2.15, this returns a `deployment_id` for the deployment record and a separate `restartJobId` for the rolling restart. Read each from the actual response: +Activate the release you want back: -```json -{ - "operation": "get_job", - "id": "" -} +```bash +harper deploy project=my-app deployment_id= restart=rolling json=true > deploy.json ``` -Poll the node that received the deployment. `get_job` returns an array; wait for its job's `status` to be `COMPLETE`, and stop on `ERROR`. An empty array means the job was not found on that node. Retry a failed request if the node is restarting. +Nothing is rebuilt or reinstalled. Wait for the job and check every node, as for any rolling deploy. A node that no longer keeps the release refuses it: [`deployment.stagingRetention.maxCount`](/reference/v5/configuration/options#deployment) sets how many replaced releases each node keeps, defaulting to `5`. See [going back to a previous release](/reference/v5/operations-api/operations#going-back-to-a-previous-release) for which releases are kept. -Use the other identifier to read the deployment details: +Before v5.3.0, or when no node keeps the release, deploy the previous release again: check out its commit and repeat the deploy, or pass the previous version as `package`. -```json -{ - "operation": "get_deployment", - "deployment_id": "" -} -``` +### From Studio -On 5.2, check its `status` and `peer_results` for install and replication failures, then confirm the serving version directly on every node. Later releases change how the rolling job applies the release; use the version-specific [CI wait procedure](../developers/deploying-from-ci.mdx#path-a-deploy-a-tagged-release-from-your-repository) with your installed version. + - - +In [Harper Studio](/reference/v5/studio/overview), open the cluster and go to **Config > Deployments**. Open the release you want back and choose **Roll back to this release**. Studio activates it on every node, one at a time, and shows the rollout's progress. Check the serving version on every node afterwards. -Prefer a pinned version or an immutable tarball over a moving branch reference. A branch moves, so you can neither audit what was running yesterday nor redeploy it. +## Limit a release's first traffic -:::tip -A pinned registry version such as `@my-org/orders-api@2.4.1` is the best-travelled form. If you pin a tarball URL instead, note that a bare `https://` URL pointing at `github.com`, `gitlab.com`, or `bitbucket.org` is treated as a git clone rather than a download, so a GitHub release asset URL will not behave the way the example above does. Host release tarballs somewhere neutral, or use a registry version. -::: +A rolling deploy changes every node. To test a release on part of your traffic first, separate installing it from exposing it, and decide at each step whether to expand. + +```mermaid +flowchart TD + Preflight --> Limit[Limit initial traffic] + Limit --> Observe + Observe --> Decide{Decide} + Decide -->|Advance| Expand[Expand traffic] + Expand -->|More stages| Observe + Expand -->|Rollout complete| Close[Verify and close] + Decide -->|Hold| Hold[Keep current exposure] + Hold --> Observe + Decide -->|Reverse| Reverse[Restore prior release or traffic] + Reverse --> Verify[Verify recovery] + Verify --> Close +``` -## What restart actually does +1. **Preflight.** Confirm the target version, component inventory, replication convergence, peer capacity with one node held out, and backups if data is at risk. Check that the prior release can still be activated or redeployed. Name the decision owner and set stop thresholds before starting. +2. **Limit.** Start with one drained node, an internal group, selected tenants, a region, or a small traffic percentage. Choose enough requests to make the comparison meaningful. +3. **Observe.** Compare old and new versions on request success, SLO latency, worker saturation, data correctness, replication, and downstream errors. +4. **Decide.** Advance, hold, or roll back against the agreed thresholds. Hold if required telemetry is missing. +5. **Expand.** Increase exposure only after minimum sample and hold conditions pass, and keep enough healthy capacity on the old version to reverse. +6. **Close.** Verify the same release and configuration on every node, restore intended traffic, and record the outcome with the measurements that supported each decision. -On Harper 5.2, the two modes behave as follows: +| Pattern | How Harper is used | Fits when, and watch out for | +| ----------------------------- | -------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | +| Isolated node validation | Drain one node, deploy with `replicated=false`, validate, re-admit a group | Strongest validation before any user sees the release. Needs spare capacity and a direct URL for each node | +| Replicated rolling deployment | `harper deploy` with `restart=rolling` | Simple once you trust the release. Each node stays in rotation while it restarts | +| Feature-targeted release | Deploy compatible code broadly, enable it by flag, header, or tenant | Best when behavior can be separated from deployment. Assign an owner and a removal date to each flag | +| Parallel environment | Deploy to a separate node pool, shift traffic after validation | Strong isolation and fast reversal. Write ownership and replication between pools need real design | +| Regional progression | Deploy the same release region by region, exposing each after its checks | Limits the blast radius. One region does not predict every traffic shape | -**`"restart": true`** starts a restart of the HTTP worker threads on the node handling the call and returns immediately, without waiting for it. A `200` means the deploy succeeded and a restart has been requested. It does not mean the new code is serving yet. Until a worker has been replaced it is still running the previous code, and on platforms where replacements share a listening port it keeps accepting connections during the changeover. +Isolated node validation needs at least three nodes, so the others can carry the traffic of the node you hold out. Take one node out of rotation with the [drain sequence](./health-checks-and-traffic-admission.mdx#drain-a-node-and-bring-it-back), log in to its own URL, and deploy to it alone: -**`"restart": "rolling"`** starts a replicated `restart_service` job. Poll its `restartJobId` with [`get_job`](/reference/v5/operations-api/operations#get_job). It sequences node restarts, but your peers still need enough capacity and your traffic-admission checks must pass. + + -:::warning -**Use `"rolling"` for cluster rollouts.** With `"restart": true`, each node restarts independently as it applies the replicated deployment. Those restarts can overlap and reduce serving capacity. Use `"restart": true` for the drained single-node validation above, or for a development instance. -::: +```bash +harper login https://node-a.example.com:9925 +harper deploy project=orders-api package=@my-org/orders-api@2.4.1 \ + replicated=false restart=true target=https://node-a.example.com:9925 +``` -Wait for the rolling job and verify the serving version on each node. A successful deployment record alone does not establish that every worker is running the release. +Run your readiness route and journey tests against that node directly, then admit a small group of users to it. -### Waiting for the restart + + + +```bash +harper deploy project=orders-api package=@my-org/orders-api@2.4.1 \ + restart=rolling json=true > deploy.json +``` - +[Wait for the job](#wait-for-every-node), then check every node. On Harper Pro 5.2, also read `harper get_deployment deployment_id="$(jq -r .deployment_id deploy.json)"` and check its `status` and `peer_results` for install and replication failures. -From v5.3.0, `"restart": true` waits for the worker restart to finish before responding, rather than returning as soon as it has been requested. The wait follows the restart's own progress rather than a fixed timeout, so the call can take tens of seconds on a slow install with many worker threads, with a hard ceiling of ten minutes. + + -A caller timeout does not stop the restart. On v5.3, the deploy response does not report restart success; check the logs for restart failures and verify the serving version through readiness or a journey test. For later versions, check the [restart and certification behavior](/reference/v5/operations-api/operations#restarting-restart) before choosing pipeline checks. +Deploy a pinned version or commit rather than a moving branch, so you can tell what ran yesterday and deploy it again. A private registry needs a [`credentials`](../developers/deploying-from-ci.mdx#give-harper-a-durable-credential-for-a-private-source) reference. -Set these parameters to match your payload size, network, and failure policy: +:::danger +**Component deployment replicates by default.** Pass `replicated=false` when validating on one node. Other operations have different [replication defaults](/reference/v5/replication/overview#replicated-transactions); check the operation's reference before an isolated rollout. +::: -- `deployment_timeout` is how long a peer waits for the replicated payload before failing, defaulting to 120000 ms. Raise it for large components or slow links. -- `ignore_replication_errors` treats a peer that fails to receive the deploy as non-fatal. By default a failed peer makes the whole operation return a non-2xx status, while the component is still deployed on the origin node. Decide which behavior you want before you need it. +To activate a release for part of your traffic without a second deploy, mount it at a path or hostname. `urlPath` mounts a component at an HTTP path and `host` serves it on a virtual hostname. Routing by either has been available since v5.1.0. What arrived in v5.2.0 is declaring that routing on the application's entry in the root `harper-config.yaml`, and the matching `host` parameter on `deploy_component`. Declared that way, the routing is persisted deployment state rather than a runtime toggle. See [HTTP middleware routing](/reference/v5/http/overview#middleware-routing). ## Configuration changes are deployments too Apply the same review, rollout, and rollback checks to configuration changes. -`set_configuration` supports `"replicated": true` to apply a change across the cluster in one call, with per-node outcomes in the response. For settings that apply on worker restart, follow with `restart_service` using `"replicated": true`, which restarts workers on each node in turn. - -Check the setting's restart requirements first. Process-level settings such as [`threads.v8Flags`](/reference/v5/configuration/options#applying-v8-flags) require a full process restart; `restart_service` does not apply them. Drain and restart each node in turn, using `restart` or your process/container manager, and verify it before advancing. The `restart` operation does not replicate. +`set_configuration` supports `replicated=true` to apply a change across the cluster in one call, with per-node outcomes in the response. For settings that apply on worker restart, follow with `restart_service`, which restarts workers on each node in turn: -```json -{ - "operation": "set_configuration", - "logging_level": "info", - "replicated": true -} +```bash +harper set_configuration logging_level=info replicated=true +harper restart_service service=http replicated=true ``` +Check the setting's restart requirements first. Process-level settings such as [`threads.v8Flags`](/reference/v5/configuration/options#applying-v8-flags) require a full process restart; `restart_service` does not apply them. Drain and restart each node in turn, using the `restart` operation or your process or container manager, and verify it before advancing. The `restart` operation does not replicate. + :::danger Only replicate cluster-appropriate parameters. Node-local values such as ports, `node.hostname`, file paths, TLS material, and `replication.hostname`, `url`, or `routes` would overwrite every peer's own values. Replicating one of these is a cluster-wide outage delivered in a single API call. ::: -`get_status.restartRequired` tracks component and code restarts, so it does not identify pending `set_configuration` changes. Record those changes yourself, use the required restart type, and read back `get_configuration`. Read-back confirms the configured value; verify the running behavior too. +`get_status.restartRequired` tracks component and code restarts, so it does not identify pending `set_configuration` changes. Record those changes yourself, use the required restart type, and read back `harper get_configuration`. Read-back confirms the configured value; verify the running behavior too. -## Rollback is a designed capability +## Classify the change before reversing it -Classify the change before choosing a reversal path, because these have different compatibility requirements, different authorities, and very different blast radii: +Different changes need different reversal paths, with different compatibility requirements and blast radii: -| Change type | Reversal | Watch out for | -| ---------------------- | ------------------------------------------------- | ------------------------------------------------------------------- | -| Component release | Deploy the previous immutable reference | The previous artifact must still be addressable | -| Feature behavior | Deactivate the flag, no deploy needed | Fastest reversal available. Requires the flag to have been built in | -| Traffic exposure | Restore prior weights or set the node unavailable | Fastest containment. Does not undo anything already written | -| Configuration | `set_configuration` back, then restart | Needs a restart, so it is not instant | -| Harper runtime version | Version-specific procedure | Mixed versions change replication and deploy behavior | -| Schema or data | Forward repair or restore | See the warning below | +| Change type | Reversal | Watch out for | +| ---------------------- | ----------------------------------------------------- | ------------------------------------------------------------------- | +| Component release | Activate the previous `deployment_id`, or redeploy it | Each node must still keep the release, or the artifact must exist | +| Feature behavior | Turn the flag off, no deploy needed | Fastest reversal available. Requires the flag to have been built in | +| Traffic exposure | Restore prior weights or set the node unavailable | Fastest containment. Does not undo anything already written | +| Configuration | `set_configuration` back, then restart | Needs a restart, so it is not instant | +| Harper runtime version | Version-specific procedure | Mixed versions change replication and deploy behavior | +| Schema or data | Forward repair or restore | See the warning below | -Practices that make each of these real: - -- **Keep the previous package addressable and rehearse redeploying it** before launch, not during an incident. -- **Prefer activating a retained release on v5.3.0 and later.** Use `deploy_component` with the previous `deployment_id` and the appropriate restart mode to restore its installed bytes without resolving or installing again. Verify that every target node still retains it: `deployment.stagingRetention.maxCount` can prune replaced releases. See [going back to a previous release](/reference/v5/operations-api/operations#going-back-to-a-previous-release). -- **Prefer expand-then-contract schema evolution.** Add the new shape, migrate, then remove the old shape in a separate change. Destructive schema behavior is operation and version dependent, so each destructive change needs an exact written procedure and verification on every node. -- **Use the same gates for rollback as for forward movement.** Availability, journey synthetic, peer stability, data validation, traffic reconciliation. A rollback is a deployment and can fail like one. +- **Rehearse rolling back before launch**, not during an incident. +- **Prefer expand-then-contract schema evolution.** Add the new shape, migrate, then remove the old shape in a separate change. Destructive schema behavior depends on the operation and version, so each destructive change needs an exact written procedure and verification on every node. +- **Use the same checks for rollback as for forward movement**: availability, journey tests, peer stability, data validation, and traffic. A rollback is a deployment and can fail like one. :::warning If a release changed the meaning of persisted data, code rollback alone will not fix it, and restoring a database to undo application code is usually the wrong move. A restore rolls back every write in the window, including all the valid ones, so it can violate your RPO in order to fix a code bug. Define forward repair or replay for these cases instead. See [Backup and Recovery](./backup-and-recovery.mdx). @@ -223,47 +337,43 @@ If a release changed the meaning of persisted data, code rollback alone will not Rehearse a stopped rollout end to end on a non-production cluster: -1. Declare a threshold before you start, for example "stop if journey success on the new cohort is more than 0.5 percent below the control cohort over five minutes." -2. Deploy a component that fails that threshold deliberately, to a bounded cohort. +1. Declare a threshold before you start, for example "stop if journey success on the new group is more than 0.5 percent below the control group over five minutes." +2. Deploy a release that fails that threshold deliberately, to a limited group. 3. Detect it through your dashboards rather than because you know what you did. -4. Reverse it, and record both the time from the reversal decision to verified service and the total time from failure to verified service. Compare the total, including detection and decision time, with your RTO. -5. Verify uniform state afterwards: `get_components` on every node, and `list_deployments` showing the reversal. - -Check every node after rollback; a cluster average can hide nodes still running the failed release. +4. Roll it back, and record both the time from the decision to verified service and the total time from failure to verified service. Compare the total, including detection and decision time, with your RTO. +5. Verify every node afterwards: `harper get_components` against each node's URL, and `harper list_deployments project=` showing the rollback. ## Operational notes -- **Version parity is an operating requirement.** Mixed Harper versions in a cluster change replication and deployment behavior, so a rollout that stalls halfway is a state you want to detect and exit, not sit in. -- **Keep deploy credentials in your delivery platform's secret store**, use TLS and least privilege, and retain the operation result as change evidence. See [secrets](/reference/v5/security/secrets). -- **Component deployment can replicate, so a deploy is a cluster event.** Keep isolated single-node deployment available for validation, because if the only deployment path you have is replicated then you have no way to test anything on one node. -- **Flag cleanup is part of the release.** A feature flag with no owner and no removal date becomes permanent configuration that nobody understands, and it will eventually be the thing nobody can explain during an incident. -- **Record who decided, not only what happened.** Named decision authority is what makes a stop gate function under pressure, and it costs nothing to write down in advance. +- **Keep Harper versions the same on every node.** Mixed versions change replication and deployment behavior, so detect and finish a rollout that stalls halfway. +- **Keep deploy credentials in your delivery platform's secret store**, or use the OIDC setup above so there is none. Use TLS and least privilege, and keep the deploy result as change evidence. See [secrets](/reference/v5/security/secrets). +- **Keep a single-node deployment path.** Every deploy replicates by default, so without `replicated=false` and a direct node URL you cannot test anything on one node. ## Readiness checklist -- [ ] Artifacts are immutable and referenced by pinned version, never by branch -- [ ] Deploy and expose are separate actions in your pipeline -- [ ] The previous known-good artifact is addressable, and redeploying it has been rehearsed -- [ ] On v5.3.0 and later, the previous `deployment_id` is still retained on every target node and activation has been rehearsed +- [ ] Each release is a pinned version, a commit, or a staged build, never a moving branch +- [ ] The rollback path is rehearsed: on v5.3.0 and later the previous `deployment_id` is kept on every node; otherwise the previous artifact is still available +- [ ] Deploy and expose are separate steps for changes that need a limited first audience - [ ] Cohort ladder defined, from smallest useful sample to full exposure - [ ] Stop thresholds and hold times declared before the rollout starts -- [ ] A named decision owner for advance, hold, stop, and roll back -- [ ] Pipeline polls the restart job rather than treating the deploy response as proof the new code is serving -- [ ] `"replicated": false` used deliberately wherever a change is meant to reach one node only, since replication is the default -- [ ] Cluster deploys use `"restart": "rolling"`, not `"restart": true` +- [ ] A named decision owner for advance, hold, and roll back +- [ ] Pipeline waits for the rolling job, then checks the serving version on every node +- [ ] Nodes that must not serve while they change are drained first, since `restart=rolling` keeps each node in rotation +- [ ] `replicated=false` used deliberately wherever a change is meant to reach one node only, since replication is the default +- [ ] Cluster deploys use `restart=rolling`, not `restart=true` - [ ] `deployment_timeout` and `ignore_replication_errors` set deliberately -- [ ] Configuration changes go through the same change record as code -- [ ] `get_configuration` read back after every configuration change +- [ ] Configuration changes go through the same change record as code, and `get_configuration` is read back after each one - [ ] Schema changes follow expand-then-contract, with destructive operations documented per node - [ ] Forward repair defined for changes that alter persisted meaning -- [ ] Reversal duration and total recovery time recorded from a rehearsal, with the total compared against RTO +- [ ] Rollback duration and total recovery time recorded from a rehearsal, with the total compared against RTO ## Additional Resources -- [Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx) for the pipeline mechanics and deploy credentials -- [`deploy_component` reference](/reference/v5/operations-api/operations#deploy_component) for every parameter including `urlPath`, `host`, and replication controls -- [Applications reference](/reference/v5/components/applications) for component structure and the full component operation set +- [Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx) for OIDC setup, private sources, and registry artifacts +- [`harper deploy`](/reference/v5/cli/commands#harper-deploy) for every CLI deploy parameter +- [`deploy_component` reference](/reference/v5/operations-api/operations#deploy_component) for the server-side behavior, including `urlPath`, `host`, and replication controls - [Deployment records](/reference/v5/operations-api/operations#deployment-operations) for `list_deployments` and `get_deployment` +- [Static files](/reference/v5/static-files/overview) for serving a built frontend - [HTTP middleware routing](/reference/v5/http/overview#middleware-routing) for `urlPath` and `host` activation - [Configuration operations](/reference/v5/configuration/operations) for `set_configuration` and restart requirements - [Database schema](/reference/v5/database/schema) for schema evolution diff --git a/learn/developers/deploying-from-ci.mdx b/learn/developers/deploying-from-ci.mdx index 140d4ff2b..053ca793b 100644 --- a/learn/developers/deploying-from-ci.mdx +++ b/learn/developers/deploying-from-ci.mdx @@ -195,7 +195,7 @@ A few details worth noting: - **`harper@^5.3`** pins the CLI's major version. 5.3 is the first CLI that authenticates with the identity token, and rerunning an old tag after a new major release still installs a 5.x CLI. - **`by_ref=true`** deploys `git+https://github.com//.git#`, where the SHA is the commit the tag points to (`GITHUB_SHA`). Every node clones that exact commit, even if the tag is moved later. Pass `ref=`, with the full commit SHA, to deploy a different commit. In a private repository, a tag resolves only if the checkout fetched it, as `fetch-depth: 0` does: the CLI looks up anything else with `git ls-remote`, which has no credential once `persist-credentials: false` keeps the token out of `.git/config`. - **`replicated` isn't needed.** On Harper Fabric and Harper Pro, a deploy goes to every node in the cluster unless you pass `replicated=false`. Each node fetches and installs the package itself, resolving the credential from the replicated secrets store. Harper core on its own does not replicate. -- **`restart=rolling`** restarts nodes one at a time so the cluster keeps serving throughout. Use `restart=true` for a single-node or dev instance. The [preview recipe](#per-pr-previews-with-shared-databases) also uses inline restarts on clusters, with an explicit allowance for concurrent initial restarts. On Harper core alone, a rolling job that only restarts — before v5.4.0, or when `certification` is `unavailable` — fails with `Replication not implemented`, which fails the wait step. +- **`restart=rolling`** restarts nodes one at a time, so they never all restart at once. Each node stays in rotation while it restarts; see [what `restart=rolling` does](../administration/safe-deployments-and-rollback.mdx#what-restartrolling-does). Use `restart=true` for a single-node or dev instance. The [preview recipe](#per-pr-previews-with-shared-databases) also uses inline restarts on clusters, with an explicit allowance for concurrent initial restarts. On Harper core alone, a rolling job that only restarts — before v5.4.0, or when `certification` is `unavailable` — fails with `Replication not implemented`, which fails the wait step. - **`json=true`** prints the deploy's result as JSON on stdout. Progress and authentication messages go to stderr, so `deploy.json` holds only the result. A deploy that fails exits non-zero, which fails the step. - **The wait step.** With `restart=rolling`, `harper deploy` exits `0` once the node you called has taken the release and started the rolling job, which keeps running after the deploy step has passed. On v5.4.0 and later, unless the deploy's `certification` is `unavailable`, the other nodes take the release in that job, one at a time, and a node that rejects it fails the job, not the deploy step; the failed job's `message` lists each node's outcome under `activated`. Otherwise every node installs the release before the deploy step passes, and the job only restarts them. The wait step polls the job's `restartJobId` with `get_job` until it ends `COMPLETE` or `ERROR`, and retries a poll that fails, such as one that reaches a node while it restarts. Each `harper get_job` authenticates through the trust policy the same way. It gives up after 60 polls, about ten minutes. Nodes take a rolling release one after another, each restarting and, on v5.4.0 and later, loading it in a canary that has up to a minute to report, so on a cluster with many nodes raise the count. With `restart=true` there is no job, and the deploy step itself fails when a node rejects the release, unless you pass `ignore_replication_errors=true`. - **The job lives on one node.** Everything else about a deploy is visible from any node, but Harper does not replicate jobs, so only the node that received the deploy knows the rolling job. Any other node answers `get_job` with an empty list, which the wait step treats as not finished and polls again. So through a load-balanced cluster URL, at least one poll has to reach the node that received the deploy. Routing that keeps the runner on one node sends its polls where the deploy went, and routing that spreads requests across nodes reaches it within a few polls. Routing that sends the polls to a different node than the deploy never does: the step fails, saying the job was not on the node that answered and its outcome is unknown, even when the rollout succeeded. If your load balancer routes that way, use one node's own URL for both the deploy and the wait, and as the trust policy's `audience`. [harper#3060](https://github.com/HarperFast/harper/issues/3060) tracks recording each node's outcome on the deployment record, which replicates and would remove this requirement. diff --git a/reference/cli/commands.md b/reference/cli/commands.md index a803f1b7f..d85cb87ad 100644 --- a/reference/cli/commands.md +++ b/reference/cli/commands.md @@ -141,7 +141,7 @@ All parameters are passed as `key=value` arguments. Every parameter is optional. - `project=` - Component project name. Defaults to the current directory's name for a directory deploy, or is derived from the package for a package deploy. - `package=` - An npm, GitHub, or tarball reference to deploy instead of the current directory (e.g. `HarperDB/app#semver:v1.0.0`). - `target=` - Remote Harper instance to deploy to. Omit to deploy to the local instance. A bare host defaults to `https://:9925`. -- `restart=true` or `restart=rolling` - Restart Harper after deploying. Use `rolling` for a staggered, zero-downtime restart across a cluster. With `rolling`, the command exits `0` once the node you called has taken the release and started the rolling restart's job; see [Waiting for a rolling deploy](#waiting-for-a-rolling-deploy). Cannot be combined with `activate=false`. +- `restart=true` or `restart=rolling` - Restart Harper after deploying. Use `rolling` to restart a cluster's nodes one at a time; each node stays in rotation while it restarts. With `rolling`, the command exits `0` once the node you called has taken the release and started the rolling restart's job; see [Waiting for a rolling deploy](#waiting-for-a-rolling-deploy). Cannot be combined with `activate=false`. - `replicated=false` - Deploy to the node you called only. On Harper Pro and Fabric, a deploy goes to every node in the cluster unless you pass this; Harper core on its own does not replicate. - `install_command=` - Override the install command run for the component. - `install_timeout=` - Maximum time, in milliseconds, to allow the install to run. From 79419e3a71243a0b9cded6365c7abec6f108e5e3 Mon Sep 17 00:00:00 2001 From: Ethan Arrowood Date: Fri, 16 Oct 2026 09:03:09 -0600 Subject: [PATCH 18/21] Address pre-push review findings on the reliability guides - Staged frontend config: keep the project's own rest, schema, resource, and static entries (TypeScript templates load resources/*.ts) and drop only the plugins whose packages the release does not install. - Configuration operations reference: replace "the cluster stays available" with the same rolling-restart limit the guides now state. - Operating replication: define the exported OpsProbe table the convergence recipe writes, and name logging.auditRetention as the bound past which a peer sends a full copy. - Node rotation: note that a worker restart can briefly run two loops on Linux, and that it ends the "not going out again" state. Co-Authored-By: Claude Opus 5.5 Dispatch-Task: fix-Ethan-Arrowood_documentation_671-1627592f --- learn/administration/operating-replication.mdx | 14 ++++++++++++-- .../safe-deployments-and-rollback.mdx | 4 ++-- learn/developers/taking-a-node-out-of-rotation.mdx | 3 ++- reference/configuration/operations.md | 2 +- 4 files changed, 17 insertions(+), 6 deletions(-) diff --git a/learn/administration/operating-replication.mdx b/learn/administration/operating-replication.mdx index 81437b750..fce8b84dd 100644 --- a/learn/administration/operating-replication.mdx +++ b/learn/administration/operating-replication.mdx @@ -92,7 +92,7 @@ There is one socket per database per peer, so the sockets present in the respons | Node registry (`hdb_nodes`) | Lives in `system`, so it propagates with it. Each node rewrites its own self-record from its `harper-config.yaml` routes on restart or component reload | Scoping applied through `add_node` alone does not survive a restart. Put durable constraints in config routes, and see below if you need the registry kept per cluster | | Destructive schema changes | `drop_database` and `drop_table` replicate by default; `drop_attribute` does not | A drop is a cluster-wide event unless you pass `"replicated": false`. Verify the result on every node | -Measure incremental catch-up and full synchronization separately. A briefly offline node can catch up from retained transaction history. If its required history has been pruned, a full copy may be needed even though the node has synchronized before. +Measure incremental catch-up and full synchronization separately. A briefly offline node can catch up from retained transaction history. If it was away longer than its peers retain that history, set by [`logging.auditRetention`](/reference/v5/logging/configuration#loggingauditretention) (default `3d`), the peer sends a full copy instead, even though the node has synchronized before. On the new node, the cost is larger than "one copy from one peer," because the full-copy decision is made per peer and per database rather than once against a bootstrap source. A joining node requests a full copy from every peer it has no resume cursor for. And because replication is bidirectional, each established peer independently decides it has no cursor for the newcomer and requests a full copy _from_ it as well. Joining a cluster of N peers is therefore N inbound transfers plus N outbound transfers of the new node's own (empty) databases, not a single stream from one source. @@ -168,7 +168,17 @@ Inventory `system.hdb_nodes` alongside this and compare it to your intended topo ## Prove convergence, not connection -Socket state cannot tell you that a specific business record is current. Write a sentinel on the source node: +Socket state cannot tell you that a specific business record is current. Add an exported probe table to your application's schema, with `database` set to the database you want to measure. Leave replication on, which is the default; a `replicate: false` table stays on the node that wrote it: + +```graphql +type OpsProbe @table(database: "data") @export { + id: ID @primaryKey + writtenAt: Long + from: String +} +``` + +Write a sentinel on the source node: ```javascript const marker = { id: 'convergence-probe', writtenAt: Date.now(), from: 'server-1' }; diff --git a/learn/administration/safe-deployments-and-rollback.mdx b/learn/administration/safe-deployments-and-rollback.mdx index 908db64c5..25e360e35 100644 --- a/learn/administration/safe-deployments-and-rollback.mdx +++ b/learn/administration/safe-deployments-and-rollback.mdx @@ -52,7 +52,7 @@ The upload includes everything in the directory except `node_modules`, including ### A frontend built once -The React and Vue templates load the `@harperfast/vite` plugin, which builds the frontend on each node when the build output is missing, so every node also carries the build toolchain. To build once and ship only the output, keep a second configuration file for the release. It serves the build output with the built-in [`static`](/reference/v5/static-files/overview) plugin and leaves out the Vite plugin: +The React and Vue templates load the `@harperfast/vite` plugin, which builds the frontend on each node when the build output is missing, so every node also carries the build toolchain. To build once and ship only the output, keep a second configuration file for the release. Copy your `config.yaml` to `deploy.config.yaml`, keep its `rest`, `graphqlSchema`, `jsResource`, and [`static`](/reference/v5/static-files/overview) entries as they are, and remove the `@harperfast/vite` and `@harperfast/schema-codegen` entries, which load packages the release does not install. For the JavaScript React template, that leaves: ```yaml # deploy.config.yaml @@ -60,7 +60,7 @@ rest: true graphqlSchema: files: 'schemas/*.graphql' jsResource: - files: 'resources/*.js' + files: 'resources/*.js' # the TypeScript templates use 'resources/*.ts' static: files: 'dist/**' ``` diff --git a/learn/developers/taking-a-node-out-of-rotation.mdx b/learn/developers/taking-a-node-out-of-rotation.mdx index d642b4f44..cbfabeb2b 100644 --- a/learn/developers/taking-a-node-out-of-rotation.mdx +++ b/learn/developers/taking-a-node-out-of-rotation.mdx @@ -439,7 +439,7 @@ export function startAvailabilityLoop(checkHealth) { A few details: -- **One loop per node.** Every worker thread loads the component, so the loop starts only where `server.workerIndex` is `0`. An [isolated application](/reference/v5/components/applications#isolated-applications) runs in one worker, whose index is `0`. +- **One loop per node.** Every worker thread loads the component, so the loop starts only where `server.workerIndex` is `0`. A worker restart can briefly overlap two; see [Caveats](#caveats). An [isolated application](/reference/v5/components/applications#isolated-applications) runs in one worker, whose index is `0`. - **A health check that throws, or does not answer within 5 seconds, counts as failing.** Either way the loop goes on looking. - **No user needed.** [`server.operation()`](/reference/v5/http/api#serveroperationoperation-context-authorize) applies no authorization unless you pass `authorize: true`, so the loop can call `set_status` without a user or a role. The `status-check` component calls it the same way. - **Recognizing itself.** `AVAILABILITY_NODES` lists every node, so the same `.env` works on all of them. The loop skips the address whose `/Health` reports this node's [`server.hostname`](/reference/v5/http/api#serverhostname). @@ -610,6 +610,7 @@ When you are done, stop Harper with `npx harper stop`, stop the stand-ins, and d - **Someone else's write pauses the loop.** After Fabric or an operator puts back a node the loop took out, the loop leaves that node alone until its health check has passed for 60 seconds, however long the node keeps failing. - **"Serving" is what this node sees.** A node that answers at its own hostname counts as serving even if the load balancer cannot reach it, and so does a node that is still running but no longer in the cluster. Keep `AVAILABILITY_NODES` current. - **The ownership check has small gaps.** The loop reads the record again just before each write, and after a failed or interrupted write it only claims an `Unavailable` written within 5 seconds of its marker. So someone else's write can be taken for the loop's if it lands a few milliseconds after the loop's own write, or within those 5 seconds when a restart interrupted the loop. Coming back has the same kind of gap: a write landing between the loop's last read and its own `Available` is overwritten. The reverse can happen too: if the loop's own write took longer than 5 seconds and a restart interrupted it, the loop leaves the node out and logs the write as someone else's. After a restart that hit a node mid-write, check its `/status`. +- **A worker restart can briefly run two loops.** On Linux, including Fabric, Harper starts a replacement worker before it stops the one it replaces, so the old worker `0` and its replacement both run the loop until the old one exits. That can take up to `replication.blobSendDrainTimeout` while it finishes sending blobs. Each loop has its own timers and can write in that window, with the same ownership gaps as above. A worker restart, such as a deploy with `restart`, also ends "not going out again until Harper restarts". After a deploy or restart, check each node's `/status`. - **Operation logging.** At log level `info` or more verbose, Harper logs every operation call, including the loop's `get_status` every 5 seconds. ## Additional Resources diff --git a/reference/configuration/operations.md b/reference/configuration/operations.md index 75d9ed695..ec521f8d4 100644 --- a/reference/configuration/operations.md +++ b/reference/configuration/operations.md @@ -72,7 +72,7 @@ The `replicated` array reports per-node outcomes. A failed peer appears as `{ "s } ``` -To restart the whole cluster afterward, follow with [restart_service](../operations-api/operations.md#restart_service) using `"replicated": true` — it restarts nodes one at a time, so the cluster stays available: +To restart the whole cluster afterward, follow with [restart_service](../operations-api/operations.md#restart_service) using `"replicated": true` — it restarts nodes one at a time, so they never all restart at once. Each node stays in rotation while it restarts, so drain a node first if it must not serve during the change: ```json { From f14d875c8467a5378d47cfb518c44aa8094cc281 Mon Sep 17 00:00:00 2001 From: Ethan Arrowood Date: Fri, 16 Oct 2026 09:17:43 -0600 Subject: [PATCH 19/21] Keep the catch-up retention wording hedged and write the probe via databases A forced full copy is not yet verified to remove rows deleted while a node was away past retention, so state only that such a node needs a full copy. Write the convergence sentinel through databases. so the recipe works for a database other than data. Co-Authored-By: Claude Opus 5.5 Dispatch-Task: fix-Ethan-Arrowood_documentation_671-1627592f --- learn/administration/operating-replication.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/learn/administration/operating-replication.mdx b/learn/administration/operating-replication.mdx index fce8b84dd..aecda8362 100644 --- a/learn/administration/operating-replication.mdx +++ b/learn/administration/operating-replication.mdx @@ -92,7 +92,7 @@ There is one socket per database per peer, so the sockets present in the respons | Node registry (`hdb_nodes`) | Lives in `system`, so it propagates with it. Each node rewrites its own self-record from its `harper-config.yaml` routes on restart or component reload | Scoping applied through `add_node` alone does not survive a restart. Put durable constraints in config routes, and see below if you need the registry kept per cluster | | Destructive schema changes | `drop_database` and `drop_table` replicate by default; `drop_attribute` does not | A drop is a cluster-wide event unless you pass `"replicated": false`. Verify the result on every node | -Measure incremental catch-up and full synchronization separately. A briefly offline node can catch up from retained transaction history. If it was away longer than its peers retain that history, set by [`logging.auditRetention`](/reference/v5/logging/configuration#loggingauditretention) (default `3d`), the peer sends a full copy instead, even though the node has synchronized before. +Measure incremental catch-up and full synchronization separately. A briefly offline node can catch up from retained transaction history, which its peers keep for [`logging.auditRetention`](/reference/v5/logging/configuration#loggingauditretention) (default `3d`). A node away longer than that needs a full copy, even though it has synchronized before. On the new node, the cost is larger than "one copy from one peer," because the full-copy decision is made per peer and per database rather than once against a bootstrap source. A joining node requests a full copy from every peer it has no resume cursor for. And because replication is bidirectional, each established peer independently decides it has no cursor for the newcomer and requests a full copy _from_ it as well. Joining a cluster of N peers is therefore N inbound transfers plus N outbound transfers of the new node's own (empty) databases, not a single stream from one source. @@ -168,7 +168,7 @@ Inventory `system.hdb_nodes` alongside this and compare it to your intended topo ## Prove convergence, not connection -Socket state cannot tell you that a specific business record is current. Add an exported probe table to your application's schema, with `database` set to the database you want to measure. Leave replication on, which is the default; a `replicate: false` table stays on the node that wrote it: +Socket state cannot tell you that a specific business record is current. Add an exported probe table to your application's schema, in the database you want to measure. Leave replication on, which is the default; a `replicate: false` table stays on the node that wrote it. For another database, change `data` here and in the write below, and give each database's probe table its own name: ```graphql type OpsProbe @table(database: "data") @export { @@ -182,7 +182,7 @@ Write a sentinel on the source node: ```javascript const marker = { id: 'convergence-probe', writtenAt: Date.now(), from: 'server-1' }; -await tables.OpsProbe.put(marker.id, marker); +await databases.data.OpsProbe.put(marker.id, marker); ``` Read it from the target node and compare `writtenAt`: From c301b3e64ba84fb9b3bc6907a5c2ac06eb4562d5 Mon Sep 17 00:00:00 2001 From: Ethan Arrowood Date: Fri, 16 Oct 2026 09:26:15 -0600 Subject: [PATCH 20/21] Tell operators to rebuild a node that was away past audit retention Deletes older than logging.auditRetention are purged from history, so the full copy a long-absent node receives can leave rows that were deleted elsewhere. Qualify the returning-node row and the replication reference by that bound, and say to rebuild such a node from empty or replace it. Co-Authored-By: Claude Opus 5.5 Dispatch-Task: fix-Ethan-Arrowood_documentation_671-1627592f --- learn/administration/operating-replication.mdx | 4 ++-- reference/replication/overview.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/learn/administration/operating-replication.mdx b/learn/administration/operating-replication.mdx index aecda8362..610605b82 100644 --- a/learn/administration/operating-replication.mdx +++ b/learn/administration/operating-replication.mdx @@ -85,14 +85,14 @@ There is one socket per database per peer, so the sockets present in the respons | Data mutations | Insert, update, upsert, delete, and bulk load replicate for tables in scope | Monitor connections, latency, and convergence for the databases that serve critical journeys | | Transactions | Replicated atomically, and a transaction may span multiple tables | Tables that need to commit together must live in the same database | | A brand new node | Requests a full copy of each in-scope database from every peer it has no resume cursor for, then enters incremental replication | Budget N transfers, not one. See the note below before sizing a join | -| A returning node | Resynchronizes automatically to catch up on the transactions it missed | This is not a full copy. A routine restart is much cheaper than a node replacement, so budget them separately | +| A returning node | Resynchronizes automatically to catch up on the transactions it missed | Not a full copy, unless the node was away longer than `logging.auditRetention`. A routine restart is much cheaper than a node replacement, so budget them separately | | Component deployment | `deploy_component` replicates by default; pass `"replicated": false` to hold it to one node | Cluster-wide is the default, so single-node validation is the case you must ask for | | Configuration | `set_configuration` supports `"replicated": true` | Never replicate node-local parameters. See the warning below | | Users and roles | Propagate when the `system` database is in replication scope, which is the default | If you narrow scope to exclude `system`, identity must be provisioned on every node by your own automation | | Node registry (`hdb_nodes`) | Lives in `system`, so it propagates with it. Each node rewrites its own self-record from its `harper-config.yaml` routes on restart or component reload | Scoping applied through `add_node` alone does not survive a restart. Put durable constraints in config routes, and see below if you need the registry kept per cluster | | Destructive schema changes | `drop_database` and `drop_table` replicate by default; `drop_attribute` does not | A drop is a cluster-wide event unless you pass `"replicated": false`. Verify the result on every node | -Measure incremental catch-up and full synchronization separately. A briefly offline node can catch up from retained transaction history, which its peers keep for [`logging.auditRetention`](/reference/v5/logging/configuration#loggingauditretention) (default `3d`). A node away longer than that needs a full copy, even though it has synchronized before. +Measure incremental catch-up and full synchronization separately. A briefly offline node can catch up from retained transaction history, which its peers keep for [`logging.auditRetention`](/reference/v5/logging/configuration#loggingauditretention) (default `3d`). A node away longer than that gets a full copy of its peers' rows instead, even though it has synchronized before. Deletes older than the retention window are gone from that history, so the copy can leave rows on the returning node that were deleted elsewhere. Rebuild such a node from empty, or replace it, rather than returning it with its old data. On the new node, the cost is larger than "one copy from one peer," because the full-copy decision is made per peer and per database rather than once against a bootstrap source. A joining node requests a full copy from every peer it has no resume cursor for. And because replication is bidirectional, each established peer independently decides it has no cursor for the newcomer and requests a full copy _from_ it as well. Joining a cluster of N peers is therefore N inbound transfers plus N outbound transfers of the new node's own (empty) databases, not a single stream from one source. diff --git a/reference/replication/overview.md b/reference/replication/overview.md index 8f19fc68b..c6d38fbb3 100644 --- a/reference/replication/overview.md +++ b/reference/replication/overview.md @@ -362,7 +362,7 @@ See [Clustering Operations](./clustering.md#cluster-status) for the full respons Initial synchronization is decided per peer and per database. A new node can request a full copy from each in-scope peer for which it has no resume cursor, so a join can load several peers at once. [`add_node` with `isLeader: true`](./clustering.md#add-node) identifies a bootstrap source; it does not prevent full-copy requests to other cursorless peers. After synchronization, replication continues incrementally. -If a node goes offline and comes back, it resynchronizes automatically to catch up on missed transactions. +If a node goes offline and comes back, it resynchronizes automatically to catch up on missed transactions. Its peers keep that history for [`logging.auditRetention`](../logging/configuration.md#loggingauditretention); a node away longer than that gets a full copy instead, which can leave rows on it that were deleted elsewhere while it was away. Rebuild such a node from empty, or replace it. You can also specify a `start_time` in the `add_node` operation to limit the initial download to data since a given point in time: From 1ef7c4ce0bc56488d1b97c925adf583850a66d0c Mon Sep 17 00:00:00 2001 From: Ethan Arrowood Date: Fri, 16 Oct 2026 09:33:09 -0600 Subject: [PATCH 21/21] Point long-absent nodes at the reconciliation policy instead of a rebuild Rebuilding from empty would discard writes only that node took, so state both sides of the hazard and send operators to the restore guide's authority and reconciliation steps, noting that routes reconnect on start. Co-Authored-By: Claude Opus 5.5 Dispatch-Task: fix-Ethan-Arrowood_documentation_671-1627592f --- learn/administration/operating-replication.mdx | 2 +- reference/replication/overview.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/learn/administration/operating-replication.mdx b/learn/administration/operating-replication.mdx index 610605b82..fb4fd33bd 100644 --- a/learn/administration/operating-replication.mdx +++ b/learn/administration/operating-replication.mdx @@ -92,7 +92,7 @@ There is one socket per database per peer, so the sockets present in the respons | Node registry (`hdb_nodes`) | Lives in `system`, so it propagates with it. Each node rewrites its own self-record from its `harper-config.yaml` routes on restart or component reload | Scoping applied through `add_node` alone does not survive a restart. Put durable constraints in config routes, and see below if you need the registry kept per cluster | | Destructive schema changes | `drop_database` and `drop_table` replicate by default; `drop_attribute` does not | A drop is a cluster-wide event unless you pass `"replicated": false`. Verify the result on every node | -Measure incremental catch-up and full synchronization separately. A briefly offline node can catch up from retained transaction history, which its peers keep for [`logging.auditRetention`](/reference/v5/logging/configuration#loggingauditretention) (default `3d`). A node away longer than that gets a full copy of its peers' rows instead, even though it has synchronized before. Deletes older than the retention window are gone from that history, so the copy can leave rows on the returning node that were deleted elsewhere. Rebuild such a node from empty, or replace it, rather than returning it with its old data. +Measure incremental catch-up and full synchronization separately. A briefly offline node can catch up from retained transaction history, which its peers keep for [`logging.auditRetention`](/reference/v5/logging/configuration#loggingauditretention) (default `3d`). A node away longer than that gets a full copy of its peers' rows instead, even though it has synchronized before. Deletes older than the retention window are gone from that history, so the copy can leave rows on the returning node that were deleted elsewhere, while keeping writes that only that node took. Before such a node reconnects, decide which data is authoritative and how to reconcile it, as for [restoring in a replicated cluster](./backup-and-recovery.mdx#restoring-in-a-replicated-cluster). Configured routes reconnect as soon as Harper starts. On the new node, the cost is larger than "one copy from one peer," because the full-copy decision is made per peer and per database rather than once against a bootstrap source. A joining node requests a full copy from every peer it has no resume cursor for. And because replication is bidirectional, each established peer independently decides it has no cursor for the newcomer and requests a full copy _from_ it as well. Joining a cluster of N peers is therefore N inbound transfers plus N outbound transfers of the new node's own (empty) databases, not a single stream from one source. diff --git a/reference/replication/overview.md b/reference/replication/overview.md index c6d38fbb3..9ae1a26d7 100644 --- a/reference/replication/overview.md +++ b/reference/replication/overview.md @@ -362,7 +362,7 @@ See [Clustering Operations](./clustering.md#cluster-status) for the full respons Initial synchronization is decided per peer and per database. A new node can request a full copy from each in-scope peer for which it has no resume cursor, so a join can load several peers at once. [`add_node` with `isLeader: true`](./clustering.md#add-node) identifies a bootstrap source; it does not prevent full-copy requests to other cursorless peers. After synchronization, replication continues incrementally. -If a node goes offline and comes back, it resynchronizes automatically to catch up on missed transactions. Its peers keep that history for [`logging.auditRetention`](../logging/configuration.md#loggingauditretention); a node away longer than that gets a full copy instead, which can leave rows on it that were deleted elsewhere while it was away. Rebuild such a node from empty, or replace it. +If a node goes offline and comes back, it resynchronizes automatically to catch up on missed transactions. Its peers keep that history for [`logging.auditRetention`](../logging/configuration.md#loggingauditretention); a node away longer than that gets a full copy instead, which can leave rows on it that were deleted elsewhere while it was away. Decide how to reconcile such a node before it reconnects. You can also specify a `start_time` in the `add_node` operation to limit the initial download to data since a given point in time: