diff --git a/fabric/load-balancing.md b/fabric/load-balancing.md new file mode 100644 index 000000000..c6da0455c --- /dev/null +++ b/fabric/load-balancing.md @@ -0,0 +1,79 @@ +--- +title: Load Balancing and Node Availability +--- + +# Load Balancing and Node Availability + +A Harper Fabric cluster serves your applications from one hostname, the cluster's Fabric hostname (for example `my-cluster.my-org.harperfabric.com`) or a [custom domain](/fabric/custom-domains) pointed at it. Fabric's load balancer picks a node of the cluster for each new connection to that hostname. This page covers how it decides which nodes may receive traffic, and how to read or change that decision for one node. + +## How Fabric decides where traffic goes + +The load balancer checks each node with a `GET /status` request to that node. A node that answers `200` may receive new connections. A node that answers anything else, or does not answer, receives none until it answers `200` again. + +`/status` is served by a component that Fabric deploys to every node of the cluster, named `status-check`. Its source is public: [HarperFast/status-check-fabric](https://github.com/HarperFast/status-check-fabric). It answers from a single record on the node, the `availability` status, which Harper's [`set_status` and `get_status`](/reference/v5/operations-api/operations#set_status--get_status--clear_status) operations write and read: + +| `availability` record | `GET /status` answers | +| --------------------- | --------------------- | +| `Available` | `200` | +| `Unavailable` | `404` | +| No record | `404` | + +Fabric writes this record itself. For example, it takes a node out while it upgrades that node, and puts it back afterwards. + +### Each node has its own record + +Harper keeps status records in the `system.hdb_status` table, which it creates with replication turned off. Every node therefore has its own `availability` record: writing it on one node changes only that node. The record is stored on disk, so it keeps its value when Harper restarts. + +Because each node answers for itself, a request meant for one node has to reach that node. Through the cluster hostname, the load balancer decides which node receives it. Use the node's own hostname, which only that node answers. + +## The `status-check` component + +### `GET /status` + +Unauthenticated. + +- `200` with the body `"Status endpoint is reporting for duty."` when the node's `availability` record is `Available`. +- `404` with an error titled `Status endpoint is reporting downtime.` otherwise, including when the node has no `availability` record. + +```bash +curl -i https:///status +``` + +### `POST /status` + +Sets the node's `availability` record to `Available`. Requires a `super_user`: the component answers `204` on success, `401` to a request without credentials, and `403` to any other user. + +### `DELETE /status` + +Sets the node's `availability` record to `Unavailable`. Authorized the same way as `POST`. + +To change the record yourself, use the Harper CLI rather than these routes, as [Taking a node out of rotation](#taking-a-node-out-of-rotation) shows. It needs no password on the command line, and from v5.3.0 the user it runs as does not have to be a `super_user`. + +:::warning +Fabric routes traffic by this component. Do not remove it, and do not deploy another resource that serves `/status`. +::: + +:::note +This is not the [`@harperdb/status-check`](https://github.com/HarperFast/status-check) npm package, which keeps its own `hdb_status` table in the `data` database. Fabric does not read that table, and the package also serves `/status`, so do not deploy it on a Fabric cluster. On a self-managed cluster, [Health Checks and Traffic Admission](/learn/administration/health-checks-and-traffic-admission) covers it. +::: + +## Taking a node out of rotation + +To stop new traffic reaching a node, set its `availability` record to `Unavailable` with the [`set_status`](/reference/v5/operations-api/operations#set_status--get_status--clear_status) operation. To let traffic back, set it to `Available`. Send each command to the node's own hostname. From your own machine, [log in](/reference/v5/cli/authentication#method-1-persistent-login-recommended-for-local-development) once and the CLI keeps the token: + +```bash +harper login https:// +harper set_status id=availability status=Unavailable target=https:// +harper get_status id=availability target=https:// +harper set_status id=availability status=Available target=https:// +``` + +`get_status` returns the record, including `__updatedtime__`, the time of its last write. + +From a GitHub Actions workflow, run the same commands with [workload identity (OIDC)](/reference/v5/cli/authentication#workload-identity-oidc), so the workflow holds no Harper credential at all. [Deploying from a CI/CD Pipeline](/learn/developers/deploying-from-ci) shows how to set up the trust policy and the user it maps to. + +That user does not need to be a `super_user`. From v5.3.0, a role whose [`operations`](/reference/v5/users-and-roles/overview#operation-permissions) list names only `set_status` and `get_status` can run those two operations and nothing else. + +Connections already open to the node stay open: the load balancer only decides where new connections go. Fabric writes the same record, so a value you set can be replaced by Fabric's own operations on the node. + +An application can also feed its own health into this record, so that a node taking errors leaves rotation while its peers keep serving. See [Taking an Unhealthy Node Out of Rotation](/learn/developers/taking-a-node-out-of-rotation). diff --git a/learn/administration/backup-and-recovery.mdx b/learn/administration/backup-and-recovery.mdx new file mode 100644 index 000000000..926252ca3 --- /dev/null +++ b/learn/administration/backup-and-recovery.mdx @@ -0,0 +1,274 @@ +--- +title: Backup and Recovery +sidebar_position: 7 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +Replication supports availability, but also propagates destructive writes, schema errors, and application corruption to peers. You need backups to recover an earlier state. + +This guide covers choosing a backup mechanism, keeping an independent copy, and testing that you can restore service within your recovery targets. + +## What You Will Learn + +- How to choose a backup mechanism for your storage engine and recovery targets +- How to copy and retain backups independently of the source node +- What `verify_backup` does and does not check +- Which databases can be restored with the server running, and which cannot +- Why restoring a database to undo a code bug is usually the wrong move + +## Prerequisites + +- A cluster with a `super_user` credential, and CLI access to at least one node +- Knowledge of your storage engine per database, from your service boundary inventory in [How Harper Runs in Production](./how-harper-runs-in-production.mdx) +- [Operating Replication](./operating-replication.mdx), because a restore in a replicated cluster is a replication event as much as a storage one +- A non-production database you are willing to destroy + +## Decide before you choose a command + +Record these decisions in your reliability plan: + +1. **Which failures must you survive?** Include node or storage loss, bad deployments, destructive operations, application corruption, regional outages, and compromised credentials. Choose a recovery mechanism for each. +2. **What are the recovery targets?** State acceptable data loss per database and time to restore each affected journey. Use those targets to set backup frequency and test the restore procedure. +3. **What granularity do you need?** Managed backups restore whole databases. To recover selected tables or records, restore into an isolated environment and plan a separate data repair. LMDB snapshot options differ; see [backup operations](/reference/v5/backups/operations#get_backup). +4. **Where must copies live?** Choose independent storage locations, retention periods, and access controls. +5. **Who may authorize and execute a restore?** Name the approver and operator, and record their actions. An in-place restore replaces current data. + +## Know which mechanism you have + +| Mechanism | Engines | What it is | +| ----------------------------------------------------- | --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | +| **Managed backups** | | Incremental, verifiable backups in a server-side repository under `storage.backupPath` | +| **Snapshot download** | | [`get_backup`](/reference/v5/backups/operations#get_backup) streams a snapshot over HTTP, with no server-side artifact | + +RocksDB managed backups and snapshot downloads include all tables, the transaction log, and file-backed blobs unless you request `exclude_blobs`. Restoring that complete backup preserves its `read_audit_log` history. + +LMDB `get_backup` streams the `.mdb` file and does not include external blob files. Table-filtered downloads include the audit store only with `include_audit: true`. If an LMDB database uses file-backed blobs, use a coordinated filesystem or volume backup of the database and every blob root with Harper stopped. An `.mdb` download alone cannot recover those blobs after the source node is lost. + +:::warning +Two limitations to check against your own deployment now rather than later: + +- **Managed backups require RocksDB.** For LMDB databases your options are `get_backup` or volume snapshots. +- **One storage root per database.** Backup is rejected when a database's tables resolve to multiple storage roots. Custom database or table paths are valid if all tables still resolve to one root. Use a separate backup strategy for databases split across roots. + +::: + +## Create and retain managed backups + + + + +```json +{ + "operation": "create_backup", + "database": "data" +} +``` + +Through a running server this returns a `job_id` immediately. Poll [`get_job`](/reference/v5/operations-api/operations#get_job) for the outcome, which includes the new `backup_id`, `size`, and `timestamp`. Treat the job result as the completion signal, not the original response. + + + + +```bash +harper create_backup database=data +harper list_backups database=data +harper purge_backups database=data keep_count=7 +``` + +Every backup operation runs from the CLI under the same name. With the server running the CLI forwards to the server; with it stopped the command operates directly on the files. + +`get_backup` requires a running server. Document how to copy the managed backup repository or take a volume snapshot if the server cannot start. + + + + +Backups of the same database share unchanged RocksDB data files, so the first one copies everything and later ones copy only what changed. Shared files are reference-counted, so deleting a backup removes only files no remaining backup references. + +:::warning +Each backup is a full logical snapshot, but its disk cost depends on file sharing: + +- RocksDB backups share unchanged data files within the same repository. +- The transaction-log snapshot is copied for each backup. Include it when measuring repository growth. +- Blob snapshots use hard links when the source and destination filesystem support them. Otherwise, they copy the files. Use `exclude_blobs: true` only if your recovery procedure can recover those blobs separately. +- Restoring a backup created with `exclude_blobs: true` requires `allow_engine_only: + true`, even into a new database. This explicitly accepts the risk of records pointing at missing or unrelated blob + files. Include separate blob recovery and validation in the drill; do not add the flag just to bypass the safety + check. +- An export that does not preserve sharing materializes those shared or linked files as separate copies. Budget for the full exported size. + +Measure physical repository growth over several backups with your retention policy and filesystem. Do not assume that a full logical snapshot consumes its full logical size, or that all snapshot files are hard-linked. +::: + +Do not use `list_backups` sizes for capacity planning. The `size` and `file_count` fields come from the RocksDB backup engine and exclude the transaction-log and blob snapshots, so each entry undercounts, while the shared files between entries mean summing them overcounts. Measure the repository directory instead. + +## Get a copy off the node + +Keep a recoverable copy outside the source node's failure and administrative boundaries. Separate credentials and retention controls should prevent a compromised node or backup writer from deleting that copy or weakening its retention policy. + +**Managed backups live on the node that created them.** The repository is a local directory, and RocksDB shares files across backup IDs, so **a backup ID is not a self-contained folder.** Two consequences: + +- A disaster-recovery copy has to take the entire per-database repository, `/`, not an individual backup. +- It has to do that while no backup operation is running, or from an atomic filesystem snapshot. A live recursive copy can race `create_backup`, `delete_backup`, or `purge_backups` and produce an unrestorable copy. + +Download a fresh snapshot from the configured node: + +```bash +harper get_backup database=data out=./data-$(date +%Y%m%dT%H%M%S).tar.gz +``` + +Or specify a remote target and write the snapshot to a local file: + +```bash +harper get_backup database=data target=https://operations.node-2.example.com out=./data.tar.gz +``` + +`target=` only changes which node the snapshot is read from. It writes a file and nothing else: it does not restore, import, or clone that node's database onto the local one. Turning the file into a live database is a separate, deliberate restore step, performed with Harper stopped. + +Note also that `get_backup` always streams the current state. It cannot download a historical managed backup, so it is a way to take a fresh off-host copy, not a way to export your retention history. + +Give each exported recovery artifact a unique, immutable identity. Include a manifest with its database, Harper version, backup timestamp, included files, blob roots, and checksums. After upload, read back the destination metadata and confirm the expected identity, size, encryption, versioning, and retention settings where supported. Checksum verification must cover the copy at the destination. + +## Verification is not optional + +```json +{ + "operation": "verify_backup", + "database": "data", + "backup_id": 1, + "verify_checksum": true +} +``` + +`verify_backup` checks the RocksDB file sizes, their checksums when `verify_checksum` is `true`, which is slower, and the framing of the transaction-log snapshot, which is always checked. + +**Blob contents are not verified.** Verification checks that a required blob snapshot is present, but does not check individual blob files for size, checksum, or readability. Validate those separately during restore testing. + +Verification checks the backup's structure. A restore test checks whether it can recover your application. + +## Know your restore constraints before the incident + +An in-place RocksDB restore requires the database to be closed: + +| Database | Online restore, server running | Offline restore, server stopped | +| ---------------------------------------------- | ------------------------------------ | ------------------------------- | +| A user database no loaded component holds open | Yes, restored in place | Yes | +| A user database a loaded component holds open | No, the job ends in `ERROR` | Yes | +| The `system` database | No, rejected before a job is created | Yes | + +Because the restore runs as a background job, a component holding the database open does not fail your request. It fails inside the job, so you find out from `get_job` rather than from the response. Harper does not track which component uses which database, so it cannot selectively stop one. + +:::tip +Safe mode skips user applications and components while keeping the built-in root component, database, Operations API, and HTTP server available: + +```bash +HARPER_SAFE_MODE=1 harper run +``` + +This can release user database handles held by application components so an online restore can proceed. It also lets you inspect or remove a component that prevents normal startup. Keep user traffic and replication isolated during recovery. Safe mode does not remove the offline requirement for restoring `system`. See [safe mode](/reference/v5/configuration/overview#harper_safe_mode). +::: + +Also include these constraints in the restore procedure: + +- **`target_database` requires the server stopped.** Restoring into a separate database rather than overwriting the source is CLI-only with Harper down. The target must not already exist or must be an empty directory. +- **An interrupted restore leaves the database unloadable.** On a crash or power loss mid-restore, Harper marks the database as incompletely restored and refuses to load it on the next start. Recover by rerunning `restore_backup` for the same database and `backup_id`. Do not try to load or hand-repair the directory. + +## Restoring in a replicated cluster + +A restored node may be behind peers that have continued accepting writes. Peers can replay newer writes, including the destructive mutations you intended to undo. Restoring one node does not make its backup authoritative; choose which data to preserve or repair and rehearse the re-entry policy before reconnecting it. + +```mermaid +flowchart TD + Live[Live cluster] --> Artifact[Create immutable recovery artifact] + Artifact --> Retained[Independent retained destination] + Retained --> Test[Restore in disposable isolated environment] + Test --> Validate[Validate storage and application] + Validate --> Policy[Choose data authority and replay policy] + Policy --> Drain[Drain user traffic from recovery node] + Drain --> Isolate[Separately isolate replication] + Isolate --> Restore[Restore and validate recovery node] + Restore --> Peers[Controlled replication re-entry] + Peers --> Traffic[Progressive traffic admission] +``` + +For an in-place recovery: + +1. Take the node out of rotation with the availability flag, per [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). Never restore a node that is serving traffic. +2. **Isolate replication separately.** For example, use a network policy that blocks peer replication in both directions and closes existing peer connections. Keep it effective across restarts, and verify that transactions cannot cross the boundary. Removing a node from the load balancer or setting `"replicated": false` on one operation does not isolate replication. +3. Confirm the data authority and replay policy: which writes to preserve, repair, or reapply, and how peers will converge. +4. Perform the restore, online or offline according to the table above. +5. Verify data expectations locally before any peer sees the node. +6. Reconnect replication under the policy you proved in an isolated multi-node drill. If that policy would reintroduce corruption from a peer, keep the affected peers isolated and reconcile their data first. Verify convergence, then complete the traffic-admission checks before restoring traffic. + +:::danger +**Draining traffic does not stop replication.** A drained node continues exchanging transactions with peers. An offline restore stops that exchange while Harper is down, but configured routes can reconnect as soon as it starts. Keep replication isolated until validation is complete and you are ready to execute the re-entry policy. +::: + +## What backups cannot fix + +If a release changed the meaning of persisted data, restoring the database is usually the wrong response. A restore rolls back every write in the window, including all the correct ones from the same period, so you can violate your recovery point objective in the course of fixing a code bug. + +For that failure class, define forward repair instead: a targeted correction, or a replay from the transaction log, that fixes the affected records and leaves the rest alone. Decide which of your failure classes get restore and which get forward repair while you are calm, and record the decision. See [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx) for how this interacts with release reversal. + +## Test the backup program + +Test the copy you would actually use after losing a node, including its destination controls: + +1. Create a backup and verify it with `verify_checksum: true`. For other mechanisms, use the corresponding integrity checks. +2. Export it using the procedure above. Record the immutable artifact ID, backup timestamp, manifest, and checksums. Read back destination metadata and check that the backup writer cannot weaken retention. +3. Restore the destination copy into a disposable environment with a compatible Harper version. Isolate it from production traffic, replication, and application side effects. +4. Validate the expected databases, tables, record counts, blob contents, and business invariants. Run the critical user journeys against the restored application. +5. Rehearse the failure-to-recovery procedure on a non-production cluster. Measure elapsed time from the simulated failure, including detection, transfer, restore, validation, and traffic admission. Compare recovered writes against the failure time to measure actual data loss. +6. Record the artifact ID, backup time, validation results, operator, approver, and achieved RPO and RTO. Rehearse an offline `system` restore separately. + +Use these results in [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx). If recovery misses a target, improve the procedure or revise the commitment. + +Monitor the age of the newest recovery point that has passed the required destination and recovery checks. A successful backup job alone does not establish a usable recovery point. + +For records requiring long-term preservation, keep a separate export in a documented format that can be read independently of Harper. Operational backups serve disaster recovery; preservation exports also need to remain usable after the original software is retired. Choose cadence, storage provider, and any physical-media process for your own environment. + +## Operational notes + +- **Retention is a policy, not a side effect of disk space.** Use `purge_backups` with an explicit `keep_count` that matches a written retention decision. +- **Backup operations are `super_user` through the server, and filesystem permissions offline.** An operator with shell access on a node can restore without an API credential, so protect the host accordingly. +- **Schedule backups per database, matched to that database's recovery point.** A single cluster-wide cadence overspends on some databases and underspends on the ones that matter. +- **Alert on job failure and recovery-point age.** Track the newest fully verified recovery point and alert before its age exceeds your RPO. +- **Record the storage engine per database in your inventory** and re-check after migrations, since your entire mechanism choice depends on it. + +## Readiness checklist + +- [ ] Failure classes enumerated, with the mechanism that addresses each +- [ ] Database recovery points and journey recovery times recorded using the [objective definitions](./engineering-rpo-rto-and-uptime.mdx#state-objectives-for-each-journey-and-its-data) +- [ ] Storage engine confirmed per database, and the mechanism chosen accordingly +- [ ] No database in scope has tables spanning more than one storage root, or the exclusion is known and accepted +- [ ] Backup cadence matches the stated recovery point per database +- [ ] Repository growth measured across successive backups, including transaction-log copies and blob hard-link behavior +- [ ] Export capacity accounts for full copies where file sharing is not preserved +- [ ] An off-host copy exists across separate failure and administrative boundaries +- [ ] Artifact identities, manifests, checksums, and destination metadata verified +- [ ] Backup writer cannot weaken retention +- [ ] Managed repository copies take the whole `/` directory, quiesced or from an atomic snapshot +- [ ] `verify_backup` runs on a schedule, with `verify_checksum` at least periodically +- [ ] Blob contents understood to be unchecked by `verify_backup`, which confirms only that the blob snapshot is present +- [ ] Restore constraints documented for user databases, component-held databases, and `system` +- [ ] Restore authority named, and restore execution logged +- [ ] Restore procedure isolates the node from replication, not just from user traffic, since the availability flag does not stop peer exchange +- [ ] Safe mode known as the way to restore a database a component holds open, and to reach a node a broken component stopped +- [ ] Destination copy restored in a disposable, isolated environment with a compatible Harper version +- [ ] Inventory, counts, blobs, business invariants, and critical journeys validated +- [ ] Artifact ID, timestamp, validation results, operator, approver, achieved RPO, and achieved RTO recorded +- [ ] Alerts use the newest fully verified recovery point +- [ ] `system` database restore rehearsed offline +- [ ] Forward repair defined for corruption caused by application code +- [ ] Separate preservation export defined where long-term access is required + +## Additional Resources + +- [Backups overview](/reference/v5/backups/overview) for how managed backups work, the full limitation list, and manual restore examples +- [Backup operations](/reference/v5/backups/operations) for every parameter of `create_backup`, `list_backups`, `verify_backup`, `delete_backup`, `purge_backups`, `restore_backup`, and `get_backup` +- [Storage configuration](/reference/v5/configuration/options#storage) for `storage.backupPath`, `storage.path`, and `storage.blobPaths` +- [Jobs](/reference/v5/operations-api/operations#jobs) and [`get_job`](/reference/v5/operations-api/operations#get_job) for tracking long-running backup operations +- [CLI operations](/reference/v5/cli/operations-api-commands) and [remote operations](/reference/v5/cli/overview#remote-operations) for the offline and `target=` forms +- [`read_audit_log`](/reference/v5/operations-api/operations#read_audit_log) for transaction history usable in forward repair +- [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx) for turning the measured numbers into commitments diff --git a/learn/administration/coming-soon.md b/learn/administration/coming-soon.md deleted file mode 100644 index d30962bdc..000000000 --- a/learn/administration/coming-soon.md +++ /dev/null @@ -1 +0,0 @@ -# Coming Soon diff --git a/learn/administration/engineering-rpo-rto-and-uptime.mdx b/learn/administration/engineering-rpo-rto-and-uptime.mdx new file mode 100644 index 000000000..b63add4ea --- /dev/null +++ b/learn/administration/engineering-rpo-rto-and-uptime.mdx @@ -0,0 +1,167 @@ +--- +title: Engineering RPO, RTO, and Uptime +sidebar_position: 8 +--- + +Define recovery time objective (RTO) and availability per critical journey, and recovery point objective (RPO) per database. Map failures to recovery mechanisms and validate the targets with measured results. + +When measured results miss a target, improve the recovery mechanism or revise the target before committing to it. + +## What You Will Learn + +- How to state recovery point, recovery time, and availability so they are testable rather than aspirational +- A scenario-to-mechanism map, with what has to be proven for each to count +- How to convert an availability target into a monthly budget, and what each tier demands of your automation +- How to reconcile stated targets against the numbers measured in earlier guides +- Why one storage setting can silently invalidate your stated recovery point +- Why declaring your degraded modes is as important as declaring your targets + +## Prerequisites + +- Measured numbers from the earlier guides: per-node capacity, convergence time, node re-entry time, release reversal time, and restore time +- [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) for the capacity model +- [Backup and Recovery](./backup-and-recovery.mdx) for the recovery mechanisms +- A named owner who can approve or reject a target, because this exercise produces decisions rather than data + +## State objectives for each journey and its data + +| Term | The question it answers | How to state it so it is testable | +| ------------------------ | ------------------------------------ | -------------------------------------------------------------------------------------------------- | +| Recovery point objective | How much recent data may be lost | Per database, in units of time, measured against your backup timestamp | +| Recovery time objective | How long until service is restored | Per journey, measured from the moment the journey starts failing, not from the moment someone acts | +| Availability target | How much unavailability is permitted | Per journey, as minutes per month, with the measurement boundary named | + +For each target, record the affected journey, when measurement starts, and where you measure it. + +Scope availability and recovery time to each journey, and record the recovery point for every database it depends on. Catalog browsing and order submission can have different targets even when they share a cluster. + +**Include detection in recovery time.** Measure from the start of the failure to verified service restoration. Starting at the alert or the operator's decision leaves out time when users were already affected. + +**Name the measurement boundary.** Availability measured at your CDN edge and availability measured at a Harper node are different quantities, and the difference is exactly the part of the stack you may not control. Whichever you choose, say so in the same sentence as the number. + +### One storage setting can override your recovery point + +Your recovery point objective is a statement about how much acknowledged data you can afford to lose. One configuration option can quietly make that statement false. + +[`storage.writeAsync`](/reference/v5/database/storage-tuning#storagewriteasync) defaults to `false`. Set it to `true` and Harper stops calling `fsync` on commit, so a write returns as soon as it is queued to the operating system's page cache. Throughput on write-heavy workloads improves substantially. The cost is that a power loss or OS crash between the commit and the flush to disk loses the most recent transactions, even though your application was told they succeeded. The database stays structurally consistent; only the newest writes are gone. + +Healthy replication and recent backups do not prove that every acknowledged write has reached durable storage. Include the possible loss of unflushed writes in your RPO. + +Leave `writeAsync` off unless you can accept the loss of unflushed writes, for example because the data can be reproduced from an upstream source or a bulk ingest can be rerun. Asynchronous replication does not guarantee a durable peer copy before the client receives success. If you enable it, record the affected databases and the accepted loss window beside their recovery objectives. + +## Map scenarios to mechanisms + +For each scenario, choose a recovery mechanism and record the evidence that it meets your target. + +| Scenario | Primary mechanism | What must be proven | +| ------------------------------------- | --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | +| Single node loss | Peer capacity plus traffic removal | `(N - F)` capacity under load, measured time to remove traffic, no user-visible errors in transition | +| Node returns after a restart | Incremental catch-up plus an admission gate | Measured convergence time, and an admission gate that does not admit before convergence completes | +| Node replacement or scale-out | Full database synchronization to the new node | Measured full-sync duration at your data volume, and the load it puts on the source node | +| Failure domain loss | Topology spread across independent domains | No shared power or network fault, and surviving capacity inside the budget | +| Region loss | Multi-region topology plus route change | Data authority per region, route propagation time, and declared degraded behavior | +| Bad release, no data change | Release reversal | Measured release reversal time, and the previous artifact still addressable | +| Bad release that changed data meaning | Forward repair, not restore | A written repair or replay procedure, rehearsed on representative data | +| Accidental destructive operation | Restore from backup | Measured restore time and actual data loss, from a real drill | +| Storage loss on one node | Off-host backup copy | A restorable off-host copy, taken by the correct procedure for your mechanism | +| Corruption propagated by replication | Restore the last good backup, plus a replication decision | A rehearsed sequence that includes what happens to peers | +| Replication certificate expiry | Certificate lifecycle management | Expiry dates tracked, with an alert far enough ahead to act | +| Downstream dependency outage | Declared degraded mode | The journey's behavior when the dependency is down, verified rather than assumed | + +Measure node replacement separately from a restart: full synchronization can take much longer than incremental catch-up. Also test how each journey behaves when a downstream dependency is unavailable. + +## Convert availability into a budget + +An availability target is a quantity of unavailability you may spend per month. + +| Monthly target | Approximate maximum unavailability | What the tier demands | +| -------------- | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | +| 99.9% | 43 min 50 sec | A human can be paged, diagnose, and act within some incidents. One untested maintenance window can consume the month | +| 99.95% | 21 min 55 sec | Automated traffic removal and a rehearsed release reversal become necessary rather than nice | +| 99.99% | 4 min 23 sec | Node failure has to be close to transparent. Detection, traffic removal, and validation must be automated, since no human response fits | + +Figures assume a 30.44 day month. Your contractual definition, exclusions, and measurement boundary may differ. + +Use the availability budget to decide how quickly detection, traffic removal, and recovery must happen. Additional nodes provide capacity, but those procedures still need to fit within the budget. + +Define when a depleted error budget should pause discretionary changes, who makes that decision, and what must improve before changes resume. + +## Reconcile targets against measurements + +Fill in both columns and investigate any measurement that misses its target. + +| Commitment | Measured from | Your target | Your measurement | +| --------------------------- | ---------------------------------------------------------------------------------------- | ----------- | ---------------- | +| Surviving capacity | The peak-load drain drill in [Sizing](./sizing-a-harper-cluster.mdx) | | | +| Time to remove traffic | The drain sequence in [Health Checks](./health-checks-and-traffic-admission.mdx) | | | +| Node re-entry time | The same drill, including convergence | | | +| Convergence time under load | The sentinel measurement in [Operating Replication](./operating-replication.mdx) | | | +| Full synchronization time | A node replacement at your data volume | | | +| Release reversal time | The stopped-rollout rehearsal in [Safe Deployments](./safe-deployments-and-rollback.mdx) | | | +| Restore time | The restore drill in [Backup and Recovery](./backup-and-recovery.mdx) | | | +| Actual data loss on restore | The same drill, against the backup timestamp | | | +| Time to detection | The fault injection drill in [Monitoring and Triage](./monitoring-and-triage.mdx) | | | + +Three reconciliations catch most problems: + +- **Backup cadence against recovery point.** If you back up every six hours, your recovery point cannot be one hour, no matter what the document says. +- **Restore time plus detection time against recovery time.** Recovery time includes noticing. A twenty minute restore behind a forty minute detection gap is a one hour recovery. +- **Convergence time against your admission gate.** Keep a returning node out of traffic until its data meets the journey's freshness requirements. + +For each missed target, improve the mechanism, revise the target, or record an accepted gap with an owner and a review date. + +## Declare your degraded modes + +Define which reduced-service behaviors are acceptable during each failure. + +For each critical journey, write down what happens when a dependency is unavailable, when a node is behind on replication, and when the cluster is below its capacity budget. Reads served from slightly stale data may be entirely acceptable for a catalog and entirely unacceptable for a balance check, and the right answer differs by journey rather than by system. + +Name who can invoke each mode and whether the transition is manual or automatic. + +### Prove it + +Two passes, and both are needed. + +**Tabletop the whole scenario table.** For each row, walk through who detects it, what they do, what mechanism carries the recovery, and what evidence confirms success. Rows where the group cannot answer without speculating are the gaps, and finding them costs an hour rather than an outage. + +**Then run one real drill per scenario class**, spread over a quarter rather than attempted in a day. The classes are: node loss, node return, node replacement, release reversal, restore, and dependency failure. Record the date, the measured numbers, what surprised you, and what you changed as a result. + +Use unexpected results to update the runbook, monitoring, or recovery mechanism before repeating the drill. + +## Operational notes + +- **Recovery numbers expire.** Every measurement here is a property of a Harper version, a data volume, a topology, and a set of components. Re-measure after version upgrades and significant data growth, and date every number you record. +- **Uptime and recovery targets belong in the same document as the mechanism.** A target stored separately from its supporting evidence drifts from reality within a quarter. +- **Do not let a single incident rewrite your targets.** Adjust targets from measurement and business need, not from the last thing that went wrong. +- **Exclusions matter as much as the number.** Planned maintenance, third-party dependency failures, and client-side problems are usually excluded from an availability calculation. Whether yours excludes them changes the number substantially, so it belongs in writing. +- **A target you cannot measure is not a target.** If nothing in your monitoring produces the number in your commitment, the first investment is measurement, not more nodes. + +## Readiness checklist + +- [ ] Recovery point stated per database, in time units +- [ ] Recovery time stated per journey, measured from the start of failure and including detection time +- [ ] Availability stated per journey, with the measurement boundary named +- [ ] Exclusions documented +- [ ] Every scenario in the map has a named mechanism +- [ ] Every mechanism has its "what must be proven" satisfied or explicitly outstanding +- [ ] Availability target converted to minutes per month +- [ ] Error budget policy states what changes when the budget runs low +- [ ] Backup cadence reconciled against recovery point +- [ ] `storage.writeAsync` confirmed off, or the databases it is on for recorded alongside their recovery point +- [ ] Restore time plus detection time reconciled against recovery time +- [ ] Convergence time reconciled against the traffic admission gate +- [ ] Every unreconciled gap has an owner and a date +- [ ] Degraded modes declared per journey, with authority to invoke them +- [ ] Tabletop completed and dated +- [ ] At least one real drill per scenario class completed and dated + +## Additional Resources + +- [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) for the capacity invariant behind the surviving-capacity commitment +- [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx) for drain, admission, and failback timing +- [Operating Replication](./operating-replication.mdx) for convergence measurement +- [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx) for release reversal timing +- [Backup and Recovery](./backup-and-recovery.mdx) for restore timing and mechanism limits +- [Monitoring and Triage](./monitoring-and-triage.mdx) for time to detection +- [Production Readiness Checklist](./production-readiness-checklist.mdx) for the gate this feeds +- [Reliability Plan Template](./reliability-plan-template.mdx) for where to record the results diff --git a/learn/administration/health-checks-and-traffic-admission.mdx b/learn/administration/health-checks-and-traffic-admission.mdx new file mode 100644 index 000000000..a2905ce2d --- /dev/null +++ b/learn/administration/health-checks-and-traffic-admission.mdx @@ -0,0 +1,291 @@ +--- +title: Health Checks and Traffic Admission +sidebar_position: 2 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +An HTTP response proves that a Harper node is running. Before sending user traffic to it, also check that its data is current and the application can serve the requested journey. + +This guide explains the checks to use and the order for draining a node, changing it, and returning it to service. + +## What You Will Learn + +- Why liveness, availability, readiness, and journey verification are four signals rather than one, and which layer consumes each +- How to control node availability on Fabric or with `@harperdb/status-check` on a self-managed cluster +- How to write an application readiness route that checks only what the routed journey needs +- The order of operations for draining a node and returning it to service +- What makes a readiness endpoint safe under load, and what makes one dangerous + +## Prerequisites + +- A Harper cluster with at least two nodes, so draining one leaves a service behind ([Fabric](/fabric) or self-managed) +- A `super_user` credential for the Operations API +- A traffic layer you can configure health checks on: a load balancer, a CDN origin group, or a service mesh +- [How Harper Runs in Production](./how-harper-runs-in-production.mdx), and your ports written down + +## Four signals, not one + +| Signal | Where it lives | What it proves | Who consumes it | +| -------------------------- | ---------------------------------------------- | ----------------------------------------------------- | ---------------------------------- | +| Process liveness | A trivial response on the node's HTTP endpoint | Harper is running | Process supervisor, restart policy | +| Node availability flag | `GET /status` on the application URL | An operator or automation says this node should serve | Traffic layer health check | +| Application readiness | A component route on the application URL | This node passes its admission checks | Operator or admission controller | +| Critical-journey synthetic | A real request through the public route | Users are actually being served | Your SLO and alerting | + +The availability flag lets an operator or automation control whether a node receives new traffic. Use it as the routing signal, and use readiness, replication checks, and journey tests to decide when to set it. + +## Install the availability flag + +On **Fabric**, the status component is already installed. It reads the node's `availability` record in `system.hdb_status`. Use the CLI to change that record, as described in [Load Balancing and Node Availability](/fabric/load-balancing#taking-a-node-out-of-rotation). Fabric also writes it during maintenance. Do not install the npm package below on Fabric: it serves the same `/status` path but uses a different table. + +On a **self-managed cluster**, [`@harperdb/status-check`](https://github.com/HarperFast/status-check) adds a `/status` route on the application port. Deploy it with: + +```json +{ + "operation": "deploy_component", + "project": "status-check", + "package": "@harperdb/status-check", + "replicated": false, + "restart": true +} +``` + +Repeat on each node, or declare it in each node's root `harper-config.yaml`: + +```yaml +status-check: + package: '@harperdb/status-check' +``` + +:::note +This entry belongs in the root `harper-config.yaml` under the Harper `rootPath`, typically `~/hdb`. An application's `config.yaml` configures its plugins and replaces the default plugin configuration. See [applications](/reference/v5/components/applications). +::: + +The self-managed component provides these routes: + +- `GET /status` returns `200` when the node is available, `404` when it is not +- `POST /status` marks the node available (authenticated) +- `DELETE /status` marks the node unavailable (authenticated) + +Use the target node's application URL. The examples assume HTTPS is configured on that URL; a default local installation uses `http://localhost:9926`. + + + + +```bash +curl -s -o /dev/null -w '%{http_code}\n' https://my-node.example.com/status + +curl -s -X DELETE https://my-node.example.com/status -u 'admin:password' + +curl -s -X POST https://my-node.example.com/status -u 'admin:password' +``` + + + + +```javascript +const base = 'https://my-node.example.com/status'; +const auth = { Authorization: 'Basic ' + btoa('admin:password') }; + +const res = await fetch(base); +console.log(res.status); + +await fetch(base, { method: 'DELETE', headers: auth }); +await fetch(base, { method: 'POST', headers: auth }); +``` + + + + +Point the traffic layer's health check at `GET /status` on each node's application URL. A `404` should stop new traffic reaching that node. Existing connections may remain open, so measure the drain before restarting it. + +:::note +Both implementations keep the flag node-local with `replicate: false`. Preserve that setting if you fork a component or store the flag elsewhere. A replicated flag could take every peer out of rotation when you drain one node. +::: + +### A note on `set_status` + +The Operations API also offers [`set_status`, `get_status`, and `clear_status`](/reference/v5/operations-api/operations#set_status--get_status--clear_status) for application-defined status values, with types for primary, maintenance, and availability. + +On Fabric, `set_status` with `id=availability` controls the record its status component reads. The self-managed npm package uses its own `data.hdb_status` table, so `set_status` does not control that package. Use its `POST /status` and `DELETE /status` routes instead. Neither flag measures application health by itself. + +## Write an application readiness route + +Liveness and the availability flag both answer questions about the node. Readiness answers a question about the journey: if traffic arrives for this route right now, will it succeed? + +Use readiness to gate admission, not as the load balancer's direct routing signal. Keep node-local failures separate from shared dependency failures: taking every node out during a shared outage removes capacity without fixing the dependency. Report shared failures as degradation and use journey tests to measure their user impact. + +Add a node-local probe table to your application's schema. Keep the `graphqlSchema` and `rest` plugins enabled so the authenticated seeding request below can reach the table: + +```graphql +type ReadinessProbe @table(replicate: false) @export { + id: ID @primaryKey + description: String +} +``` + +Add a resource to your application's `resources.js`: + +```javascript +import { Resource, tables } from 'harper'; + +let cachedChecks; +let cacheUntil = 0; +let inFlight; + +async function checkReadiness() { + const checks = {}; + + // Use a fresh server-side context for this fixed, non-sensitive read. + try { + const sentinel = await tables.ReadinessProbe.get('readiness-probe-sentinel', {}); + checks.data = sentinel ? 'ok' : 'failed'; + } catch (error) { + checks.data = 'failed'; + } + + // A shared dependency outage is degradation, not a reason to drain every node. + try { + const res = await fetch('https://pricing.internal.example.com/health', { + signal: AbortSignal.timeout(500), + }); + checks.pricing = res.ok ? 'ok' : 'degraded'; + } catch (error) { + checks.pricing = 'degraded'; + } + return checks; +} + +function getChecks() { + if (Date.now() < cacheUntil) return Promise.resolve(cachedChecks); + return (inFlight ??= checkReadiness() + .then((checks) => { + cachedChecks = checks; + cacheUntil = Date.now() + 2000; + return checks; + }) + .finally(() => { + inFlight = undefined; + })); +} + +export class Readiness extends Resource { + static async get() { + const checks = await getChecks(); + const ready = checks.data === 'ok'; + return new Response(JSON.stringify({ ready, checks, version: process.env.APP_VERSION }), { + status: ready ? 200 : 503, + headers: { 'Content-Type': 'application/json' }, + }); + } +} +``` + +The `jsResource` plugin loads the exported class. Keep it enabled if your application has an explicit `config.yaml`. At the default application mount, the route is `/Readiness`; adjust the probe URL if you configured `urlPath` or `host` routing. See [resource routing](/reference/v5/resources/overview#exporting-resources-as-endpoints) for the path rules. + +:::note +Because the handler is a `static` method, it replaces Harper's built-in dispatch along with its authorization check, so `GET /Readiness` is unauthenticated. Keep the response to check names and outcomes, never connection strings, credentials, or internal hostnames. The shared in-flight check and two-second cache bound repeated reads and outbound requests per worker; also restrict probe access or rate-limit it at your traffic boundary. + +If you rewrite this as an instance `get()` instead, authorization comes back and defaults to `super_user` only. Configure explicit authorization for your probe caller rather than assuming that the route remains public. +::: + +Create the sentinel record on each node before enabling the probe. The example assumes `ReadinessProbe` is exported at the server root: + +```bash +curl -s -X PUT https://my-node.example.com/ReadinessProbe/readiness-probe-sentinel \ + -H 'Content-Type: application/json' \ + -u 'admin:password' \ + -d '{"description":"readiness probe sentinel - do not delete"}' +``` + +Keep the probe table node-local and exclude it from application cleanup jobs. A shared pricing failure reports `degraded` but does not make every node unavailable. A `200` therefore establishes node-local readiness, not that every journey dependency is healthy; check the response and the journey synthetic as well. + +Bound downstream checks with a timeout. Set `APP_VERSION` in your deployment so the response identifies the component build answering during a rollout. + +## Drain a node and bring it back + +Keep the node unavailable until all return checks pass. On Fabric, set `availability` through the CLI as linked above; the `DELETE /status` and `POST /status` calls below are the self-managed package's controls. + +```mermaid +flowchart TD + Serving --> Unavailable[Declare unavailable] + Unavailable --> Drained[Measured user traffic is zero] + Drained --> Change[Change or restart] + Change --> Live{Process live?} + Live -->|Yes| Synced{Replication converged?} + Synced -->|Yes| Ready{Readiness and journey test pass?} + Ready -->|Yes| Available[Declare available] + Available --> Restored[Restore traffic progressively] + Live -->|No| Hold[Hold unavailable and investigate] + Synced -->|No| Hold + Ready -->|No| Hold + Hold --> Live +``` + + + + +1. **Check peer capacity.** Confirm that the remaining nodes can carry the expected traffic within their capacity budget. If they cannot, postpone the change or add capacity. See [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx). +2. **Declare it unavailable.** `DELETE /status` on the target node. +3. **Verify traffic actually stopped.** Watch request volume on the target fall to zero and rise on its peers. Do not trust the configured weight; check the measured request count from [analytics](/reference/v5/analytics/overview) or your traffic layer's own metrics. Health check intervals, DNS TTLs, and client-side connection reuse all add delay here, and the delay is yours to measure. +4. **Watch the peers absorb it.** If they are going outside budget despite step 1, abort: `POST /status` to put this node back in rotation, and re-plan the change for a lower-traffic window or with added capacity. +5. **Do the work.** Restart, upgrade, reconfigure, or investigate. + + + + +1. **Confirm the databases are current.** `cluster_status` should show the expected peer sockets connected for every database this node serves, and convergence should be complete rather than in progress. +2. **Confirm readiness passes locally.** Call your `/Readiness` route directly against the node, bypassing the traffic layer. +3. **Run the journey synthetic against the node directly.** A real read, or a safe write, through the same path a user would take. +4. **Declare it available.** `POST /status`. +5. **Hold before restoring full weight.** Give it a stability window at partial traffic and watch error rate and latency against its peers before treating it as fully back. + + + + +Set the initial traffic weight before declaring the node available, where your traffic layer supports it. Increase it only while error rate and latency remain acceptable. If a return check fails, leave the node unavailable and investigate. + +## Readiness hygiene + +- **Keep the response cheap and bounded.** A readiness check that performs a broad scan or writes data will amplify an incident, because it runs at probe frequency across every node at exactly the moment the system is already struggling. +- **Never make it the only routing signal.** Liveness plus availability plus readiness, consumed at the right layers. +- **Probe from more than one location** where your traffic layer supports it. A single probe point cannot distinguish a network path problem from a node problem. +- **Make it observable.** Record response code, latency, the reason for a failure, and the node and component version. A readiness check whose failures you cannot explain after the fact is a check you will end up ignoring. +- **Version the contract.** If you change what readiness means, that is a change to the traffic admission policy, and it deserves the same care as a code release. + +### Prove it + +On a non-production cluster, restart one node under representative read and write load, using the full drain and return sequence above. Record three numbers: how long from `DELETE /status` until measured traffic on that node reaches zero, whether any user-visible errors occurred during the transition, and how long from process start until the node legitimately passed all three return gates. + +Record the third number as node re-entry time, including convergence and application validation. + +## Operational notes + +- **A returning node is not the same as a new node.** A node whose databases have never synchronized downloads them in full. A node that was offline and comes back catches up on the transactions it missed. Both need to finish before traffic arrives, but they take very different amounts of time, so do not budget for the first when you are planning a routine restart. +- **Set unavailable before recovery work, not after.** Any operation that touches data on a node, including a restore, should happen with the node out of rotation. +- **Configuration changes need a restart to take effect**, so a node that has been reconfigured but not restarted is running the old configuration while reporting the new one. Sequence the restart into the same maintenance window. +- **Fabric routes by its status component.** It does not automatically consume your readiness route. For application-driven availability, see [Taking an Unhealthy Node Out of Rotation](../developers/taking-a-node-out-of-rotation.mdx). + +## Readiness checklist + +- [ ] Fabric's existing status component retained, or `@harperdb/status-check` configured on each self-managed node +- [ ] Traffic layer health check points at `GET /status` on each node's application URL +- [ ] Availability flag storage still declares `replicate: false`, if you forked the component or persist it yourself +- [ ] An application readiness route exists, scoped to one journey's dependencies +- [ ] Every downstream call in the readiness route has a timeout +- [ ] Readiness response includes the component version +- [ ] A critical-journey synthetic runs against the public route, separately from the liveness probe +- [ ] Drain sequence documented, with measured time-to-zero-traffic +- [ ] Return sequence documented, with a stability window before full weight +- [ ] Measured node re-entry time recorded, including convergence + +## Additional Resources + +- [`@harperdb/status-check`](https://github.com/HarperFast/status-check) component source and options +- [Operations API operations](/reference/v5/operations-api/operations) for `cluster_status`, `system_information`, and the status operations +- [Components overview](/reference/v5/components/overview) for the full list of Harper-maintained components, including the Prometheus exporter +- [Analytics overview](/reference/v5/analytics/overview) for per-node request and latency data +- [Harper Applications in Depth](../developers/harper-applications-in-depth.mdx) for custom resources and the `jsResource` plugin +- [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) for whether your peers can absorb a drained node diff --git a/learn/administration/how-harper-runs-in-production.mdx b/learn/administration/how-harper-runs-in-production.mdx new file mode 100644 index 000000000..38554ae8b --- /dev/null +++ b/learn/administration/how-harper-runs-in-production.mdx @@ -0,0 +1,187 @@ +--- +title: How Harper Runs in Production +sidebar_position: 1 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +Harper runs your application code, HTTP server, local database, and replication client on the same node. To operate it in production, you need to know how much traffic each node can handle, which data it serves, and how to move traffic when it needs maintenance. + +This guide covers what a single Harper node actually contains, and how to understand your deployment before you design anything around it. + +## What You Will Learn + +- What a Harper node contains, and which port carries which kind of traffic +- Why the node is your unit of failure and your unit of scale, and what that adds/removes from your operating burden +- The three ways a Harper runbook differs from a two-tier runbook +- How to inventory your service boundary with the Operations API, so later guides have something concrete to work from + +## Prerequisites + +- A running Harper instance, either a [Harper Fabric](/fabric) cluster or a [local installation](../getting-started/install-and-connect-harper.mdx) +- A `super_user` credential for the Operations API +- An application deployed to it ([Create your First Application](../getting-started/create-your-first-application.mdx)) + +## What one node contains + +A Harper node is a single process running your components, an HTTP server, a local storage engine, and peer replication. There is no network hop between your application code and the data it reads, no separate cache tier to keep coherent, and no connection pool to tune between tiers. + +Record the ports used by your deployment: + +| Port | Serves | Who should reach it | +| ---------------------------------- | --------------------------------------------------------------------------- | --------------------------------- | +| `9926` | Application traffic: REST, WebSocket, MQTT-over-WebSocket, component routes | Your traffic layer and your users | +| `9925` | The [Operations API](/reference/v5/operations-api/overview) | Operators and your pipeline only | +| Configured secure replication port | Secure peer [replication](/reference/v5/replication/overview) | Other nodes in the cluster only | + +`9925` and `9926` are the default HTTP ports. HTTPS uses `operationsApi.network.securePort` and `http.securePort`, or a TLS-terminating proxy. Use the node's configured URL in the examples throughout this track. + +Resolve the effective replication port from `replication.securePort`, falling back to `operationsApi.network.securePort` when it is unset. An upgrade retains the existing configuration, so do not assume that a newly installed node and an upgraded one use the same port. + +Read the configuration from each node and record its HTTP, HTTPS, and replication ports: + +```json +{ "operation": "get_configuration" } +``` + +:::tip +Restrict the Operations API to operators and deployment automation. It can deploy components, read logs, and manage configuration. See [security overview](/reference/v5/security/overview). +::: + +## A node is the unit of both failure and scale + +Losing a node removes both its request-handling capacity and access to its local data. Plan for the remaining nodes to serve the application while it is unavailable. + +Keeping application code and data together removes some of the work associated with separate tiers: + +- No network latency on data access, and no tail latency from a saturated connection pool between tiers +- No cache invalidation problem between an application cache and a database of record (because they are the same thing) +- No separate application-to-database connection to monitor +- Application and data access measured together in your capacity tests + +Choose how many simultaneous node failures your service must tolerate, then check whether the survivors can carry peak traffic. [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx) walks through that calculation. + +Adding a node also moves data. Allow time for the databases it serves to synchronize before sending it user traffic. + +## Three differences that change your runbook + +### A process that is up is not a node that should take traffic + +A Harper process will answer a TCP connection and return an HTTP response before it is a good place to send a user request. A new or replacement node has to synchronize the databases it serves before its answers are correct. A returning node has to catch up on transactions it missed. + +Keep a starting node out of traffic until its data and application checks pass. [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx) describes the checks and the return sequence. + +### Replication is peer-to-peer and scoped, so verify your own scope + +Harper peers exchange data over WebSockets with mTLS on the secure replication port and discover each other through configured routes. There is no primary. Data mutations and transactions replicate; some things do not, and the scope is configurable per database and per table. + +Read the live replication state on each node: + +```json +{ + "operation": "cluster_status" +} +``` + +The response lists each peer connection and, within it, one socket per database per peer. That tells you which databases are actually flowing, which is the question that matters during an incident. What is in scope, what is deliberately out of it, and how to prove convergence rather than just connection are covered in [Operating Replication](./operating-replication.mdx). + +### Reversal is a redeploy, not an infrastructure event + +Your application ships as a component, deployed with [`deploy_component`](/reference/v5/operations-api/operations#deploy_component) from an immutable reference. Rolling back means deploying the previous immutable reference. There is no image to rebuild, no instance to replace, and no cluster to rebuild to undo a bad release. + +Immutable component references make code rollback repeatable. Configuration rollback and data recovery need separate procedures because they change different parts of the service. See [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx). + +## Inventory your service boundary + +Collect the following information from each node and keep it with your runbooks. + +The examples use `https://operations.my-node.example.com/` for a configured Operations API endpoint. Replace it with the node's own Operations API URL; a default local installation uses `http://localhost:9925/`. + + + + +```bash +curl -s -X POST https://operations.my-node.example.com/ \ + -H 'Content-Type: application/json' \ + -u 'admin:password' \ + -d '{"operation":"system_information","attributes":["system","cpu","memory","disk","threads"]}' +``` + + + + +```javascript +await fetch('https://operations.my-node.example.com/', { + method: 'POST', + headers: { + 'Content-Type': 'application/json', + 'Authorization': 'Basic ' + btoa('admin:password'), + }, + body: JSON.stringify({ + operation: 'system_information', + attributes: ['system', 'cpu', 'memory', 'disk', 'threads'], + }), +}); +``` + + + + +:::warning +The `attributes` array silently drops names it does not recognize, so a typo returns a smaller response rather than an error. The valid values are `system`, `time`, `cpu`, `memory`, `disk`, `network`, `harperdb_processes`, `table_size`, `metrics`, and `threads`. If a response is missing a section you asked for, check the spelling before you check the node. +::: + +Then: + +- [`get_components`](/reference/v5/components/applications#get_components) for what is deployed, including which extensions are present +- [`list_deployments`](/reference/v5/operations-api/operations#list_deployments) for what changed and when +- [`registration_info`](/reference/v5/operations-api/operations#registration_info) for the Harper version, which `system_information` does not report (it returns the node and npm versions, not Harper's) +- `cluster_status` for the peers and databases actually connected + +Fill in a boundary record you can keep: + +| Item | Value for your deployment | +| ----------------------- | ----------------------------------------------------- | +| Harper version | From `registration_info` | +| Node count and roles | From `cluster_status` and your topology intent | +| Ports in use | From `get_configuration` | +| Databases | Which exist, and which replicate | +| Storage engine | Per database, since it constrains your backup options | +| Components deployed | From `get_components`, with the version of each | +| Critical journeys | The user-facing paths that must work, named | +| Downstream dependencies | Anything Harper calls that can fail independently | + +Include application journeys and downstream dependencies in your monitoring. Node health alone will not reveal incorrect responses or failures in services the application calls. + +### Prove it + +On a non-production cluster, stop one node. Confirm that peers keep serving, note when `cluster_status` shows the change, and check whether the traffic layer stops routing to the node. Compare the result with your inventory and topology plan. + +## Operational notes + +- **Fabric and self-managed differ in what you own, not in how Harper behaves.** On [Fabric](/fabric), cluster creation, certificates, and the metrics pipeline are managed for you. The failure unit, the replication model, and the admission problem are identical. +- **Version parity across nodes is an operating requirement, not a nicety.** Mixed versions in a cluster change replication and deployment behavior. Record the version per node in your boundary inventory and alert on drift. +- **Configuration changes need the restart type required by the setting.** Some apply on worker restart; process-level settings need a full process restart. Track pending changes and follow the [configuration rollout procedure](./safe-deployments-and-rollback.mdx#configuration-changes-are-deployments-too). +- **`get_status` reports a `restartRequired` flag, but it tracks component and code restarts rather than configuration changes.** Do not rely on it to tell you a configuration change is still pending. + +## Readiness checklist + +- [ ] Ports confirmed from `get_configuration`, not assumed from documentation +- [ ] Operations API on `9925` is not reachable from the public internet +- [ ] Harper version recorded per node, with an alert on drift +- [ ] Databases listed, with replication scope confirmed via `cluster_status` +- [ ] Storage engine recorded per database +- [ ] Components and versions recorded from `get_components` +- [ ] Critical user journeys named and written down +- [ ] Downstream dependencies named, with their own failure behavior understood + +## Additional Resources + +- [HTTP server reference](/reference/v5/http/overview) for the application port and server architecture +- [Operations API overview](/reference/v5/operations-api/overview) and the [full operation list](/reference/v5/operations-api/operations) +- [Replication overview](/reference/v5/replication/overview) for the peer model, mTLS, routes, and scope +- [Components overview](/reference/v5/components/overview) for the component model and the available extensions +- [Database overview](/reference/v5/database/overview) for storage engines and transaction boundaries +- [Configuration overview](/reference/v5/configuration/overview) for `harper-config.yaml` and restart requirements +- [Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx) for how application code reaches these nodes diff --git a/learn/administration/monitoring-and-triage.mdx b/learn/administration/monitoring-and-triage.mdx new file mode 100644 index 000000000..464881984 --- /dev/null +++ b/learn/administration/monitoring-and-triage.mdx @@ -0,0 +1,254 @@ +--- +title: Monitoring and Triage +sidebar_position: 5 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +Monitor the parts of the service you can investigate or change: the public route, individual Harper nodes, replication, downstream dependencies, and recent deployments. + +This guide identifies useful signals at each boundary and gives you a procedure for narrowing an incident. + +## What You Will Learn + +- The layers a Harper symptom can live in, and the minimum signal for each +- Which Harper metrics are worth an alert, by name, and which are only worth a dashboard +- How to get metrics out of Harper, on Fabric and self-managed +- How to preload an APM or tracing agent so it can instrument your application +- How to write log entries that are still useful during an incident +- A triage sequence that narrows a Harper incident in about five minutes + +## Prerequisites + +- A cluster with your application deployed and taking traffic +- A `super_user` credential for the Operations API +- Somewhere to send metrics: [Grafana](/fabric/grafana-integration) on Fabric, or a Prometheus-compatible system for self-managed +- [Operating Replication](./operating-replication.mdx), since replication signals are half of what you will watch + +## Monitor at the boundaries you operate + +| Layer | Minimum signals | Where they come from | +| ------------------ | ------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------- | +| User and edge | Success, latency, correctness, broken out by geography and cohort | Your synthetic journey, real-user telemetry, CDN and traffic layer logs | +| HTTP and component | Request rate, error rate, latency percentiles, active component version, availability flag | `success` and `duration` metrics, `GET /status`, your readiness route | +| Node resources | CPU, memory, worker utilization, task queue latency, disk, restarts | `system_information`, `utilization`, `main-thread-utilization`, host or container metrics | +| Replication | Peer connections, per-database sockets, latency, convergence lag | `cluster_status`, `replication-latency`, `system.hdb_nodes` | +| Data | Read and write rate, transaction commit time, queue depth, freshness | `transaction-commit-time`, write and read transaction queue depth, your own sentinel | +| Storage | Database size, volume free space, table growth | `database-size`, `storage-volume`, `table-size` | +| Change | Component and config version, cohort, operator, start and end, outcome | `list_deployments`, `get_deployment`, `get_components`, `get_configuration` read-back | +| Recovery | Backup age, job state, verification result, last restore drill | `list_backups`, `verify_backup`, `get_job` | + +Show deployments and configuration changes alongside request errors and latency. Their timing helps you identify whether a release affected one node, one version, or the whole cluster. + +## The signals worth an alert + +Harper records a large standard metric set automatically. Most of it belongs on a dashboard. This much belongs on a pager: + +| Alert on | Metric or source | Why this one | +| -------------------------------------- | ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | +| Journey success rate below SLO | `success`, by resource path and method | The only signal that maps directly to user impact | +| Journey latency at your SLO percentile | `duration`, by resource path and method | Alert on the percentile in your SLO, never on the mean | +| Worker saturation | `utilization`, per worker thread | Percentage of time the worker was processing requests. This is your real headroom signal | +| Event loop backpressure | `main-thread-utilization`, the `taskQueueLatency` attribute | Rises before throughput drops, so it is an early warning rather than a postmortem input | +| Replication convergence lag | `replication-latency`, source timeline, and sentinel checks | Connection health alone does not establish that a node's data is current | +| Write commit time | `transaction-commit-time` | Storage-level degradation shows here before it shows in request latency | +| Write queue depth growing | `write-transaction-queue-depth` | A growing queue means the node is accepting work faster than it can commit it | +| Storage headroom | `storage-volume` free, and `database-size` | Disk exhaustion is an outage with no graceful degradation | +| Missing peer | `cluster_status` | Covered in [Operating Replication](./operating-replication.mdx) | +| Verified recovery point older than RPO | Backup manifest and destination/restore validation records | A successful job alone does not establish that the copy can restore service | + +Alert on the latency percentile in your SLO. Set saturation thresholds from your measurements in [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx), where you determine the load at which latency becomes unacceptable. + +## Get the metrics out + +Harper stores raw analytics in `hdb_raw_analytics` and aggregates them into `hdb_analytics`. Query these tables or export metrics to your monitoring system. + + + + +Use the [Grafana integration](/fabric/grafana-integration). It ships dashboards over Harper's analytics without you building a pipeline, which is the fastest path to the alert list above. + + + + +Deploy [`@harperdb/prometheus-exporter`](https://github.com/HarperFast/prometheus-exporter) and scrape it into whatever you already run: + +```json +{ + "operation": "deploy_component", + "project": "prometheus-exporter", + "package": "@harperdb/prometheus-exporter", + "restart": "rolling" +} +``` + +Once deployed, the scrape endpoint is `//metrics` on the application port, so the call above exposes it at `/prometheus-exporter/metrics`. The route is authorized, so give your scraper a credential rather than expecting it to be open. + +:::warning +**The exporter does not cover every metric in the alert table above.** It translates the request and replication metrics, including `success`, `duration`, and `replication-latency`, but it does not currently export `main-thread-utilization` and its `taskQueueLatency` attribute, `transaction-commit-time`, `write-transaction-queue-depth`, `database-size`, `storage-volume`, or `table-size`. The `utilization` it does expose is a thread utilization figure drawn from `system_information`, which is not the same as the analytics `utilization` metric. + +For the saturation, commit-time, and storage rows you will need to query `hdb_analytics` directly or collect `system_information` on your own schedule. Check what your version actually exports before you build a dashboard on the assumption that everything above arrives in Prometheus. +::: + + + + +To find out what is actually available on your version rather than guessing from documentation: + +```json +{ + "operation": "list_metrics", + "metric_types": ["builtin", "custom"] +} +``` + +Then [`describe_metric`](/reference/v5/analytics/operations#describe_metric) for the shape of any one of them. + +Your application can add its own metrics with [`server.recordAnalytics()`](/reference/v5/http/api#serverrecordanalyticsvalue-metric-path-method-type), which is how you get business-level signals such as checkout completion onto the same dashboard as node saturation. That correlation is usually what tells you whether a technical symptom matters. + +:::tip +Check `analytics.replicate` in the [analytics configuration](/reference/v5/analytics/overview#analytics-configuration) before building dashboards. Whether metrics stay node-local or replicate changes what a cluster-wide query means, and it is easier to decide that deliberately than to discover it after building panels on the wrong assumption. +::: + +## Run an APM or tracing agent + +Harper's own metrics tell you how the node is behaving. They do not give you distributed traces across your application code and its downstream calls, which is what you want when a journey is slow and you need to know where the time went. That comes from an instrumentation agent, and an agent has to load before the code it instruments. + +Two configuration keys put a module on each worker thread's startup, ahead of Harper's own modules and yours: + +```yaml +threads: + preloadRequire: dd-trace/init # the entry that calls init() + preload: dd-trace/register.js # ESM loader hooks for automatic instrumentation +``` + +`threads.preload` loads a module via Node's `--import`, which is how an agent installs the loader hooks that let it instrument modules imported later. `threads.preloadRequire` uses `--require`, which runs the module body and is typically how an agent's initialization entry actually starts it. Both are documented under [threads configuration](/reference/v5/configuration/options#threads). + +Agent entry points vary: some install loader hooks and start the agent together; others need separate `preload` and `preloadRequire` entries. Loading hooks without starting the agent can produce no-op spans and no exports. + +Both keys apply only to worker threads and require Node.js. Bare specifiers resolve against installed components' `node_modules`, so the agent can be a component dependency. + +Whatever agent you use, follow its own worker-thread documentation, and finish by confirming spans actually arrive at your collector rather than assuming the configuration took. + +## Logs you can use during an incident + +Harper's `logger` global takes a message plus a context object from component code. Call this helper after a successful submission, passing the request context and measured duration in milliseconds: + +```javascript +import { server } from 'harper'; + +function logOrderSubmitted(context, elapsed) { + logger.info('order submitted', { + component: 'orders-api', + version: process.env.APP_VERSION, + node: server.hostname, + requestId: context.requestId, + operation: 'submit', + durationMs: elapsed, + outcome: 'ok', + }); +} +``` + +Include the component and its version, the node, a request or trace identifier, the operation, its duration, the outcome, and a safe error class. That field set is what lets you answer "was this only the new version" and "was this only one node" without guessing, and those are the first two questions in almost every incident. + +Three details in that example need care: + +- **The node name comes from `server.hostname`**, not an environment variable. Harper does not set a node-name variable in the process environment, and `server.hostname` is the same identity analytics uses, so it is what correlates with your metrics. +- **`context.requestId` is only populated when `http.logging.id` is enabled**, and HTTP request logging is off by default. Without it the field is `undefined` and you silently lose your correlation id. Either enable it in [logging configuration](/reference/v5/logging/configuration) or generate an id in your own code. +- **`APP_VERSION` is yours to inject.** Harper does not provide it. Set it in your deployment so the log line can name the build. + +:::warning +Harper's logger is built on Node's `Console`, so this renders as a formatted text line, not JSON. The context object is inspected into the message rather than emitted as separate fields: + +```text +[main/3] [info]: order submitted { component: 'orders-api', version: '2.4.1', ... } +``` + +That is fine for reading during an incident, but a log pipeline cannot key on `component` or `outcome` without parsing the line. If you need queryable fields, serialize the context yourself and log a single JSON string. +::: + +Levels run `trace`, `debug`, `info`, `warn`, `error`, `fatal`, and `notify`. The default is `warn`, which drops the example's `info` message. Set [`logging.external.level`](/reference/v5/logging/configuration#loggingexternal) to `info` to retain it, and check the resulting volume. `notify` is always logged. + +`console.log` output does not reach the log files unless `logging.console` is enabled, so unstructured console output is not a production record. Centralize logs off the node, because the node you most need logs from is the one you are about to restart. + +Read logs through the API when you need them from a specific node: + +```json +{ + "operation": "read_log", + "limit": 200, + "level": "error" +} +``` + +## The five-minute triage sequence + +Use these checks to locate the affected layer, then contain the failure before investigating its cause. + +1. **Confirm and bound the impact.** What journey, starting exactly when, in which geography or cohort, and which component version is live. Without a start time you cannot correlate anything. + +2. **Compare the public route against a direct node call.** Request the same journey through your traffic layer, then directly against each node on `9926`. If direct calls succeed and the public route fails, you are looking at the traffic layer or the availability flag, not at Harper. + +3. **Check admission state on every node.** `GET /status` on `9926` and your readiness route. A node advertising unavailable is a node deliberately or accidentally out of rotation, and finding that here saves a lot of time. + +4. **Compare a suspect node against a healthy peer.** Same call, both nodes, then diff: + + ```json + { + "operation": "system_information", + "attributes": ["cpu", "memory", "threads", "harperdb_processes"] + } + ``` + + Differences between peers are more informative than absolute values, because they tell you whether this is one node or the whole cluster. + +5. **Check replication.** Inspect the expected peer/database sockets in `cluster_status`, sustained latency, and `lastCommitConfirmed` progress while writes continue. Compare `lastReceivedRemoteTime` with the expected source timeline and verify freshness with a sentinel or journey test. Timestamp differences alone do not prove convergence. + +6. **Correlate with the last change.** `list_deployments` for what shipped and when, and `get_configuration` read back against what you believe is configured. Remember that a configuration change applied without a restart leaves a node running something other than its stated configuration. + +7. **Contain with the smallest reversible action.** Stop a ramp, drain one node, deactivate a feature, restore prior traffic weights, or isolate suspected data. Record the decision, who made it, and the next decision deadline. + +If required telemetry is missing, investigate the gap before declaring the service healthy. Record containment actions so the next operator can see what changed. + +### Prove it + +Pick a fault and inject it on a non-production cluster, then time yourself. Good candidates: saturate one node's workers, block the replication port on one peer, make a downstream dependency return errors, or deploy a component that fails on one route. + +Record time to alert, time to identify the affected layer, and any misleading or missing signals. Update the procedure before repeating the drill. + +## Operational notes + +- **Node-level dashboards, not cluster averages.** A cluster average conceals the single node doing twice the work, which is the most common cause of a latency complaint that looks like nothing on a dashboard. +- **Watch measured request distribution, not configured weights.** Per-node request counts are the ground truth. Sticky sessions, DNS caching, and connection reuse all skew actual distribution away from intent. +- **Annotate deployments onto your graphs.** If your monitoring supports annotations, feed `list_deployments` into them. This single change resolves more incidents faster than any additional metric. +- **`read_audit_log` needs transaction logging enabled** and is a heavier tool for reconstructing what changed in a table. Know before an incident whether you have it on, because turning it on afterwards does not help. +- **Protect your telemetry surfaces.** `read_log` and `get_components` can expose configuration and source detail, so they are `super_user` operations for a reason. Restrict and log their use. + +## Readiness checklist + +- [ ] A dashboard exists that can place a symptom at edge, traffic layer, node, replication, data, or change +- [ ] Alerts on journey success and latency at the SLO percentile, not the mean +- [ ] Alerts on worker `utilization` and `taskQueueLatency`, thresholds derived from measured per-node capacity +- [ ] Alerts on replication convergence lag and missing peers +- [ ] Alerts on storage headroom and the newest fully verified recovery point's age +- [ ] Metrics exported off the node, via Grafana on Fabric or the Prometheus exporter self-managed +- [ ] If you run an APM agent, both `threads.preload` and `threads.preloadRequire` set as that agent requires, with spans confirmed at the collector +- [ ] `analytics.replicate` setting known and deliberate +- [ ] Structured logging includes component, version, node, request id, operation, duration, and outcome +- [ ] Logs centralized off the node +- [ ] Deployments annotated onto dashboards +- [ ] Triage sequence written down where on-call can find it +- [ ] Fault injection drill completed, with time-to-detection recorded + +## Additional Resources + +- [Analytics overview](/reference/v5/analytics/overview) for the full standard metric catalog and configuration options +- [Analytics operations](/reference/v5/analytics/operations) for `list_metrics` and `describe_metric` +- [Grafana integration](/fabric/grafana-integration) for Fabric dashboards +- [`@harperdb/prometheus-exporter`](https://github.com/HarperFast/prometheus-exporter) for self-managed metric scraping +- [Logging overview](/reference/v5/logging/overview), [configuration](/reference/v5/logging/configuration), and [operations](/reference/v5/logging/operations) +- [HTTP API reference](/reference/v5/http/api) for `server.recordAnalytics()` +- [Fabric logging](/fabric/logging) for log access on Fabric +- [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx) for the gates these signals feed diff --git a/learn/administration/operating-replication.mdx b/learn/administration/operating-replication.mdx new file mode 100644 index 000000000..fb4fd33bd --- /dev/null +++ b/learn/administration/operating-replication.mdx @@ -0,0 +1,254 @@ +--- +title: Operating Replication +sidebar_position: 4 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +Harper peers exchange data over WebSockets and catch up after interruptions. This guide explains how to monitor replication and verify convergence in production. For configuration, see the [replication reference](/reference/v5/replication/overview). + +A connected socket proves that peers can communicate. Before admitting user traffic, also verify that the data needed by the application has arrived. + +## What You Will Learn + +- How to determine what your cluster actually replicates, rather than assuming a default +- What moves between peers automatically, what does not, and which of those will surprise you +- How to read `cluster_status` as an operator, including which timing field actually indicates a node is behind +- How to prove convergence with a sentinel rather than inferring it from connection state +- Which application behaviors replication cannot make safe, and what to do about them instead + +## Prerequisites + +- A cluster of at least three nodes, so you can interrupt one peer and still observe the others +- A `super_user` credential for the Operations API +- [How Harper Runs in Production](./how-harper-runs-in-production.mdx) and your service boundary inventory +- Familiarity with how your cluster was joined, either through `harper-config.yaml` routes or the [clustering operations](/reference/v5/replication/clustering) + +## Know your own scope + +By default Harper replicates all data in all databases. Scope can be narrowed two ways: per database in configuration, and per table in the schema. + +```yaml +replication: + databases: + - data + - system +``` + +```graphql +type LocalTableForNode @table(replicate: false) { + id: ID! @primaryKey + name: String! +} +``` + +All tables in a replicated database replicate unless the table opts out. So the scope you are operating is the product of a config list, a set of schema directives, and any directional routes someone added later. Do not reconstruct it from memory. Read it off the running cluster: + + + + +```bash +curl -s -X POST https://operations.my-node.example.com/ \ + -H 'Content-Type: application/json' \ + -u 'admin:password' \ + -d '{"operation":"cluster_status"}' +``` + + + + +```javascript +await fetch('https://operations.my-node.example.com/', { + method: 'POST', + headers: { + 'Content-Type': 'application/json', + 'Authorization': 'Basic ' + btoa('admin:password'), + }, + body: JSON.stringify({ operation: 'cluster_status' }), +}); +``` + + + + +There is one socket per database per peer, so the sockets present in the response are the ground truth about what is flowing. Run this from every node, not one. Replication direction can be constrained per route, so node A's view of the cluster is not necessarily node B's view, and a one-sided picture is how directional configuration mistakes survive into production. + +:::warning +**`system` replicates by default.** It contains users, roles, and encrypted secret rows, so trust every node that receives it with those contents. Confirm the scope in your configuration and live connections before changing it. See [controlled system replication](/reference/v5/replication/overview#replicating-the-system-database-with-controlled-flow) for directional routes and table exclusions. +::: + +## What moves automatically, and what does not + +| Object or change | Behavior | What it means for you | +| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Data mutations | Insert, update, upsert, delete, and bulk load replicate for tables in scope | Monitor connections, latency, and convergence for the databases that serve critical journeys | +| Transactions | Replicated atomically, and a transaction may span multiple tables | Tables that need to commit together must live in the same database | +| A brand new node | Requests a full copy of each in-scope database from every peer it has no resume cursor for, then enters incremental replication | Budget N transfers, not one. See the note below before sizing a join | +| A returning node | Resynchronizes automatically to catch up on the transactions it missed | Not a full copy, unless the node was away longer than `logging.auditRetention`. A routine restart is much cheaper than a node replacement, so budget them separately | +| Component deployment | `deploy_component` replicates by default; pass `"replicated": false` to hold it to one node | Cluster-wide is the default, so single-node validation is the case you must ask for | +| Configuration | `set_configuration` supports `"replicated": true` | Never replicate node-local parameters. See the warning below | +| Users and roles | Propagate when the `system` database is in replication scope, which is the default | If you narrow scope to exclude `system`, identity must be provisioned on every node by your own automation | +| Node registry (`hdb_nodes`) | Lives in `system`, so it propagates with it. Each node rewrites its own self-record from its `harper-config.yaml` routes on restart or component reload | Scoping applied through `add_node` alone does not survive a restart. Put durable constraints in config routes, and see below if you need the registry kept per cluster | +| Destructive schema changes | `drop_database` and `drop_table` replicate by default; `drop_attribute` does not | A drop is a cluster-wide event unless you pass `"replicated": false`. Verify the result on every node | + +Measure incremental catch-up and full synchronization separately. A briefly offline node can catch up from retained transaction history, which its peers keep for [`logging.auditRetention`](/reference/v5/logging/configuration#loggingauditretention) (default `3d`). A node away longer than that gets a full copy of its peers' rows instead, even though it has synchronized before. Deletes older than the retention window are gone from that history, so the copy can leave rows on the returning node that were deleted elsewhere, while keeping writes that only that node took. Before such a node reconnects, decide which data is authoritative and how to reconcile it, as for [restoring in a replicated cluster](./backup-and-recovery.mdx#restoring-in-a-replicated-cluster). Configured routes reconnect as soon as Harper starts. + +On the new node, the cost is larger than "one copy from one peer," because the full-copy decision is made per peer and per database rather than once against a bootstrap source. A joining node requests a full copy from every peer it has no resume cursor for. And because replication is bidirectional, each established peer independently decides it has no cursor for the newcomer and requests a full copy _from_ it as well. Joining a cluster of N peers is therefore N inbound transfers plus N outbound transfers of the new node's own (empty) databases, not a single stream from one source. + +A node join can load every existing peer, so include it in cluster capacity planning. Passing `isLeader: true` to `add_node` identifies a bootstrap source, but does not prevent full-copy requests to other peers that lack a resume cursor. Do not use it to budget for a single transfer. + +Each node derives its advertised `hdb_nodes` record from its configuration. A restart or component reload can replace constraints applied only through `add_node`; keep durable constraints in config routes. + +Registry rows can replicate with `system`. Advertised records describe directional routes, so that exchange can preserve a constrained topology. To keep registries separate across a migration bridge, explicitly exclude `hdb_nodes` on the relevant routes while allowing users and roles through. See [controlled system replication](/reference/v5/replication/overview#replicating-the-system-database-with-controlled-flow). + +:::danger +When replicating configuration, only send cluster-appropriate parameters. Replicating a node-local value such as a port, `node.hostname`, a file path, TLS material, or `replication.hostname`, `url`, or `routes` overwrites every peer's own local value. Use `set_configuration` with `"replicated": true` for cluster-wide parameters. If the setting applies on worker restart, follow with `restart_service` using `"replicated": true`. For settings that require a full process restart, drain and restart each node in turn; see [configuration changes](./safe-deployments-and-rollback.mdx#configuration-changes-are-deployments-too). +::: + +## Read `cluster_status` like an operator + +A trimmed response, with the fields that matter: + +```json +{ + "type": "cluster-status", + "node_name": "server-1.example.net", + "is_enabled": true, + "connections": [ + { + "url": "wss://server-2.example.net:", + "name": "server-2.example.net", + "database_sockets": [ + { + "database": "data", + "connected": true, + "latency": 0.7, + "lastCommitConfirmed": "Wed, 12 Feb 2025 19:09:34 GMT", + "lastReceivedRemoteTime": "Wed, 12 Feb 2025 16:49:29 GMT", + "lastReceivedLocalTime": "Wed, 12 Feb 2025 19:09:31 GMT" + } + ] + } + ] +} +``` + +What each field is telling you: + +- **`connected`** is the liveness of this one database's socket to this one peer. A missing peer, or a peer present with `connected: false`, is actionable before users notice. +- **`latency`** is the round trip to that peer in milliseconds. Alert on sustained growth rather than on a single sample. +- **`lastCommitConfirmed`** is the last time this peer acknowledged receiving one of your commits. If it stops advancing while you are still writing, your writes are not landing on that peer. +- **`lastReceivedRemoteTime`** is the source node's timestamp on the newest transaction you have received. +- **`lastReceivedLocalTime`** is your own clock when you received it. + +Check connection state and sustained latency alongside commit progress during writes. Compare `lastReceivedRemoteTime` with the source's expected write timeline, then use a sentinel or journey test to verify freshness. `sendingMessage` appears only while a transaction is being sent; its absence on an idle socket is normal. + +The receive-time gap can help diagnose delay, with two cautions: + +- **It describes received transactions.** On an idle or stalled connection, the timestamps stop moving. Their difference does not measure how far the receiver is behind the latest source write. +- **It includes clock skew.** The timestamps come from different nodes. Compare against a healthy baseline with synchronized clocks. + +Use these fields as supporting evidence, rather than as the sole admission check. + +:::tip +While a database is taking a full copy, these timing fields render as the literal string `"Copying"` rather than a date. Anything that parses them as timestamps will fail on a joining node, which is exactly the node you most want to be watching. Handle that value explicitly. +::: + +Inventory `system.hdb_nodes` alongside this and compare it to your intended topology. Configuration intent and live peer state should agree, and the node's own row is in there too, not just its peers. + +### What to alert on + +- A peer missing entirely from `connections`, or `connected: false` on a database that serves a critical journey +- Sustained growth in `latency`, judged against your own baseline +- `lastCommitConfirmed` not advancing on a peer while writes are occurring +- Received transaction progress falling behind the expected source timeline, confirmed by a stale sentinel or journey result +- Repeated reconnects, which are visible in the logs even when a point-in-time status check looks healthy +- Version or configuration drift between peers + +## Prove convergence, not connection + +Socket state cannot tell you that a specific business record is current. Add an exported probe table to your application's schema, in the database you want to measure. Leave replication on, which is the default; a `replicate: false` table stays on the node that wrote it. For another database, change `data` here and in the write below, and give each database's probe table its own name: + +```graphql +type OpsProbe @table(database: "data") @export { + id: ID @primaryKey + writtenAt: Long + from: String +} +``` + +Write a sentinel on the source node: + +```javascript +const marker = { id: 'convergence-probe', writtenAt: Date.now(), from: 'server-1' }; +await databases.data.OpsProbe.put(marker.id, marker); +``` + +Read it from the target node and compare `writtenAt`: + +```bash +curl -s https://server-2.example.net/OpsProbe/convergence-probe \ + -u 'admin:password' +``` + +The interval between the write and the moment the peer returns the new value is your measured convergence time for that database, under whatever load the cluster is carrying at the time. Run it under load, not on an idle cluster, and record the result. That number is what your node admission gate should be compared against, and it is an input to the RPO work in [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx). + +The `replication-latency` metric from [analytics](/reference/v5/analytics/overview#replication-metrics) gives you the continuous version of the same measurement: the difference between the source commit timestamp and local time, reported per node, database, and table. Use the sentinel for a definitive answer during a change, and the metric for a dashboard. + +`replication-latency` is not recorded for `system`. Check its peer sockets and commit progress, then verify that expected identity changes have reached each node. Use a dedicated sentinel if your runbook needs a repeatable test of `system` convergence. + +## Consistency belongs in the application design + +Replication makes data available on every peer. It does not make every peer agree at every instant, and a distributed decision made on a node that has not yet converged can observe stale state and act on it. + +Concurrent read-then-write operations on different nodes need additional coordination when correctness requires a single decision: + +- Claim-once actions: redeeming a code, assigning a unique handle, awarding a one-per-customer offer +- Hard floors: inventory that must not go negative, a balance that must not overdraw +- Global limits: a rate limit or quota enforced across the whole cluster +- State machine transitions where two nodes could both believe they are making the same transition + +For each of these, choose one of three designs: route all decisions for a given key to a single owner, use a serialization mechanism so the conflict is resolved in one place, or delegate to an external coordinator. Then test your read-after-write expectations through the actual public route rather than against a single node, because a single node always looks consistent to itself. + +### Prove it + +On a non-production cluster under write load: + +1. Record baseline `cluster_status` timing fields on every node. +2. Interrupt replication to one peer, by stopping the node or blocking the replication port, while writes continue elsewhere. +3. Watch what your monitoring reports, and how long it takes to say anything. Note whether `connected` flipped, whether latency alerted, and how long until a human would have known. +4. Verify your application's actual behavior on the isolated node. Does the critical journey fail, serve stale data, or serve correctly? All three are possible and you should know which. +5. Restore the peer, then measure convergence with the sentinel until it is current. +6. Compare the convergence time against the admission gate in [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). If your gate would have admitted the node before step 5 finished, the gate is wrong. + +## Operational notes + +- **mTLS is required on the secure replication port and cannot be disabled.** Certificate expiry is therefore a replication outage, so treat certificate lifetime as an operational deadline with its own alert. See [certificate management](/reference/v5/security/certificate-management). +- **Gossip discovery means one route can join a whole cluster.** A node that connects to one peer discovers the rest. That is convenient and it means an accidental route can widen your topology more than you intended. +- **Sharding is a separate scope control.** If you use [sharding](/reference/v5/replication/sharding), not every node holds every record, so "this node is converged" and "this node can answer this query locally" become different questions. +- **Analytics data has its own replication setting.** See `analytics.replicate` in the [analytics configuration](/reference/v5/analytics/overview#analytics-configuration) before you build dashboards that assume metrics are or are not cluster-wide. +- **Keep node-local operational state out of scope.** Availability flags, maintenance markers, and anything else that describes one node rather than the cluster should not replicate, or a routine drain propagates to peers. + +## Readiness checklist + +- [ ] Replication scope read off the running cluster, from every node, not reconstructed from memory +- [ ] Whether the `system` database is in scope is a documented, deliberate decision +- [ ] Identity provisioning procedure exists and matches that decision +- [ ] Tables requiring one transaction boundary confirmed to be in one database +- [ ] Durable topology constraints live in `harper-config.yaml` routes, not only in `add_node` calls +- [ ] Measured convergence time recorded under load, per critical database +- [ ] Alerts configured on missing peers, sustained latency growth, stalled commit progress during writes, and stale sentinel values +- [ ] Non-commutative operations identified and given an owner, a serialization point, or an external coordinator +- [ ] Replication certificate expiry dates tracked with an alert +- [ ] Replication interruption and recovery exercise completed and dated + +## Additional Resources + +- [Replication overview](/reference/v5/replication/overview) for routes, scope, controlled flow, and securing connections +- [Clustering operations](/reference/v5/replication/clustering) for `add_node`, `set_node`, and the `cluster_status` response in full +- [Sharding](/reference/v5/replication/sharding) for controlling how many nodes hold a given record +- [Replication metrics](/reference/v5/analytics/overview#replication-metrics) for `replication-latency` and byte counters +- [Certificate management](/reference/v5/security/certificate-management) and [certificate verification](/reference/v5/security/certificate-verification) +- [Database schema](/reference/v5/database/schema) for the `replicate` table directive +- [Monitoring and Triage](./monitoring-and-triage.mdx) for putting these signals on a dashboard diff --git a/learn/administration/production-readiness-checklist.mdx b/learn/administration/production-readiness-checklist.mdx new file mode 100644 index 000000000..af12d1adf --- /dev/null +++ b/learn/administration/production-readiness-checklist.mdx @@ -0,0 +1,199 @@ +--- +title: Production Readiness Checklist +sidebar_position: 9 +--- + +Use this checklist to review a service before launch. It brings together the checks from the Administration guides, with links to the procedures and explanations. + +Copy it into your runbook and adapt it to your deployment. + +## How to use this + +Complete each item with evidence: a dated measurement, a written procedure, a configured alert, or a completed drill. + +Mark items that do not apply with a reason, so the reviewer can see which decisions were deliberate. + +Before you start, agree on: + +- **Who signs off.** Name the person responsible for accepting the evidence. +- **What "outstanding" means.** Some items can be accepted as gaps with an owner and a date. Decide in advance which ones cannot. +- **When you run it again.** This is not a one-time launch artifact. Re-run it after a Harper version upgrade, a topology change, or significant data growth, since most of the measured numbers expire. + +## Service definition + +From [How Harper Runs in Production](./how-harper-runs-in-production.mdx). + +- [ ] Critical user journeys named, with an owner for each +- [ ] Ports confirmed from `get_configuration` rather than assumed +- [ ] Operations API not reachable from the public internet +- [ ] Harper version recorded per node, with an alert on drift +- [ ] Databases listed, with replication scope confirmed from `cluster_status` +- [ ] Storage engine recorded per database +- [ ] Components and versions recorded from `get_components` +- [ ] Downstream dependencies named, with their failure behavior understood +- [ ] Fabric or self-managed responsibilities understood and divided + +## Capacity and topology + +From [Sizing a Harper Cluster](./sizing-a-harper-cluster.mdx). + +- [ ] `F`, the tolerated simultaneous node loss, declared explicitly and visible to change operators +- [ ] Per-node throughput measured with your own application code, data shape, and query mix +- [ ] Measurement taken with a peer synchronizing, not on an idle cluster +- [ ] Target utilization derived from latency at your SLO percentile +- [ ] `(N - F) x per-node x utilization >= peak` verified with real numbers +- [ ] Measurement conditions recorded alongside the number, and dated +- [ ] Topology's "what must be proven" satisfied, including independence of failure domains +- [ ] Maintenance policy states what happens when a drain would breach `F` +- [ ] Storage growth thresholds set, separately from request-rate thresholds + +## Health and traffic admission + +From [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). + +- [ ] Liveness, availability flag, application readiness, and journey synthetic all exist as separate signals +- [ ] Fabric's existing status component retained, or `@harperdb/status-check` configured for a self-managed cluster +- [ ] Traffic layer health check points at `GET /status` on the application port +- [ ] Availability flag storage still declares `replicate: false`, if you forked the component or persist it yourself +- [ ] Readiness route scoped to one journey's dependencies, with a timeout on every downstream call +- [ ] Readiness response includes the component version +- [ ] Drain sequence documented, with measured time to zero traffic +- [ ] Return sequence documented, including convergence verification and a stability window before full weight +- [ ] Measured node re-entry time recorded + +## Replication + +From [Operating Replication](./operating-replication.mdx). + +- [ ] Replication scope read off the running cluster, from every node +- [ ] Whether the `system` database is in scope is a deliberate, documented decision +- [ ] Identity provisioning procedure matches that decision +- [ ] Tables that must commit together confirmed to be in one database +- [ ] Durable topology constraints live in `harper-config.yaml` routes, not only in `add_node` calls +- [ ] Measured convergence time recorded under load, per critical database +- [ ] Alerts on missing peers, sustained latency growth, stalled commit progress during writes, and stale sentinel values +- [ ] Non-commutative operations identified and given an owner, a serialization point, or an external coordinator +- [ ] Replication certificate expiry tracked with an alert +- [ ] Replication interruption and recovery exercise completed and dated + +## Deployment and rollback + +From [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx). + +- [ ] Artifacts immutable, referenced by pinned version rather than by branch +- [ ] Deploy and expose are separate actions in the pipeline +- [ ] Previous known-good artifact addressable, and redeploying it rehearsed +- [ ] On v5.3.0 and later, the previous `deployment_id` still retained on every target node and activation rehearsed +- [ ] Cohort ladder defined, from smallest useful sample to full exposure +- [ ] Stop thresholds and hold times declared before a rollout begins +- [ ] Named decision owner for advance, hold, stop, and roll back +- [ ] Pipeline checks deployment details and waits for the restart job, then verifies the serving version on every node +- [ ] Nodes that must not serve while they change are drained first, since `restart=rolling` keeps each node in rotation +- [ ] `deployment_timeout` and `ignore_replication_errors` set deliberately +- [ ] Configuration changes go through the same change record as code +- [ ] `get_configuration` read back after every configuration change +- [ ] Node-local parameters never replicated +- [ ] Schema changes follow expand then contract, with destructive operations documented and verified per node +- [ ] Forward repair defined for changes that alter persisted meaning +- [ ] Measured release reversal time recorded from a rehearsal + +## Observability + +From [Monitoring and Triage](./monitoring-and-triage.mdx). + +- [ ] A dashboard exists that can place a symptom at edge, traffic layer, node, replication, data, or change +- [ ] Alerts on journey success and latency at the SLO percentile, not the mean +- [ ] Alerts on worker utilization and task queue latency, with thresholds derived from measured capacity +- [ ] Alerts on replication convergence lag +- [ ] Alerts on storage headroom and the age of the newest fully verified recovery point +- [ ] Metrics exported off the node +- [ ] `analytics.replicate` setting known and deliberate +- [ ] Structured logs include component, version, node, request id, operation, duration, and outcome +- [ ] Logs centralized off the node +- [ ] Deployments annotated onto dashboards +- [ ] Triage sequence written where on-call can find it +- [ ] Fault injection drill completed, with time to detection recorded + +## Backup and recovery + +From [Backup and Recovery](./backup-and-recovery.mdx). + +- [ ] Failure classes enumerated, each with the mechanism that addresses it +- [ ] Database recovery points and journey recovery times recorded using the [objective definitions](./engineering-rpo-rto-and-uptime.mdx#state-objectives-for-each-journey-and-its-data) +- [ ] Mechanism chosen per database according to its storage engine +- [ ] No database in scope has tables spanning multiple storage roots, or the exclusion is known and accepted +- [ ] Backup cadence matches the stated recovery point +- [ ] Repository capacity measured across successive backups, including transaction-log copies and blob hard links +- [ ] Export capacity accounts for full copies where the destination does not preserve file sharing +- [ ] Off-host copy exists in a separate failure and administrative boundary +- [ ] Backup artifacts have immutable identities, manifests, and checksums +- [ ] Destination encryption, versioning, and retention checked after upload where applicable +- [ ] Backup writer cannot weaken the destination's retention policy +- [ ] Managed repository copies take the whole per-database directory, quiesced or from an atomic snapshot +- [ ] `verify_backup` runs on a schedule +- [ ] Blob integrity understood to be unverified by `verify_backup` +- [ ] Restore constraints documented for user databases, component-held databases, and `system` +- [ ] Restore authority named, and restore execution logged +- [ ] Restore procedure isolates the node from replication, not just from user traffic +- [ ] Restore tested in a disposable, isolated environment using a compatible Harper version +- [ ] Restored inventory, counts, business invariants, and critical journeys validated +- [ ] Artifact ID, backup time, validation results, operator, approver, achieved RPO, and achieved RTO recorded +- [ ] `system` database restore rehearsed offline +- [ ] Preservation export planned separately where records require long-term, software-independent access + +## Objectives + +From [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx). + +- [ ] Objectives recorded using the [definitions](./engineering-rpo-rto-and-uptime.mdx#state-objectives-for-each-journey-and-its-data), with each journey's measurement boundary and dependent databases named +- [ ] Exclusions documented +- [ ] Every scenario in the scenario map has a named mechanism +- [ ] Availability target converted to minutes per month +- [ ] Error budget policy states what changes when the budget runs low +- [ ] Backup cadence reconciled against recovery point +- [ ] Restore time plus detection time reconciled against recovery time +- [ ] Convergence time reconciled against the admission gate +- [ ] Every unreconciled gap has an owner and a date +- [ ] Degraded modes declared per journey, with authority to invoke them + +## Security and access + +- [ ] Least privilege enforced for operators, applications, and automation +- [ ] `super_user` credentials inventoried, with a rotation procedure +- [ ] Operations API restricted to administrative networks +- [ ] TLS material inventoried, with expiry alerts for both application and replication certificates +- [ ] Deploy credentials held in a secret store, not in pipeline configuration +- [ ] Administrative access logged, including CLI access on the hosts +- [ ] Filesystem access on nodes understood to permit offline restore without an API credential + +## Ownership and runbooks + +- [ ] On-call rotation and escalation path defined +- [ ] Runbooks exist for drain, return, deploy, reverse, restore, and cluster expansion +- [ ] Every runbook has been executed by someone other than its author +- [ ] Change record location known, and used +- [ ] Incident and postmortem process defined +- [ ] A named owner for this checklist, and a date for the next review + +## Exercises completed + +Record the date of the most recent run of each. An undated exercise is an undocumented one. + +| Exercise | Guide | Last run | Result | +| ----------------------------------------- | ------------------------------------------------------------------------ | -------- | ------ | +| Service boundary inventory | [How Harper Runs in Production](./how-harper-runs-in-production.mdx) | | | +| Drain and return under load | [Health Checks](./health-checks-and-traffic-admission.mdx) | | | +| Peak-load drain and rejoin | [Sizing](./sizing-a-harper-cluster.mdx) | | | +| Replication interruption and recovery | [Operating Replication](./operating-replication.mdx) | | | +| Fault injection and time to detection | [Monitoring and Triage](./monitoring-and-triage.mdx) | | | +| Stopped rollout and reversal | [Safe Deployments](./safe-deployments-and-rollback.mdx) | | | +| Restore drill, user database | [Backup and Recovery](./backup-and-recovery.mdx) | | | +| Restore drill, `system` database, offline | [Backup and Recovery](./backup-and-recovery.mdx) | | | +| Scenario tabletop | [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx) | | | + +## Additional Resources + +- [Reliability Plan Template](./reliability-plan-template.mdx) for the document that holds the answers this checklist asks for +- Every guide in this section, linked per group above +- [Security overview](/reference/v5/security/overview) and [certificate management](/reference/v5/security/certificate-management) +- [Configuration options](/reference/v5/configuration/options) for the settings referenced throughout diff --git a/learn/administration/reliability-plan-template.mdx b/learn/administration/reliability-plan-template.mdx new file mode 100644 index 000000000..4c59030fd --- /dev/null +++ b/learn/administration/reliability-plan-template.mdx @@ -0,0 +1,194 @@ +--- +title: Reliability Plan Template +sidebar_position: 10 +--- + +The [Production Readiness Checklist](./production-readiness-checklist.mdx) asks the questions. This is the document that holds the answers. + +Keep one plan per service, somewhere your on-call team can reach during an incident. Replace the example values with your deployment's measurements and procedures. + +Example values are included in italics to show the intended level of specificity. Replace them. + +--- + +## 1. Service identity + +| Field | Value | +| ------------------ | ---------------------------------------------------------- | +| Service name | _orders-api_ | +| Business owner | _name, team_ | +| Technical owner | _name, team_ | +| On-call rotation | _link to schedule_ | +| Escalation path | _first, second, and decision authority for a stop_ | +| Harper deployment | _Fabric cluster, or self-managed, with cluster identifier_ | +| Harper version | _5.2.x, per node_ | +| Plan last reviewed | _date_ | +| Next review due | _date_ | + +## 2. Critical journeys + +List the user-facing paths that must work, including the databases and dependencies each one needs. + +| Journey | Description | Databases and tables involved | Downstream dependencies | Owner | +| ---------------- | ----------------------------- | ----------------------------- | ----------------------- | ----- | +| _submit order_ | _POST through the public API_ | _orders.orders, orders.items_ | _pricing, payments_ | | +| _browse catalog_ | _cached read_ | _catalog.products_ | _none_ | | + +## 3. Objectives + +Use the [objective definitions](./engineering-rpo-rto-and-uptime.mdx#state-objectives-for-each-journey-and-its-data): availability and recovery time per journey, recovery point per database. + +| Journey | Availability target | Measurement boundary | Recovery time objective | Recovery point objective by database | Exclusions | +| ------- | ------------------- | -------------------- | ------------------------------ | ------------------------------------ | ------------------------------------------ | +| | _99.95%_ | _at the CDN edge_ | _15 min from start of failure_ | _orders: 5 min; catalog: 1 hour_ | _planned maintenance, third-party outages_ | + +Monthly unavailability budget: _21 min 55 sec at 99.95%_ + +Error budget policy: _what changes when the remaining budget falls below a threshold, and who decides_ + +## 4. Architecture and topology + +- Node count and `F`, the tolerated simultaneous loss: _5 nodes, F = 1_ +- Failure domain layout: _how nodes are distributed, and the proof that domains are independent_ +- Traffic layer and routing: _what fronts Harper, and how it selects nodes_ +- Replication scope: _which databases, which tables excluded, and whether `system` is in scope_ +- Sharding in use: _yes or no, and where_ +- Storage engine per database: _per database_ + +## 5. Capacity + +| Measurement | Value | Conditions | Date measured | +| -------------------------- | ----- | ------------------------------------------------ | ------------- | +| Peak required throughput | | _per journey_ | | +| Tested per-node throughput | | _data volume, query mix, peer synchronizing y/n_ | | +| Target utilization | | _derived from latency at the SLO percentile_ | | +| Surviving capacity at `F` | | _measured in the drain drill, not calculated_ | | + +Maintenance policy when a drain would breach `F`: _pause, add capacity, or accept with documented duration_ + +## 6. Traffic admission + +- Liveness probe: _endpoint, interval, threshold_ +- Availability flag: _`status-check` deployed how, and who may set it_ +- Readiness route: _path, and the dependencies it checks_ +- Journey synthetic: _what it does, from where, how often_ +- Drain sequence: _link to runbook_ +- Return sequence, including stability window: _link to runbook_ +- Measured time to zero traffic after drain: _value, date_ +- Measured node re-entry time including convergence: _value, date_ + +## 7. Replication + +- Measured convergence time under load, per critical database: _value, date_ +- Measured full synchronization time for a new node: _value, date_ +- Admission gate threshold, and how it compares to convergence: _value_ +- Non-commutative operations and their handling: _operation, and the owner, serialization point, or coordinator_ +- Certificate expiry dates and alert lead time: _dates_ + +## 8. Deployment and change + +- Artifact source and reference format: _registry or git, pinned version format_ +- Cohort ladder: _one node, internal, 5 percent, 25 percent, full_ +- Stop thresholds and hold times: _stated numerically_ +- Decision authority for advance, hold, stop, roll back: _name or role_ +- Restart mode used, and how completion is confirmed: _`rolling`, polling the job_ +- Rollback reference and retention check: _pinned package, or retained `deployment_id` on every node for v5.3.0 and later_ +- Measured release reversal time: _value, date_ +- Forward repair procedures for data-affecting releases: _link_ +- Change record location: _link_ + +## 9. Observability + +- Dashboard location: _link_ +- Alerts configured, with thresholds and destinations: _table or link_ +- Metric export path: _Grafana on Fabric, or Prometheus scrape_ +- Log destination and retention: _system, retention period_ +- Deployment annotations enabled: _yes or no_ +- Triage runbook: _link_ +- Measured time to detection, per drill: _value, date_ + +## 10. Backup and recovery + +| Database | Mechanism | Cadence | Retention | Off-host destination | Last verified | Last restore drill | Measured restore time | Measured data loss | +| -------- | --------- | ------- | --------- | -------------------- | ------------- | ------------------ | --------------------- | ------------------ | +| | | | | | | | | | + +- Restore authority: _who may execute a restore_ +- Restore constraints per database: _online, or offline only because a component holds it open_ +- `system` database restore procedure: _offline only, link to runbook_ +- Failure classes assigned to restore versus forward repair: _table_ +- Artifact record: _immutable ID, backup timestamp, manifest and checksum locations_ +- Destination controls: _failure and administrative boundaries, encryption, versioning, retention, read-back evidence_ +- Backup writer permissions: _how retention changes are prevented_ +- Newest fully verified recovery point: _artifact ID, timestamp, validation results, operator, approver_ +- Restore validation: _database/table inventory, counts, business invariants, critical journeys, compatible Harper version_ +- Achieved RPO and RTO: _measured data loss and time from failure to verified service_ +- Preservation export, if required: _format, reader, and retention independent of Harper_ + +## 11. Scenario responses + +| Scenario | Mechanism | Runbook | What proves it works | Last exercised | +| ---------------------------------- | --------- | ------- | -------------------- | -------------- | +| _single node loss_ | | | | | +| _node return after restart_ | | | | | +| _node replacement_ | | | | | +| _failure domain loss_ | | | | | +| _region loss_ | | | | | +| _bad release, no data change_ | | | | | +| _bad release that changed data_ | | | | | +| _accidental destructive operation_ | | | | | +| _storage loss on one node_ | | | | | +| _certificate expiry_ | | | | | +| _downstream dependency outage_ | | | | | + +## 12. Degraded modes + +| Journey | Condition | Declared behavior | Who may invoke | Automatic or manual | +| ------- | ------------------------------- | ------------------------------------------- | -------------- | ------------------- | +| | _pricing service unavailable_ | _serve last known price, flag the response_ | | | +| | _node behind on replication_ | _remove from rotation_ | | | +| | _cluster below capacity budget_ | _shed non-critical traffic_ | | | + +## 13. Security and access + +- Operator roles and privileges: _who has what_ +- `super_user` credential inventory and rotation: _link, schedule_ +- Operations API network restriction: _how enforced_ +- Certificate inventory and expiry alerts: _link_ +- Deploy credential custody: _secret store, rotation method_ +- Administrative access logging, including host-level: _system_ + +## 14. Open risks + +Record accepted gaps with their impact, owner, and review date. + +| Risk | Impact if realized | Why accepted | Owner | Review date | +| ---- | ------------------ | ------------ | ----- | ----------- | +| | | | | | + +## 15. Revision history + +| Date | Author | Change | Trigger | +| ---- | ------ | ------ | ---------------------------------------------------- | +| | | | _launch, version upgrade, topology change, incident_ | + +--- + +## Keeping this current + +Most of the numbers in this plan are properties of a Harper version, a data volume, a topology, and a set of components. All four change, so the numbers expire. Re-measure and revise after: + +- A Harper version upgrade +- A topology change, including adding or removing nodes +- Significant data growth +- Any incident that produced a surprise + +Date each measurement so operators can judge whether it still describes the current service. + +## Additional Resources + +- [Production Readiness Checklist](./production-readiness-checklist.mdx) for the gate that populates this plan +- [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx) for how to derive sections 3 and 11 +- [Backup and Recovery](./backup-and-recovery.mdx) for section 10 +- [Safe Deployments and Rollback](./safe-deployments-and-rollback.mdx) for section 8 +- [Monitoring and Triage](./monitoring-and-triage.mdx) for section 9 diff --git a/learn/administration/safe-deployments-and-rollback.mdx b/learn/administration/safe-deployments-and-rollback.mdx new file mode 100644 index 000000000..25e360e35 --- /dev/null +++ b/learn/administration/safe-deployments-and-rollback.mdx @@ -0,0 +1,380 @@ +--- +title: Safe Deployments and Rollback +sidebar_position: 6 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +This guide deploys a Harper application to a cluster, waits until every node runs it, and rolls it back. It then covers limiting a release's first traffic and deciding when to expand or reverse it. The commands use the Harper CLI and match projects created with `npm create harper@latest`. [Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx) covers pipeline authentication, private sources, and registry artifacts in more depth. + +## What You Will Learn + +- How to deploy a Harper backend, or a frontend built once, from your machine or from GitHub Actions +- What `restart=rolling` does and does not do, and how to wait for it +- How to roll back from the CLI or from Studio +- How to limit a release's first traffic, and decide when to advance, hold, or reverse +- How to classify a change so you reverse the right thing, and why reversing the wrong thing can cause data loss + +## Prerequisites + +- A Harper Fabric cluster, or a self-managed Harper Pro cluster. Rolling back to a kept release needs v5.3.0 or later. +- The Harper CLI on your machine: `npm install -g harper` +- An application, for example one created with `npm create harper@latest` +- [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx), for taking a node out of rotation +- [Monitoring and Triage](./monitoring-and-triage.mdx), for comparing versions during a rollout + +## Deploy a release + +### Log in + +Set the cluster's Operations API URL once per shell, then log in with the cluster's admin username and password: + +```bash +export HARPER_CLI_TARGET=https://my-cluster.example.com:9925 +harper login "$HARPER_CLI_TARGET" +``` + +The CLI saves a token for that URL. Later `harper` commands in the shell use `HARPER_CLI_TARGET` as their target. + +### A Harper backend + +An application that runs from source, such as a `create-harper` project without a frontend build, deploys straight from its directory: + +```bash +cd my-app +harper deploy project=my-app restart=rolling json=true > deploy.json +``` + +`create-harper` projects run the same deploy, without `json=true`, as `npm run deploy`. The CLI packages the directory and uploads it, and Harper installs its dependencies. On Fabric and Harper Pro, a deploy reaches every node unless you pass `replicated=false`. + +The upload includes everything in the directory except `node_modules`, including `.git` and any `.env` file. Deploy from a clean checkout, or stage only the files you want to ship, as the next section does. + +### A frontend built once + +The React and Vue templates load the `@harperfast/vite` plugin, which builds the frontend on each node when the build output is missing, so every node also carries the build toolchain. To build once and ship only the output, keep a second configuration file for the release. Copy your `config.yaml` to `deploy.config.yaml`, keep its `rest`, `graphqlSchema`, `jsResource`, and [`static`](/reference/v5/static-files/overview) entries as they are, and remove the `@harperfast/vite` and `@harperfast/schema-codegen` entries, which load packages the release does not install. For the JavaScript React template, that leaves: + +```yaml +# deploy.config.yaml +rest: true +graphqlSchema: + files: 'schemas/*.graphql' +jsResource: + files: 'resources/*.js' # the TypeScript templates use 'resources/*.ts' +static: + files: 'dist/**' +``` + +Build, copy the release into its own directory, and deploy that directory: + +```bash +npm ci +npm run build +rm -rf deploy && mkdir deploy +cp -R dist schemas resources deploy/ +cp deploy.config.yaml deploy/config.yaml +(cd deploy && harper deploy project=my-app restart=rolling json=true) > deploy.json +``` + +The staged directory has no `package.json`, so the nodes skip the install step. If your resources import npm packages, add a `package.json` to `deploy/` that lists only those runtime dependencies. Harper Studio [deploys itself this way](https://github.com/HarperFast/studio/blob/stage/.github/actions/studio-deploy/action.yaml). + +### Wait for every node + +With `restart=rolling`, the deploy returns once the node you called has the release. The other nodes take it in a job that keeps running afterwards. Poll it until it prints `COMPLETE`: + +```bash +harper get_job id="$(jq -r '.restartJobId' deploy.json)" json=true | jq -r '.[0].status' +``` + +`ERROR` means the rollout failed, and `harper get_job` without the `jq` filter shows the job's `message`. `null` means the node that answered does not have the job: Harper does not replicate jobs, so poll the node that received the deploy. See [the job lives on one node](../developers/deploying-from-ci.mdx#path-a-deploy-a-tagged-release-from-your-repository). When the job completes, check the serving version on every node through your readiness route or a journey test. + +### Deploy from GitHub Actions + +`create-harper` projects include `.github/workflows/deploy.yaml`. It tests every pull request and, when a change merges to `main`, deploys with `restart=rolling` and waits for the job. It authenticates with GitHub's OIDC token, so the repository stores no Harper credential. Set up the cluster side once from your machine, logged in as a `super_user`: + +```bash +npm run deploy:setup-ci +``` + + + +The script runs [`harper deploy setup=true provider=github-actions`](/reference/v5/cli/commands#github-actions-deploys-oidc). It creates a deploy-only user and a trust policy pinned to `main` and the `production` environment, and sets the repository's `HARPER_CLI_TARGET` variable. With an earlier CLI, create the user and policy by hand, as [Let the workflow authenticate with OIDC](../developers/deploying-from-ci.mdx#let-the-workflow-authenticate-with-oidc) describes, pinning `workflow_ref` to `main`. Trust policies need a v5.3.0 or later cluster. + +For a frontend built once, replace the template's workflow with one that builds in one job and deploys in another: + +```yaml +# .github/workflows/deploy.yaml +name: Deploy to Harper +on: + push: + branches: [main] + +permissions: + contents: read + +concurrency: + group: deploy-${{ github.ref }} + +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + - uses: actions/setup-node@v4 + with: + node-version: 22 + - run: npm ci + - run: npm test + - run: npm run build + - name: Stage the release + run: | + mkdir deploy + cp -R dist schemas resources deploy/ + cp deploy.config.yaml deploy/config.yaml + - uses: actions/upload-artifact@v4 + with: + name: release + path: deploy/ + + deploy: + needs: build + runs-on: ubuntu-latest + environment: production + permissions: + id-token: write + env: + HARPER_CLI_TARGET: ${{ vars.HARPER_CLI_TARGET }} + steps: + - uses: actions/download-artifact@v4 + with: + name: release + path: deploy + - uses: actions/setup-node@v4 + with: + node-version: 22 + - run: npm install --global harper@^5.3 + - name: Deploy the release + working-directory: deploy + run: harper deploy project=my-app restart=rolling json=true > "$RUNNER_TEMP/deploy.json" + - name: Wait for every node to take the release + run: | + JOB_ID=$(jq -r '.restartJobId // empty' "$RUNNER_TEMP/deploy.json") + [ -n "$JOB_ID" ] || { echo "The rolling deploy returned no restartJobId"; exit 1; } + LAST="could not be read" + for attempt in $(seq 60); do + sleep 10 + harper get_job id="$JOB_ID" json=true > "$RUNNER_TEMP/job.json" || continue + case $(jq -r '.[0].status // "MISSING"' "$RUNNER_TEMP/job.json") in + COMPLETE) exit 0 ;; + ERROR) jq -r '.[0].message' "$RUNNER_TEMP/job.json"; exit 1 ;; + MISSING) LAST="was not on the node that answered; its outcome is unknown" ;; + *) LAST="was still running" ;; + esac + done + echo "After 60 polls, job $JOB_ID $LAST" + exit 1 +``` + +The `build` job runs your dependencies' install scripts and build tools, and cannot request an identity token. The `deploy` job can, and it runs none of your repository's code: it installs only the Harper CLI and uploads the built release. [Path B](../developers/deploying-from-ci.mdx#path-b-build-in-ci-publish-to-github-packages-deploy-the-artifact) explains this split, and publishes the build to a registry instead, which gives the release a version you can redeploy from any machine. + +## What `restart=rolling` does + +`restart=rolling` restarts the node you called, then each other node in turn. It waits for one node's restart to finish before it starts the next, and returns a `restartJobId` to poll. It does not: + +- **Take a node out of rotation.** Each node keeps taking traffic while its workers are replaced a few at a time. Harper does not mark the node unavailable or drain its connections. [harper#2975](https://github.com/HarperFast/harper/issues/2975) tracks load-balancer-aware rolling deploys. +- **Run your checks between nodes.** It moves on when a node's restart finishes, not when your readiness route or journey tests pass. From v5.4.0, [canary certification](/reference/v5/operations-api/operations#certifying-a-release-in-a-canary-worker) rejects a release that fails to load, but it does not run your checks. +- **Prove that every node restarted.** Without canary certification (before v5.4.0, or when the deploy's `certification` is `unavailable`), the job records an error for a node it cannot reach and continues to the next one. A `COMPLETE` job can still leave a node on the old release. + +To keep traffic off a node while it changes, drain it first, as in [isolated node validation](#limit-a-releases-first-traffic) or the [drain and return sequence](./health-checks-and-traffic-admission.mdx#drain-a-node-and-bring-it-back). + +`restart=true` restarts each node as it applies the replicated deploy, so restarts can overlap and reduce serving capacity. Use it for one drained node, a single-node instance, or development. On v5.2, it returns as soon as the restart is requested. From v5.3.0, it waits for the restart on the node you called, up to ten minutes. A caller timeout does not stop the restart, and the response does not report restart success, so check the serving version. For later versions, see [restart and certification behavior](/reference/v5/operations-api/operations#restarting-restart). + +Two deploy parameters control replication failures: + +- `deployment_timeout` is how long a peer waits for the replicated payload before failing, defaulting to 120000 ms. Raise it for large components or slow links. +- `ignore_replication_errors` treats a peer that fails to receive the deploy as non-fatal. By default a failed peer makes the whole operation return a non-2xx status, while the component is still deployed on the origin node. Decide which behavior you want before you need it. + +## Roll back + +### From the CLI + + + +When a deploy replaces a release, each node keeps the replaced one under its `deployment_id`. List recent releases, newest first: + +```bash +harper list_deployments project=my-app status=success limit=5 json=true \ + | jq -r '.deployments[] | "\(.deployment_id) \(.started_at / 1000 | floor | todate)"' +``` + +Activate the release you want back: + +```bash +harper deploy project=my-app deployment_id= restart=rolling json=true > deploy.json +``` + +Nothing is rebuilt or reinstalled. Wait for the job and check every node, as for any rolling deploy. A node that no longer keeps the release refuses it: [`deployment.stagingRetention.maxCount`](/reference/v5/configuration/options#deployment) sets how many replaced releases each node keeps, defaulting to `5`. See [going back to a previous release](/reference/v5/operations-api/operations#going-back-to-a-previous-release) for which releases are kept. + +Before v5.3.0, or when no node keeps the release, deploy the previous release again: check out its commit and repeat the deploy, or pass the previous version as `package`. + +### From Studio + + + +In [Harper Studio](/reference/v5/studio/overview), open the cluster and go to **Config > Deployments**. Open the release you want back and choose **Roll back to this release**. Studio activates it on every node, one at a time, and shows the rollout's progress. Check the serving version on every node afterwards. + +## Limit a release's first traffic + +A rolling deploy changes every node. To test a release on part of your traffic first, separate installing it from exposing it, and decide at each step whether to expand. + +```mermaid +flowchart TD + Preflight --> Limit[Limit initial traffic] + Limit --> Observe + Observe --> Decide{Decide} + Decide -->|Advance| Expand[Expand traffic] + Expand -->|More stages| Observe + Expand -->|Rollout complete| Close[Verify and close] + Decide -->|Hold| Hold[Keep current exposure] + Hold --> Observe + Decide -->|Reverse| Reverse[Restore prior release or traffic] + Reverse --> Verify[Verify recovery] + Verify --> Close +``` + +1. **Preflight.** Confirm the target version, component inventory, replication convergence, peer capacity with one node held out, and backups if data is at risk. Check that the prior release can still be activated or redeployed. Name the decision owner and set stop thresholds before starting. +2. **Limit.** Start with one drained node, an internal group, selected tenants, a region, or a small traffic percentage. Choose enough requests to make the comparison meaningful. +3. **Observe.** Compare old and new versions on request success, SLO latency, worker saturation, data correctness, replication, and downstream errors. +4. **Decide.** Advance, hold, or roll back against the agreed thresholds. Hold if required telemetry is missing. +5. **Expand.** Increase exposure only after minimum sample and hold conditions pass, and keep enough healthy capacity on the old version to reverse. +6. **Close.** Verify the same release and configuration on every node, restore intended traffic, and record the outcome with the measurements that supported each decision. + +| Pattern | How Harper is used | Fits when, and watch out for | +| ----------------------------- | -------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | +| Isolated node validation | Drain one node, deploy with `replicated=false`, validate, re-admit a group | Strongest validation before any user sees the release. Needs spare capacity and a direct URL for each node | +| Replicated rolling deployment | `harper deploy` with `restart=rolling` | Simple once you trust the release. Each node stays in rotation while it restarts | +| Feature-targeted release | Deploy compatible code broadly, enable it by flag, header, or tenant | Best when behavior can be separated from deployment. Assign an owner and a removal date to each flag | +| Parallel environment | Deploy to a separate node pool, shift traffic after validation | Strong isolation and fast reversal. Write ownership and replication between pools need real design | +| Regional progression | Deploy the same release region by region, exposing each after its checks | Limits the blast radius. One region does not predict every traffic shape | + +Isolated node validation needs at least three nodes, so the others can carry the traffic of the node you hold out. Take one node out of rotation with the [drain sequence](./health-checks-and-traffic-admission.mdx#drain-a-node-and-bring-it-back), log in to its own URL, and deploy to it alone: + + + + +```bash +harper login https://node-a.example.com:9925 +harper deploy project=orders-api package=@my-org/orders-api@2.4.1 \ + replicated=false restart=true target=https://node-a.example.com:9925 +``` + +Run your readiness route and journey tests against that node directly, then admit a small group of users to it. + + + + +```bash +harper deploy project=orders-api package=@my-org/orders-api@2.4.1 \ + restart=rolling json=true > deploy.json +``` + +[Wait for the job](#wait-for-every-node), then check every node. On Harper Pro 5.2, also read `harper get_deployment deployment_id="$(jq -r .deployment_id deploy.json)"` and check its `status` and `peer_results` for install and replication failures. + + + + +Deploy a pinned version or commit rather than a moving branch, so you can tell what ran yesterday and deploy it again. A private registry needs a [`credentials`](../developers/deploying-from-ci.mdx#give-harper-a-durable-credential-for-a-private-source) reference. + +:::danger +**Component deployment replicates by default.** Pass `replicated=false` when validating on one node. Other operations have different [replication defaults](/reference/v5/replication/overview#replicated-transactions); check the operation's reference before an isolated rollout. +::: + +To activate a release for part of your traffic without a second deploy, mount it at a path or hostname. `urlPath` mounts a component at an HTTP path and `host` serves it on a virtual hostname. Routing by either has been available since v5.1.0. What arrived in v5.2.0 is declaring that routing on the application's entry in the root `harper-config.yaml`, and the matching `host` parameter on `deploy_component`. Declared that way, the routing is persisted deployment state rather than a runtime toggle. See [HTTP middleware routing](/reference/v5/http/overview#middleware-routing). + +## Configuration changes are deployments too + +Apply the same review, rollout, and rollback checks to configuration changes. + +`set_configuration` supports `replicated=true` to apply a change across the cluster in one call, with per-node outcomes in the response. For settings that apply on worker restart, follow with `restart_service`, which restarts workers on each node in turn: + +```bash +harper set_configuration logging_level=info replicated=true +harper restart_service service=http replicated=true +``` + +Check the setting's restart requirements first. Process-level settings such as [`threads.v8Flags`](/reference/v5/configuration/options#applying-v8-flags) require a full process restart; `restart_service` does not apply them. Drain and restart each node in turn, using the `restart` operation or your process or container manager, and verify it before advancing. The `restart` operation does not replicate. + +:::danger +Only replicate cluster-appropriate parameters. Node-local values such as ports, `node.hostname`, file paths, TLS material, and `replication.hostname`, `url`, or `routes` would overwrite every peer's own values. Replicating one of these is a cluster-wide outage delivered in a single API call. +::: + +`get_status.restartRequired` tracks component and code restarts, so it does not identify pending `set_configuration` changes. Record those changes yourself, use the required restart type, and read back `harper get_configuration`. Read-back confirms the configured value; verify the running behavior too. + +## Classify the change before reversing it + +Different changes need different reversal paths, with different compatibility requirements and blast radii: + +| Change type | Reversal | Watch out for | +| ---------------------- | ----------------------------------------------------- | ------------------------------------------------------------------- | +| Component release | Activate the previous `deployment_id`, or redeploy it | Each node must still keep the release, or the artifact must exist | +| Feature behavior | Turn the flag off, no deploy needed | Fastest reversal available. Requires the flag to have been built in | +| Traffic exposure | Restore prior weights or set the node unavailable | Fastest containment. Does not undo anything already written | +| Configuration | `set_configuration` back, then restart | Needs a restart, so it is not instant | +| Harper runtime version | Version-specific procedure | Mixed versions change replication and deploy behavior | +| Schema or data | Forward repair or restore | See the warning below | + +- **Rehearse rolling back before launch**, not during an incident. +- **Prefer expand-then-contract schema evolution.** Add the new shape, migrate, then remove the old shape in a separate change. Destructive schema behavior depends on the operation and version, so each destructive change needs an exact written procedure and verification on every node. +- **Use the same checks for rollback as for forward movement**: availability, journey tests, peer stability, data validation, and traffic. A rollback is a deployment and can fail like one. + +:::warning +If a release changed the meaning of persisted data, code rollback alone will not fix it, and restoring a database to undo application code is usually the wrong move. A restore rolls back every write in the window, including all the valid ones, so it can violate your RPO in order to fix a code bug. Define forward repair or replay for these cases instead. See [Backup and Recovery](./backup-and-recovery.mdx). +::: + +### Prove it + +Rehearse a stopped rollout end to end on a non-production cluster: + +1. Declare a threshold before you start, for example "stop if journey success on the new group is more than 0.5 percent below the control group over five minutes." +2. Deploy a release that fails that threshold deliberately, to a limited group. +3. Detect it through your dashboards rather than because you know what you did. +4. Roll it back, and record both the time from the decision to verified service and the total time from failure to verified service. Compare the total, including detection and decision time, with your RTO. +5. Verify every node afterwards: `harper get_components` against each node's URL, and `harper list_deployments project=` showing the rollback. + +## Operational notes + +- **Keep Harper versions the same on every node.** Mixed versions change replication and deployment behavior, so detect and finish a rollout that stalls halfway. +- **Keep deploy credentials in your delivery platform's secret store**, or use the OIDC setup above so there is none. Use TLS and least privilege, and keep the deploy result as change evidence. See [secrets](/reference/v5/security/secrets). +- **Keep a single-node deployment path.** Every deploy replicates by default, so without `replicated=false` and a direct node URL you cannot test anything on one node. + +## Readiness checklist + +- [ ] Each release is a pinned version, a commit, or a staged build, never a moving branch +- [ ] The rollback path is rehearsed: on v5.3.0 and later the previous `deployment_id` is kept on every node; otherwise the previous artifact is still available +- [ ] Deploy and expose are separate steps for changes that need a limited first audience +- [ ] Cohort ladder defined, from smallest useful sample to full exposure +- [ ] Stop thresholds and hold times declared before the rollout starts +- [ ] A named decision owner for advance, hold, and roll back +- [ ] Pipeline waits for the rolling job, then checks the serving version on every node +- [ ] Nodes that must not serve while they change are drained first, since `restart=rolling` keeps each node in rotation +- [ ] `replicated=false` used deliberately wherever a change is meant to reach one node only, since replication is the default +- [ ] Cluster deploys use `restart=rolling`, not `restart=true` +- [ ] `deployment_timeout` and `ignore_replication_errors` set deliberately +- [ ] Configuration changes go through the same change record as code, and `get_configuration` is read back after each one +- [ ] Schema changes follow expand-then-contract, with destructive operations documented per node +- [ ] Forward repair defined for changes that alter persisted meaning +- [ ] Rollback duration and total recovery time recorded from a rehearsal, with the total compared against RTO + +## Additional Resources + +- [Deploying from a CI/CD Pipeline](../developers/deploying-from-ci.mdx) for OIDC setup, private sources, and registry artifacts +- [`harper deploy`](/reference/v5/cli/commands#harper-deploy) for every CLI deploy parameter +- [`deploy_component` reference](/reference/v5/operations-api/operations#deploy_component) for the server-side behavior, including `urlPath`, `host`, and replication controls +- [Deployment records](/reference/v5/operations-api/operations#deployment-operations) for `list_deployments` and `get_deployment` +- [Static files](/reference/v5/static-files/overview) for serving a built frontend +- [HTTP middleware routing](/reference/v5/http/overview#middleware-routing) for `urlPath` and `host` activation +- [Configuration operations](/reference/v5/configuration/operations) for `set_configuration` and restart requirements +- [Database schema](/reference/v5/database/schema) for schema evolution +- [Multiple Applications on One Cluster](../developers/multiple-applications.mdx) for running more than one component side by side diff --git a/learn/administration/sizing-a-harper-cluster.mdx b/learn/administration/sizing-a-harper-cluster.mdx new file mode 100644 index 000000000..d4998acc8 --- /dev/null +++ b/learn/administration/sizing-a-harper-cluster.mdx @@ -0,0 +1,173 @@ +--- +title: Sizing a Harper Cluster +sidebar_position: 3 +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +Size a Harper cluster so that, after losing `F` nodes, the survivors can carry peak demand within your utilization limit. + +This guide works through the capacity calculation, the load tests it depends on, and the headroom needed for maintenance and recovery. + +## What You Will Learn + +- The capacity invariant that sizes for the failure state, with a worked example +- What workload and recovery conditions to include in a per-node throughput measurement +- Why maintenance is a planned failure, and what that means for change windows +- How to choose a topology from your objectives rather than from a diagram +- How to convert an availability SLO into minutes per month, and what each tier demands of your automation + +## Prerequisites + +- [How Harper Runs in Production](./how-harper-runs-in-production.mdx), and your service boundary inventory +- [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx), because capacity headroom is meaningless if traffic cannot be moved off a node +- A load generation tool and a non-production cluster you can push to saturation +- A stated peak throughput requirement for your critical journeys + +## Size for the failure state + +The rule: + +```text +(N - F) x tested per-node throughput x target utilization >= required peak throughput +``` + +Where `N` is your node count, `F` is the largest simultaneous node loss the design tolerates, and target utilization is the fraction of measured capacity you are willing to run at, which should leave room for latency to stay acceptable rather than merely for requests to complete. + +Worked example. Suppose your critical journey needs 12,000 requests per second at peak, a node sustains 5,000 requests per second in your own test with acceptable latency, you will run at 70 percent utilization, and you want to survive losing one node: + +```text +Required surviving capacity = 12,000 / 0.7 = 17,143 req/s +Surviving nodes needed = 17,143 / 5,000 = 3.43 -> 4 +N = 4 + F (1) = 5 nodes +``` + +This requires five nodes. Three would handle the peak only while all remain available, without the chosen utilization headroom. + +In this example, tolerating two simultaneous failures requires six nodes. The 70 percent utilization limit also matters: it leaves the surviving nodes room to handle variable load while keeping latency acceptable. + +## Measure per-node capacity + +Use production-like conditions when measuring throughput: + +- **Representative application code, data shape, and query mix.** A Harper node runs your component logic in the same process as the data access, so your code is part of the capacity measurement in a way it would not be for a standalone database. A synthetic key-value benchmark tells you very little about the node's capacity to serve your journey. +- **Realistic downstream latency.** If your resource calls an upstream pricing service, its latency occupies worker capacity on the node. +- **Replication catch-up load.** A node that is feeding a recovering peer is doing work that does not appear in its own request metrics. Measure with a peer synchronizing, because that is precisely the state you will be in when you are already down a node. +- **Traffic imbalance.** Configured weights express intent. Actual request distribution is what consumes capacity, and it is rarely even. + +Measure each node separately rather than dividing a cluster total by node count. An average conceals the one node that is about to tip. + + + + +```bash +curl -s -X POST https://operations.my-node.example.com/ \ + -H 'Content-Type: application/json' \ + -u 'admin:password' \ + -d '{"operation":"system_information","attributes":["cpu","memory","disk","network","threads","harperdb_processes"]}' +``` + + + + +```javascript +await fetch('https://operations.my-node.example.com/', { + method: 'POST', + headers: { + 'Content-Type': 'application/json', + 'Authorization': 'Basic ' + btoa('admin:password'), + }, + body: JSON.stringify({ + operation: 'system_information', + attributes: ['cpu', 'memory', 'disk', 'network', 'threads', 'harperdb_processes'], + }), +}); +``` + + + + +Watch CPU, memory, event loop and worker pressure from the `threads` attribute, disk, and network, and pair them with request rate and latency from [analytics](/reference/v5/analytics/overview). The number you want is not the point where requests start failing. It is the point where your journey's latency leaves your SLO, which arrives earlier. + +:::tip +Record the conditions alongside the number: Harper version, node size, data volume, query mix, and whether a peer was synchronizing. A per-node capacity figure without its conditions is not reusable, and six months later nobody will remember whether the test included replication load. +::: + +## Maintenance is a planned failure + +Draining a node for a deploy, an upgrade, or an investigation consumes exactly the same headroom that a node failure does. So a cluster sized for `F = 1` is running at `F = 0` for the duration of every maintenance window, with no tolerance left. + +The operating rule that follows: if draining one node means the service can no longer tolerate its declared `F`, pause the change. Either wait for lower traffic, or add capacity for the window, or accept and document the reduced tolerance for a bounded period. What you should not do is treat the maintenance window as free because nothing is technically broken. + +Sizing for `F = 2` can let you drain one node for maintenance and still tolerate another node's failure at peak load. + +## Choose a topology from objectives + +Pick the pattern that matches the failures you actually need to survive, and know what you have to prove for it to count. + +| Pattern | When it fits | What you must prove | +| ------------------------------------ | ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | +| Single region, multiple peers | Node and host failures dominate, and low inter-node latency matters | `N - F` capacity under load, peer catch-up time, traffic removal, backup independence | +| Multiple failure domains, one region | Zone or rack isolation is available from your provider | No shared power or network fault, acceptable replication latency across domains, balanced traffic | +| Multiple regions | Regional impairment or user locality is a real requirement | Data authority per region, route propagation time, convergence, degraded-mode behavior | +| Edge or on-premises peers | Data locality or residency constrains where data can live | Replication scope per location, identity provisioning, backup path from each site | + +Test the failures your topology is meant to survive. Confirm that supposedly independent domains do not share a power supply, network path, or other dependency that could take them down together. + +Note that adding a node is a data movement event as well as a capacity event, since a Harper node carries data along with request handling. A node whose databases have never synchronized downloads them in full before it is useful, so scaling out is not instantaneous and cannot be your response to an unexpected traffic spike. Size ahead of demand. + +## Turn the SLO into a budget + +An availability target is a quantity of unavailability you are permitted to spend per month. Written that way it becomes an operating constraint rather than an aspiration. + +| Monthly SLO | Approximate maximum unavailability | What it demands | +| ----------- | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 99.9% | 43 min 50 sec | Manual diagnosis can fit inside some incidents, but one untested maintenance window can consume the month | +| 99.95% | 21 min 55 sec | Automated node removal and a rehearsed release reversal become necessary | +| 99.99% | 4 min 23 sec | Node failure must be close to transparent. Detection, traffic removal, and validation have to be automated, because no human response fits in the budget | + +Figures assume a 30.44 day month. Your contractual definition, exclusions, and measurement boundary may differ, and the boundary matters more than the number: availability measured at your CDN edge and availability measured at the Harper node are different quantities. + +Compare your measured detection and recovery times with the budget. A manual response that fits a 99.9 percent target may consume the entire budget at 99.99 percent. + +### Prove it + +Run the failure state rather than calculating it. On a cluster carrying representative load at your expected peak: + +1. Record per-node request rate, latency, and saturation as a baseline. +2. Drain one node using the sequence from [Health Checks and Traffic Admission](./health-checks-and-traffic-admission.mdx). +3. Measure what the survivors do: request rate, latency at your SLO percentile, CPU and worker pressure, and whether any user-visible errors appeared during the transition. +4. Keep the drained node out of user traffic while it rejoins replication. Measure the survivors while they carry peak traffic and feed the recovering peer. +5. Compare measured surviving capacity against your invariant. If step 4 pushed latency out of SLO, your target utilization is too high or your `N` is too low. + +## Operational notes + +- **Utilization targets belong per journey, not per cluster.** A write-heavy journey and a cached read journey consume very different resources on the same node, so a single cluster-wide utilization figure will be wrong for both. +- **Re-measure after a version upgrade.** Per-node capacity is a property of a Harper version, your component code, and your data volume. All three change. +- **Watch measured distribution, not configured weights.** Per-node request counts from analytics are the ground truth. A misconfigured weight, a sticky session policy, or DNS caching can leave one node doing far more work than the topology claims. +- **Storage growth is a capacity dimension too.** Disk headroom, compaction behavior, and backup space all scale with data volume rather than with request rate, so they need their own thresholds. See [compaction](/reference/v5/database/compaction). +- **On Fabric, cluster shape is managed but the invariant is unchanged.** You are still choosing `N` and living with `F`. + +## Readiness checklist + +- [ ] `F` declared explicitly, and written down where change operators will see it +- [ ] Per-node throughput measured with your own application code and data shape +- [ ] Measurement taken with a peer synchronizing, not on an idle cluster +- [ ] Target utilization set from latency at your SLO percentile, not from request success +- [ ] `(N - F) x per-node x utilization >= peak` verified with real numbers +- [ ] Measurement conditions recorded alongside the capacity figure +- [ ] Maintenance policy states what happens when a drain would breach `F` +- [ ] Topology's "what you must prove" column exercised, not assumed +- [ ] SLO converted to minutes per month, with the measurement boundary named +- [ ] Peak-load drain and rejoin exercise completed and dated + +## Additional Resources + +- [Analytics overview](/reference/v5/analytics/overview) and [analytics operations](/reference/v5/analytics/operations) for per-node request and latency metrics +- [Grafana integration](/fabric/grafana-integration) for dashboards on Fabric +- [`@harperdb/prometheus-exporter`](https://github.com/HarperFast/prometheus-exporter) for self-managed metrics scraping +- [Storage tuning](/reference/v5/database/storage-tuning) for the durability and throughput trade-offs available per database +- [Compaction](/reference/v5/database/compaction) for storage growth behavior +- [Replication overview](/reference/v5/replication/overview) for peer synchronization behavior when adding or returning a node +- [Engineering RPO, RTO, and Uptime](./engineering-rpo-rto-and-uptime.mdx) for scenario-level targets built on this capacity model diff --git a/learn/developers/deploying-from-ci.mdx b/learn/developers/deploying-from-ci.mdx index 140d4ff2b..053ca793b 100644 --- a/learn/developers/deploying-from-ci.mdx +++ b/learn/developers/deploying-from-ci.mdx @@ -195,7 +195,7 @@ A few details worth noting: - **`harper@^5.3`** pins the CLI's major version. 5.3 is the first CLI that authenticates with the identity token, and rerunning an old tag after a new major release still installs a 5.x CLI. - **`by_ref=true`** deploys `git+https://github.com//.git#`, where the SHA is the commit the tag points to (`GITHUB_SHA`). Every node clones that exact commit, even if the tag is moved later. Pass `ref=`, with the full commit SHA, to deploy a different commit. In a private repository, a tag resolves only if the checkout fetched it, as `fetch-depth: 0` does: the CLI looks up anything else with `git ls-remote`, which has no credential once `persist-credentials: false` keeps the token out of `.git/config`. - **`replicated` isn't needed.** On Harper Fabric and Harper Pro, a deploy goes to every node in the cluster unless you pass `replicated=false`. Each node fetches and installs the package itself, resolving the credential from the replicated secrets store. Harper core on its own does not replicate. -- **`restart=rolling`** restarts nodes one at a time so the cluster keeps serving throughout. Use `restart=true` for a single-node or dev instance. The [preview recipe](#per-pr-previews-with-shared-databases) also uses inline restarts on clusters, with an explicit allowance for concurrent initial restarts. On Harper core alone, a rolling job that only restarts — before v5.4.0, or when `certification` is `unavailable` — fails with `Replication not implemented`, which fails the wait step. +- **`restart=rolling`** restarts nodes one at a time, so they never all restart at once. Each node stays in rotation while it restarts; see [what `restart=rolling` does](../administration/safe-deployments-and-rollback.mdx#what-restartrolling-does). Use `restart=true` for a single-node or dev instance. The [preview recipe](#per-pr-previews-with-shared-databases) also uses inline restarts on clusters, with an explicit allowance for concurrent initial restarts. On Harper core alone, a rolling job that only restarts — before v5.4.0, or when `certification` is `unavailable` — fails with `Replication not implemented`, which fails the wait step. - **`json=true`** prints the deploy's result as JSON on stdout. Progress and authentication messages go to stderr, so `deploy.json` holds only the result. A deploy that fails exits non-zero, which fails the step. - **The wait step.** With `restart=rolling`, `harper deploy` exits `0` once the node you called has taken the release and started the rolling job, which keeps running after the deploy step has passed. On v5.4.0 and later, unless the deploy's `certification` is `unavailable`, the other nodes take the release in that job, one at a time, and a node that rejects it fails the job, not the deploy step; the failed job's `message` lists each node's outcome under `activated`. Otherwise every node installs the release before the deploy step passes, and the job only restarts them. The wait step polls the job's `restartJobId` with `get_job` until it ends `COMPLETE` or `ERROR`, and retries a poll that fails, such as one that reaches a node while it restarts. Each `harper get_job` authenticates through the trust policy the same way. It gives up after 60 polls, about ten minutes. Nodes take a rolling release one after another, each restarting and, on v5.4.0 and later, loading it in a canary that has up to a minute to report, so on a cluster with many nodes raise the count. With `restart=true` there is no job, and the deploy step itself fails when a node rejects the release, unless you pass `ignore_replication_errors=true`. - **The job lives on one node.** Everything else about a deploy is visible from any node, but Harper does not replicate jobs, so only the node that received the deploy knows the rolling job. Any other node answers `get_job` with an empty list, which the wait step treats as not finished and polls again. So through a load-balanced cluster URL, at least one poll has to reach the node that received the deploy. Routing that keeps the runner on one node sends its polls where the deploy went, and routing that spreads requests across nodes reaches it within a few polls. Routing that sends the polls to a different node than the deploy never does: the step fails, saying the job was not on the node that answered and its outcome is unknown, even when the rollout succeeded. If your load balancer routes that way, use one node's own URL for both the deploy and the wait, and as the trust policy's `audience`. [harper#3060](https://github.com/HarperFast/harper/issues/3060) tracks recording each node's outcome on the deployment record, which replicates and would remove this requirement. diff --git a/learn/developers/taking-a-node-out-of-rotation.mdx b/learn/developers/taking-a-node-out-of-rotation.mdx new file mode 100644 index 000000000..cbfabeb2b --- /dev/null +++ b/learn/developers/taking-a-node-out-of-rotation.mdx @@ -0,0 +1,624 @@ +--- +title: Taking an Unhealthy Node Out of Rotation +--- + +Harper Fabric's load balancer sends traffic to every node of a cluster whose `GET /status` answers `200`. That answer comes from a per-node record that Fabric maintains, so a node keeps receiving traffic while it is running, even when your application on that node cannot serve it. This guide shows how an application can write that record too, from its own health: a node that keeps failing takes itself out of rotation while another node serves, and puts itself back once it recovers, leaving alone what Fabric or an operator has set, outside the narrow windows described under [Caveats](#caveats). + +## What You Will Learn + +- How Fabric's load balancing uses the `status-check` component and the node's `availability` record +- When an application should take its node out of rotation, and when it should not +- The rules that keep an automated loop from fighting Fabric, flapping, or taking the last node out +- A small, generic component that implements those rules +- How to test it locally against a throwaway Harper with the real `status-check` component loaded + +## Prerequisites + +- A Harper Fabric cluster with at least two nodes, and a [Harper application](./harper-applications-in-depth.mdx) deployed to it +- Each node's own hostname, which only that node answers, as opposed to the cluster hostname +- For local testing: Node.js, and npm to install a throwaway copy of Harper + +## How Fabric decides where traffic goes + +Fabric deploys a component named `status-check` to every node. It serves `GET /status`, unauthenticated, and answers `200` when the node's `availability` record is `Available`, and `404` when the record is `Unavailable` or missing. The load balancer only sends new connections to nodes that answer `200`. See [Load Balancing and Node Availability](/fabric/load-balancing) for the component's routes. + +The `availability` record is read and written with the [`get_status` and `set_status`](/reference/v5/operations-api/operations#set_status--get_status--clear_status) operations. Three properties of it shape everything below: + +- **Each node has its own.** Harper stores it in `system.hdb_status`, which does not replicate, so a write on one node changes only that node. +- **It holds nothing but a status.** `set_status` stores the `status` value and drops any other field, so the record cannot also record who wrote it. `get_status` returns `__updatedtime__`, the time of the last write, which every write moves. +- **Fabric writes it too.** Fabric takes a node out while it upgrades it and puts it back afterwards. Operators can do the same with the Harper CLI, from their own machine or from a CI workflow; see [Taking a node out of rotation](/fabric/load-balancing#taking-a-node-out-of-rotation). + +## When to do this + +Take a node out when it fails in a way that is local to that node and that the load balancer cannot see, because `/status` keeps answering `200`. For example: + +- A dependency only this node reaches, such as a regional service, is unreachable from it. +- Something the application loads per node, such as a cache, a model, or a data file, failed to load here. +- This node has fallen behind on data that its requests need to be current. + +Do not use it for: + +- **Failures every node shares**, such as a central origin being down. Taking nodes out does not help, and the rule against taking the last node out stops it anyway. Leave shared dependencies out of the health check, or give them their own handling. +- **A node that is down altogether.** If Harper is not running, the loop is not either, but the node's `/status` does not answer, so the load balancer already leaves it out. +- **Planned maintenance.** Set the record yourself, or let Fabric do it. + +## The rules + +A loop that writes the `availability` record shares it with Fabric and with operators, and its mistakes take capacity out of the cluster. These rules keep it safe. Each one is in the [sample code](#build-it). + +### Wait before acting, longer before coming back + +The loop looks every 5 seconds. It takes the node out only after the health check has failed on every look for 30 seconds, and puts it back only after the check has passed on every look for 60 seconds. One look the other way restarts the count. + +**Why:** a node fails its health check for a few seconds while it starts, for example when a deploy restarts it. Waiting longer than that means a deploy never takes a node out. Requiring longer to come back than to go out keeps a node whose health check is flaky from flapping in and out of rotation. + +### Never undo someone else's write + +The loop only puts back a node it took out itself, and only while the record is still the one it wrote. Before it writes `Unavailable`, it stores a marker in a small table of its own. After the write, it adds the record's `__updatedtime__` to the marker. While the node is out, the record still having that `__updatedtime__` proves nobody has written it since. If anybody has (Fabric, or an operator putting the node back by hand), the loop leaves the record as they set it. It replaces its marker with a note that the record is theirs, and does not take the node out again until its health check has passed for the full 60 seconds. The note persists, so a restart, such as the one an upgrade brings, does not clear it. `set_status` has no compare-and-swap, so these checks leave small windows; [Caveats](#caveats) lists them. + +It never touches a node that is `Unavailable` without its marker, or a node that has no `availability` record. Asking the other nodes can take a few seconds, so the loop reads the record again just before each write, and leaves the write to its next look if anyone has written the record in between. + +**Why:** Fabric and operators have reasons the application cannot see. A loop that put back a node Fabric had taken out for an upgrade would send traffic to a node in the middle of one. The marker has to live outside the record, because `set_status` keeps only the status. + +When someone puts a node in while it is failing, the 30 seconds count from that write, not from when the failing began. Otherwise a node Fabric has just put back after an upgrade would be taken straight out again. + +### Never take the last node out + +Before it goes out, the loop asks every other node whether it is serving: its health check passes, and its `/status` answers `200`. It asks each one at that node's own hostname. If no other node is serving, it stays in and logs a warning. + +While it is out, it keeps asking. If no other node has served for 30 seconds, it comes back even though it is failing. + +**Why:** a node that fails some requests is better than no node at all. The question has to go to each node's own hostname: through the cluster hostname, the load balancer might send it to the node that is asking. + +### Look again straight after going out + +The loop looks again 1.5 seconds after it goes out. If no other node is serving at that look, or at any look in its first 10 seconds out, it comes straight back, and does not go out again until it restarts. + +**Why:** two nodes that start failing together can each see the other serving, and go out at the same moment. This catches that, and anything else that takes the other nodes out together with this one, such as a status record that replicates when it should not. + +### Survive a restart + +The marker table persists, so a loop that restarts while its node is out still knows the node is its to put back, and one that restarts while it leaves someone else's write in place still knows to leave it. The loop writes the marker before the record, and deletes it if the write fails. If the process stops between the two writes, the next look finds a marker without an `__updatedtime__`, and claims the record only if it went `Unavailable` within 5 seconds of the marker. + +**Why:** the `availability` record also persists. A node the loop took out stays out across a restart, and without the marker nothing would ever put it back. + +### Say what it does, and keep a switch + +Every write the loop makes is logged at `warn` with the reason, and so is every time it holds back for one of the rules above: staying in because no other node is serving, or leaving someone else's write in place. One environment variable turns the loop off. + +**Why:** a node leaving rotation changes where users' traffic goes. Whoever investigates later needs to find out why from the log, and needs a way to stop the loop without a code change. + +## Build it + +The sample is an application component with four parts: + +- `config.yaml` +- a marker table in `schema.graphql` +- a health check and a `/Health` endpoint in `resources.js` +- the loop in `availability.js` + +Replace the health check with your own. The rest is generic. + +### Configuration + +```yaml +# config.yaml +loadEnv: + files: '.env' +rest: true +graphqlSchema: + files: 'schema.graphql' +jsResource: + files: 'resources.js' +``` + +The loop reads its settings from the environment. [`loadEnv`](/reference/v5/environment-variables/overview) has to come first, so the variables are set before `resources.js` loads. + +```bash +# .env +# Every node's own base URL, this node's included, separated by commas. +AVAILABILITY_NODES=https://node-1.example.com,https://node-2.example.com +# Set to "off" to stop the loop. +AVAILABILITY_LOOP=on +# Used by this sample's health check; replace it with your own. +HEALTH_DEPENDENCY_URL=https://dependency.example.com/health +``` + +### The marker table + +```graphql +# schema.graphql +# What this application has done to this node's availability record, so that a +# restart doesn't forget it. Never replicated: like the record, it describes one node. +type RotationMark @table(replicate: false) { + id: ID @primaryKey + outAt: Float # when the application took the node out (ms since epoch) + statusAt: Float # the availability record's __updatedtime__ after that write + reason: String + asideAt: Float # when it found someone else had written the record since +} +``` + +[`replicate: false`](/reference/v5/replication/overview) keeps each node's marker on that node, so every node uses the same row id, `this-node`. A row with `outAt` means the loop has the node out; a row with `asideAt` means it is leaving someone else's write in place. The table has no `@export`, so it is not reachable over REST. + +### The health check and `/Health` + +```javascript +// resources.js +import { Resource, server } from 'harper'; +import { startAvailabilityLoop } from './availability.js'; + +// What "this node can serve" means for your application. Check what can fail +// on this node alone; this example asks a dependency the node reaches. +const DEPENDENCY_URL = process.env.HEALTH_DEPENDENCY_URL; + +async function checkDependency() { + if (!DEPENDENCY_URL) return { ok: true }; + try { + const response = await fetch(DEPENDENCY_URL, { + signal: AbortSignal.timeout(2000), + }); + return response.ok ? { ok: true } : { ok: false, reason: `dependency answered ${response.status}` }; + } catch { + // Fetch errors can include credentials from the URL. Keep them out of logs and markers. + return { ok: false, reason: 'dependency unreachable' }; + } +} + +// At most one check every HEALTH_CACHE_MS, its answer shared, so traffic to the +// unauthenticated /Health can't multiply the requests this node makes. +const HEALTH_CACHE_MS = 2000; +let latest; +function checkHealth() { + if (!latest || Date.now() - latest.at >= HEALTH_CACHE_MS) { + latest = { at: Date.now(), result: checkDependency() }; + } + return latest.result; +} + +// GET /Health: unauthenticated, so the other nodes can ask it. `node` lets the +// availability loop recognize this node's own address in its list of nodes. The +// failure reason stays in the loop's log: anyone can read this response. +export class Health extends Resource { + static async get() { + const { ok } = await checkHealth(); + return { + status: ok ? 200 : 503, + headers: { 'Cache-Control': 'no-store' }, + data: { ok, node: server.hostname }, + }; + } +} + +startAvailabilityLoop(checkHealth); +``` + +Keep the check fast and bounded by a timeout: the loop runs it every 5 seconds, and the other nodes call `/Health` before they go out. The sample shares one answer for 2 seconds, so requests to `/Health` can't multiply the checks. `/Health` answers anyone, so it returns only whether the node is healthy and its name; the reason stays in the loop's log. + +### The loop + +```javascript +// availability.js +import { server, tables } from 'harper'; + +const RECORD = 'availability'; +const ROW = 'this-node'; // RotationMark doesn't replicate, so each node has this one row +const TICK_MS = 5_000; +const RECHECK_MS = 1_500; +const OUT_AFTER_MS = 30_000; +const BACK_AFTER_MS = 60_000; +const ALONE_MS = 30_000; +const JUST_OUT_MS = 10_000; +const WRITE_MS = 5_000; // the loop's own Unavailable lands this soon after its row +const HEALTH_MS = 5_000; +const ASK_MS = 3_000; + +const log = logger.withTag('availability'); +const { RotationMark } = tables; + +// With authorization off (the default), a component can call these without a user. +const getStatus = () => server.operation({ operation: 'get_status', id: RECORD }); +const setStatus = (status) => server.operation({ operation: 'set_status', id: RECORD, status }); +const writtenAt = (record) => Number(record?.__updatedtime__) || 0; +const secs = (ms) => `${Math.round(ms / 1000)} s`; +const messageOf = (error) => error?.message ?? String(error); + +// The health check, bounded: one that throws or doesn't answer counts as failing. +function healthNow(checkHealth) { + let timer; + const timeout = new Promise((resolve) => { + timer = setTimeout(resolve, HEALTH_MS, { + ok: false, + reason: 'health check timed out', + }); + }); + return Promise.race([Promise.resolve().then(checkHealth), timeout]) + .catch((error) => ({ + ok: false, + reason: `health check threw: ${messageOf(error)}`, + })) + .finally(() => clearTimeout(timer)); +} + +async function ask(base, path) { + try { + const response = await fetch(new URL(path, base), { + signal: AbortSignal.timeout(ASK_MS), + }); + return { + status: response.status, + body: await response.json().catch(() => null), + }; + } catch (error) { + return { status: 0, error: messageOf(error) }; + } +} + +async function otherNodesServing(nodes, me) { + const answers = await Promise.all( + nodes.map(async (base) => { + if (URL.canParse(base) && new URL(base).hostname === me) return null; // this node + const health = await ask(base, '/Health'); + if (health.body?.node === me) return null; // this node, under another name + if (health.status !== 200) return { base, why: `/Health ${health.status || health.error}` }; + const status = await ask(base, '/status'); + if (status.status !== 200) return { base, why: `/status ${status.status || status.error}` }; + return { base, serving: true }; + }) + ); + const others = answers.filter(Boolean); + return { + serving: others.filter((node) => node.serving).map((node) => node.base), + why: + others + .filter((node) => !node.serving) + .map((node) => `${node.base} ${node.why}`) + .join('; ') || 'no other node listed', + }; +} + +export function startAvailabilityLoop(checkHealth) { + if (server.workerIndex !== 0) return; + // Every node's own base URL, this one's included. Never the cluster's shared + // hostname: the load balancer could route the question back to this node. + const nodes = (process.env.AVAILABILITY_NODES ?? '') + .split(',') + .map((url) => url.trim()) + .filter(Boolean); + if (process.env.AVAILABILITY_LOOP === 'off' || nodes.length < 2) { + log.notify?.('loop off'); + return; + } + const me = server.hostname; + let failingSince = null; + let passingSince = null; + let aloneSince = null; + let stopped = false; // the others stopped serving as this node went out + let warnedAlone = false; + + async function goBack(record, why) { + // Leave it to the next look if someone has written the record since this one read it. + if (writtenAt(await getStatus()) !== writtenAt(record)) return; + await setStatus('Available'); + log.warn?.(`Available, back in rotation: ${why}`); + await RotationMark.delete(ROW); + aloneSince = null; + warnedAlone = false; + } + + async function tick() { + const now = Date.now(); + const health = await healthNow(checkHealth); + if (health.ok) { + failingSince = null; + passingSince ??= now; + warnedAlone = false; + } else { + passingSince = null; + failingSince ??= now; + } + const passedFor = passingSince === null ? 0 : now - passingSince; + + const record = await getStatus(); + const row = await RotationMark.get(ROW); + + if (row?.asideAt) { + // Someone else wrote the record after this loop took the node out. It is + // theirs until the health check has passed for BACK_AFTER_MS. + if (passedFor >= BACK_AFTER_MS) await RotationMark.delete(ROW); + return; + } + + if (row?.outAt) { + // Out by this loop's write, unless someone has written the record since. + // statusAt is 0 if the process stopped between the loop's two writes; the + // record is then the loop's only if it went Unavailable right after the row + // was written (Harper's clock can trail Date.now() slightly). + const unavailable = record?.status === 'Unavailable'; + const at = writtenAt(record); + const ours = + unavailable && (row.statusAt ? at === row.statusAt : at >= row.outAt - 1_000 && at <= row.outAt + WRITE_MS); + if (!ours) { + if (!unavailable && !row.statusAt && writtenAt(record) < row.outAt) { + await RotationMark.delete(ROW); // the loop's write never happened + return; + } + await RotationMark.put({ id: ROW, asideAt: now }); + log.warn?.( + `${record?.status ?? 'no record'}, written by someone else after this node took itself out: leaving it` + ); + return; + } + if (!row.statusAt) { + await RotationMark.put({ + id: ROW, + outAt: row.outAt, + statusAt: writtenAt(record), + reason: row.reason, + }); + } + if (passedFor >= BACK_AFTER_MS) return goBack(record, `health has passed for ${secs(passedFor)}`); + const others = await otherNodesServing(nodes, me); + if (others.serving.length) { + aloneSince = null; + return; + } + if (now - row.outAt < JUST_OUT_MS) { + stopped = true; // as if they went out with it: stay in until this node restarts + return goBack( + record, + `no other node is serving just after this one went out (${others.why}); not going out again until Harper restarts` + ); + } + aloneSince ??= now; + if (now - aloneSince >= ALONE_MS) { + return goBack(record, `no other node has served for ${secs(now - aloneSince)} (${others.why})`); + } + return; + } + + if (health.ok || stopped) return; + if (record?.status !== 'Available') return; // out by someone else, or never put in + // Count from when failing began or from the record's last write, whichever + // is later: a node someone has just put in gets the full time again. + if (now - Math.max(failingSince, writtenAt(record)) < OUT_AFTER_MS) return; + const others = await otherNodesServing(nodes, me); + if (!others.serving.length) { + if (!warnedAlone) + log.warn?.(`health failing (${health.reason}), but no other node is serving (${others.why}): staying in`); + warnedAlone = true; + return; + } + // Leave it to the next look if someone wrote the record while the others were asked. + if (writtenAt(await getStatus()) !== writtenAt(record)) return; + // The row first: if the process stops between the two writes, the next + // look still knows whether the Unavailable record is this loop's. + const outAt = Date.now(); + await RotationMark.put({ + id: ROW, + outAt, + statusAt: 0, + reason: health.reason, + }); + try { + await setStatus('Unavailable'); + } catch (error) { + await RotationMark.delete(ROW); + throw error; + } + log.warn?.( + `Unavailable, out of rotation: health has failed for ${secs(now - failingSince)} (${health.reason}); serving: ${others.serving.join(', ')}` + ); + await RotationMark.put({ + id: ROW, + outAt, + statusAt: writtenAt(await getStatus()), + reason: health.reason, + }); + return 'out'; + } + + let errorLoggedAt = 0; + async function loop() { + let next = TICK_MS; + try { + if ((await tick()) === 'out') next = RECHECK_MS; + } catch (error) { + if (Date.now() - errorLoggedAt > 60_000) { + errorLoggedAt = Date.now(); + log.warn?.(`could not check: ${messageOf(error)}`); + } + } finally { + setTimeout(loop, next).unref(); + } + } + setTimeout(loop, TICK_MS).unref(); + log.notify?.(`loop on; asks ${nodes.join(' ')}`); +} +``` + +A few details: + +- **One loop per node.** Every worker thread loads the component, so the loop starts only where `server.workerIndex` is `0`. A worker restart can briefly overlap two; see [Caveats](#caveats). An [isolated application](/reference/v5/components/applications#isolated-applications) runs in one worker, whose index is `0`. +- **A health check that throws, or does not answer within 5 seconds, counts as failing.** Either way the loop goes on looking. +- **No user needed.** [`server.operation()`](/reference/v5/http/api#serveroperationoperation-context-authorize) applies no authorization unless you pass `authorize: true`, so the loop can call `set_status` without a user or a role. The `status-check` component calls it the same way. +- **Recognizing itself.** `AVAILABILITY_NODES` lists every node, so the same `.env` works on all of them. The loop skips the address whose `/Health` reports this node's [`server.hostname`](/reference/v5/http/api#serverhostname). +- **The numbers are a starting point.** 30 seconds out and 60 seconds back suit a node that restarts in a few seconds. If your nodes take longer to start, raise `OUT_AFTER_MS` above that time. + +## Test it locally + +Run the loop against a throwaway Harper with Fabric's real `status-check` component, a stand-in for a second node, and a stand-in for the dependency. Use scratch values for the home directory, root path, credentials, and ports, so the test touches nothing of an existing Harper install. + +### Start a throwaway Harper + +In an empty directory: + +```bash +npm install harper@5.3.2 +export HOME="$PWD/home" ROOTPATH="$PWD/root" TC_AGREEMENT=yes \ + HDB_ADMIN_USERNAME=admin HDB_ADMIN_PASSWORD=scratch-password \ + OPERATIONSAPI_NETWORK_PORT=9965 HTTP_PORT=9966 +mkdir -p "$HOME" +npx harper install +``` + +`npx harper` runs the copy you just installed. The scratch `HOME` keeps `harper install` from finding an existing installation in your own home directory, and `harper login` from touching your own saved credentials. + +Point the application at the stand-ins. Variables already set in Harper's environment take precedence over the application's `.env`, because `loadEnv` does not override them by default. Then start Harper: + +```bash +export AVAILABILITY_NODES=http://127.0.0.1:9966,http://127.0.0.1:9967 \ + HEALTH_DEPENDENCY_URL=http://127.0.0.1:9968/ +npx harper start +``` + +[Log in](/reference/v5/cli/authentication#method-1-persistent-login-recommended-for-local-development) with the admin username and password you chose. The CLI keeps the token, and the commands below use it: + +```bash +npx harper login http://127.0.0.1:9965 +``` + +### Start the stand-ins + +The application's health check and its list of nodes point at two stand-ins. Save this as `stand-ins.mjs` and run it with `node stand-ins.mjs` in a second terminal. It answers as another node on port 9967 and as the dependency on port 9968, and you change their answers with `POST` requests: + +```javascript +// Stand-ins for a local test: another node on 9967, and the dependency on 9968. +// Flip them with POSTs: /Health/503 or /status/404 on the node, /down or /up on the dependency. +import http from 'node:http'; + +const node = { Health: 200, status: 200 }; +let dependencyUp = true; + +http + .createServer((request, response) => { + const [, path, code] = request.url.split('/'); + if (request.method === 'POST' && path in node) node[path] = Number(code); + const body = path === 'Health' ? { ok: node.Health === 200, node: 'other-node.localhost' } : null; + response.writeHead(node[path] ?? 404, { 'content-type': 'application/json' }).end(JSON.stringify(body)); + }) + .listen(9967, '127.0.0.1'); + +http + .createServer((request, response) => { + if (request.method === 'POST') dependencyUp = request.url === '/up'; + response.writeHead(dependencyUp ? 200 : 503).end(); + }) + .listen(9968, '127.0.0.1'); +``` + +### Deploy `status-check` and the application + +Deploy Fabric's `status-check` component from its public repository, then the application from its directory: + +```bash +npx harper deploy project=status-check package=HarperFast/status-check-fabric restart=true +cd my-app && npx harper deploy restart=true && cd .. +``` + +A fresh install has no `availability` record, so `/status` answers `404`, and the loop leaves the node alone. Put the node in, as Fabric would: + +```bash +npx harper set_status id=availability status=Available +curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:9966/status +``` + +That prints `200`. Harper's log, `root/log/hdb.log`, shows that the loop is on. The log lines in this guide omit each line's timestamp and thread: + +``` +[notify] [availability]: loop on; asks http://127.0.0.1:9966 http://127.0.0.1:9967 +``` + +### Run the drill + +Follow the loop in the log while you work through the steps below: + +```bash +tail -f root/log/hdb.log | grep availability +``` + +1. **Out.** Make the health check fail: + + ```bash + curl -X POST http://127.0.0.1:9968/down + ``` + + About 30 seconds later, `/status` answers `404`, `npx harper get_status id=availability` reports `Unavailable`, and the log says why: + + ``` + [warn] [availability]: Unavailable, out of rotation: health has failed for 30 s (dependency answered 503); serving: http://127.0.0.1:9967 + ``` + +2. **Back.** Let it recover with `curl -X POST http://127.0.0.1:9968/up`. About 60 seconds later, `/status` answers `200` again: + + ``` + [warn] [availability]: Available, back in rotation: health has passed for 60 s + ``` + +3. **Never the last node.** Take the other node out, then fail the health check: + + ```bash + curl -X POST http://127.0.0.1:9967/status/404 + curl -X POST http://127.0.0.1:9968/down + ``` + + The node stays in, and warns once: + + ``` + [warn] [availability]: health failing (dependency answered 503), but no other node is serving (http://127.0.0.1:9967 /status 404): staying in + ``` + + Put the other node back with `curl -X POST http://127.0.0.1:9967/status/200`, and this one goes out at its next look. + +4. **Someone else's write.** While the loop has the node out, put it back by hand, as an operator would: + + ```bash + npx harper set_status id=availability status=Available + ``` + + The loop leaves it in, even though the health check still fails and even if Harper restarts, until the check has passed for 60 seconds: + + ``` + [warn] [availability]: Available, written by someone else after this node took itself out: leaving it + ``` + + The same happens when Fabric takes the node out itself. Write `status=Unavailable` while the loop has the node out, then let the dependency recover: the node stays out. + +5. **A restart.** With the node out and the dependency back up, restart Harper, in the terminal where you exported the variables: + + ```bash + npx harper stop + npx harper start + ``` + + The record is still `Unavailable` after the restart. The loop finds its marker, and puts the node back about 60 seconds after Harper starts. + +6. **Straight back.** Fail the health check and wait for the node to go out, then within five seconds take the other node out with `curl -X POST http://127.0.0.1:9967/status/404`. At its next look, the loop comes back, and it does not go out again until Harper restarts: + + ``` + [warn] [availability]: Available, back in rotation: no other node is serving just after this one went out (http://127.0.0.1:9967 /status 404); not going out again until Harper restarts + ``` + +When you are done, stop Harper with `npx harper stop`, stop the stand-ins, and delete the directory. + +## Caveats + +- **Open connections stay.** The load balancer only decides where new connections go. Clients already connected to a node that goes out, including WebSocket and MQTT clients, stay on it until they disconnect. +- **Turning the loop off leaves its node as it is.** The `availability` record persists, so a node the loop had taken out stays out after you set `AVAILABILITY_LOOP=off` or remove the application. Before you do either, check each node's `/status`, and put back, with `harper set_status`, any node whose `RotationMark` row has an `outAt`. +- **The health check decides.** A check that fails when the node could serve takes capacity out for nothing. A check that depends on something every node shares makes every node fail together, and then the loop keeps them all in. +- **A node that cannot be asked counts as not serving.** If one node cannot reach another at its own hostname, or that node's `/Health` needs credentials, the loop treats it as not serving and keeps its own node in. The loop errs toward keeping nodes in, never toward taking them out. +- **Someone else's write pauses the loop.** After Fabric or an operator puts back a node the loop took out, the loop leaves that node alone until its health check has passed for 60 seconds, however long the node keeps failing. +- **"Serving" is what this node sees.** A node that answers at its own hostname counts as serving even if the load balancer cannot reach it, and so does a node that is still running but no longer in the cluster. Keep `AVAILABILITY_NODES` current. +- **The ownership check has small gaps.** The loop reads the record again just before each write, and after a failed or interrupted write it only claims an `Unavailable` written within 5 seconds of its marker. So someone else's write can be taken for the loop's if it lands a few milliseconds after the loop's own write, or within those 5 seconds when a restart interrupted the loop. Coming back has the same kind of gap: a write landing between the loop's last read and its own `Available` is overwritten. The reverse can happen too: if the loop's own write took longer than 5 seconds and a restart interrupted it, the loop leaves the node out and logs the write as someone else's. After a restart that hit a node mid-write, check its `/status`. +- **A worker restart can briefly run two loops.** On Linux, including Fabric, Harper starts a replacement worker before it stops the one it replaces, so the old worker `0` and its replacement both run the loop until the old one exits. That can take up to `replication.blobSendDrainTimeout` while it finishes sending blobs. Each loop has its own timers and can write in that window, with the same ownership gaps as above. A worker restart, such as a deploy with `restart`, also ends "not going out again until Harper restarts". After a deploy or restart, check each node's `/status`. +- **Operation logging.** At log level `info` or more verbose, Harper logs every operation call, including the loop's `get_status` every 5 seconds. + +## Additional Resources + +- [Load Balancing and Node Availability](/fabric/load-balancing) - the `status-check` component's routes and authorization +- [`set_status`, `get_status`, and `clear_status`](/reference/v5/operations-api/operations#set_status--get_status--clear_status) - the operations behind the `availability` record +- [`server.operation()`](/reference/v5/http/api#serveroperationoperation-context-authorize) - calling operations from a component +- [Replication: excluding a table](/reference/v5/replication/overview) - `@table(replicate: false)` +- [Environment Variables](/reference/v5/environment-variables/overview) - `loadEnv` +- [Logging API](/reference/v5/logging/api) - `logger` and tagged loggers +- [HarperFast/status-check-fabric](https://github.com/HarperFast/status-check-fabric) - the `status-check` component's source diff --git a/redirects.ts b/redirects.ts index c6eb20368..7e06c8f3b 100644 --- a/redirects.ts +++ b/redirects.ts @@ -358,6 +358,7 @@ const currentRedirects: RedirectRule[] = [ // ── Learn ───────────────────────────────────────────────────────────────── { from: '/learn/developers/coming-soon', to: '/learn' }, + { from: '/learn/administration/coming-soon', to: '/learn/administration/how-harper-runs-in-production' }, // ── Fabric ──────────────────────────────────────────────────────────────── { from: '/fabric/rest-api', to: '/fabric' }, diff --git a/reference/backups/operations.md b/reference/backups/operations.md index 3abe60d37..c59dd65f4 100644 --- a/reference/backups/operations.md +++ b/reference/backups/operations.md @@ -33,7 +33,7 @@ Offline invocation matters most for restore: an in-place `restore_backup` of the Creates an incremental directory backup of the database under the configured backup root ([`storage.backupPath`](../configuration/options.md#storage), default `/backup`). Through a running server this runs as a background [job](../operations-api/operations.md#jobs): the operation returns a `job_id` immediately, and [`get_job`](../operations-api/operations.md#get_job) reports the outcome including the new `backup_id`, `size`, and `timestamp`. -Backups of the same database share unchanged RocksDB data files, so the second and subsequent backups only copy the data that changed. The transaction-log snapshot — and the blob snapshot, if the database has file-backed blobs — is copied in full on every backup, so those portions are not incremental; with a large audit-retention window, frequent backups are not free for them. Pass `exclude_blobs: true` to skip the blob snapshot. Use [`purge_backups`](#purge_backups) to manage retention. +Backups of the same database share unchanged RocksDB data files within the repository. Each backup also copies the transaction-log snapshot. Blob snapshots use hard links when the source and destination filesystem support them, and copy the files otherwise. Pass `exclude_blobs: true` to skip the blob snapshot. Measure physical repository growth to plan capacity, and use [`purge_backups`](#purge_backups) to manage retention. ```json { "operation": "create_backup", "database": "data" } @@ -68,7 +68,13 @@ harper list_backups database=data -Verifies a managed backup's RocksDB file sizes — and their checksums when `verify_checksum` is `true` (slower) — together with the framing of its transaction-log snapshot (always checked). The blob snapshot is not verified. Through a running server this runs as a background [job](../operations-api/operations.md#jobs). `backup_id` is required. +Verifies a managed backup. `backup_id` is required. + +- **RocksDB files:** checks file sizes and, when `verify_checksum` is `true`, checksums. Checksum verification takes longer. +- **Transaction log:** always checks the snapshot's framing. +- **Blob snapshot:** checks that a snapshot is present if the backup recorded blobs. A missing snapshot fails verification. Individual blob sizes, checksums, and readability are not checked, so damaged blob contents can pass verification. + +Through a running server, verification runs as a background [job](../operations-api/operations.md#jobs). Poll `get_job` for its result. ```json { "operation": "verify_backup", "database": "data", "backup_id": 1, "verify_checksum": true } @@ -112,6 +118,8 @@ harper purge_backups database=data keep_count=3 Restores a database in place from a managed backup. `backup_id` defaults to the latest backup. The audit/transaction log is restored alongside the data, and — for a database with file-backed blobs — the blob roots are purged and rewritten from the backup's blob snapshot. + `allow_engine_only` (optional, default `false`) permits restoring a backup created with `exclude_blobs: true`. Without it, that restore is refused, including when using `target_database`. Enabling it accepts the risk that restored records reference missing or unrelated blob files; recover and validate those blobs separately. + Through a running server this runs as a background [job](../operations-api/operations.md#jobs): Harper closes the database across all worker threads, restores it, and reloads it. This works only when no loaded component is holding the database open — if one does, the job ends in `ERROR` (surfaced by [`get_job`](../operations-api/operations.md#get_job)) telling you to restore offline. Restoring the `system` database is rejected up front, before a job is created, as is `target_database` while the server is running. These cases require running the command from the CLI with the server stopped. See [when can a database be restored?](./overview.md#when-can-a-database-be-restored) As of v5.3.1, subscriptions opened before a restore do not carry over to the restored data. When Harper reloads the database, each one ends, and its last message is a `DatabaseGenerationChangedError` (status code 409, code `DATABASE_GENERATION_CHANGED`). Resubscribe to resynchronize against the restored state. A restore that fails before it changes anything reloads the database as it was; those subscriptions end instead with a retryable `DatabaseClosingError` (status code 503, code `DATABASE_CLOSING`); resubscribe to continue. @@ -137,7 +145,7 @@ harper restore_backup database=data backup_id=1 target_database=data_restored Streams a full snapshot of the specified database in the HTTP response for download — there is no server-side artifact and nothing to clean up. The server must be running; this is the one backup operation with no offline form. Behavior depends on the storage engine: - **RocksDB** — streams a `tar` archive of the current database state — all tables, the transaction log, and any file-backed blobs — gzipped by default (`gzip` is a RocksDB-only option and compresses the snapshot substantially). Pass `"gzip": false` for a plain `tar`, or `"exclude_blobs": true` to leave blobs out. The snapshot is always the current state; downloading a specific historical `backup_id` is not supported — to move a retained managed backup off-host, copy its whole per-database backup repository (see [Limitations](./overview.md#limitations)). Restoring a RocksDB snapshot by hand takes an extra step when the database has blobs — see [download a snapshot and restore it manually](./overview.md#example-download-a-snapshot-and-restore-it-manually). -- **LMDB** — streams the `.mdb` file. Specify `"table"` for a single table or `"tables"` for a set, and `"include_audit": true` to include the audit store. These options are LMDB-only. +- **LMDB** — streams the `.mdb` file, without external blob files. Specify `"table"` for a single table or `"tables"` for a set; a table-filtered download includes the audit store only with `"include_audit": true`. These options are LMDB-only. For databases with file-backed blobs, use a coordinated filesystem or volume backup of the database and all blob roots with Harper stopped; see [LMDB recovery](./overview.md#lmdb-). ```json { "operation": "get_backup", "database": "data" } diff --git a/reference/backups/overview.md b/reference/backups/overview.md index 3a0ab21dd..c97754f86 100644 --- a/reference/backups/overview.md +++ b/reference/backups/overview.md @@ -5,12 +5,12 @@ title: Backups -Harper can back up and restore its databases natively. A backup is a whole-database copy — all tables, the audit/transaction log, and any file-backed blobs — so a restored database is a consistent point-in-time image of everything in it. (`get_backup` on an LMDB database is the exception: it can stream a subset of tables, and includes the audit store only when requested.) +Harper can back up and restore its databases natively. RocksDB managed backups and snapshot downloads include all tables, the audit/transaction log, and file-backed blobs unless `exclude_blobs` is requested. LMDB `get_backup` streams only the `.mdb` file, optionally filtered by table; external blob files need a separate, coordinated backup. Two complementary mechanisms are available: - **Managed backups** — incremental, verifiable backups kept in a server-side repository. Harper creates, lists, verifies, deletes, purges, and restores them through the [backup operations](./operations.md). -- **Snapshot download** — [`get_backup`](./operations.md#get_backup) streams a full snapshot of a database over HTTP for storage off-host, with no server-side artifact. +- **Snapshot download** — [`get_backup`](./operations.md#get_backup) streams a RocksDB archive or an LMDB `.mdb` file over HTTP for storage off-host, with no server-side artifact. Every backup operation is available through the [Operations API](../operations-api/overview.md) and as a [CLI command](./operations.md#running-backup-operations-from-the-cli) under the same operation name. @@ -20,7 +20,7 @@ Managed backups use the RocksDB backup engine. Each backup is a directory under Every managed backup includes the database's transaction log — so a restored database keeps its `read_audit_log` history as of the backup point — and, for a database with file-backed blobs, a snapshot of those blobs (skip it with `exclude_blobs`). -Only the RocksDB data files are shared and incremental. The transaction log and the blob snapshot are each copied in full on every backup, so with a large audit-retention window or many blobs, frequent backups cost more than the data-only view suggests. +Each backup copies the transaction-log snapshot. Blob snapshots use hard links when supported on the same filesystem, and copy the files otherwise. Measure physical repository growth over successive backups, including the log copies and blob snapshots. Exports that do not preserve file sharing require space for the full materialized copies. The long-running operations (`create_backup`, `verify_backup`, `restore_backup`) run as background [jobs](../operations-api/operations.md#jobs) when invoked through a running server: the operation returns a `job_id` immediately, and [`get_job`](../operations-api/operations.md#get_job) reports the outcome. @@ -33,8 +33,8 @@ The offline path matters for restore. RocksDB is single-writer, so an in-place r ## Limitations - **Managed backups require the RocksDB storage engine.** For LMDB databases, use [`get_backup`](./operations.md#get_backup) or [volume snapshots](../cli/commands.md#how-backups-work). -- **Whole-database granularity.** There is no per-table backup or restore. (The one exception: `get_backup` on an LMDB database can stream individual tables, and includes the audit store only with `include_audit`.) -- **One storage root per database.** A database whose tables use per-table `path` storage configs spans multiple root stores and cannot be backed up with these operations. +- **Whole-database granularity.** There is no per-table backup or restore. The exception is `get_backup` on LMDB, which can stream selected tables; a table-filtered download includes the audit store only with `include_audit`. +- **One storage root per database.** Backup is rejected if the database's tables resolve to multiple storage roots. A custom database-level `storage.path`, or per-table paths that still resolve to a single root, are supported. - **Backups live on the node that created them.** The backup repository is a local directory. RocksDB shares files across backup IDs (a backup ID is not a self-contained folder), so disaster-recovery copies must take the entire per-database repository — `/` — not an individual backup, and must do so while no backup operation is running, or from an atomic filesystem snapshot. A live recursive copy can race `create_backup`/`delete_backup`/`purge_backups` and produce an unrestorable copy. Alternatively, use `get_backup` to pull a snapshot from a running server. - **`get_backup` always streams the current state.** It cannot download a historical managed backup; to move a retained backup off-host, copy its whole per-database repository as above. - **A restore is a point-in-time rollback.** In a replicated cluster, coordinate a restore with replication before bringing the node back. @@ -63,7 +63,7 @@ Create a managed backup of the `data` database (the default): harper create_backup database=data ``` -The first backup copies the entire database; each one after that only copies the RocksDB data files that changed, so re-backing-up the data is cheap. (The transaction-log snapshot, and the blob snapshot for databases with file-backed blobs, are copied in full each time — see [how managed backups work](#how-managed-backups-work).) Schedule `create_backup` as often as your recovery point requires, and manage retention with `purge_backups`: +Later backups share unchanged RocksDB data files, while transaction-log copies and blob snapshots have their own storage cost; see [how managed backups work](#how-managed-backups-work). Schedule `create_backup` as often as your recovery point requires, and manage retention with `purge_backups`: ```bash harper purge_backups database=data keep_count=7 @@ -93,7 +93,7 @@ For a RocksDB database the stream is a `tar` archive, gzipped by default. It con ```bash harper get_backup database=data out=./data.tar.gz -# or pull from another node — this clones that node's database onto the current one +# or read the snapshot from another node, still writing it to a file here harper get_backup database=data target=https://node-2.example.com:9925 out=./data.tar.gz ``` @@ -125,6 +125,10 @@ The target node must have at least as many blob roots configured as the snapshot ### LMDB +:::warning +This stream does not include external blob files. For LMDB databases with file-backed blobs, use a coordinated filesystem or volume backup of the `.mdb` file and every configured blob root with Harper stopped. Restore that matching set and validate blob reads before serving traffic; restoring the `.mdb` file alone leaves its external blob references unresolved if the source files are lost. +::: + For an LMDB database the stream is the database's single `.mdb` file: ```bash diff --git a/reference/cli/commands.md b/reference/cli/commands.md index a803f1b7f..d85cb87ad 100644 --- a/reference/cli/commands.md +++ b/reference/cli/commands.md @@ -141,7 +141,7 @@ All parameters are passed as `key=value` arguments. Every parameter is optional. - `project=` - Component project name. Defaults to the current directory's name for a directory deploy, or is derived from the package for a package deploy. - `package=` - An npm, GitHub, or tarball reference to deploy instead of the current directory (e.g. `HarperDB/app#semver:v1.0.0`). - `target=` - Remote Harper instance to deploy to. Omit to deploy to the local instance. A bare host defaults to `https://:9925`. -- `restart=true` or `restart=rolling` - Restart Harper after deploying. Use `rolling` for a staggered, zero-downtime restart across a cluster. With `rolling`, the command exits `0` once the node you called has taken the release and started the rolling restart's job; see [Waiting for a rolling deploy](#waiting-for-a-rolling-deploy). Cannot be combined with `activate=false`. +- `restart=true` or `restart=rolling` - Restart Harper after deploying. Use `rolling` to restart a cluster's nodes one at a time; each node stays in rotation while it restarts. With `rolling`, the command exits `0` once the node you called has taken the release and started the rolling restart's job; see [Waiting for a rolling deploy](#waiting-for-a-rolling-deploy). Cannot be combined with `activate=false`. - `replicated=false` - Deploy to the node you called only. On Harper Pro and Fabric, a deploy goes to every node in the cluster unless you pass this; Harper core on its own does not replicate. - `install_command=` - Override the install command run for the component. - `install_timeout=` - Maximum time, in milliseconds, to allow the install to run. diff --git a/reference/components/applications.md b/reference/components/applications.md index 2cf06274a..88bc0da81 100644 --- a/reference/components/applications.md +++ b/reference/components/applications.md @@ -466,7 +466,7 @@ Creates a new component project in the component root directory using a template - `install_command` _(optional)_ — Install command. Defaults to `npm install` - `install_timeout` _(optional)_ — Install timeout in milliseconds. Defaults to `300000` (5 minutes) - `install_allow_scripts` _(optional)_ — Allow install scripts to run. Defaults to `false`, which causes `--ignore-scripts` to be passed to the install command (this is ignored with `install_command`). -- `replicated` _(optional)_ — Replicate to all cluster nodes +- `replicated` _(optional)_ — On Harper Pro and Fabric, replicate to all cluster nodes by default; pass `false` to apply on this node only. Harper core on its own does not replicate ```json { @@ -570,7 +570,7 @@ On a `.env` file it replaces the whole file with the payload, so saving back the - `file` _(required)_ — Path relative to project folder - `payload` _(required)_ — File content to write - `encoding` _(optional)_ — File encoding. Defaults to `utf8` -- `replicated` _(optional)_ — Replicate update to all cluster nodes +- `replicated` _(optional)_ — On Harper Pro and Fabric, replicate the update to all cluster nodes by default; pass `false` to apply on this node only. Harper core on its own does not replicate ```json { @@ -593,7 +593,7 @@ For deploying from private repositories, SSH keys must be registered on the Harp - `host` _(required)_ — Host alias for SSH config (used in `package` URL): a single alias, not a pattern - `hostname` _(required)_ — Actual domain (e.g., `github.com`) - `known_hosts` _(optional)_ — Public SSH keys of the host. Auto-retrieved for `github.com` -- `replicated` _(optional)_ — Replicate to all cluster nodes +- `replicated` _(optional)_ — On Harper Pro and Fabric, replicate to all cluster nodes by default; pass `false` to apply on this node only. Harper core on its own does not replicate A `key`, `host` or `hostname` that ssh couldn't use is refused with a `400` naming the problem; see [what `key`, `host` and `hostname` must be](../operations-api/operations.md#what-key-host-and-hostname-must-be). Each key's settings live in a block of `/ssh/config` that Harper owns; you can edit the file outside those blocks (see [the key's block in the ssh config](../operations-api/operations.md#the-keys-block-in-the-ssh-config)). diff --git a/reference/configuration/operations.md b/reference/configuration/operations.md index 75d9ed695..ec521f8d4 100644 --- a/reference/configuration/operations.md +++ b/reference/configuration/operations.md @@ -72,7 +72,7 @@ The `replicated` array reports per-node outcomes. A failed peer appears as `{ "s } ``` -To restart the whole cluster afterward, follow with [restart_service](../operations-api/operations.md#restart_service) using `"replicated": true` — it restarts nodes one at a time, so the cluster stays available: +To restart the whole cluster afterward, follow with [restart_service](../operations-api/operations.md#restart_service) using `"replicated": true` — it restarts nodes one at a time, so they never all restart at once. Each node stays in rotation while it restarts, so drain a node first if it must not serve during the change: ```json { diff --git a/reference/configuration/options.md b/reference/configuration/options.md index d12128410..47841fa85 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -232,7 +232,7 @@ logging: level: warn root: ~/hdb/log stdStreams: false - auditLog: false + auditLog: true rotation: interval: 1D maxSize: 100M @@ -243,8 +243,8 @@ logging: - `root` — Log directory; _Default_: `/log` - `path` — Explicit log file path (overrides `root`) - `stdStreams` — Write to stdout/stderr; _Default_: `false` -- `console` — Include `console.*` output; _Default_: `true` -- `auditLog` — Enable table transaction audit logging; _Default_: `false` +- `console` — Include `console.*` output in the log file; _Default_: `false` +- `auditLog` — Enable table transaction audit logging; _Default_: `true` - `auditRetention` — Audit log retention duration; _Default_: `3d` - `external` — Logging for components using the logger API; sub-options: `level`, `path` - `rotation.enabled` / `rotation.compress` / `rotation.interval` / `rotation.maxSize` / `rotation.path` — Log file rotation (activates when `interval` or `maxSize` is set) diff --git a/reference/configuration/overview.md b/reference/configuration/overview.md index 112327b24..c2fc1e65b 100644 --- a/reference/configuration/overview.md +++ b/reference/configuration/overview.md @@ -299,7 +299,7 @@ logging: Setting `HARPER_SAFE_MODE` to any value starts Harper without loading user applications or components. Harper's core services (database, operations API, HTTP server) start normally, but no applications from the components directory are loaded and no package-based extensions are initialized. ```bash -HARPER_SAFE_MODE=1 harperdb +HARPER_SAFE_MODE=1 harper run ``` This is useful when a broken or misbehaving component prevents Harper from starting. Safe mode lets you access the operations API to inspect, repair, or remove the problematic component without needing to manually edit files on disk. diff --git a/reference/http/api.md b/reference/http/api.md index 4d0646210..ec7b5186a 100644 --- a/reference/http/api.md +++ b/reference/http/api.md @@ -40,7 +40,7 @@ To continue the middleware chain, call `next(request)`. To short-circuit, return ### `HttpOptions` - (`name`, `before`, `after`, `urlPath`, and `host`) + (`name`, `before`, `after`, `urlPath`, and `host`) | Property | Type | Default | Description | | ------------ | ------- | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | diff --git a/reference/http/overview.md b/reference/http/overview.md index a33be64a4..3c7c09a3e 100644 --- a/reference/http/overview.md +++ b/reference/http/overview.md @@ -85,7 +85,7 @@ A mount is a routing prefix, not an isolation boundary: Host matching ignores the port and is case-insensitive. IPv6 hosts are configured as a bare literal (`::1`), not bracketed. -Custom components can configure the same behavior programmatically with `server.http(listener, { host, urlPath })`. See [`HttpOptions`](./api#httpoptions) for matching priority and middleware ordering options. +Custom components can configure handler routing programmatically with `server.http(listener, { host, urlPath })`. These handler options have been available since v5.1.0; the root-configuration and deployment options above were added in v5.2.0. See [`HttpOptions`](./api#httpoptions) for matching priority and middleware ordering options. ## Protocols Served diff --git a/reference/logging/configuration.md b/reference/logging/configuration.md index f204b9325..a607d10c7 100644 --- a/reference/logging/configuration.md +++ b/reference/logging/configuration.md @@ -106,7 +106,7 @@ See [Database / Transaction Logging](../database/transaction.md) for details on ```yaml logging: - auditLog: false + auditLog: true ``` ### `logging.auditRetention` @@ -339,7 +339,7 @@ logging: file: true stdStreams: false console: false - auditLog: false + auditLog: true auditRetention: 3d rotation: enabled: true diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 21815c98a..caf233ed8 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -116,7 +116,7 @@ Creates a new database. ### `drop_database` -Drops a database and all its tables/records. Supports `"replicated": true` to propagate to all cluster nodes. +Drops a database and all its tables/records. **Replicates to all cluster nodes by default** — pass `"replicated": false` to drop it on this node only. ```json { "operation": "drop_database", "database": "dev" } @@ -137,7 +137,7 @@ Creates a new table. Optional fields: `database` (defaults to `data`), `attribut ### `drop_table` -Drops a table and all associated records. Supports `"replicated": true`. +Drops a table and all associated records. **Replicates to all cluster nodes by default** — pass `"replicated": false` to drop it on this node only. ```json { "operation": "drop_table", "database": "dev", "table": "dog" } @@ -1730,14 +1730,14 @@ Detailed documentation: [WAF Operations and Rule Schema](../web-application-fire Operations for restarting Harper and managing system state. -| Operation | Description | Role Required | -| -------------------- | ----------------------------------------------------- | ------------- | -| `restart` | Restarts the Harper instance | super_user | -| `restart_service` | Restarts a specific Harper service | super_user | -| `system_information` | Returns detailed host system metrics | super_user | -| `set_status` | Sets an application-specific status value (in-memory) | super_user | -| `get_status` | Returns one status value, or the instance summary | super_user | -| `clear_status` | Removes a status entry | super_user | +| Operation | Description | Role Required | +| -------------------- | ------------------------------------------------- | ------------- | +| `restart` | Restarts the Harper instance | super_user | +| `restart_service` | Restarts a specific Harper service | super_user | +| `system_information` | Returns detailed host system metrics | super_user | +| `set_status` | Sets an application-specific status value | super_user | +| `get_status` | Returns one status value, or the instance summary | super_user | +| `clear_status` | Removes a status entry | super_user | ### `restart` @@ -1767,7 +1767,11 @@ Set `scope` to an [isolated application's name](../components/applications.md#is ### `system_information` -Returns system metrics including CPU, memory, disk, network, and Harper process info. Optionally filter by `attributes` array (e.g., `["cpu", "memory", "replication"]`). With `attributes: ["threads"]`, the response includes a `threads` array of workers. Each entry has a `threadId`; a dedicated isolated worker also has `application: ""`. These describe the node that answers, not its peers. +Returns system metrics including CPU, memory, disk, network, and Harper process info. Optionally filter by an `attributes` array (e.g., `["cpu", "memory", "threads"]`). With `attributes: ["threads"]`, the response includes a `threads` array of workers. Each entry has a `threadId`; a dedicated isolated worker also has `application: ""`. These describe the node that answers, not its peers. + +Valid attribute names are `system`, `time`, `cpu`, `memory`, `disk`, `network`, `harperdb_processes`, `table_size`, `metrics`, and `threads`. Omitting `attributes` returns all of them. + +> **Unrecognized attribute names are silently ignored**, not rejected. A typo produces a smaller response rather than an error, so if a section you asked for is missing, check the spelling before you check the node. ```json { "operation": "system_information" } @@ -1775,7 +1779,13 @@ Returns system metrics including CPU, memory, disk, network, and Harper process ### `set_status` / `get_status` / `clear_status` -Manage in-memory application status values. Status types: `primary`, `maintenance`, `availability` (availability only accepts `'Available'` or `'Unavailable'`). Status is not persisted across restarts. +Manage application status values. The status types are fixed — `primary`, `maintenance`, `availability` — while the value you store against each is yours to define (availability only accepts `'Available'` or `'Unavailable'`). `set_status` stores only `status`; any other field in the request is ignored. + +Values are **persisted**, stored in the `hdb_status` table in the `system` database, so they survive a restart. That table is declared `replicate: false`, so a status set on one node stays on that node and does not propagate to peers — which is what makes these values usable as node-local coordination markers. + +These are a coordination primitive for your own automation: Harper core stores and returns them but does not change its own behavior based on them, so they are not a substitute for a real health check. A `get_status` call with no `id` additionally returns Harper-derived state, including a `restartRequired` flag that tracks pending component and code restarts (not configuration changes). + +On Harper Fabric, the `status-check` component answers the load balancer from the `availability` record, so that record decides whether a node receives traffic. See [Load Balancing and Node Availability](/fabric/load-balancing). ```json { "operation": "set_status", "id": "primary", "status": "active" } diff --git a/reference/replication/clustering.md b/reference/replication/clustering.md index 194418bb5..29c8955bd 100644 --- a/reference/replication/clustering.md +++ b/reference/replication/clustering.md @@ -27,6 +27,7 @@ Adds a new Harper instance to the cluster. If `subscriptions` are provided, it c - `revoked_certificates` _(optional)_ — array of revoked certificate serial numbers that will not be accepted for any connections - `shard` _(optional)_ — shard number for this node. Only needed when using sharding - `start_time` _(optional)_ — ISO 8601 UTC datetime. If set, only data after this time is downloaded during initial synchronization instead of the entire database +- `isLeader` _(optional)_ — identify the added node as a bootstrap source for this node. This does not grant primary ownership or write authority, or prevent full-copy requests to other peers for which this node has no resume cursor. Subsequent replication still follows the configured routes - `subscriptions` _(optional)_ — explicit table-level replication relationships. This is optional (and discouraged). Each subscription is an object with: - `database` — database name - `table` — table name @@ -172,12 +173,12 @@ Returns an array of status objects from the cluster, including active WebSocket "database": "data", "connected": true, "latency": 0.7, - "thread_id": 1, + "threadId": 1, "nodes": ["server-2.domain.com"], "lastCommitConfirmed": "Wed, 12 Feb 2025 19:09:34 GMT", "lastReceivedRemoteTime": "Wed, 12 Feb 2025 16:49:29 GMT", "lastReceivedLocalTime": "Wed, 12 Feb 2025 16:50:59 GMT", - "lastSendTime": "Wed, 12 Feb 2025 16:50:59 GMT" + "sendingMessage": "Wed, 12 Feb 2025 16:50:59 GMT" } ] } @@ -196,6 +197,8 @@ Returns an array of status objects from the cluster, including active WebSocket | `lastReceivedLocalTime` | Local time when the last transaction was received. A gap between this and `lastReceivedRemoteTime` suggests the node is catching up | | `sendingMessage` | Timestamp of the transaction actively being sent. Absent when waiting for the next transaction | +While a database is taking its initial full copy, these timing fields hold the literal string `"Copying"` rather than a timestamp. Anything that parses them as dates needs to handle that value, since a node taking a full copy is exactly the node you are most likely to be watching. + --- ### Configure Cluster diff --git a/reference/replication/overview.md b/reference/replication/overview.md index 107e52ecf..9ae1a26d7 100644 --- a/reference/replication/overview.md +++ b/reference/replication/overview.md @@ -360,9 +360,9 @@ See [Clustering Operations](./clustering.md#cluster-status) for the full respons ## Initial Synchronization and Resynchronization -When a new node is added and its database has not been previously synced, Harper downloads the full database from the first node it connects to. After the initial sync completes, the node enters replication mode and receives incremental updates. +Initial synchronization is decided per peer and per database. A new node can request a full copy from each in-scope peer for which it has no resume cursor, so a join can load several peers at once. [`add_node` with `isLeader: true`](./clustering.md#add-node) identifies a bootstrap source; it does not prevent full-copy requests to other cursorless peers. After synchronization, replication continues incrementally. -If a node goes offline and comes back, it resynchronizes automatically to catch up on missed transactions. +If a node goes offline and comes back, it resynchronizes automatically to catch up on missed transactions. Its peers keep that history for [`logging.auditRetention`](../logging/configuration.md#loggingauditretention); a node away longer than that gets a full copy instead, which can leave rows on it that were deleted elsewhere while it was away. Decide how to reconcile such a node before it reconnects. You can also specify a `start_time` in the `add_node` operation to limit the initial download to data since a given point in time: @@ -384,11 +384,16 @@ The following data operations are replicated across the cluster: - Delete - Bulk loads (CSV data load, CSV file load, CSV URL load, import from S3) -**Destructive schema operations are not replicated**: `drop_database`, `drop_table`, and `drop_attribute` must be run on each node independently. +**Destructive schema operations differ from each other, so check before you run one:** -Users and roles are not replicated across the cluster by default. They do propagate when the `system` database (where `hdb_user` and `hdb_role` live) is included in replication; as of v5.2 this no longer forces a full mesh — see [Replicating the `system` database with controlled flow](#replicating-the-system-database-with-controlled-flow). +- `drop_database` and `drop_table` **replicate to the whole cluster by default** on Pro and Fabric. Pass `"replicated": false` to drop on one node only. +- `drop_attribute` is not replicated and must be run on each node independently. -Certain management operations — including component deployment and rolling restarts — can also be replicated across the cluster. +Data mutations, `drop_database`, `drop_table`, and component deployment replicate by default on Pro and Fabric; pass `"replicated": false` to limit them to one node. [`set_configuration`](../configuration/operations.md#set-configuration) and [`restart_service`](../operations-api/operations.md#restart_service) are opt-in: pass `"replicated": true` for a cluster-wide configuration update or rolling restart. + +Users and roles **do** propagate by default. They live in the `system` database, and the default replication scope is every database, so `system` is in scope unless you have narrowed it. As of v5.2 replicating `system` no longer forces a full mesh — see [Replicating the `system` database with controlled flow](#replicating-the-system-database-with-controlled-flow). If you narrow the scope to exclude `system`, users and roles stop propagating and must be provisioned on every node by your own automation. + +Check each management operation's replication default before using it in a single-node procedure. ## Inspecting Cluster Configuration diff --git a/release-notes/v5-lincoln/5.2.md b/release-notes/v5-lincoln/5.2.md index a189ac0df..ef1a16c0d 100644 --- a/release-notes/v5-lincoln/5.2.md +++ b/release-notes/v5-lincoln/5.2.md @@ -10,7 +10,7 @@ All patch release notes for 5.2.x are available on the [releases page](https://g ## Backup & Restore -New managed-backup operations for RocksDB databases: `create_backup`, `list_backups`, `verify_backup`, `delete_backup`, `purge_backups`, and `restore_backup`. Managed backups are a server-side repository under the configured backup root (`storage.backupPath`, default `/backup`); the RocksDB data files are shared and incremental across backups, while the audit/transaction log — and, for databases with file-backed blobs, the blobs (skip with `exclude_blobs`) — are captured in full on every backup. `verify_backup` checks the RocksDB files (checksums with `verify_checksum`) and the transaction-log snapshot; it does not verify blobs. `restore_backup` restores a user database in place without stopping Harper, as long as no loaded component is holding that database open; a database held open by a component — and always the `system` database — is restored offline by running `restore_backup` from the CLI with the server stopped. Each of these operations can be run from the CLI under its operation name (e.g. `harper create_backup database=data`): while Harper is running the operation is forwarded to the server, and while it is stopped the command operates directly on the files. See [Backups](/reference/v5/backups/overview) and [Backup Operations](/reference/v5/backups/operations). +New managed-backup operations for RocksDB databases: `create_backup`, `list_backups`, `verify_backup`, `delete_backup`, `purge_backups`, and `restore_backup`. Managed backups are a server-side repository under the configured backup root (`storage.backupPath`, default `/backup`); the RocksDB data files are shared and incremental across backups, while the audit/transaction-log snapshot is copied on every backup. Blob snapshots use hard links when supported on the same filesystem and copies otherwise (skip with `exclude_blobs`). `verify_backup` checks the RocksDB files (checksums with `verify_checksum`) and the transaction-log snapshot; it requires a blob snapshot directory when the backup recorded blobs, but does not validate individual blob contents. `restore_backup` restores a user database in place without stopping Harper, as long as no loaded component is holding that database open; a database held open by a component — and always the `system` database — is restored offline by running `restore_backup` from the CLI with the server stopped. Each of these operations can be run from the CLI under its operation name (e.g. `harper create_backup database=data`): while Harper is running the operation is forwarded to the server, and while it is stopped the command operates directly on the files. See [Backups](/reference/v5/backups/overview) and [Backup Operations](/reference/v5/backups/operations). `get_backup` now understands RocksDB: it streams a full-snapshot `tar` of the database — including any file-backed blobs (pass `exclude_blobs: true` to omit them), gzipped by default (pass `gzip: false` for a plain `tar`) — instead of failing. The LMDB behavior (streaming the `.mdb`, with `table`/`tables`/`include_audit`) is unchanged. @@ -86,9 +86,9 @@ Because Harper keeps one refresh-token hash per user, issuing a token revokes th ### Middleware routing and ordering -Applications can now be routed by virtual hostname, URL prefix, or both, with no dispatch code. Declare `host` and `urlPath` on the application's entry in the root `harper-config.yaml` — or pass them to `deploy_component` — and every handler the application registers is served under that hostname and path. Because the routing lives in the root config, the same application can be mounted differently per environment without editing it. A plugin's own `urlPath` still positions it within the application, and the application's mount is prefixed onto it. +Applications can now declare virtual-hostname and URL-prefix routing in the root configuration or at deployment time. Declare `host` and `urlPath` on the application's entry in the root `harper-config.yaml` — or pass them to `deploy_component` — and every handler the application registers is served under that hostname and path. Because the routing lives in the root config, the same application can be mounted differently per environment without editing it. A plugin's own `urlPath` still positions it within the application, and the application's mount is prefixed onto it. -Components can also pass `host` and `urlPath` directly to `server.http()`, `server.ws()`, and `server.upgrade()`. The new `name`, `before`, and `after` options provide explicit middleware ordering. See [HTTP middleware routing](/reference/v5/http/overview#middleware-routing) and [`HttpOptions`](/reference/v5/http/api#httpoptions). +Components can still pass `host` and `urlPath` directly to `server.http()`, `server.ws()`, and `server.upgrade()`, as in v5.1. The `name`, `before`, and `after` ordering options are also available since v5.1. See [HTTP middleware routing](/reference/v5/http/overview#middleware-routing) and [`HttpOptions`](/reference/v5/http/api#httpoptions). ## Security diff --git a/sidebarsFabric.ts b/sidebarsFabric.ts index 529e87780..61413a3f8 100644 --- a/sidebarsFabric.ts +++ b/sidebarsFabric.ts @@ -32,6 +32,11 @@ const sidebarsFabric: SidebarsConfig = { id: 'custom-domains', label: 'Custom Domains', }, + { + type: 'doc', + id: 'load-balancing', + label: 'Load Balancing & Node Availability', + }, { type: 'doc', id: 'managing-applications', diff --git a/sidebarsLearn.ts b/sidebarsLearn.ts index 721b1690c..6282dbd6f 100644 --- a/sidebarsLearn.ts +++ b/sidebarsLearn.ts @@ -71,6 +71,11 @@ const sidebarsLearn: SidebarsConfig = { id: 'developers/deploying-from-ci', label: 'Deploying from a CI/CD Pipeline', }, + { + type: 'doc', + id: 'developers/taking-a-node-out-of-rotation', + label: 'Taking an Unhealthy Node Out of Rotation', + }, ], }, {