Document Fabric load balancing, status-check, and taking an unhealthy node out of rotation

[dawsontoth]

⊙ Problem

Fabric's load balancer sends new connections to every node whose GET /status answers 200. That answer comes from the status-check component Fabric deploys, which reads the node's availability record. So a node keeps getting traffic while its application fails, and no page covers the component or a safe way out.

❓ Your call: is a guide with a sample loop warranted?

Yes: the pattern already runs in production on Fabric, and without its rules a loop can fight Fabric or take every node out. The alternatives are documenting only the manual drain, or shipping the loop as a published component so copies get fixes; both are easy to move to later.

💡 Solution

• A Learn guide shows an application how to leave rotation while failing and return once recovered.
• Six rules keep it from undoing Fabric's writes or taking the last node out. A sample component follows them; a local drill tests each.
• A Fabric page documents status-check, and shows how to take a node out with the Harper CLI, from a laptop or GitHub Actions with OIDC, without a super user.
• The set_status reference now says it stores only status, and links the Fabric page.

❓ Your call: fix #671's Administration page there or here?

#671's traffic-admission guide says nothing in Harper acts on set_status, and has Fabric readers deploy the npm @harperdb/status-check, which collides with Fabric's /status. This PR only reconciles from the Fabric side. I'd scope those lines to self-managed clusters in #671; editing them in this stacked PR is the alternative.

❓ Your call: are the Fabric facts right?

The pages say only what I could verify in source or on a live Harper, or was told. A Fabric owner should confirm these before the PR leaves draft.

Questions for a Fabric owner

1. The load balancer's check: interval, timeout, how many failures take a node out, and whether anything but 200 passes. The guide's 30 s and 60 s don't allow for it.
2. Is the balanced name DNS-based? Then cached answers keep clients on a node that went out until the TTL ends; the caveat names only open connections.
3. Where customers find each node's own hostname, and whether its Operations API answers on :9925, where the CLI sends a target with no port.
4. Does Fabric set Available on new nodes, and after an upgrade even when the node was Unavailable before it?
5. Does status-check show in Studio's Applications list, where a customer could remove it?
6. Should server.workerIndex, which the guide uses, get an entry in reference/http/api.md?
7. The public status-check-fabric README says table writers may POST and DELETE; only super users can (a non-super user gets 403). Fix it upstream?
8. Two harper CLI faults on 5.3.2, worked around here: harper restart printed "Harper is already running" and restarted nothing, and harper get_status id=availability on a node with no record errors after printing null.

🔧 Changes

• learn/developers/taking-a-node-out-of-rotation.mdx: the guide. The rules, the sample, the drill and the caveats.
• fabric/load-balancing.md: the status-check reference, its routes, a warning not to remove it, and the CLI and OIDC route.
• reference/operations-api/operations.md: two additions on top of Add production reliability guides for Harper operators #671's rewrite of this paragraph, and "Harper core" where it says Harper does not act on status values.
• sidebarsFabric.ts and sidebarsLearn.ts: register the two pages.

✅ Verification

• Checked against harper source at v5.3.2: the record's table is persisted with replicate: false, set_status stores only { status }, server.operation() skips authorization by default, and the set_status role grant first shipped in v5.3.0.
• Ran the guide's code, copied from the formatted page, on a throwaway Harper 5.3.2 beside the real status-check, including the harper login flow and a role limited to set_status. Every drill step behaved as the guide says (details below).
• npm run format:check is clean. npm run build fails only on Add production reliability guides for Harper operators #671's own TODO link; with that link patched locally, it passes and every new link resolves.

Live results on Harper 5.3.2

• /status: 404 with no record, 200 with Available. DELETE: 401 anonymous, 403 non-super user, 204 super user. The record survived a full stop and start.
• A role whose operations list is set_status and get_status ran set_status; list_users and drop_user were refused, and its POST /status got 403.
• The loop ran on one worker of two. It went out 30-35 s after the dependency failed, and came back 60-66 s after it recovered.
• With the other node's /status at 404, it stayed in and warned once; it went out at the next look once that node returned.
• An operator's Available while out was left alone, including across a stop and start (46 s, still failing). An operator's Unavailable was never undone.
• When the other node went out with it, it came back 1.5 s later, logged that it would stay in until a restart, and did.
• A pending marker claimed an Unavailable written 0.5 s after it, and left alone one written 10 s after it.
• 200 concurrent /Health requests reached the dependency once.

_{🤖 Generated by Claude Opus 5.5 via Claude Code; posted via @dawsontoth.}

Related PRs: #671 overlaps (this PR is stacked on it; same set_status paragraph), 10 others independent
Complexity: medium

_{Review-Coverage: authored=claude; ran=cursor-composer,codex,gemini,cursor-muse; adjudicated=domain; declined=cursor-grok,cursor-kimi; rounds=4; full=2 @ eed055a}

_{Review-Attention: study ~9m (decisions: do-less-alternative, ownership-by-timestamp-inference, aside-suspends-loop, health-cache-shape, app-writes-lb-record, document-external-component-contract; raised: open major) @ eed055a}

[gemini-code-assist]

Code Review

This pull request introduces new documentation and guides regarding load balancing, node availability, and taking unhealthy nodes out of rotation in Harper Fabric, complete with a sample JavaScript implementation for an automated availability loop. It also updates the operations API reference to clarify that status records are persisted in the system.hdb_status table and survive restarts. Feedback on the code suggests optimizing the node-checking logic by parsing and comparing hostnames locally to avoid redundant self-requests.

[Ethan-Arrowood]

Looks to be some conflicts.

[Ethan-Arrowood]

Otherwise great work on this. I like it!

[github-actions]

🧹 Preview Cleanup

The preview deployment for this PR has been removed.