diff --git a/reference/configuration/options.md b/reference/configuration/options.md index 7416746a..888fc6c3 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -469,16 +469,18 @@ Some behavior worth knowing: ## `deployment` -Retention for what `deploy_component` leaves behind. See [`deploy_component`](../operations-api/operations.md#deploy_component). +Retention for what `deploy_component` leaves behind, and how long startup waits for component installs. See [`deploy_component`](../operations-api/operations.md#deploy_component). ```yaml deployment: + startupInstallTimeout: 600000 payloadRetention: maxSize: 10485760 stagingRetention: maxCount: 5 ``` +- `startupInstallTimeout` — Milliseconds. The longest startup waits for component installs before it opens listeners. While waiting, it logs every 60 seconds which ones remain outstanding — including throughout `0`'s unbounded wait — unless every install settles first, which always ends the wait before that first log, whatever the timeout; a failed install logs immediately regardless of the timeout. A component whose install is still running at the deadline keeps installing in the background, and the node serves the version already installed meanwhile, or nothing for a component that was never installed. When that install succeeds, the swap lands on disk immediately and Harper flags a restart as needed (`restartRequired` in [`get_status`](../operations-api/operations.md#set_status--get_status--clear_status) called with no `id`) — any one component's late success sets it, even while others are still installing. Code already loaded keeps running until that restart, but some component plugins apply an added or changed file on their own rather than waiting for one; content served straight from disk, such as static files, can also reflect the new version before the restart. Outside `harper dev`, which restarts automatically soon after any restart request, Harper does not restart on its own. An install that fails after the deadline does not set the flag; Harper only logs an error naming the component. `0` waits indefinitely, which was the behavior before v5.3.0. An empty value uses the default. A negative, non-finite, or otherwise unparseable value also falls back to it, with a warning in the log; _Default_: `600000` (10 minutes) - `payloadRetention.maxSize` — Bytes. After a successful deploy, a payload larger than this has its stored tarball (`payload_blob`) dropped from the `hdb_deployment` row; the row and its metadata stay. Set it very high to retain every payload; _Default_: `10485760` (10 MiB) - `stagingRetention.maxCount` — How many complete builds that are not live may remain per component under `/.deploy-staging`: builds staged with `activate: false`, and [releases replaced by a later deploy](../operations-api/operations.md#going-back-to-a-previous-release), which `deployment_id` can make live again. The newest survive; older ones are removed when that component next deploys and at startup. `drop_component` removes all of them. Each is a full installed copy of the component, `node_modules` included. `0` keeps none, and no replaced release is kept. This bounds dormant builds only, not a disk quota: staging directories that belong to an in-flight or unsettled deploy are never touched; _Default_: `5` diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 9e6f25b8..a2330a51 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -1133,6 +1133,8 @@ Two kinds of deploy set it: - A component that had no directory before this deploy. It has never been loaded, so its routes cannot be live until Harper restarts. - A redeploy whose package metadata changed (v5.2.1) — a dependency or module-entry change invalidates loaded code, and package metadata sits outside most plugin file globs, so the component's own watcher does not see it. +A third source is not a deploy at all: a component install still running when [`deployment.startupInstallTimeout`](../configuration/options.md#deployment) expires at startup sets the flag if it later succeeds (v5.3.0). + An ordinary redeploy sets nothing: the component's watched files are handled by its file watcher, which requests a restart only when the update needs one. The flag is evaluated per node. A peer applying the replicated deploy checks its own directory state, since whether the component was already active can differ from node to node. @@ -1793,12 +1795,12 @@ With an `id`, it returns that one status record: { "operation": "get_status" } ``` -| Field | Description | -| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `systemStatus` | Every status record set with `set_status` | -| `componentStatus` | Per-component health, aggregated across threads: `name` plus a `status` of `healthy`, `warning`, `error`, `loading`, or `unknown` | -| `restartRequired` | Whether a restart is pending on this node — set by a change that needs one to take effect, such as a deploy that did not restart (see [Deploying without a restart](#deploying-without-a-restart)) or an edit to a loaded `.env` file (see [Environment File Operations](#environment-file-operations)), and cleared on restart | -| `middlewareChains` | The resolved HTTP, upgrade, and WebSocket middleware order (v5.2.0). Present only when the request passes `middleware: true` | +| Field | Description | +| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `systemStatus` | Every status record set with `set_status` | +| `componentStatus` | Per-component health, aggregated across threads: `name` plus a `status` of `healthy`, `warning`, `error`, `loading`, or `unknown` | +| `restartRequired` | Whether a restart is pending on this node — set by a deploy that did not restart (see [Deploying without a restart](#deploying-without-a-restart)) or by a component install that succeeded after startup stopped waiting for it (v5.3.0; see [`deployment.startupInstallTimeout`](../configuration/options.md#deployment)), and cleared on restart | +| `middlewareChains` | The resolved HTTP, upgrade, and WebSocket middleware order (v5.2.0). Present only when the request passes `middleware: true` | `restartRequired` is per node: it reports the node that served the request, not the cluster. diff --git a/release-notes/v5-lincoln/5.3.md b/release-notes/v5-lincoln/5.3.md index b28b0a9a..98490c15 100644 --- a/release-notes/v5-lincoln/5.3.md +++ b/release-notes/v5-lincoln/5.3.md @@ -132,6 +132,16 @@ Keys already stored are not re-checked, and a value already sealed as `enc:v1:` Each SSH key's block in `/ssh/config` now runs from its `#` line through a `# END harper ssh key ` line, with a `# BEGIN harper ssh key ` line directly under the first. `get_ssh_key`, `list_ssh_keys` and `delete_ssh_key` read those lines instead of guessing where a block ends, so a line you add outside a key's lines is never removed with the key or read as its setting. Each node adds the lines to its existing config when it starts, without changing what ssh resolves, and a node rolled back to an earlier version still reads and deletes the blocks as it did before. See [The key's block in the ssh config](/reference/v5/operations-api/operations#the-keys-block-in-the-ssh-config). +### Bounded Startup Install Wait + +Startup no longer waits indefinitely for component installs. Previously, listeners opened only after every startup component install finished, so one install that stalled — a git fetch that stopped responding, a process stuck in I/O — kept the whole node down with nothing logged after the install began. Startup now waits at most `deployment.startupInstallTimeout` (default 10 minutes; `0` restores the unbounded wait) and logs the components it is still waiting for every 60 seconds ([harper#2072](https://github.com/HarperFast/harper/issues/2072)). + +An install still running at the deadline is not cancelled. It keeps running in the background while the node serves the version already installed, or runs without the component if it was never installed. + +**During that window the node reports up while that application returns 404 or runs its previous version.** When the install succeeds, `get_status` called with no `id` reports `restartRequired: true`. Restart the node to load it. An install that fails after the deadline does not set the flag; Harper only logs the error. + +**After upgrading, a node whose first install of a component takes longer than 10 minutes starts without that component.** Raise the timeout, or set it to `0` to keep the previous behavior. See [Configuration Options](/reference/v5/configuration/options#deployment). + ## Backups ### Subscriptions End When Their Database Is Restored (5.3.1)