diff --git a/reference/components/applications.md b/reference/components/applications.md index 07fb7db33..2cf06274a 100644 --- a/reference/components/applications.md +++ b/reference/components/applications.md @@ -534,6 +534,8 @@ Packages a project folder as a base64-encoded `.tar` string. Returns all local component files, folders, and configuration from `harper-config.yaml`. The response includes an `entries` array; each direct child's `name` identifies a component. CI cleanup can inspect `entries[].name` to check whether the project is present on this node. + An entry for a `.env` file carries `protected: true`, unless it is a template such as `.env.example`. See [Environment File Operations](../operations-api/operations.md#environment-file-operations). + ```json { "operation": "get_components" @@ -544,6 +546,8 @@ Returns all local component files, folders, and configuration from `harper-confi Returns the contents of a file within a component project. + A `.env` file is returned **masked**: one `KEY=********` line per key, never a value. Template files such as `.env.example` are returned as they are. See [Environment File Operations](../operations-api/operations.md#environment-file-operations). + - `project` _(required)_ — Project name - `file` _(required)_ — Path relative to project folder - `encoding` _(optional)_ — File encoding. Defaults to `utf8` @@ -560,6 +564,8 @@ Returns the contents of a file within a component project. Creates or updates a file within a component project. +On a `.env` file it replaces the whole file with the payload, so saving back the masked body from `get_component_file` sets every value to `********`. To change one key and keep the others, use `set_env_value`; see [Environment File Operations](../operations-api/operations.md#environment-file-operations). + - `project` _(required)_ — Project name - `file` _(required)_ — Path relative to project folder - `payload` _(required)_ — File content to write diff --git a/reference/environment-variables/encrypted-values.md b/reference/environment-variables/encrypted-values.md new file mode 100644 index 000000000..724c72209 --- /dev/null +++ b/reference/environment-variables/encrypted-values.md @@ -0,0 +1,119 @@ +--- +id: encrypted-values +title: Encrypted Environment Values +--- + + + +An individual value in a component's `.env` file can be stored as **ciphertext** instead of plaintext. Encrypted values carry the literal marker `enc:v1:` and are decrypted only in memory, at the moment Harper loads the file into `process.env`: + +```bash +# .env +LOG_LEVEL=debug +STRIPE_KEY=enc:v1:eyJraWQiOiJhYzNm... +``` + +Plaintext and encrypted values coexist in the same file — encryption is per-value and opt-in. + +Because a client can encrypt the value **before** sending it, the plaintext never reaches the operations API, its logs, the replication payload, or the on-disk file. The same `enc:v1:` envelope backs the [secrets store](../security/secrets.md), so a value encrypted once works with either. + +:::tip +For production credentials, the [secrets store](../security/secrets.md) is the better mechanism: it is replicated, audited, rotatable, and scopable per component, and it never puts the ciphertext in a file that can be committed to version control. Reach for encrypted `.env` values when you already have a `.env` workflow and want to keep individual values out of plaintext on disk. +::: + +## Requirements + +Decryption requires **secret custody** — the cluster's secrets private key — which is provided by a Harper Pro component. Open-source core recognizes the `enc:v1:` marker but ships no decryptor. + +| Node | Behavior at load | +| -------------------------------- | ------------------------------------------------------------------------------------ | +| Custody registered (Harper Pro) | The value is decrypted, then loaded like a plaintext value. | +| No custody (core, or key absent) | The value is **skipped** with an error logged, and queued for a decryptor. | +| Custody registers later | Queued values are decrypted and loaded as soon as it registers. | +| Value cannot be decrypted | Skipped with an error logged — a wrong key, a tampered envelope, or a malformed one. | + +A decrypted value is loaded under the same [override rule](./overview.md#override-behavior) as a plaintext one, so by default it does not replace a variable that is already set. + +A skipped value is never written to `process.env`, so the ciphertext never becomes the value. If nothing else set that variable, it is **absent**, and your component fails on a missing variable rather than silently receiving an unusable string. If the variable was already set, by the shell or container environment or by an earlier `.env` file, it keeps that value. + +Skipping is never fatal to the node. It still boots, so you can correct a bad value with `set_env_value` and restart the workers, and a node without Pro is not crashed by a replicated encrypted value. + +A decryptor that comes up after component `.env` loading replays the values it missed into `process.env`. Replaying sets the variable but does not rerun code that has already read it. A component that reads the value while it initializes, before custody registers, finds it missing and keeps whatever it did with that result. For such a component, read the variable when you use it rather than at startup, or restart the workers once custody is available. Each deferred value is logged as an error that ends in `deferring`. + +## Encrypting a value + +The flow is the same one the secrets store uses for client-side encryption. + +### 1. Fetch the cluster public key + +`get_secrets_public_key` is `super_user` only and requires custody on the answering node: + +```json +{ "operation": "get_secrets_public_key" } +``` + +Response: + +```json +{ + "public_key": "-----BEGIN PUBLIC KEY-----\n...", + "fingerprint": "" +} +``` + +Cache the key by its `fingerprint`; that value is the `kid` you seal into the envelope. + +### 2. Build the envelope + +An encrypted value is the literal prefix `enc:v1:` followed by the base64url encoding of a JSON envelope. Hybrid encryption is used — AES-256-GCM encrypts the value, RSA-OAEP (SHA-256) wraps the AES key — because the RSA key is too small to directly encrypt multi-line secrets such as PEM keys: + +``` +enc:v1: +``` + +```jsonc +{ + "kid": "", // which key encrypted this + "k": "", + "iv": "", + "ct": "", + "tag": "", +} +``` + +A reference Node.js implementation is in [Secrets — Reference client](../security/secrets.md#reference-client-nodejs); the function it defines returns exactly the string to use as the `value` below. + +`kid` lets multiple keypairs coexist during a rotation: the decryptor selects the matching private key, or rejects the value if it holds no key for that `kid`. + +### 3. Write it with `set_env_value` + +Send the envelope as an ordinary value. Harper stores the string verbatim: + +```json +{ + "operation": "set_env_value", + "project": "my-app", + "key": "STRIPE_KEY", + "value": "enc:v1:" +} +``` + +Like any edit to a `.env` file a component has already loaded, the new value reaches running code when the node's worker threads restart. See [Environment File Operations](../operations-api/operations.md#environment-file-operations). + +:::note +Unlike `set_secret`, `set_env_value` does **not** validate the envelope — it is just a string to the `.env` writer, and there is no server-side check of its structure or `kid`. A malformed or wrongly-keyed envelope is only discovered when the file is loaded, as a decrypt error in the log. `set_secret` performs those checks on ingest, which is one more reason to prefer the [secrets store](../security/secrets.md) for credentials. +::: + +## What stays hidden + +`get_env_keys` and `get_component_file` report key **names** only and mask `.env` values, so an encrypted value is protected on the read surface twice over. See [Environment File Operations](../operations-api/operations.md#environment-file-operations). + +**Protects against:** theft of the on-disk `.env` file, the editor and operations read surface, the value appearing in operation logs and replication payloads, and an operator observing traffic at the TLS-terminating layer. + +**Does not protect against:** a fully compromised running node. A node that holds custody necessarily holds the private key, and the decrypted value is in `process.env` where any component in the process — and any child process it spawns — can read it. This is defense in depth for accidental and at-rest exposure, not protection from a compromised host. + +## Related + +- [Environment Variables](./overview.md) — the `loadEnv` plugin and `.env` loading +- [Secrets](../security/secrets.md) — the encrypted, replicated secrets store +- [Environment File Operations](../operations-api/operations.md#environment-file-operations) — `get_env_keys`, `set_env_value`, `delete_env_value` diff --git a/reference/environment-variables/overview.md b/reference/environment-variables/overview.md index 2cbe03aa1..9e056e71d 100644 --- a/reference/environment-variables/overview.md +++ b/reference/environment-variables/overview.md @@ -14,7 +14,7 @@ If you are looking for information on how to configure your Harper installation ::: :::tip -For production credentials, prefer the encrypted, replicated [secrets store](../security/secrets.md) over a committed `.env` file. Secrets are stored as ciphertext, delivered to components via `process.env` or a per-component `secrets` accessor, and never appear in operation logs or replication payloads. The same `enc:v1:` envelope format also lets you encrypt individual `.env` values at rest. +For production credentials, prefer the encrypted, replicated [secrets store](../security/secrets.md) over a committed `.env` file. Secrets are stored as ciphertext, delivered to components via `process.env` or a per-component `secrets` accessor, and never appear in operation logs or replication payloads. The same `enc:v1:` envelope format also lets you [encrypt individual `.env` values](./encrypted-values.md) at rest. ::: ## Configuration @@ -60,6 +60,14 @@ Harper's own instance-wide configuration is composed once at startup, **before** To change Harper's configuration, set it in the [configuration file](../configuration/overview.md) or export the variable in the real process/container environment **before** Harper starts — not through `loadEnv`. ::: +## Managing `.env` Files Remotely + + + +A component's `.env` file can be edited through the Operations API without exposing the values already in it. `get_env_keys` lists key names only, `set_env_value` writes one or many keys while preserving every other line, and `delete_env_value` removes keys. `get_component_file` returns a `.env` file **masked** — key names and one `KEY=********` line per key, never a value. An edit to a file `loadEnv` has already loaded reaches running code only after the node's worker threads restart. See [Environment File Operations](../operations-api/operations.md#environment-file-operations). + +Individual values can also be stored encrypted, so the plaintext never reaches the API, the logs, the replication payload, or the file on disk. See [Encrypted Environment Values](./encrypted-values.md). + ## Override Behavior By default, `loadEnv` follows the standard dotenv convention: **existing environment variables take precedence** over values in `.env` files. This means variables already set in the shell or container environment will not be overwritten. @@ -94,5 +102,7 @@ Files are loaded in the order specified. ## Related +- [Encrypted Environment Values](./encrypted-values.md) +- [Secrets](../security/secrets.md) - [Components Overview](../components/overview.md) - [Configuration](../configuration/overview.md) diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 23e1e330b..a0c8fbd75 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -854,6 +854,9 @@ Detailed documentation: [Components Overview](../components/overview.md) | `get_components` | Lists all component files and config | super_user | | `get_component_file` | Returns the contents of a file within a component | super_user | | `set_component_file` | Creates or updates a file within a component | super_user | +| `get_env_keys` | Lists the key names in a component's `.env` file (never values) | super_user | +| `set_env_value` | Sets one or many `.env` keys, preserving the rest of the file | super_user | +| `delete_env_value` | Removes one or many `.env` keys, leaving the rest intact | super_user | | `list_deployments` | Lists deployment records with optional filters | super_user | | `get_deployment` | Fetches a single deployment record by ID; supports SSE streaming | super_user | | `get_deployment_payload` | Returns the tarball stored for a deployment | super_user | @@ -1175,6 +1178,114 @@ From v5.4.0, the `load` phase brackets the [canary's decision](#certifying-a-rel Every event but `done` is also appended to the deployment's `event_log`, which [`get_deployment`](#get_deployment) replays as a stream. A replayed log can also contain `truncated`, `{ dropped_events }`, where the log reached its 200-entry bound and dropped entries from its middle, keeping the start and the most recent, and a `payload_dropped` carrying `deleted_by` instead of `max_size` when [`delete_deployment_payload`](#delete_deployment_payload) reclaimed the tarball. +### Environment File Operations + + + +Three operations edit a component's `.env` file without ever returning a value. They let an editor (Studio, CI, your own tooling) rotate one key without seeing — or accidentally clobbering — the other secrets in the file. + +| Operation | Description | Role Required | +| ------------------ | --------------------------------------------------------------- | ------------- | +| `get_env_keys` | Lists the key names in a component's `.env` file — never values | super_user | +| `set_env_value` | Sets one or many keys, preserving the rest of the file | super_user | +| `delete_env_value` | Removes one or many keys, leaving the rest intact | super_user | + +All three take a `project` (the component directory name) and an optional `file`, which defaults to `.env`. `file` must name an env file — `.env` or `.env.` such as `.env.local` — and is rejected otherwise. Key names must match `[A-Za-z0-9_.-]+`, the character set the `.env` parser accepts. + +`set_env_value` and `delete_env_value` replicate. The node that receives the request edits its own file, then sends the edit once to each peer. The response's `replicated` array has one entry per peer, naming it in `node`. A peer that could not be reached, or that answered with an error, is listed with `"status": "failed"` and a `reason`. Harper does not retry it, so that node's file may not have the edit. The request still succeeds, so check `replicated` for a `failed` entry. Once that peer is back, send the affected keys' current state again: their current values, or a delete for keys that should still be absent. A resend reaches every node, so resending the original request is safe only if no later edit changed those keys. Deleting a key that is not present is not an error. + +An edit changes the file, not the running components. [`loadEnv`](../environment-variables/overview.md) reads a `.env` file into `process.env` when the component loads, and reads only the files its `files` option names. When a file it has already loaded changes, it does not reload the values. It flags the node as needing a restart instead, so `get_status` reports `restartRequired: true`. Until the node's worker threads restart, running code still sees the old value of a changed key, and still sees a deleted key. To apply an edit, restart the workers on every node, for example with [`restart_service`](#restart_service) using `"service": "http_workers"` and `"replicated": true`, which restarts the nodes one at a time. When `harper dev` runs an application directory, it restarts the workers on its own when the flag is set. + +:::note +`get_component_file` on a `.env` file returns a **masked** body: `protected: true`, the `keys` array, and a `message` of one `KEY=********` line per key. `get_components` likewise flags such files with `protected: true`. Template files — `.env.example`, `.env.sample`, `.env.template` — hold placeholders rather than secrets, so they are returned verbatim and are not masked. + +`set_component_file` is **not** blocked on a `.env` file. It overwrites the file verbatim with whatever payload you send. Use `set_env_value` when you want to change one key and keep the others. + +This is disclosure protection for the editor surface, not a security boundary — component code can still read the real values from `process.env`. +::: + +#### `get_env_keys` + +```json +{ + "operation": "get_env_keys", + "project": "my-app", + "file": ".env" +} +``` + +Response: + +```json +{ + "file": ".env", + "keys": ["API_KEY", "DB_URL"], + "size": 62, + "mtime": "2026-02-11T18:04:22.000Z" +} +``` + +Returns an error if the file does not exist. + +#### `set_env_value` + +Supply exactly one of `key` + `value` (a single entry) or `values` (a key-to-value map). Every other line in the file — comments, blank lines, untouched keys, and their formatting — is left exactly as it was; existing keys are replaced in place and new keys are appended. The file (and the project directory) is created if it does not exist. + +```json +{ + "operation": "set_env_value", + "project": "my-app", + "key": "API_KEY", + "value": "rotated-value" +} +``` + +```json +{ + "operation": "set_env_value", + "project": "my-app", + "values": { "API_KEY": "rotated-value", "DB_URL": "postgres://user:pass@host/db" } +} +``` + +Response: + +```json +{ + "message": "Successfully set env value(s) in .env", + "keys": ["API_KEY", "DB_URL"] +} +``` + +`keys` is the file's key list after the write — the request value is never echoed back. With Harper Pro, the response of both `set_env_value` and `delete_env_value` also carries the [`replicated` array](#environment-file-operations). + +Harper quotes the value as needed so the `.env` parser reads back exactly what you sent, including spaces, `#`, quotes, backslashes, and newlines. A carriage return is the exception: the parser reads `\r\n` and a lone `\r` back as `\n`. One combination cannot be represented: a value containing a single quote (`'`) together with a double quote, a backslash, or a carriage return is rejected with a `400`. + +A value may also be an `enc:v1:` ciphertext envelope rather than plaintext — see [Encrypted Environment Values](../environment-variables/encrypted-values.md). + +#### `delete_env_value` + +Supply exactly one of `key` (a single name) or `keys` (an array of names). + +```json +{ + "operation": "delete_env_value", + "project": "my-app", + "keys": ["API_KEY", "DB_URL"] +} +``` + +Response: + +```json +{ + "message": "Successfully deleted env value(s) from .env", + "keys": [] +} +``` + +Returns an error if the file does not exist. Removing a key that is not present is not an error. + ### Deployment Operations Harper records every `deploy_component` call in the `system.hdb_deployment` table, capturing the full lifecycle of a deployment including phase transitions (prepare → load → replicate → restart → success/failed), per-node outcomes, and a bounded event log of install output. @@ -1682,12 +1793,12 @@ With an `id`, it returns that one status record: { "operation": "get_status" } ``` -| Field | Description | -| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `systemStatus` | Every status record set with `set_status` | -| `componentStatus` | Per-component health, aggregated across threads: `name` plus a `status` of `healthy`, `warning`, `error`, `loading`, or `unknown` | -| `restartRequired` | Whether a restart is pending on this node — set by a deploy that did not restart (see [Deploying without a restart](#deploying-without-a-restart)) and cleared on restart | -| `middlewareChains` | The resolved HTTP, upgrade, and WebSocket middleware order (v5.2.0). Present only when the request passes `middleware: true` | +| Field | Description | +| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `systemStatus` | Every status record set with `set_status` | +| `componentStatus` | Per-component health, aggregated across threads: `name` plus a `status` of `healthy`, `warning`, `error`, `loading`, or `unknown` | +| `restartRequired` | Whether a restart is pending on this node — set by a change that needs one to take effect, such as a deploy that did not restart (see [Deploying without a restart](#deploying-without-a-restart)) or an edit to a loaded `.env` file (see [Environment File Operations](#environment-file-operations)), and cleared on restart | +| `middlewareChains` | The resolved HTTP, upgrade, and WebSocket middleware order (v5.2.0). Present only when the request passes `middleware: true` | `restartRequired` is per node: it reports the node that served the request, not the cluster. diff --git a/reference/security/secrets.md b/reference/security/secrets.md index 0b9a5c4cd..f7950e72f 100644 --- a/reference/security/secrets.md +++ b/reference/security/secrets.md @@ -260,7 +260,7 @@ enc:v1: `kid` lets multiple keypairs coexist during rotation: the decryptor selects the matching private key, or rejects the value if it holds no key for that `kid`. -The same `enc:v1:` envelope format is used for encrypted `.env` values, so a value encrypted once can be used with either the secrets store or `loadEnv`. +The same `enc:v1:` envelope format is used for [encrypted `.env` values](../environment-variables/encrypted-values.md), so a value encrypted once can be used with either the secrets store or `loadEnv`. ### Reference client (Node.js) diff --git a/sidebarsReference.ts b/sidebarsReference.ts index 23e140c15..f997c1810 100644 --- a/sidebarsReference.ts +++ b/sidebarsReference.ts @@ -588,6 +588,11 @@ const sidebars: SidebarsConfig = { id: 'environment-variables/overview', label: 'Overview', }, + { + type: 'doc', + id: 'environment-variables/encrypted-values', + label: 'Encrypted Values', + }, ], }, {