Skip to content

docs(env): env-file operations and encrypted .env values - #676

Merged
Ethan-Arrowood merged 8 commits into
mainfrom
kris/env-secrets
Oct 13, 2026
Merged

Ethan-Arrowood merged 8 commits into
mainfrom
kris/env-secrets

Conversation

@kriszyp

@kriszyp kriszyp commented Sep 22, 2026 •

Copy link
Copy Markdown
Member

Documents the component .env surface of Harper's secret handling, which had no page anywhere in the docs. The secrets store itself was already covered by reference/security/secrets.md and the Secrets section of the operations reference; this fills in the file-based half that shares its enc:v1: envelope.

What is documented

reference/operations-api/operations.md — new ### Environment File Operations section (in Components, before Deployment Operations), plus three rows in the Components summary table.

  • get_env_keys, set_env_value, delete_env_value: request and response shapes, project + optional file (defaults to .env, must be .env or .env.<suffix>), the [A-Za-z0-9_.-]+ key character set, and the fact that the two writers replicate.
  • set_env_value's key+value vs values forms, in-place preservation of comments/formatting/untouched keys, file creation, and the one unrepresentable quoting combination.
  • A note covering the masking behavior that was also undocumented: get_component_file on a protected .env returns protected: true, keys, and a KEY=******** body; get_components flags such files; .env.example / .sample / .template are not masked; set_component_file is not blocked and writes verbatim.

reference/environment-variables/encrypted-values.md — new page (sidebar entry added under Environment Variables).

  • The enc:v1: marker, per-value opt-in, coexistence with plaintext.
  • The Harper Pro custody requirement, and the four load-time outcomes (decrypt / skip-and-defer with no custody / replay when custody registers late / skip on decrypt failure), including that a skipped value is absent rather than set to ciphertext and that it never crashes the node.
  • The fetch-encrypt-write client flow, with the get_secrets_public_key response shape.
  • The threat model, and a note that set_env_value does not validate the envelope the way set_secret does.

reference/environment-variables/overview.md — a short "Managing .env Files Remotely" section pointing at both, plus Related links.

reference/security/secrets.md — one line: its existing mention of encrypted .env values now links to the new page.

Envelope internals and the reference Node.js client stay in secrets.md and are linked rather than duplicated.

Verified against code

All behavior was read from HarperFast/harper origin/main (the working checkout's relevant files are byte-identical to origin/main — confirmed by diff), not from the prior internal write-up:

  • Operation handlers and response shapes: components/operations.js (getEnvKeys, setEnvValue, deleteEnvValue, getComponentFile), components/operationsValidation.js (getEnvKeysValidator, setEnvValueValidator, deleteEnvValueValidator).
  • .env parsing, masking, quoting rules: utility/envFile.ts (isEnvFile, isProtectedEnvFile, isExampleEnvFile, formatEnvValue, upsertEnvValues, removeEnvKeys, ENV_KEY_REGEX, ENV_ENCRYPTED_PREFIX).
  • Load-time decrypt / skip / defer: resources/loadEnv.ts, resources/secretDecryptor.ts.
  • Envelope format and kid semantics: utility/secretEnvelope.ts.
  • get_secrets_public_key response is { public_key, fingerprint }: components/secretOperations.ts.
  • Pro custody (cluster-shared RSA-4096 keypair, leader generates on first boot, clone fetches it): harper-pro security/keyCustody.ts, security/fileKeyCustody.ts.
  • Response shapes and masking cross-checked against unitTests/components/envOperations.test.js.

Version badge. v5.2.0 on both new sections. Verified by feature presence at tags rather than ancestry: the three operation names in utility/hdbTerms.ts, components/secretOperations.ts, utility/envFile.ts, isProtectedEnvFile in components/operations.js, and isEncryptedEnvValue + deferEncryptedEnvValue in resources/loadEnv.ts are all present at v5.2.0 and all absent at v5.1.28.

Corrections to the prior internal write-up (docs/env-secret-encryption.md, commit 56511db95), which this PR does not reuse verbatim:

  • It documents get_secrets_public_key as returning { publicKey, fingerprint, scheme, algorithm }. The code returns { public_key, fingerprint } — snake_case, and no scheme or algorithm field. The new page uses the real shape.
  • Its claim that the keypair is "distributed the same way the JWT keypair is" is not stated here; the new page says custody is a Harper Pro component without describing the distribution mechanism, since the docs already cover custody at that level in secrets.md.

For the human reviewer

Review round on 2026-10-01 (Ethan's requested changes plus the pre-push review). What changed and what was left alone:

  • When an edit takes effect. An edit changes the file, not the running components: an edit to a file loadEnv has already loaded only sets restartRequired (resources/loadEnv.ts → requestRestart). Running workers keep the old values, including a deleted key, until they restart. restart_service with "replicated": true is the documented way to apply an edit. Only harper dev running an application directory restarts on its own (autoReload is set only for RUN_HDB_APP). The get_status restartRequired row now mentions .env edits.
  • Replication outcome. replicateOperation (harper-pro replication/replicator.ts) sends to each peer once and does not retry. Each peer appears in the response's replicated array, and a peer that could not be reached or answered with an error is "status": "failed". The request still succeeds, so the page says to check replicated and resend the affected keys' current state, not the original request, which would undo a later edit on every node. This replaces the earlier "an edit reaches every node", and settles the replication-response question the first draft listed as unverified. The restart-semantics question from that list is settled by the bullet above.
  • Encrypted values. A skipped value is never written to process.env: a skipped value leaves an already-set variable in place. A decrypted value, live or replayed, follows the plaintext override rule. A replay does not rerun code that already read the variable.
  • Sibling page agrees. reference/components/applications.md now notes the protected flag in get_components, the masked body from get_component_file, and that set_component_file replaces a .env file whole, so saving back a masked body writes ******** as every value.
  • Carriage returns. A value containing \r is accepted but reads back with \r\n or \r turned into \n (dotenv normalizes line endings before it parses). The page now says so.

Declined here as core defects. All four are reproduced or traced in HarperFast/harper, and the fix belongs in core, not in a docs caveat. Deciding whether to add a temporary caveat until core is fixed is yours:

  1. utility/envFile.ts closesQuote/valueEndLine disagree with dotenv about where a quoted value ends. Given WIN_PATH="C:\tools\" followed by API_KEY=abc, or an unterminated A="foo, set_env_value on that key deletes the following keys on every node. Reproduced with the production helper. This contradicts the page's "every other line … is left exactly as it was".
  2. ASSIGNMENT_LINE matches only KEY=value, but dotenv also accepts KEY: value. On such a line, delete_env_value reports success and leaves the key, and set_env_value appends a duplicate. Reproduced.
  3. set_env_value and delete_env_value read, modify, and write the file with no lock and no ordering across nodes. Overlapping edits can drop a key or leave nodes with different files.
  4. The deferred replay in resources/secretDecryptor.ts skips loadEnv's config-shaping guard. An encrypted HARPER_SET_CONFIG loaded before custody registers ends up in process.env.

Kept as written: set_component_file stays documented as an unguarded verbatim write to .env, and set_env_value as not validating enc:v1: envelopes. Both are accurate descriptions of current core behavior. Whether core should block either one is a product call.

Merged main (2026-10-13, owner-authorized). The one conflict, in reference/operations-api/operations.md, was positional only. main's deploy subsections (Checking what each node installed, Streaming deploy progress) stay at the end of the deploy section, and ### Environment File Operations follows them, before ### Deployment Operations. The restart guidance still holds after main added restart_service's scope: with no scope, a restart covers every worker, including isolated applications' (harper bin/restart.ts, manageThreads.js decodeRestartScope).

Overlapping open PRs. docs(learn): add Administration production reliability track edits the operations reference; check it keeps the .env clause in the restartRequired row and does not contradict the restart guidance. docs(deployment): document deployment.startupInstallTimeout edits the deploy subsections directly above the new section, so whichever merges second resolves a positional conflict.

Verification

  • npm run format:check — clean.
  • npm run build — succeeds, no broken-link errors (onBrokenLinks: 'throw').
  • Both cross-page anchors confirmed against the built HTML: id="environment-file-operations" in build/reference/v5/operations-api/operations.html and id="reference-client-nodejs" in build/reference/v5/security/secrets.html.
  • New page renders at build/reference/v5/environment-variables/encrypted-values.html and appears in the sidebar.

Uncertain / not verified

  • enc:v1: value size. set_secret caps value/envelope at 256 KB; set_env_value has no such cap in its validator. I did not document a limit for env values, but I also did not test a large one against the file writer.
  • Studio coverage. secrets.md says Studio provides a UI for the secrets store. I did not claim anything about Studio's .env editing surface, having no way to check it here.
  • Release notes. release-notes/v5-lincoln/5.2.md does not mention the env operations or .env masking. I left it alone — adding a retroactive entry felt out of scope for this PR, but it may be worth a follow-up.

— Claude Opus 5.5 (review round)

🤖 Generated with Claude Code

Related PRs: #671 overlaps, #683 overlaps, 6 others independent

Review-Coverage: authored=claude; ran=gemini,codex; adjudicated=domain; declined=cursor-grok,cursor-composer,cursor-kimi,cursor-muse; rounds=8; full=3 @ b4c39c9

Review-Attention: read ~3m (decisions: mask-writeback-doc-vs-core-guard, concurrency-caveat-vs-core-lock, envelope-spec-duplicated) @ b4c39c9

The three operations that edit a component's `.env` file - `get_env_keys`,
`set_env_value`, `delete_env_value` - had no page anywhere in the docs, and
neither did the `enc:v1:` encrypted-value form that `loadEnv` decrypts at
startup. The secrets store was already documented; this fills in the file-based
half that shares its envelope format.

Adds an Environment File Operations section to the operations reference,
covering request and response shapes, the `.env`-only `file` constraint, the key
character set, replication, and the masking `get_component_file` and
`get_components` apply to protected `.env` files (while `set_component_file`
still writes them verbatim).

Adds a sibling reference page for encrypted values: what the `enc:v1:` marker
means, the Harper Pro custody requirement and the deferral behavior when no
decryptor is registered yet, the fetch-encrypt-write client flow, and the threat
model. Envelope internals and the reference client stay in the secrets page and
are linked rather than duplicated.

All of this shipped in v5.2.0, verified by the presence of the operations and
the loader path at the v5.2.0 tag and their absence at v5.1.28.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces comprehensive documentation for Encrypted Environment Values and remote .env file management operations (v5.2.0), including new Operations API endpoints such as get_env_keys, set_env_value, and delete_env_value. The feedback suggests improving readability in the Operations API documentation by presenting critical fallback behaviors in separate, distinct sentences rather than combining them with semicolons.

Comment thread reference/operations-api/operations.md Outdated
@github-actions
github-actions Bot temporarily deployed to pr-676 September 22, 2026 00:18 Inactive
@github-actions

Copy link
Copy Markdown

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-676

This preview will update automatically when you push new commits.

@kriszyp
kriszyp marked this pull request as ready for review September 29, 2026 13:27
@kriszyp
kriszyp requested a review from a team as a code owner September 29, 2026 13:27

@Ethan-Arrowood Ethan-Arrowood left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Reviewed with Codex

Comment thread reference/operations-api/operations.md
Comment thread reference/environment-variables/encrypted-values.md Outdated
Comment thread reference/environment-variables/encrypted-values.md Outdated
kriszyp and others added 3 commits October 1, 2026 11:42
… leaves behind

An edit through set_env_value or delete_env_value changes the file only:
loadEnv flags restartRequired on a change to a loaded file and the running
workers keep the old values until they restart. A skipped encrypted value
leaves an already-set variable in place, a decrypted one follows the same
override rule as plaintext, and a late decryptor's replay does not rerun
code that already read the variable.

Dispatch-Task: fix-kriszyp_documentation_676-7119e128
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…a carriage return reads back as a newline

The replicated response lists each peer, and an unreachable one is marked
failed and not retried. dotenv normalizes \r\n and \r to \n, so a value
with a carriage return does not round-trip. The harper dev auto-restart
applies only when it runs an application directory.

Dispatch-Task: fix-kriszyp_documentation_676-7119e128
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… listed as failed too

Dispatch-Task: fix-kriszyp_documentation_676-7119e128
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-676

This preview will update automatically when you push new commits.

@Ethan-Arrowood Ethan-Arrowood left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - fix the merge conflict however you best see fit and you're welcome to squash merge without another review (I trust you wont dramatically change this PR when dealing with the conflict). Thanks!

🤖 Reviewed with Codex

kriszyp and others added 4 commits October 12, 2026 18:13
Resolve the one positional conflict in reference/operations-api/operations.md:
main's deploy subsections ("Checking what each node installed", "Streaming
deploy progress") stay at the end of the deploy section, and the new
"Environment File Operations" section follows them, before "Deployment
Operations".

Dispatch-Task: fix-kriszyp_documentation_676-7119e128
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nding it again is safe

Dispatch-Task: fix-kriszyp_documentation_676-7119e128
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…quest, and note replicated beside the responses

Resending an older set_env_value would undo a later rotation on every node,
and an older delete would remove a key that was set again since.

Dispatch-Task: fix-kriszyp_documentation_676-7119e128
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…asking

get_components flags a .env entry protected, get_component_file returns it
masked, and set_component_file replaces it whole, so the sibling page no
longer contradicts the operations reference.

Dispatch-Task: fix-kriszyp_documentation_676-7119e128
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions
github-actions Bot temporarily deployed to pr-676 October 13, 2026 00:57 Inactive
@github-actions

Copy link
Copy Markdown

🚀 Preview Deployment

Your preview deployment is ready!

🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-676

This preview will update automatically when you push new commits.

@Ethan-Arrowood
Ethan-Arrowood merged commit f194af2 into main Oct 13, 2026
15 checks passed
@Ethan-Arrowood
Ethan-Arrowood deleted the kris/env-secrets branch October 13, 2026 19:05
@github-actions

Copy link
Copy Markdown

🧹 Preview Cleanup

The preview deployment for this PR has been removed.

This branch was previously deployed

1 inactive deployment
pr-676 — b4c39c90 Deployed Oct 13, 2026 by github-actions[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants