feat(api): spec.nodeConfig serves config.toml and app.toml verbatim from ConfigMaps

[bdchatham]

What

A SeiNode can take its config.toml and app.toml verbatim from ConfigMaps:

spec:
  nodeConfig:
    configRef: {name: rpc-config-v1}   # config.toml
    appRef:    {name: rpc-app-v1}      # app.toml

Both are required at admission and may name the same ConfigMap. They mount read-only over the seid config directory. The controller adds nothing and validates nothing beyond what it needs to avoid destroying a healthy pod.

Why the plan changes, not just the pod spec

A mount at the config path is not stable while anything else writes there. Tested on Linux 6.12.68: a rename onto a bind-mounted file from another mount namespace succeeds and detaches the mount — is_local_mountpoint() is namespace-scoped and vfs_rename() calls detach_mounts(). seid then reads the writer's file with nothing reporting the swap. A same-namespace rename does return EBUSY, which is why the opposite conclusion looks right.

Every sidecar writer of these files commits with os.Rename. So a node with nodeConfig gets a plan with no config task in it:

init:    ensure-data-pvc, [validate-*], apply-rbac-proxy-config,
         apply-statefulset, apply-service, configure-genesis, mark-ready
update:  [validate-*], apply-statefulset, apply-service,
         replace-pod, observe-image, mark-ready

staticConfigPlanner wraps the mode planner, so each mode keeps its own Validate and init plan. MountsNodeConfig — the union of spec and observed stamp — is the predicate everywhere, because a node mid-revert still has the mount. mountedConfigWriterInPlan is the single fail-closed gate.

Clearing nodeConfig rolls the pod and then writes the controller-managed base, so the node does not keep seid init defaults.

Rejected and refused

Rejected at admission, each because its only route to seid was a config task: configValues, overrides, peers, externalAddress. peers is the sharpest — status.resolvedPeers would keep looking correct while config.toml stayed as written. Put p2p.persistent-peers in the ConfigMap.

Refused by staticConfigPlanner.Validate, each because it learns config at run time:

Shape	Run-time discovery
bootstrap Job	its pod holds the same PVC and rewrites config.toml there
state-sync source	trust height and hash, from live witnesses
genesis ceremony	the founding validator set
Autobahn	controller-derived engine keys

The bootstrap case would have been silent: reconcileStatefulSet applies unconditionally, so the production pod holds the mounts while the Job pod runs config-apply, and the claim is ReadWriteOnce, not ReadWriteOncePod.

Notes for review

• The sidecar carries the mounts deliberately. A rename from a container without the mount detaches it silently; with it, the same rename fails EBUSY. Removing it would turn every loud failure into a silent one.
• An in-place ConfigMap edit does not reach a running pod — kubelet pins subPath mounts at pod start. The references are the unit of change.
• Residual: the StatefulSet carries the references before any plan runs, so an unresolved one leaves the template unmountable. replace-pod refuses to delete into that, but a drain or eviction recreates the pod into ContainerCreating. A NodeConfigReady condition plus a render-time hold is the fix, deferred.
• Served schema: spec.nodeConfig.{configRef,appRef}.name and status.currentNodeConfig. A third seiRef drops in beside them when seid moves to a single sei.toml.

🤖 Generated with Claude Code

[cursor]

PR Summary

High Risk
Changes how seid configuration is delivered and blocks config-writing tasks on mounted nodes; misconfigured ConfigMaps or incompatible node shapes can wedge rolls or leave nodes on unintended config until revert plans complete.

Overview
Adds spec.nodeConfig so operators can supply config.toml and app.toml verbatim from ConfigMaps (configRef / appRef), mounted read-only on the seid pod (and sidecar for mount-safety). Admission rejects combining this with controller-driven config (configValues, overrides, peers, externalAddress).

The controller tracks status.currentNodeConfig and treats ConfigMap reference changes as pod-template drift (roll via replace-pod, which pre-validates ConfigMap presence and TOML). A staticConfigPlanner wraps mode planners: init/update plans omit sidecar tasks that rename/write those files (and config-validate), with a fail-closed check on any plan that still carries writers. Revert to controller-managed config rolls first, then runs config-apply and sets plan.clearsNodeConfig. Bootstrap jobs, state-sync snapshot sources, genesis ceremony, Autobahn, and workflows are refused when static config applies.

^{Reviewed by Cursor Bugbot for commit 707e8d1. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Stale Bugbot comment from a previous run.

[seidroid]

Since my last read the only change is that noderesource now exports ConfigTomlKey/AppTomlKey (the unexported mount keys alias them) and replace-pod's guard references those instead of literals, plus test fixture constants — a tidy-up that single-sources the key names and addresses none of my findings. The revert path still rolls a node back to controller management without ever writing controller config, so the verdict stands; from the other reading I kept the ConfigFileRef.Name validation gap and dropped its observe-image stamping claim, because the rollout gate (ObservedGeneration plus UpdatedReplicas against the current revision) cannot report complete for a pod at an older template, so newer refs cannot be stamped over an older pod — the real consequence of that window is the indefinite poll I list as pre-existing.

Blocking

1 finding on the changed lines, as inline comments.

Non-blocking

2 findings on the changed lines, as inline comments.

• Nothing stops spec.nodeConfig being set on a SeiNetwork-owned child, and the parent's unconditional sync of spec.overrides / spec.configValues (internal/controller/seinetwork/nodes.go) then fails the new CEL rules on every reconcile. Because ensureSeiNode issues one Update carrying the whole spec, that also takes image, label and sidecar propagation down for the network — the same blast radius the spec.resources comment there already warns about. Either reject nodeConfig on a controlled child or skip those syncs when it is set.
• The four staticConfigPlanner.Validate refusals are plan-time, but reconcileStatefulSet applies the template unconditionally and earlier in the reconcile, so setting nodeConfig on a state-sync, Autobahn, bootstrap or ceremony node mutates the pod template before the refusal is reached. OnDelete spares the live pod, but the next eviction or manual delete brings up a pod mounting operator config on exactly the shapes the refusal exists to prevent. CEL cannot express these, so a render-time hold (as with the unresolved-reference residual) is the lever.

Pre-existing

Already true on the base branch, not introduced here.

• suggestion — internal/task/observe_image.go polls with no deadline and the executor has no task timeout, so if the pod template changes after replace-pod but before observe-image completes, UpdatedReplicas never reaches Replicas under OnDelete and the plan stays Active forever — no new plan is built and NodeUpdateInProgress never clears. True on base for an image edit mid-roll; nodeConfig makes it far more reachable, since republishing under a new name is the documented way to change config.

_{seidroid review · decision request_changes · session 67e448f06da64676ac86fd92accb0b2a · turn resp_claude_cd2c925a838be810f524b8c8bda3655b · item 51bcea1323a95938b2df78727f4a4667}

Findings: 1 blocking | 4 non-blocking | 3 posted inline | 1 pre-existing

[seidroid]

blocker — The revert plan rolls the pod but never restores controller-managed config, so a node born with nodeConfig comes back on bare seid init output. config-apply never ran on such a node (withoutManagedConfigTasks stripped it from the init plan) and the ConfigMap content only ever existed as a bind mount, so the PVC's config.toml is the seid-init default — RPC laddr on 127.0.0.1, no persistent-peers, no mode/pruning in app.toml. After this plan completes, observe-image stamps currentNodeConfig=nil, the node returns to the mode planner, podTemplateDrifted is false and no further plan is built (pinned by TestNodeConfigRevertRollsBeforeAnyConfigWrite), so the seid container fails its RPC StartupProbe indefinitely while the plan reports success. Note the fix cannot be an extra task here: mountedConfigWriterInPlan keys on the union including the stamp, so it would refuse this very plan — the write has to come from a follow-up plan once the stamp has cleared.

[seidroid]

suggestion — Name carries only length bounds, so a value that is not a DNS subdomain passes admission and then renders a StatefulSet the API server rejects, leaving apply-statefulset failing every reconcile. The sibling object-name fields in this file (DataVolumeImport.PVCName, volumeAttributesClassName) both pin the DNS pattern; adding it here rejects the typo at kubectl apply. (Raised by codex and confirmed.)

[seidroid]

suggestion — This comment names withoutMountedConfigWriters, which does not exist, and its claim is wrong in a load-bearing way: only buildBasePlan runs the filter. The bootstrap and genesis-ceremony progressions (bootstrap.go:132, bootstrap.go:170) are never filtered — they are safe because Validate refuses those shapes. A reader adding a new non-base builder would take this comment as coverage it does not have.

[seidroid]

review found something blocking. The findings are on the lines they are about, and the summary is in this tool's verdict on this pull request.

[seidroid]

This revision answers my earlier blocker: a revert plan now appends config-apply and config-validate after replace-pod, and mountedConfigWriterInPlan exempts tasks past the replacement, so a reverted node no longer keeps seid init defaults on disk. It is still incomplete — the write lands on a pod whose seid already started and nothing restarts it — which is codex's first finding, kept after checking the seid container's healthz-only start gate and the repo's own config-update progression; I also kept codex's ConfigFileRef.Name validation gap and again dropped its observe-image stamping claim, since the rollout gate cannot report complete for a pod at an older revision and the real consequence of that window is the indefinite poll I list as pre-existing.

Blocking

1 finding on the changed lines, as inline comments.

Non-blocking

2 findings on the changed lines, as inline comments.

• Nothing stops spec.nodeConfig being set on a SeiNetwork-owned child, and the parent's unconditional sync of spec.overrides / spec.configValues (internal/controller/seinetwork/nodes.go) then fails the new CEL rules on every reconcile. Because ensureSeiNode issues one Update carrying the whole spec, that also takes image, label and sidecar propagation down for the network — the same blast radius the spec.resources comment there already warns about. Either reject nodeConfig on a controlled child or skip those syncs when it is set.
• The four staticConfigPlanner.Validate refusals are plan-time, but reconcileStatefulSet applies the template unconditionally and earlier in the reconcile, so setting nodeConfig on a state-sync, Autobahn, bootstrap or ceremony node mutates the pod template before the refusal is reached. OnDelete spares the live pod, but the next eviction or manual delete brings up a pod mounting operator config on exactly the shapes the refusal exists to prevent. CEL cannot express these, so a render-time hold (as with the unresolved-reference residual) is the lever.

Pre-existing

Already true on the base branch, not introduced here.

• suggestion — internal/task/observe_image.go polls with no deadline and the executor has no task timeout, so if the pod template changes after replace-pod but before observe-image completes, UpdatedReplicas never reaches Replicas under OnDelete and the plan stays Active forever — no new plan is built and NodeUpdateInProgress never clears. True on base for an image edit mid-roll; nodeConfig makes it far more reachable, since republishing under a new name is the documented way to change config.

_{seidroid review · decision request_changes · session 67e448f06da64676ac86fd92accb0b2a · turn resp_claude_acd236cd361893a6eda5274b0efd99c1 · item 0f4e66156f69536b8630c93967ef61e3}

Findings: 1 blocking | 4 non-blocking | 3 posted inline | 1 pre-existing

[seidroid]

blocker — config-apply lands after the replacement pod is already running, and nothing restarts seid, so the node is marked ready on the configuration this write replaced. The seid container's start gate only polls the sidecar's /v0/healthz (sidecarWaitCommand), not mark-ready, so seid execs within seconds of the pod starting — long before observe-image completes and this task runs — and it re-reads config.toml only on restart. The repo's own running-node config path does this correctly: buildConfigUpdatePlan uses config-patch, config-validate, mark-ready, restart-seid, with the comment that approval must precede restart so seid passes the healthz gate. Appending sidecar.TaskTypeRestartSeid here closes it; as written, a node reverted after previously having controller config keeps running the stale file indefinitely while the plan reports success (TestNodeConfigRevertRollsBeforeAnyConfigWrite pins mark-ready as the last task). (Raised by codex and confirmed.)

[bdchatham]

Rejecting this one — the premise is wrong, and I checked it in the sidecar rather than reasoning from the progression.

/v0/healthz is gated on mark-ready. handleHealthz returns 503 unless engine.Healthz() (sidecar/server/server.go:101-107), Healthz() is return e.ready.Load() (sidecar/engine/engine.go:409-411), and the only writers of ready are the TaskMarkReady / TaskMarkNotReady arms at engine.go:399-403. TestLivezReturns200BeforeReady pins it: if eng.Healthz() { t.Fatal("expected healthz=false before mark-ready") } (sidecar/server/server_test.go:81-84).

So on a replacement pod the sidecar process is new, ready starts false, healthz answers 503, and the seid container sits in its wait loop. seid does not exec "within seconds of the pod starting" — it execs when mark-ready runs, which is the last task in this plan, after config-apply. restart-seid would be restarting a process that has not started.

That is also why buildConfigUpdatePlan needs restart-seid and this plan does not: that path has no pod replacement, so seid is already running and past its gate. This path replaces the pod, so the gate has not opened yet. The same reasoning is why every mode's NodeUpdate plan ends at mark-ready with no restart.

I have put that in the code comment so the next reader does not have to re-derive it.

Your framing did surface two real problems next to it, both fixed in 707e8d1 — see the replies on the Bugbot threads.

[seidroid]

suggestion — Name carries only length bounds, so a value that is not a DNS subdomain passes admission and then renders a StatefulSet the API server rejects, leaving apply-statefulset failing every reconcile. The sibling object-name fields in this file (DataVolumeImport.PVCName, volumeAttributesClassName) both pin the DNS pattern; adding it here rejects the typo at kubectl apply. (Raised by codex and confirmed.)

[bdchatham]

Fixed in 707e8d1 — pinned to the same DNS-subdomain pattern the sibling object-name fields use. The field carried a pattern before the rename to configRef/appRef and I dropped it in the move.

[seidroid]

suggestion — This comment names withoutMountedConfigWriters, which does not exist, and its claim is wrong in a load-bearing way: only buildBasePlan runs the filter. The bootstrap and genesis-ceremony progressions (bootstrap.go:132, bootstrap.go:170) are never filtered — they are safe because Validate refuses those shapes. A reader adding a new non-base builder would take this comment as coverage it does not have.

[bdchatham]

Both correct, fixed in 707e8d1. The name was stale after the rename to withoutManagedConfigTasks, and the coverage claim was worse than stale — it read as exhaustive when only buildBasePlan runs the filter. The comment now says what is actually true: the bootstrap and ceremony progressions are never filtered, Validate refuses those shapes, and mountedConfigWriterInPlan refuses any plan that slips through anyway.

[seidroid]

review found something blocking. The findings are on the lines they are about, and the summary is in this tool's verdict on this pull request.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 53aa878. Configure here.}

[bdchatham]

Review pass — 707e8d1

Applied (4): the revert stamp now waits for the restore (ClearsNodeConfig on the plan, applied by the executor on completion, mirroring ConfigValuesHash); the revert plan finishes through withConfigValues so it carries the overlay and establishes the baseline; ConfigFileRef.Name pins the DNS-subdomain pattern; the stale withoutMountedConfigWriters comment and its overstated coverage are corrected. Details on each thread.

Rejected (1) — the blocker. /v0/healthz is gated on mark-ready: handleHealthz → engine.Healthz() → e.ready.Load(), and only the TaskMarkReady/TaskMarkNotReady arms write that flag (sidecar/engine/engine.go:399-411). TestLivezReturns200BeforeReady pins healthz false before mark-ready. A replacement pod runs a new sidecar process with ready false, so seid stays in its wait loop until mark-ready — the last task in this plan, after config-apply. restart-seid would restart a process that has not started. Full reasoning on the thread.

test-integration is a pre-existing flake, and I have the control run

TestGenesisCeremony_ValidatorLostMidCeremony_Recovers failed twice here. I said I would not call it a flake without evidence, so I ran the control: a clean worktree at origin/main (af5f6f9), same envtest assets, -count=4.

--- FAIL: TestGenesisCeremony_ValidatorLostMidCeremony_Recovers (121.65s)
    ceremony_validator_lost_test.go:116: timeout after 2m0s waiting for:
        the rebuilt genesis ceremony completes over the whole set

It failed on the first of four runs on unmodified main, with the identical message and the identical 121s timeout. The stall is ensure-data-pvc refusing the deleted node's leftover PVC (already exists and is not owned by SeiNode) until the 2m budget runs out — a path this PR does not touch. Not mine to fix here; worth its own issue.

Two non-blocking findings I have not actioned, and why

SeiNetwork child sync. Real, and I would rather fix it deliberately than fold it in. Note the shape is narrower than the summary suggests: generateSeiNode stamps GenesisCeremony on every child, and staticConfigPlanner.Validate refuses nodeConfig on a ceremony node — so a controlled child can never usefully carry it. The failure mode you describe is the parent's whole-spec Update tripping the new CEL rules, which is about the sync, not the feature. Rejecting nodeConfig on a controlled child at admission is the cleaner lever. Happy to add it here if you want it in this PR.

Render-time hold for the four refused shapes. Agreed, and it is the same lever as the unresolved-reference residual already documented in the field doc: reconcileStatefulSet applies unconditionally and earlier than any plan-time refusal, so the template can carry mounts for a shape Validate will refuse. Both want one render-time gate, and I would rather add that once, with a NodeConfigReady condition behind it, than bolt on a partial version now. Flagged in the PR description as the known residual.

Pre-existing observe-image poll with no deadline — agreed on both counts, including that nodeConfig makes it more reachable. Out of scope here.

[seidroid]

review found nothing blocking.

Superseded: the latest review found nothing blocking in this change.

[seidroid]

This revision completes the revert path: the plan now carries config-apply, the configValues overlay and config-validate ahead of mark-ready, a new TaskPlan.clearsNodeConfig defers the stamp clear to plan completion so a failure retries on the writer-exempt planner, and ConfigFileRef.Name gained the DNS pattern. Both of my blockers are addressed — I verified the author's start-gate claim in the sidecar (Engine.Healthz is e.ready, flipped only by mark-ready, and the seid container blocks on healthz 200), so seid does not read config before the write lands and no restart-seid is needed; codex contributed the name-validation finding, now fixed, and I again dropped its observe-image stamping claim, since the rollout gate cannot report complete for a pod at an older revision and the real consequence of that window is the indefinite poll I keep as pre-existing.

Non-blocking

• Nothing stops spec.nodeConfig being set on a SeiNetwork-owned child, and the parent's unconditional sync of spec.overrides / spec.configValues (internal/controller/seinetwork/nodes.go, untouched here) then fails the new CEL rules on every reconcile. Because ensureSeiNode issues one Update carrying the whole spec, that also takes image, label and sidecar propagation down for the network — the same blast radius the spec.resources comment there already warns about. Either reject nodeConfig on a controlled child or skip those syncs when it is set.
• The four staticConfigPlanner.Validate refusals are plan-time, but reconcileStatefulSet applies the template unconditionally and earlier in the reconcile, so setting nodeConfig on a state-sync, Autobahn, bootstrap or ceremony node mutates the pod template before the refusal is reached. OnDelete spares the live pod, but the next eviction or manual delete brings up a pod mounting operator config on exactly the shapes the refusal exists to prevent. CEL cannot express these, so a render-time hold (as with the unresolved-reference residual) is the lever.

Pre-existing

Already true on the base branch, not introduced here.

• suggestion — internal/task/observe_image.go polls with no deadline and the executor has no task timeout, so if the pod template changes after replace-pod but before observe-image completes, UpdatedReplicas never reaches Replicas under OnDelete and the plan stays Active forever — no new plan is built and NodeUpdateInProgress never clears. True on base for an image edit mid-roll; nodeConfig makes it more reachable, since republishing under a new name is the documented way to change config.

_{seidroid review · decision approve · session 67e448f06da64676ac86fd92accb0b2a · turn resp_claude_3397c23c27969ae030eca2b2cbf6d004 · item b315bc5d8ce456ecbe294d3ecab2209b}

Findings: 0 blocking | 2 non-blocking | 0 posted inline | 1 pre-existing