Skip to content

Workloads can reference ConfigMaps and Secrets, delivered to every POP cell#129

Open
scotwells wants to merge 8 commits into
mainfrom
feat/configmap-secret-mounts-federated
Open

Workloads can reference ConfigMaps and Secrets, delivered to every POP cell#129
scotwells wants to merge 8 commits into
mainfrom
feat/configmap-secret-mounts-federated

Conversation

@scotwells

@scotwells scotwells commented Jun 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Workloads can reference ConfigMaps and Secrets from their templates, and the platform delivers that data to whichever location runs the workload. Instances wait to start until their configuration is present, instead of launching with missing env vars or empty mounts, and WorkloadDeployment status explains what's still pending and why. The instance gating is behind a feature flag, off by default, so merging this is safe before the rollout completes.

How it works

The management plane resolves each referenced ConfigMap/Secret into a copy that travels with the deployment to its assigned location. New Instances get a scheduling gate that is cleared once the copies have arrived, and per-Instance progress rolls up to the WorkloadDeployment so users see why replicas are pending. Invalid or unauthorized references are rejected at admission time.

Review map

8 commits, best reviewed one at a time; every commit builds and tests green standalone.

  1. API: envFrom support + the shared status/labeling vocabulary; CRD updates.
  2. Reference collection, copy naming, source readers, metrics (promotable to a shared library).
  3. Webhook: validate references and check the author can read each one (create and update).
  4. Project-side resolver: read sources, enforce size limits, create the copies.
  5. Federation: deliver copies to the right location alongside their deployment.
  6. Scheduling gate on new Instances (feature-flagged, default off).
  7. Cell-side gate clearing once the data lands.
  8. WorkloadDeployment status rollup.

Related

Known limitations

Testing

Unit suites cover every stage, including fail-before transition tests; end-to-end coverage is deferred to #149.

🤖 Generated with Claude Code

@scotwells scotwells force-pushed the feat/configmap-secret-mounts-federated branch from e0b8ee7 to 9a064cc Compare June 2, 2026 00:02
This was referenced Jun 2, 2026
Open
Open
Open
Open
Open
Open
Open
Open
Open
scotwells added a commit that referenced this pull request Jun 3, 2026
@scotwells @claude
docs: write guide as shipped behavior, link merged examples
1e70de5 
Drop the pre-merge dependency framing (the #129 'not yet on main' note)
and repoint example references from PR URLs to relative paths
(examples/serverless-js-configmap, examples/config-secret-probe), as if
the referenced-data delivery and example PRs are merged. Keep the
upstream base/base-compat ROM-enablement limitation, which is a Unikraft
runtime matter independent of these PRs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
scotwells added a commit that referenced this pull request Jun 4, 2026
@scotwells @claude
feat(cli): add UP-TO-DATE column to compute workloads
135681b 
Surface rolling-update / restart progress in `datumctl compute workloads` by
showing updated/desired replica counts next to ready. UP-TO-DATE counts
instances on the latest template revision (status.updatedReplicas), so a roll
is visible as the count dips below desired and then recovers.

Includes a byte-identical copy of the UpdatedReplicas/ObservedGeneration
WorkloadDeployment status fields in api/v1alpha so the plugin can read them.
These fields are defined identically on the controller branch (PR #129); the
duplicate resolves cleanly once both land on main.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@scotwells scotwells mentioned this pull request Jun 4, 2026
Merged
@scotwells scotwells force-pushed the feat/configmap-secret-mounts-federated branch from d981d02 to e420096 Compare June 4, 2026 23:06
@scotwells scotwells changed the base branch from feat/federated-deployment-scheduling to split/federation-core-bundled June 4, 2026 23:06
@scotwells scotwells changed the title feat: Deliver referenced ConfigMaps and Secrets to instances feat: deliver referenced ConfigMaps and Secrets to instances (core) Jun 4, 2026
@scotwells scotwells mentioned this pull request Jun 4, 2026
Open
@scotwells scotwells changed the title feat: deliver referenced ConfigMaps and Secrets to instances (core) Workloads can reference ConfigMaps and Secrets, delivered to every POP cell Jun 4, 2026
@scotwells scotwells force-pushed the split/federation-core-bundled branch from 0920e3e to 741fb15 Compare June 5, 2026 01:42
@scotwells scotwells force-pushed the feat/configmap-secret-mounts-federated branch from e420096 to b11ab65 Compare June 5, 2026 01:42
@scotwells scotwells force-pushed the split/federation-core-bundled branch from 741fb15 to 119ddfd Compare June 5, 2026 15:13
@scotwells scotwells force-pushed the feat/configmap-secret-mounts-federated branch from b11ab65 to e0df8b4 Compare June 5, 2026 15:13
@scotwells scotwells force-pushed the split/federation-core-bundled branch from 119ddfd to 5997231 Compare June 5, 2026 15:25
@scotwells scotwells force-pushed the feat/configmap-secret-mounts-federated branch from e0df8b4 to 5ec3c00 Compare June 5, 2026 15:25
@scotwells scotwells force-pushed the split/federation-core-bundled branch from 5997231 to 1d0fca7 Compare June 5, 2026 16:19
@scotwells scotwells force-pushed the feat/configmap-secret-mounts-federated branch from 5ec3c00 to bc253fd Compare June 5, 2026 16:19
@scotwells scotwells force-pushed the split/federation-core-bundled branch from 1d0fca7 to a67b32c Compare June 5, 2026 16:46
@scotwells scotwells force-pushed the feat/configmap-secret-mounts-federated branch from bc253fd to 643da4d Compare June 5, 2026 16:46
@scotwells scotwells changed the base branch from split/federation-core-bundled to feat/federated-deployment-scheduling June 5, 2026 17:30
An error occurred while trying to automatically change base from split/federation-core-bundled to feat/federated-deployment-scheduling June 5, 2026 17:30
@scotwells scotwells force-pushed the feat/federated-deployment-scheduling branch from 71e388c to 5718fbb Compare June 10, 2026 19:26
@scotwells scotwells force-pushed the feat/configmap-secret-mounts-federated branch from b821986 to ab8ee24 Compare June 10, 2026 19:26
@scotwells scotwells force-pushed the feat/federated-deployment-scheduling branch from 5718fbb to cad32c4 Compare June 10, 2026 20:08
@scotwells scotwells force-pushed the feat/configmap-secret-mounts-federated branch 2 times, most recently from 9a1e70b to 34f3a24 Compare June 10, 2026 22:33
@scotwells scotwells force-pushed the feat/federated-deployment-scheduling branch from cad32c4 to 0cca93e Compare June 10, 2026 22:33
@scotwells scotwells force-pushed the feat/configmap-secret-mounts-federated branch from 34f3a24 to 9a2a132 Compare June 10, 2026 22:57
@scotwells scotwells force-pushed the feat/federated-deployment-scheduling branch 2 times, most recently from 9542455 to 615ef54 Compare June 10, 2026 23:33
@scotwells scotwells force-pushed the feat/configmap-secret-mounts-federated branch 2 times, most recently from 06edcdc to b80c452 Compare June 10, 2026 23:51
@scotwells scotwells force-pushed the feat/federated-deployment-scheduling branch 2 times, most recently from 5b638bb to 7dc94a0 Compare June 11, 2026 00:56
@scotwells scotwells force-pushed the feat/configmap-secret-mounts-federated branch from b80c452 to 74999ba Compare June 11, 2026 00:56
Base automatically changed from feat/federated-deployment-scheduling to main June 11, 2026 01:46
scotwells and others added 4 commits June 10, 2026 22:08
@scotwells @claude
feat(api): ConfigMap and Secret references with referenced-data vocab…
0c9ce59 
…ulary

Workload templates can now populate container environments from ConfigMaps
and Secrets via envFrom, alongside the existing volume and env.valueFrom
references. The API also gains the shared vocabulary the referenced-data
pipeline communicates with across planes:

- ReferencedDataReady condition and its reason set (Resolving,
  AwaitingPropagation, terminal Source* errors, Ready) on both
  WorkloadDeployment and Instance.
- Annotations bridging the federation boundary: the expected-companion
  manifest, the terminal resolver error, the gate-start timestamp, and the
  rolling-restart trigger.
- The referenced-data label that marks companion objects for propagation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@scotwells @claude
feat(referenceddata): reference collection, companion naming, and pro…
0c9adcb 
…ject readers

Introduce the capability seam for resolving ConfigMaps and Secrets referenced
by workload templates, designed to be promotable into a shared platform
library:

- A collector that walks a workload template (volumes, env.valueFrom,
  envFrom) and returns a deduplicated, deterministically ordered reference
  set.
- Companion naming: cell-side copies are named after their source objects so
  consumer references in the template resolve unchanged, with a kind-prefixed
  token format for set-membership checks.
- ProjectConfigSecretReader interface with a local-cluster implementation,
  leaving room for a cross-project reader when the multicluster mode needs
  one.
- Prometheus metrics covering resolver outcomes, companion delivery, and
  gate-wait duration.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@scotwells @claude
feat(webhook): validate ConfigMap and Secret references on workload w…
7742d2d 
…rites

Reject invalid references at admission time instead of letting them fail
deep in the delivery pipeline: referenced names must be valid object names,
envFrom prefixes valid C identifiers, and the requesting user must be
authorized (via SubjectAccessReview) to read each referenced ConfigMap and
Secret, so a workload cannot be used to exfiltrate objects its author cannot
see.

Workload updates now run the same validation as creates, closing the gap
where a valid workload could be mutated into an invalid or unauthorized one
after admission.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@scotwells @claude
feat(controller): resolve referenced data into labeled companion objects
9a23cd7 
Add the ReferencedDataController, a management-plane reconciler that watches
WorkloadDeployments referencing ConfigMaps or Secrets and materialises
labeled companion copies for delivery to cells:

- Sources are read from the project namespace, enforced against per-object
  and aggregate size limits (RAM-backed cell filesystems make oversized data
  a boot failure, not a disk problem), and copied to companions named after
  their source so consumer references resolve unchanged.
- Companions are reference-counted via a conflict-retried finalizer scheme so
  shared sources survive until the last referencing deployment goes away.
- The resolver owns the WorkloadDeployment's ReferencedDataReady condition
  and records the expected-companion manifest annotation the cell checks
  against. Terminal source errors (not found, unauthorized, too large) are
  additionally stamped as an annotation so the verdict survives the
  federation hop, where status conditions do not propagate hub-to-cell.
- When a federation hub is configured, companions are written to the
  project's downstream namespace for propagation; otherwise they land in the
  project namespace directly (single-cluster path).
- New field indexes let source ConfigMap/Secret rotation re-queue exactly the
  deployments that reference the changed object.

The controller RBAC gains read access to ConfigMaps and Secrets, plus write
access for materialised companions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@scotwells scotwells force-pushed the feat/configmap-secret-mounts-federated branch from 5b8d521 to ead160c Compare June 11, 2026 03:15
scotwells and others added 4 commits June 10, 2026 22:26
@scotwells @claude
feat(federation): deliver companions to cells alongside their deployment
2a06ed4 
Extend the WorkloadDeployment federator so referenced data actually reaches
the cells that need it:

- Each city's PropagationPolicy now selects companion ConfigMaps and Secrets
  by the referenced-data label in addition to the city's WorkloadDeployments,
  so companions co-arrive with the deployments that mount them.
- Controller-managed annotations (the expected-companion manifest) are copied
  onto the downstream WorkloadDeployment; the cell gate-clears instances by
  checking arrived companions against this manifest.
- Status sync from the cell merges rather than overwrites: the resolver owns
  the ReferencedDataReady condition on the project object, so the federator
  re-applies it over the downstream status and retries on conflict instead of
  clobbering the resolver's verdict.

The compute-manager hub role gains get/list/watch on ConfigMaps and Secrets;
without this grant Karmada silently skips the companion kinds and mounts
never materialise on edge cells.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@scotwells @claude
feat(controller): gate new instances on referenced data behind a feat…
d02c3fa 
…ure flag

Instances whose template references ConfigMaps or Secrets now receive a
ReferencedData scheduling gate at creation, so the provider does not launch
them before their mounted data exists on the cell. Launching ungated would
surface as containers missing env vars or mount sources rather than a clear
pending state.

Stamping is controlled by the new enableReferencedDataGate feature flag,
default off: the gate must not be introduced until the cell gate-clearing
reconciler and provider gate-honoring are deployed everywhere, otherwise
gated instances would stall indefinitely or launch without their data.
Templates with no references never receive the gate regardless of the flag.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@scotwells @claude
feat(controller): clear the ReferencedData gate when companions land …
04d8e7d 
…on the cell

The cell InstanceReconciler now owns the second half of the gate contract:
it removes the ReferencedData scheduling gate once every companion the
resolver promised has arrived, letting the provider launch the instance.

- The expected-companion manifest annotation on the owner WorkloadDeployment
  is checked against labeled companions actually present in the namespace —
  a pure set-membership test, no name recomputation on the cell.
- Progress is reflected on the Instance's ReferencedDataReady condition:
  AwaitingPropagation while companions are in flight, Ready when complete.
  Terminal resolver verdicts (source not found, unauthorized, too large) are
  promoted from the WD's error annotation — the only signal that crosses the
  federation boundary — so users see the root cause on the Instance instead
  of an indefinite wait.
- Companion Create events in a namespace re-queue that namespace's
  instances, so gates clear promptly on arrival rather than on the next
  periodic resync.
- Gate-clearing latency, expected/present companion counts, condition
  transitions, and cleared events are all observable; the gate-start
  timestamp annotation feeds the wait-duration histogram.

Quota and ReferencedData gate removal are unified in a single
reconcileSchedulingGates pass so one status write covers both.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@scotwells @claude
feat(controller): surface referenced-data progress on deployment status
e9d0519 
Roll the per-Instance ReferencedDataReady signal up to the
WorkloadDeployment so users see why replicas are pending: ReplicasReady
reports how many desired replicas are waiting for referenced data
(after quota, which takes precedence as the earlier blocker).

Rolling up a condition the resolver also writes makes reconcile loops the
main hazard, so the reconciler gains two complementary guards:

- The WorkloadDeployment watch only re-enqueues on spec changes or
  ReferencedDataReady transitions, filtering out this reconciler's own
  Available/ReplicasReady writes.
- Status is only written when it actually changed, skipping the superfluous
  API call (and the resourceVersion bump that would otherwise fire another
  event) on no-op passes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@scotwells scotwells force-pushed the feat/configmap-secret-mounts-federated branch from ead160c to e9d0519 Compare June 11, 2026 03:31
This was referenced Jun 11, 2026
Open
Open
@scotwells scotwells requested review from a team and savme June 11, 2026 13:40
@scotwells scotwells marked this pull request as ready for review June 11, 2026 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@savme savme Awaiting requested review from savme

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@scotwells