Skip to content

infra: CI validations, delete Neo4j, drop dead Grafana password - #37

Open
spashii wants to merge 1 commit into
mainfrom
infra/consolidate-gitops
Open

infra: CI validations, delete Neo4j, drop dead Grafana password#37
spashii wants to merge 1 commit into
mainfrom
infra/consolidate-gitops

Conversation

@spashii

@spashii spashii commented Aug 13, 2026

Copy link
Copy Markdown
Member

Three of the four things from today's review. The Argo consolidation is deliberately not here: it needs a topology decision first (see below).

What is in this PR

CI validations.github/workflows/validate.yml

Argo syncs main straight to the clusters with prune: true and selfHeal: true. Anything merged here deploys. Nothing checked that the charts render.

  • helm lint and helm template for all five combinations in argo/*.yaml. The matrix mirrors what Argo actually renders, so if you add an Application you add a row.
  • kubeconform on the rendered output, pinned to the DOKS 1.32.5 in infra/main.tf.
  • terraform fmt -check and terraform validate for infra, ai-infra/state, ai-infra/vertex.
  • Every Argo Application must track main. A feature branch pinned in an Application CR is exactly the drift in docs/triage-2026-05-14.md, where echo-monitoring-prod sat eight months behind.

It caught a real bug on the first run. The monitoring ingress emitted cert-manager.io/cluster-issuer and nginx.ingress.kubernetes.io/ssl-redirect twice, once hardcoded in the template and once from ingress.annotations. Annotations are now built as a single dict, so values override defaults and no key can repeat. The basicAuth.enabled=true path was smoke-tested too.

Neo4j deleted. Deployment, service, PVC, and the echo-critical PriorityClass that existed only for it and is referenced by nothing. Verified zero references across server/, agent/ and frontend/ first. Also removes a plaintext neo4j password committed in all three values files. Frees 2.7Gi memory and a 40Gi PVC per cluster.

Grafana adminPassword deleted from both monitoring values files. No template read it, the live value comes from the SealedSecret. It rendered to nothing and only read like a live credential in a public repo.

Docs. AGENTS.md and .agents/architecture.md still described Neo4j as deployed. Corrected, and AGENTS.md now carries a note that it is a generated snapshot which goes stale, with the comment block in infra/main.tf named as the deployment source of truth.

Verification

All five renders were diffed against main. The only resources removed are the four Neo4j ones. Monitoring resources are byte-identical apart from the neo4j regex alternations and the ingress fix.

echo-prod              Valid: 18, Invalid: 0, Errors: 0, Skipped: 1
echo-dev               Valid: 18, Invalid: 0, Errors: 0, Skipped: 1
echo-testing           Valid: 18, Invalid: 0, Errors: 0, Skipped: 1
echo-monitoring-prod   Valid: 38, Invalid: 0, Errors: 0, Skipped: 0
echo-monitoring-dev    Valid: 38, Invalid: 0, Errors: 0, Skipped: 0

The skipped resource is the cert-manager ClusterIssuer CRD, expected under -ignore-missing-schemas.

Deploy note

Argo runs with prune: true, so on sync it will delete the Neo4j Deployment, Service, PVC and the echo-critical PriorityClass. The 40Gi PVC goes with it. Take a neo4j-admin database dump first if you want the data, per the note in docs/migration-plan.md.

Not in this PR

  • One Argo / app-of-apps. Needs a decision between hub-and-spoke and per-cluster roots. Worth noting docs/migration-plan.md decision 6 ("don't adopt app-of-apps") is void: it rested on this repo being deleted within 2-3 months from 2026-05-15, and the repo is staying with OVHcloud as the new target.
  • Branch protection on main. A GitHub setting, not a file. Still unprotected.
  • imagePullPolicy: Always while tags are immutable git SHAs.

Argo syncs main straight to the clusters with prune and selfHeal, so anything
merged here deploys. Until now nothing checked that the charts even render.

CI (.github/workflows/validate.yml)
- helm lint + helm template for all five combinations in argo/*.yaml, so the
  matrix mirrors what Argo actually renders. Add an Application, add a row.
- kubeconform against the rendered output, pinned to the DOKS 1.32.5 version
  in infra/main.tf.
- terraform fmt -check and validate for infra, ai-infra/state, ai-infra/vertex.
- every Argo Application must track main. A feature branch pinned in an
  Application is the exact drift in docs/triage-2026-05-14.md, where
  echo-monitoring-prod sat eight months behind.

It found a real bug immediately: the monitoring ingress emitted
cert-manager.io/cluster-issuer and nginx.ingress.kubernetes.io/ssl-redirect
twice, once hardcoded and once from ingress.annotations. Annotations are now
built as one dict, so values override defaults and no key repeats.

Neo4j
Deleted the deployment, service and PVC, plus the echo-critical PriorityClass
that existed only for it and is referenced by nothing. Confirmed zero
references across server/, agent/ and frontend/ before removing. Also removes
a plaintext neo4j password that was committed in all three values files.
Frees 2.7Gi memory and a 40Gi PVC per cluster.

Grafana
Removed adminPassword from both monitoring values files. No template read it;
the live value comes from the monitoring-secrets SealedSecret. It rendered to
nothing and only read like a live credential in a public repo.

Docs
AGENTS.md and .agents/architecture.md still described Neo4j as deployed.
Corrected, and AGENTS.md now says what it is: a generated snapshot that goes
stale, with the comment block in infra/main.tf as the deployment source of
truth.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant