infra: CI validations, delete Neo4j, drop dead Grafana password

.agents/architecture.md

@@ -3,7 +3,7 @@
 ## Core Systems
 - Terraform in `infra/` provisions the DigitalOcean VPC, Kubernetes cluster, managed Postgres, Redis, Spaces, and optional registry before seeding namespaces and registry credentials (infra/main.tf:80-195).
 - Argo CD Applications (`echo-*`, `echo-monitoring-*`) track this repository’s `main` branch with automated prune/self-heal and namespace creation (argo/echo-dev.yaml:1-23, argo/echo-monitoring-prod.yaml:1-23).
-- The `helm/echo` chart deploys the API server, worker tiers (worker, workerCpu, workerScheduler), Directus, and Neo4j along with shared env configuration and ingress/rollout settings (helm/echo/values.yaml:1-166).
+- The `helm/echo` chart deploys the API server, worker tiers (worker, workerCpu, workerScheduler) and Directus along with shared env configuration and ingress/rollout settings (helm/echo/values.yaml:1-166).
 - `helm/monitoring` delivers Prometheus, Grafana, Loki, promtail, node-exporter, blackbox checks, and alertmanager with storage and ingress defaults (helm/monitoring/values.yaml:1-112).
 - Sealed secrets house sensitive config for each namespace, edited locally via `secret-manager.sh` before sealing (secret-manager.sh:104-189, secrets/sealed-backend-secrets-dev.yaml:1-18).
 - `ai-infra` bootstraps a GCS-backed Terraform state bucket and Vertex AI endpoint plus service-account IAM for Gemini usage (ai-infra/state/main.tf:1-31, ai-infra/vertex/main.tf:1-20, ai-infra/README.md:5-30).

.github/workflows/validate.yml

@@ -0,0 +1,148 @@
+name: validate
+
+# Why this exists: Argo CD syncs `main` straight to the clusters with
+# prune + selfHeal. Anything that merges here is deployed. A chart that does not
+# render, or a manifest the API server rejects, becomes a production problem with
+# no gate in between.
+#
+# Precedent: a Helm/Python `{{ }}` brace collision in cronjob-warning-digest.yaml
+# silently blocked the whole monitoring chart from rendering for weeks. A plain
+# `helm template` would have caught it on the pull request.
+#
+# The render matrix below MUST mirror the valueFiles in argo/*.yaml. If you add
+# an Argo Application or change its valueFiles, add it here in the same commit.
+
+on:
+  pull_request:
+  push:
+    branches: [main]
+
+permissions:
+  contents: read
+
+jobs:
+  helm:
+    name: helm render (${{ matrix.name }})
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          # mirrors argo/echo-prod.yaml
+          - name: echo-prod
+            chart: helm/echo
+            values: "-f helm/echo/values-prod.yaml -f helm/echo/values-extended-env.yaml"
+          # mirrors argo/echo-dev.yaml
+          - name: echo-dev
+            chart: helm/echo
+            values: "-f helm/echo/values.yaml -f helm/echo/values-extended-env.yaml -f helm/echo/values-echo-next.yaml"
+          # mirrors argo/echo-testing.yaml
+          - name: echo-testing
+            chart: helm/echo
+            values: "-f helm/echo/values-testing.yaml -f helm/echo/values-extended-env.yaml"
+          # mirrors argo/echo-monitoring-prod.yaml
+          - name: echo-monitoring-prod
+            chart: helm/monitoring
+            values: "-f helm/monitoring/values-prod.yaml"
+          # mirrors argo/echo-monitoring-dev.yaml
+          - name: echo-monitoring-dev
+            chart: helm/monitoring
+            values: "-f helm/monitoring/values.yaml"
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: azure/setup-helm@v4
+        with:
+          version: v3.16.3
+
+      - name: helm lint
+        run: helm lint ${{ matrix.chart }} ${{ matrix.values }}
+
+      - name: helm template
+        run: |
+          mkdir -p rendered
+          helm template ${{ matrix.name }} ${{ matrix.chart }} ${{ matrix.values }} \
+            > rendered/${{ matrix.name }}.yaml
+          # An empty or near-empty render means the chart silently produced nothing.
+          lines=$(wc -l < rendered/${{ matrix.name }}.yaml)
+          echo "rendered $lines lines"
+          test "$lines" -gt 50
+
+      - name: install kubeconform
+        run: |
+          curl -sSL -o /tmp/kubeconform.tar.gz \
+            https://github.com/yannh/kubeconform/releases/download/v0.6.7/kubeconform-linux-amd64.tar.gz
+          tar -xzf /tmp/kubeconform.tar.gz -C /tmp kubeconform
+          sudo mv /tmp/kubeconform /usr/local/bin/
+
+      - name: kubeconform
+        # Kubernetes version is pinned to the DOKS version in infra/main.tf.
+        # Bump both together.
+        run: |
+          kubeconform \
+            -kubernetes-version 1.32.5 \
+            -strict \
+            -ignore-missing-schemas \
+            -summary \
+            rendered/${{ matrix.name }}.yaml
+
+  argo-manifests:
+    name: argo application manifests
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: install pyyaml
+        run: python3 -m pip install --quiet pyyaml
+
+      - name: parse every Argo Application
+        run: |
+          python3 - <<'PY'
+          import glob, sys, yaml
+          bad = 0
+          for path in sorted(glob.glob("argo/*.yaml")):
+              try:
+                  docs = [d for d in yaml.safe_load_all(open(path)) if d]
+              except yaml.YAMLError as e:
+                  print(f"FAIL {path}: {e}"); bad = 1; continue
+              for d in docs:
+                  if d.get("kind") != "Application":
+                      continue
+                  spec = d.get("spec", {})
+                  src = spec.get("source", {})
+                  rev = src.get("targetRevision")
+                  # A feature branch pinned here is exactly the drift that left
+                  # echo-monitoring-prod eight months behind main.
+                  if rev != "main":
+                      print(f"FAIL {path}: targetRevision is {rev!r}, expected 'main'")
+                      bad = 1
+                  if not spec.get("destination", {}).get("namespace"):
+                      print(f"FAIL {path}: no destination.namespace")
+                      bad = 1
+                  print(f"ok   {path} -> {src.get('path')} @ {rev}")
+          sys.exit(bad)
+          PY
+
+  terraform:
+    name: terraform ${{ matrix.dir }}
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        dir: [infra, ai-infra/state, ai-infra/vertex]
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: hashicorp/setup-terraform@v3
+        with:
+          terraform_version: 1.9.8
+          terraform_wrapper: false
+
+      - name: terraform fmt
+        run: terraform -chdir=${{ matrix.dir }} fmt -check -recursive
+
+      - name: terraform init (no backend)
+        run: terraform -chdir=${{ matrix.dir }} init -backend=false -input=false
+
+      - name: terraform validate
+        run: terraform -chdir=${{ matrix.dir }} validate

AGENTS.md

@@ -1,5 +1,15 @@
 # AGENTS.md
 
+> **This file is a generated snapshot and it goes stale.** It has been wrong
+> before: it described Neo4j as a deployed workload after it stopped being used,
+> and it says "no CI workflows tracked in repo" while `dembrane/echo`'s GitHub
+> Actions commit image tags into this repo several times a day.
+>
+> **For deployments, the comment block at the top of `infra/main.tf` is the
+> source of truth.** That is the runbook actually in use (workspace selection,
+> which tfvars, the `doctl` kubeconfig lines). Prefer code and config over any
+> prose in this file, and fix this file when you find it wrong.
+
 ## 1) Snapshot
 - Project: Dembrane ECHO GitOps • Repo type: mono (Terraform + Helm + Argo CD GitOps) (README.md:40, infra/main.tf:80)
 - Entrypoints: Terraform stacks (`infra/`, `ai-infra/`), Argo CD apps (`echo-*`, `echo-monitoring-*`), Helm charts (`helm/echo`, `helm/monitoring`) (infra/main.tf:80-195, ai-infra/README.md:16-24, argo/echo-dev.yaml:1-23, helm/monitoring/Chart.yaml:1-6)
@@ -11,12 +21,12 @@
 
 ## 2) Tech & Tooling
 - Runtimes & package managers: Terraform ≥1.0, kubectl, Helm 3, kubeseal, doctl per prerequisites; Python 3 + `requests` for Loki tooling (README.md:84-103, scripts/LOKI_LOG_QUERY.md:5-17).
-- Core libraries by role: DigitalOcean Terraform resources provision VPC/K8s/DB/cache/object storage (infra/main.tf:80-145); Argo CD Applications drive GitOps sync (argo/echo-dev.yaml:1-23); `helm/echo` manages API, Directus, worker tiers, and Neo4j (helm/echo/values.yaml:42-166); `helm/monitoring` delivers Prometheus/Grafana/Loki stack (helm/monitoring/values.yaml:1-61); `ai-infra` provisions Vertex AI endpoints and IAM (ai-infra/vertex/main.tf:1-20).
+- Core libraries by role: DigitalOcean Terraform resources provision VPC/K8s/DB/cache/object storage (infra/main.tf:80-145); Argo CD Applications drive GitOps sync (argo/echo-dev.yaml:1-23); `helm/echo` manages API, Directus and worker tiers (helm/echo/values.yaml:42-166); `helm/monitoring` delivers Prometheus/Grafana/Loki stack (helm/monitoring/values.yaml:1-61); `ai-infra` provisions Vertex AI endpoints and IAM (ai-infra/vertex/main.tf:1-20).
 - Scripts you’ll actually use: `secret-manager.sh` for base64 edits, batch updates, and compares (secret-manager.sh:4-193); `scripts/query_logs.py` wraps Loki queries with chunking/pagination (scripts/LOKI_LOG_QUERY.md:29-132); `scripts/k6/sendChunks.js` replays participant uploads via k6 (scripts/k6/README.md:11-35).
 - Code style (lint/format/type): Terraform providers are version-locked by `.terraform.lock.hcl`; run CLI formatters (`terraform fmt`, `helm lint`) locally as needed (infra/.terraform.lock.hcl:1-33).
 
 ## 3) Architecture (mental model)
-- Modules/services & responsibilities: Terraform builds DigitalOcean infra then seeds namespaces/secrets; Helm deploys application workloads (API, workers, Directus, Neo4j) and monitoring stack (infra/main.tf:80-195, helm/echo/values.yaml:42-166, helm/monitoring/values.yaml:1-112).
+- Modules/services & responsibilities: Terraform builds DigitalOcean infra then seeds namespaces/secrets; Helm deploys application workloads (API, workers, Directus) and monitoring stack (infra/main.tf:80-195, helm/echo/values.yaml:42-166, helm/monitoring/values.yaml:1-112).
 - Data & external surfaces: Postgres, Redis, and Spaces are managed services; ingress exposes `directus`/`api` hostnames with TLS and monitoring endpoints with optional auth (infra/main.tf:101-145, helm/echo/values.yaml:143-159, helm/monitoring/values.yaml:1-60).
 - Notable patterns: Argo CD auto-prune/self-heal enforces drift control; HPAs and priority classes tune scaling for core workloads (argo/echo-dev.yaml:18-23, helm/echo/templates/hpa-api-server.yaml:1-32, helm/echo/templates/priorityclass-echo-critical.yaml:1-11).
 - Diagram → see `.agents/architecture.md`.

NEED_HELP.md

@@ -1,5 +1,27 @@
 @Dembrane, @spashii needs your help!
 
+## done (2026-08-13)
+- ~~CI validations~~ → `.github/workflows/validate.yml`: `helm lint` and
+  `helm template` for all five Argo render combinations, `kubeconform` on the
+  output, `terraform fmt`/`validate`, and a check that every Argo Application
+  tracks `main`. It caught a duplicate-annotation bug in the monitoring ingress
+  on its first run.
+- ~~Neo4j~~ deleted (deployment, service, PVC, the `echo-critical` PriorityClass
+  that existed only for it, and its committed password). Zero references in
+  `server/`, `agent/` or `frontend/` confirmed before removal.
+- ~~Grafana `adminPassword: "admin"` in values~~ deleted. It was dead config; the
+  live value comes from the SealedSecret.
+
+## still open, highest value first
+- **protect `main`** (see the section below; still unprotected as of 2026-08-13
+  while Argo syncs prod from it with prune + selfHeal)
+- **one Argo, app-of-apps** — Application CRs are still applied by hand, which is
+  the drift bug from `docs/triage-2026-05-14.md`. Note `docs/migration-plan.md`
+  decision 6 ("don't adopt app-of-apps") is void: the repo is staying and the
+  target is now OVHcloud, not GCP Cloud Run.
+- `imagePullPolicy: Always` on every deployment while image tags are immutable
+  git SHAs. Puts the registry in the pod startup path for no benefit.
+
 ## infra
 - add Azure LLMs
 - add Runpod Servvice (or use Az Serverless / Google Cloud Run)

helm/echo/values-prod.yaml

@@ -126,25 +126,6 @@ workerScheduler:
       cpu: "800m"
       memory: "1Gi"
 
-neo4j:
-  image:
-    repository: "neo4j"
-    tag: "5.18.0-community"
-  password: "admin@dembrane"
-  storage:
-    size: "40Gi"
-  config:
-    pagecacheSize: "2G"
-    heapInitialSize: "2G"
-    heapMaxSize: "4G"
-  resources:
-    requests:
-      cpu: "2"
-      memory: "2Gi"
-    limits:
-      cpu: "4"
-      memory: "8Gi"
-
 ingress:
   enabled: true
   className: "nginx"

helm/echo/values-testing.yaml

@@ -134,24 +134,6 @@ workerScheduler:
       cpu: "600m"
       memory: "1Gi"
 
-neo4j:
-  image:
-    repository: "neo4j"
-    tag: "5.18.0-community"
-  password: "admin@dembrane"
-  storage:
-    size: "5Gi"  # Smaller than dev
-  config:
-    pagecacheSize: "256M"
-    heapSize: "256M"
-  resources:
-    requests:
-      cpu: "300m"
-      memory: "512Mi"
-    limits:
-      cpu: "800m"
-      memory: "2Gi"
-
 ingress:
   enabled: true
   className: "nginx"