Skip to content

Precompute dashboard metrics and tune production workload - #1

Merged
McanKul merged 11 commits into
mainfrom
agent/precompute-dashboard-snapshots
Sep 9, 2026
Merged

McanKul merged 11 commits into
mainfrom
agent/precompute-dashboard-snapshots

Conversation

@McanKul

@McanKul McanKul commented Jul 29, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • Add persistent PostgreSQL snapshots for query metrics and global/database trends, plus a background refresh worker.
  • Serve dashboard windows from precomputed data and keep bounded cache/fallback behavior.
  • Add production Caddy/Compose bootstrap files and variable workload pacing.
  • Harden source-registration shell quoting and make fresh-stack validation wait for/refresh the snapshots its fixtures consume.
  • Bump the product to 1.1.1 with repository migration 0016 and update migration/rollback documentation.

Validation

  • GitHub CI: repository contracts, backend, frontend, and fresh PostgreSQL stack all passed.
  • Backend: 243 tests passed.
  • Frontend: 66 tests passed; production build passed.
  • Deployment components: join snapshotter 17, workload 31, role reconciler 9, ERP benchmark 64 tests passed.
  • Migration runner/manifest: 6 tests passed; release consistency passed.
  • PostgreSQL 18 migration smoke: migrations 0015/0016, first population, and concurrent refresh passed for metric and trend snapshots.
  • Fresh-stack integration: clean volumes, health/readiness, migration idempotency, persisted 0013→0016 upgrade, role drift repair, full verifier, 30d replay snapshot, disposable clone runtime/policy, and cleanup passed.
  • Production: all 10 services are running; API/repository/collector are healthy; application 1.1.1 and repository 0016/16 are current.
  • The live large-data verifier's final data-dependent candidate scan found no currently beneficial single-column index and correctly returned NO_IMPROVEMENT; persisted HypoPG validation passed and no unsafe DDL or false recommendation was produced.

McanKul and others added 11 commits July 30, 2026 01:42
Only QueryMetricsSnapshotWarming had a handler, so the four remaining
domain exceptions raised by the repository escaped as unhandled 500s.
Since sql/017 ships the matviews WITH NO DATA, this was the normal
cold-start path on every fresh deploy.

Backoff exceptions now return 503 with Retry-After; capacity exceptions
return 503 without it, because retrying cannot fix an exceeded limit.

The blanket `except Exception` in both refresh paths also caught the
warming signal and armed the failure backoff, escalating an unpopulated
matview into a refresh backoff. Warming is now re-raised ahead of it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The generated admin token was 44 base64 chars with no prefix, but
security.py fullmatches ^adv_pat_v1_[A-Za-z0-9_-]{43}$ before comparing
hashes, so every bootstrapped deployment had unusable admin credentials
and only the SHA-256 was kept.

Also removes the hardcoded deploy host, ACME email and basic-auth user
in favour of DASHBOARD_DOMAIN/ACME_EMAIL/DASHBOARD_BASIC_AUTH_USER,
matching the placeholder style the Caddyfile already used; drops the ERP
workload defaults and the workload restart override that would have run
a 500-table load generator against a production source; sets
REGISTER_DEMO_SOURCE=false per the installation guide; and validates the
credentials directory before .env is written, since failing afterwards
left an .env the guard refuses to regenerate.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Drops two tracked PDFs (600 KB), the PDF review whose checks are already
automated in scripts/verify.sh, and the dated iteration gap analysis.
The one permanent fact in the gap analysis -- which fields PoWA 5.2
statement history carries and which it does not -- moves to the known
limits section of docs/ARCHITECTURE.md.

erp-load-acceptance.yml targets a self-hosted runner label that has no
registered runner; its last six scheduled runs were all cancelled on the
24h queue timeout without executing a step.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@McanKul
McanKul marked this pull request as ready for review September 9, 2026 00:40
@McanKul
McanKul merged commit 85f0d75 into main Sep 9, 2026
4 checks passed
@McanKul
McanKul deleted the agent/precompute-dashboard-snapshots branch September 9, 2026 09:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant