diff --git a/CLAUDE.md b/CLAUDE.md index 4e5fc865..70286f32 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -480,10 +480,7 @@ Full CI/CD pipeline documented in `totalreclaw-internal/docs/ci-cd-pipeline.md`. 2. **E2E tests gate production promotion.** At minimum, relay smoke tests must pass against staging before promoting. 3. **npm packages tested against staging before publish.** Run `tests/verify-publish.sh` after every publish. 4. **Subgraph deploys to staging first.** Verify indexing before deploying to production subgraph. -5. **After every subgraph deploy, immediately update the relay's `SUBGRAPH_ENDPOINT` env var on Railway.** The relay queries the subgraph via this URL — a stale version means the relay reads from an outdated schema. Update both services: - - **Staging**: `railway variables set "SUBGRAPH_ENDPOINT=https://api.studio.thegraph.com/query/41768/total-reclaw-gnosis/" -s totalreclaw` - - **Production** (production runs TWO endpoint vars — free `SUBGRAPH_ENDPOINT` + `PRO_SUBGRAPH_ENDPOINT` — update BOTH): `railway variables --set "SUBGRAPH_ENDPOINT=https://api.studio.thegraph.com/query/41768/total-reclaw-gnosis/" --set "PRO_SUBGRAPH_ENDPOINT=https://api.studio.thegraph.com/query/41768/total-reclaw-gnosis/" -s totalreclaw-production` - Setting env vars triggers an automatic Railway redeploy. Verify the relay restarts cleanly by checking `/health`. NOTE: `railway variables --set` may time out on the response yet still apply — re-read with `railway variables -s --json` to confirm before retrying. Also: deploy the prod subgraph from `subgraph/subgraph-gnosis-mainnet.yaml` (the default `subgraph.yaml` is a stale base-sepolia manifest — a bare `graph deploy` targets the wrong chain). +5. **After every subgraph deploy, repoint the relay's subgraph endpoints on Railway.** Relays read through **The Graph Gateway** (staging since 2026-09-27, prod since 2026-10-01) with **Studio as fallback**: `SUBGRAPH_ENDPOINT` + `PRO_SUBGRAPH_ENDPOINT` = `https://gateway.thegraph.com/api/deployments/id/` (pinned deployment ID), `SUBGRAPH_FALLBACK_ENDPOINT` = the Studio query URL, `GRAPH_API_KEY` = one Gateway key per environment (Bearer header only; never in a URL). So a new subgraph version means: deploy to Studio → **publish to the network** → wait until the Gateway serves the new deployment at head → set all three vars on the matching service in one command. Full commands, subgraph IDs and rollback: [`docs/guides/deployment.md`](docs/guides/deployment.md) §subgraph. Setting env vars triggers an automatic Railway redeploy **from GitHub `main`**, so never change prod env vars while `main` carries unpromoted relay code. Verify with `/health/deep` (`checks.subgraph.upstream == "gateway"`). NOTE: `railway variables --set` may time out on the response yet still apply; re-read with `railway variables -s --json` before retrying. Deploy the prod subgraph from `subgraph/subgraph-gnosis-mainnet.yaml` (the default `subgraph.yaml` is a stale base-sepolia manifest). 6. **After every release event, update the release pipeline tracker (MANDATORY).** The single source of truth for "what's live / next RC / what's been QA'd" is **`totalreclaw-internal/docs/release-pipeline.md`**. The session that runs the event updates it **the same session** — never defer to a follow-up: - **RC publish** (workflow success) → update Latest RC + Status + history. @@ -504,7 +501,7 @@ For deployment procedures, follow the canonical runbook **[`docs/guides/deployme **IMPORTANT: All tests (E2E, integration, smoke, cross-client) MUST hit the staging relay (`api-staging.totalreclaw.xyz`), NEVER production.** Staging runs single-chain Gnosis (chain 100) for both tiers — same routing as production post-ops-1 (#283, closed 2026-06-05). Staging is on-chain **isolated**: it writes to its own DataEdge (`0xE7a4D2677B686e13775Ba9092631089e35F0BB91`) + dedicated `total-reclaw-gnosis-staging` subgraph, so staging E2E txs never touch production data. Test registrations send `X-TotalReclaw-Test: true` header and show as `[TEST]` in Telegram notifications. -**⚠️ SHARED DATABASE: staging + production relay share ONE Postgres (deliberate for now — Pedro, 2026-07-29).** The on-chain layer is isolated (above); the DB layer is NOT — one `tiers`/`subscriptions`/`users` table serves both environments. Consequences: (a) staging test rows live in the prod DB; (b) `tiers` is **prod-owned** — staging tier-sync (test-mode Stripe) is guarded to no-op so it can't overwrite the rows prod serves; (c) any "staging" migration or destructive write hits prod data, so every schema change must stay idempotent (`IF NOT EXISTS` / `ON CONFLICT DO NOTHING`). **Re-evaluate (split the DB) before full product launch.** Full rationale, guardrails, and split plan: `totalreclaw-internal/docs/specs/ops/shared-database-decision.md`. +**⚠️ SHARED DATABASE SERVER, SEPARATE SCHEMAS: staging + production relay share ONE Postgres instance (deliberate for now — Pedro, 2026-07-29), isolated by schema** — prod runs with `DATABASE_SCHEMA=production` (its tables live in `production.*`), staging uses the default `public` schema (verified 2026-09-24/26). So the *tables* are not shared, but the server, credentials and backups are. Consequences: (a) migrations run at boot per service against their own schema, and every migration must stay idempotent and schema-aware (catalog checks must filter on `table_schema`; an unfiltered `information_schema` check once skipped a prod column because staging had it); (b) any ad-hoc SQL must name the schema explicitly (`production.subscriptions` vs `public.subscriptions`), or you will read or write the wrong environment; (c) `tiers` is **prod-owned**: staging tier-sync (test-mode Stripe) is guarded to no-op. **Re-evaluate (split the DB) before full product launch.** Full rationale, guardrails, and split plan: `totalreclaw-internal/docs/specs/ops/shared-database-decision.md`. ### Relay Quick Reference diff --git a/docs/guides/deployment.md b/docs/guides/deployment.md index ea00138a..1ab2b92a 100644 --- a/docs/guides/deployment.md +++ b/docs/guides/deployment.md @@ -152,17 +152,39 @@ npm run deploy:prod # = graph deploy --studio total-reclaw-gnosis subgraph- ``` Pass `--version-label v` (bump from the currently-served version). -**After ANY subgraph deploy, repoint BOTH endpoint env vars on the matching relay -service** — each service runs `SUBGRAPH_ENDPOINT` AND `PRO_SUBGRAPH_ENDPOINT` (both -point at the same subgraph per env, since both tiers share one chain): +**Relays read through The Graph Gateway (since 2026-09-27 staging / 2026-10-01 prod), +with Studio as fallback.** Each relay service has three endpoint vars: `SUBGRAPH_ENDPOINT` and +`PRO_SUBGRAPH_ENDPOINT` (the primary, identical per env since both tiers share one chain), plus +`SUBGRAPH_FALLBACK_ENDPOINT` (Studio, used for one retry only on Gateway upstream errors and +never sent the key). The Gateway key is `GRAPH_API_KEY`: one key per environment, sent only as +an `Authorization: Bearer` header to Gateway hosts. **Never put the key in an endpoint URL;** the +relay refuses key-in-URL endpoints at startup and redacts them in logs. + +| Env | Network subgraph ID | Primary (pinned deployment) | Fallback | +|---|---|---|---| +| staging (`totalreclaw`) | `4SPfU4DuuiFG4zsfDuAoRexrxN2XxTuCP5beZSgKFtCU` | `https://gateway.thegraph.com/api/deployments/id/` | `https://api.studio.thegraph.com/query/41768/total-reclaw-gnosis-staging/` | +| production (`totalreclaw-production`) | `AWYQJaB4jUjEm49BPdYbVn7RP8v8usRyqTET2rRHCFTZ` | `https://gateway.thegraph.com/api/deployments/id/` | `https://api.studio.thegraph.com/query/41768/total-reclaw-gnosis/` | + +**After ANY subgraph deploy:** (1) `graph deploy` to Studio as above; (2) **publish the new +version to the network** from Studio; (3) wait until the Gateway serves the new deployment ID +at the chain head (`{ _meta { deployment block { number } hasIndexingErrors } }` with the env's +key); (4) repoint all three vars on the matching service in one command (one redeploy): ```bash # staging: -railway variables --set "SUBGRAPH_ENDPOINT=https://api.studio.thegraph.com/query/41768/total-reclaw-gnosis-staging/" \ - --set "PRO_SUBGRAPH_ENDPOINT=https://api.studio.thegraph.com/query/41768/total-reclaw-gnosis-staging/" -s totalreclaw -# production: -railway variables --set "SUBGRAPH_ENDPOINT=https://api.studio.thegraph.com/query/41768/total-reclaw-gnosis/" \ - --set "PRO_SUBGRAPH_ENDPOINT=https://api.studio.thegraph.com/query/41768/total-reclaw-gnosis/" -s totalreclaw-production +railway variables --set "SUBGRAPH_ENDPOINT=https://gateway.thegraph.com/api/deployments/id/" \ + --set "PRO_SUBGRAPH_ENDPOINT=https://gateway.thegraph.com/api/deployments/id/" \ + --set "SUBGRAPH_FALLBACK_ENDPOINT=https://api.studio.thegraph.com/query/41768/total-reclaw-gnosis-staging/" -s totalreclaw +# production (same shape, prod deployment ID + total-reclaw-gnosis/): +railway variables --set "SUBGRAPH_ENDPOINT=https://gateway.thegraph.com/api/deployments/id/" \ + --set "PRO_SUBGRAPH_ENDPOINT=https://gateway.thegraph.com/api/deployments/id/" \ + --set "SUBGRAPH_FALLBACK_ENDPOINT=https://api.studio.thegraph.com/query/41768/total-reclaw-gnosis/" -s totalreclaw-production ``` +We pin the **deployment-ID** URL (not `/api/subgraphs/id/…`) so a newly published version can't take +over reads before it's indexed. Verify with `GET /health/deep`: `checks.subgraph.upstream` must +be `gateway` and `status` `ok` (it also flags Gateway lag vs Studio beyond `SUBGRAPH_MAX_LAG_BLOCKS`). +**Rollback:** set `SUBGRAPH_ENDPOINT`/`PRO_SUBGRAPH_ENDPOINT` back to the Studio URL. Both +deployments currently have 0 GRT signal and are served by the upgrade indexer only; Studio +fallback covers that indexer going away. ### Dead-shard failure mode (Graph Studio, seen 2026-07-02) A Studio deployment can die **server-side**: queries return @@ -215,4 +237,4 @@ publish manually.** CI must be green first. Order: Core → MCP → Plugin (npm) - Production NEVER auto-deploys. - Always assert `/health.version` == the SHA you shipped. `dev` = SHA not stamped. - E2E gates prod promotion. -- After a subgraph deploy, update `SUBGRAPH_ENDPOINT` on both relay services. +- After a subgraph deploy, publish the new version to the network, then repoint `SUBGRAPH_ENDPOINT` + `PRO_SUBGRAPH_ENDPOINT` (Gateway, pinned deployment ID) and `SUBGRAPH_FALLBACK_ENDPOINT` (Studio) on the matching relay service.