Skip to content

docs: Graph Gateway subgraph runbook + correct shared-DB note #672

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
p-diogo merged 1 commit into from
Oct 1, 2026
Merged

docs: Graph Gateway subgraph runbook + correct shared-DB note #672

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 2 additions & 5 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -480,10 +480,7 @@ Full CI/CD pipeline documented in `totalreclaw-internal/docs/ci-cd-pipeline.md`.
2. **E2E tests gate production promotion.** At minimum, relay smoke tests must pass against staging before promoting.
3. **npm packages tested against staging before publish.** Run `tests/verify-publish.sh` after every publish.
4. **Subgraph deploys to staging first.** Verify indexing before deploying to production subgraph.
5. **After every subgraph deploy, immediately update the relay's `SUBGRAPH_ENDPOINT` env var on Railway.** The relay queries the subgraph via this URL — a stale version means the relay reads from an outdated schema. Update both services:
- **Staging**: `railway variables set "SUBGRAPH_ENDPOINT=https://api.studio.thegraph.com/query/41768/total-reclaw-gnosis/<new-version>" -s totalreclaw`
- **Production** (production runs TWO endpoint vars — free `SUBGRAPH_ENDPOINT` + `PRO_SUBGRAPH_ENDPOINT` — update BOTH): `railway variables --set "SUBGRAPH_ENDPOINT=https://api.studio.thegraph.com/query/41768/total-reclaw-gnosis/<new-version>" --set "PRO_SUBGRAPH_ENDPOINT=https://api.studio.thegraph.com/query/41768/total-reclaw-gnosis/<new-version>" -s totalreclaw-production`
Setting env vars triggers an automatic Railway redeploy. Verify the relay restarts cleanly by checking `/health`. NOTE: `railway variables --set` may time out on the response yet still apply — re-read with `railway variables -s <svc> --json` to confirm before retrying. Also: deploy the prod subgraph from `subgraph/subgraph-gnosis-mainnet.yaml` (the default `subgraph.yaml` is a stale base-sepolia manifest — a bare `graph deploy` targets the wrong chain).
5. **After every subgraph deploy, repoint the relay's subgraph endpoints on Railway.** Relays read through **The Graph Gateway** (staging since 2026-09-27, prod since 2026-10-01) with **Studio as fallback**: `SUBGRAPH_ENDPOINT` + `PRO_SUBGRAPH_ENDPOINT` = `https://gateway.thegraph.com/api/deployments/id/<Qm…>` (pinned deployment ID), `SUBGRAPH_FALLBACK_ENDPOINT` = the Studio query URL, `GRAPH_API_KEY` = one Gateway key per environment (Bearer header only; never in a URL). So a new subgraph version means: deploy to Studio → **publish to the network** → wait until the Gateway serves the new deployment at head → set all three vars on the matching service in one command. Full commands, subgraph IDs and rollback: [`docs/guides/deployment.md`](docs/guides/deployment.md) §subgraph. Setting env vars triggers an automatic Railway redeploy **from GitHub `main`**, so never change prod env vars while `main` carries unpromoted relay code. Verify with `/health/deep` (`checks.subgraph.upstream == "gateway"`). NOTE: `railway variables --set` may time out on the response yet still apply; re-read with `railway variables -s <svc> --json` before retrying. Deploy the prod subgraph from `subgraph/subgraph-gnosis-mainnet.yaml` (the default `subgraph.yaml` is a stale base-sepolia manifest).

6. **After every release event, update the release pipeline tracker (MANDATORY).** The single source of truth for "what's live / next RC / what's been QA'd" is **`totalreclaw-internal/docs/release-pipeline.md`**. The session that runs the event updates it **the same session** — never defer to a follow-up:
- **RC publish** (workflow success) → update Latest RC + Status + history.
Expand All @@ -504,7 +501,7 @@ For deployment procedures, follow the canonical runbook **[`docs/guides/deployme

**IMPORTANT: All tests (E2E, integration, smoke, cross-client) MUST hit the staging relay (`api-staging.totalreclaw.xyz`), NEVER production.** Staging runs single-chain Gnosis (chain 100) for both tiers — same routing as production post-ops-1 (#283, closed 2026-06-05). Staging is on-chain **isolated**: it writes to its own DataEdge (`0xE7a4D2677B686e13775Ba9092631089e35F0BB91`) + dedicated `total-reclaw-gnosis-staging` subgraph, so staging E2E txs never touch production data. Test registrations send `X-TotalReclaw-Test: true` header and show as `[TEST]` in Telegram notifications.

**⚠️ SHARED DATABASE: staging + production relay share ONE Postgres (deliberate for now — Pedro, 2026-07-29).** The on-chain layer is isolated (above); the DB layer is NOT — one `tiers`/`subscriptions`/`users` table serves both environments. Consequences: (a) staging test rows live in the prod DB; (b) `tiers` is **prod-owned** — staging tier-sync (test-mode Stripe) is guarded to no-op so it can't overwrite the rows prod serves; (c) any "staging" migration or destructive write hits prod data, so every schema change must stay idempotent (`IF NOT EXISTS` / `ON CONFLICT DO NOTHING`). **Re-evaluate (split the DB) before full product launch.** Full rationale, guardrails, and split plan: `totalreclaw-internal/docs/specs/ops/shared-database-decision.md`.
**⚠️ SHARED DATABASE SERVER, SEPARATE SCHEMAS: staging + production relay share ONE Postgres instance (deliberate for now — Pedro, 2026-07-29), isolated by schema** — prod runs with `DATABASE_SCHEMA=production` (its tables live in `production.*`), staging uses the default `public` schema (verified 2026-09-24/26). So the *tables* are not shared, but the server, credentials and backups are. Consequences: (a) migrations run at boot per service against their own schema, and every migration must stay idempotent and schema-aware (catalog checks must filter on `table_schema`; an unfiltered `information_schema` check once skipped a prod column because staging had it); (b) any ad-hoc SQL must name the schema explicitly (`production.subscriptions` vs `public.subscriptions`), or you will read or write the wrong environment; (c) `tiers` is **prod-owned**: staging tier-sync (test-mode Stripe) is guarded to no-op. **Re-evaluate (split the DB) before full product launch.** Full rationale, guardrails, and split plan: `totalreclaw-internal/docs/specs/ops/shared-database-decision.md`.

### Relay Quick Reference

Expand Down
40 changes: 31 additions & 9 deletions docs/guides/deployment.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,17 +152,39 @@ npm run deploy:prod # = graph deploy --studio total-reclaw-gnosis subgraph-
```
Pass `--version-label v<X.Y.Z>` (bump from the currently-served version).

**After ANY subgraph deploy, repoint BOTH endpoint env vars on the matching relay
service** — each service runs `SUBGRAPH_ENDPOINT` AND `PRO_SUBGRAPH_ENDPOINT` (both
point at the same subgraph per env, since both tiers share one chain):
**Relays read through The Graph Gateway (since 2026-09-27 staging / 2026-10-01 prod),
with Studio as fallback.** Each relay service has three endpoint vars: `SUBGRAPH_ENDPOINT` and
`PRO_SUBGRAPH_ENDPOINT` (the primary, identical per env since both tiers share one chain), plus
`SUBGRAPH_FALLBACK_ENDPOINT` (Studio, used for one retry only on Gateway upstream errors and
never sent the key). The Gateway key is `GRAPH_API_KEY`: one key per environment, sent only as
an `Authorization: Bearer` header to Gateway hosts. **Never put the key in an endpoint URL;** the
relay refuses key-in-URL endpoints at startup and redacts them in logs.

| Env | Network subgraph ID | Primary (pinned deployment) | Fallback |
|---|---|---|---|
| staging (`totalreclaw`) | `4SPfU4DuuiFG4zsfDuAoRexrxN2XxTuCP5beZSgKFtCU` | `https://gateway.thegraph.com/api/deployments/id/<Qm…>` | `https://api.studio.thegraph.com/query/41768/total-reclaw-gnosis-staging/<v>` |
| production (`totalreclaw-production`) | `AWYQJaB4jUjEm49BPdYbVn7RP8v8usRyqTET2rRHCFTZ` | `https://gateway.thegraph.com/api/deployments/id/<Qm…>` | `https://api.studio.thegraph.com/query/41768/total-reclaw-gnosis/<v>` |

**After ANY subgraph deploy:** (1) `graph deploy` to Studio as above; (2) **publish the new
version to the network** from Studio; (3) wait until the Gateway serves the new deployment ID
at the chain head (`{ _meta { deployment block { number } hasIndexingErrors } }` with the env's
key); (4) repoint all three vars on the matching service in one command (one redeploy):
```bash
# staging:
railway variables --set "SUBGRAPH_ENDPOINT=https://api.studio.thegraph.com/query/41768/total-reclaw-gnosis-staging/<v>" \
--set "PRO_SUBGRAPH_ENDPOINT=https://api.studio.thegraph.com/query/41768/total-reclaw-gnosis-staging/<v>" -s totalreclaw
# production:
railway variables --set "SUBGRAPH_ENDPOINT=https://api.studio.thegraph.com/query/41768/total-reclaw-gnosis/<v>" \
--set "PRO_SUBGRAPH_ENDPOINT=https://api.studio.thegraph.com/query/41768/total-reclaw-gnosis/<v>" -s totalreclaw-production
railway variables --set "SUBGRAPH_ENDPOINT=https://gateway.thegraph.com/api/deployments/id/<newQm>" \
--set "PRO_SUBGRAPH_ENDPOINT=https://gateway.thegraph.com/api/deployments/id/<newQm>" \
--set "SUBGRAPH_FALLBACK_ENDPOINT=https://api.studio.thegraph.com/query/41768/total-reclaw-gnosis-staging/<v>" -s totalreclaw
# production (same shape, prod deployment ID + total-reclaw-gnosis/<v>):
railway variables --set "SUBGRAPH_ENDPOINT=https://gateway.thegraph.com/api/deployments/id/<newQm>" \
--set "PRO_SUBGRAPH_ENDPOINT=https://gateway.thegraph.com/api/deployments/id/<newQm>" \
--set "SUBGRAPH_FALLBACK_ENDPOINT=https://api.studio.thegraph.com/query/41768/total-reclaw-gnosis/<v>" -s totalreclaw-production
```
We pin the **deployment-ID** URL (not `/api/subgraphs/id/…`) so a newly published version can't take
over reads before it's indexed. Verify with `GET /health/deep`: `checks.subgraph.upstream` must
be `gateway` and `status` `ok` (it also flags Gateway lag vs Studio beyond `SUBGRAPH_MAX_LAG_BLOCKS`).
**Rollback:** set `SUBGRAPH_ENDPOINT`/`PRO_SUBGRAPH_ENDPOINT` back to the Studio URL. Both
deployments currently have 0 GRT signal and are served by the upgrade indexer only; Studio
fallback covers that indexer going away.

### Dead-shard failure mode (Graph Studio, seen 2026-07-02)
A Studio deployment can die **server-side**: queries return
Expand Down Expand Up @@ -215,4 +237,4 @@ publish manually.** CI must be green first. Order: Core → MCP → Plugin (npm)
- Production NEVER auto-deploys.
- Always assert `/health.version` == the SHA you shipped. `dev` = SHA not stamped.
- E2E gates prod promotion.
- After a subgraph deploy, update `SUBGRAPH_ENDPOINT` on both relay services.
- After a subgraph deploy, publish the new version to the network, then repoint `SUBGRAPH_ENDPOINT` + `PRO_SUBGRAPH_ENDPOINT` (Gateway, pinned deployment ID) and `SUBGRAPH_FALLBACK_ENDPOINT` (Studio) on the matching relay service.
Loading