Skip to content

HELD: Light OAuth for staff on production, with the hosted MCP exchange - #3247

Draft
jeremy wants to merge 5 commits into
mainfrom
oauth-light-production-staff
Draft

jeremy wants to merge 5 commits into
mainfrom
oauth-light-production-staff

Conversation

@jeremy

@jeremy jeremy commented Oct 26, 2026 •

Copy link
Copy Markdown
Member

Held: don't merge until the prerequisites below are done, on Jeremy's word.

Turns Fizzy's OAuth authorization server on in production for 37signals staff only, and lights the hosted MCP server's delegation exchange.

Change

  • Production app env:
    • OAUTH_ACCEPTANCE_ENABLED: discovery, consent and token endpoints answer.
    • OAUTH_CIMD_ENABLED: Client ID Metadata Documents from the hosts already allowlisted in deploy.yml.
    • FIZZY_OAUTH_MCP_RESOURCES: grants can be audienced to https://mcp.fizzy.do/mcp.
  • fizzy_mcp accessory:
    • The exchange's token URL and client id.
    • Its client secret, read from the deploy secrets.
    • FIZZY_MCP_UPSTREAM=http://host.docker.internal with --add-host host.docker.internal:host-gateway, which resolves the TODO that stood in its place. The exchange and the API calls after it go to the app on the accessory's own iad web host, through that host's kamal-proxy, over the plain http the balancers already use. They stay in iad without depending on any one balancer. The issuer, token endpoint and API base keep their https URLs; only the connection moves (fizzy-mcp-server#19).
    • app.fizzy.do:host-gateway as well, so an image without FIZZY_MCP_UPSTREAM fails its startup metadata check rather than reaching Fizzy outside iad.

OAuth stays limited to staff by OAUTH_STAFF_ONLY (#3246); this PR doesn't change that.

Prerequisites, in order

  1. The OAuth train, including the staff gate (Limit OAuth to 37signals staff while it's piloted #3246), is on main and deployed to production.
  2. mcp.fizzy.do resolves and serves HTTPS.
  3. The production fizzy-mcp-server client is provisioned with bin/rails oauth:provision_first_party_clients, run once with FIZZY_OAUTH_MCP_RESOURCES=https://mcp.fizzy.do/mcp passed explicitly, because production only gets that variable when this PR lands. Its secret is then minted with bin/rails "oauth:mint_client_secret[fizzy-mcp-server]" into the deploy secrets. Until that secret exists, any production deploy from this branch fails at secret fetch, which is why this PR is held.
  4. Done: fizzy-mcp-server#19 is released as v0.2.1, and fizzy_mcp's image in deploy.yml is pinned to that release's digest on main (Pin fizzy-mcp-server v0.2.1 #3254).
  5. Done: Sweep dynamic registrations nobody authorized after a day #3255 is merged to main. It prunes dynamically registered clients that were never authorized after 24 hours, and runs cleanup_oauth_clients hourly.

After merge

  1. Deploy production.
  2. Reboot fizzy_mcp one host at a time (saas/bin/mcp-rolling-reboot).

@jeremy

jeremy commented Oct 28, 2026

Copy link
Copy Markdown
Member Author

@codex review

Merged main into this branch (cc84818). One conflict, in saas/.kamal/secrets.production: main added Production/OAUTH_TOKEN_PEPPER to the fetch list, so this branch's Production/FIZZY_MCP_EXCHANGE_CLIENT_SECRET now follows it. deploy.production.yml merged cleanly and keeps OAUTH_STAFF_ONLY: "true" and the kamal-proxy v0.10.2 pin.

@jeremy
jeremy requested a balanced review from Copilot October 28, 2026 02:56
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-29T05:20:44.959418Z 53b3a0c Manual request
🔒 Security Review ✅ Completed 2026-10-29T05:22:15.614933Z 53b3a0c Manual request

Security findings

Advisory findings (1)

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The hosts-file override pins both MCP replicas to one load balancer, preventing DNS failover and causing both to lose Fizzy connectivity if that balancer fails.

1 open issue
What changed in this PR

Enables Fizzy’s production OAuth server and hosted MCP delegation exchange while retaining the staff-only gate from #3246. The PR remains held pending deployment, HTTPS, and client-secret provisioning prerequisites.

Changes:

  • Enables OAuth endpoints, allowlisted client metadata documents, and the hosted MCP resource.
  • Configures MCP exchange credentials and pins outbound Fizzy traffic to an iad load balancer.
  • Adds the exchange client secret to production deploy-secret fetching and extraction.

[!TIP]
If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or run gh pr ready --undo.
Click "Ready for review" or run gh pr ready to reengage.

File Description
saas/​config/​deploy.production.yml Enables OAuth and configures the MCP exchange, secret injection, and iad routing.
saas/​.kamal/​secrets.production Fetches and extracts the MCP exchange client secret.

Review effort: Balanced

Comment thread saas/config/deploy.production.yml Outdated
@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: cc8481855f

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: cc8481855f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@jeremy

jeremy commented Oct 28, 2026

Copy link
Copy Markdown
Member Author

State on cc84818: Codex code and security reviews found nothing. Copilot left one thread. The SaaS test jobs fail, and they failed the same way on eadbce0 before the merge. Both problems come from the add-host ERB, and both need a decision before this can land.

  • CI: Test (SaaS) / Tests (SQLite) has 6 errors in HotcellAccessoryTest. That test loads saas/config/deploy.yml for every destination through Kamal::Configuration.create_from, which evaluates the ERB, and Resolv.getaddress raises Resolv::ResolvError on a runner that can't resolve the internal name. The MySQL SaaS job was cancelled after that failure.
  • Copilot (HELD: Light OAuth for staff on production, with the hosted MCP exchange #3247 (comment)): pinning app.fizzy.do to one balancer gives both MCP replicas a single point of failure. Main's TODO named "iad's load balancers", plural.

I haven't changed anything here. Ways to fix it: resolve at deploy time through something other than config-read ERB; have the test stub the lookup; or point the exchange and API at an iad-local name that covers both balancers, which would also answer Copilot. That last one is a networking change for Ops. The thread stays open until someone decides.

The exchange and the API calls after it go to the app on the accessory's own
host through Docker's host-gateway and that host's kamal-proxy, over the plain
http the load balancer already uses to reach it. They stay in iad with no
dependency on a single balancer, and reading the config no longer resolves a
hostname, which CI's runners can't.

The issuer's name maps to the host too, so an image without FIZZY_MCP_UPSTREAM
fails its startup metadata check rather than reaching Fizzy outside iad.
@jeremy

jeremy commented Oct 28, 2026

Copy link
Copy Markdown
Member Author

@codex review

d0df105 replaces the ERB add-host, which resolved one iad balancer's address when the config was read. Each fizzy_mcp copy now reaches the app on its own iad web host instead, through Docker's host-gateway and that host's kamal-proxy, over plain http (FIZZY_MCP_UPSTREAM, fizzy-mcp-server#19). The config no longer resolves anything when it's read, so the SaaS test jobs should pass again. The Copilot thread is answered on its line.

This stays a held draft. The new prerequisite 4 in the description says why: the image pin has to move to a fizzy-mcp-server release that carries #19.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: d0df10506f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Production rollout requires operator confirmation of DNS/TLS, client and secret provisioning, and a pinned MCP release, while the unavailable upstream PR leaves routing and token exchange needing deployment validation.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: d0df10506f

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Production activation awaits operator confirmation of deployment, DNS/TLS, and client-secret provisioning, while the inaccessible MCP implementation leaves same-host routing and exchange requiring deployment validation.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@jeremy

jeremy commented Oct 29, 2026

Copy link
Copy Markdown
Member Author

@codex review
Merged main (v0.2.1 pin, #3254) at 1ff574c; please review this head.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: 1ff574c443

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛡️ Codex Security Review · Automatically triggered

Here are some automated security review suggestions for this pull request.

Reviewed commit: 1ff574c443

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

Comment thread saas/config/deploy.production.yml
@jeremy

jeremy commented Oct 29, 2026

Copy link
Copy Markdown
Member Author

@codex review
Re-requesting the security review on 1ff574c: the code review receipt is in, the security receipt is not.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: 1ff574c443

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@jeremy

jeremy commented Oct 29, 2026

Copy link
Copy Markdown
Member Author

State on 1ff574c, which merges main (#3240, and #3254's fizzy-mcp-server v0.2.1 pin) into this branch. The merge had no conflicts, and the diff against main is still exactly this PR's production env, fizzy_mcp exchange, upstream and add-host entries, and secret fetch. Prerequisite 4 in the body is now marked done.

  • CI: green on 1ff574c.
  • Codex code review: no major issues on 1ff574c.
  • Copilot: reviewed 1ff574c and found no open issues. Its overview only notes that the production prerequisites need an operator, which the body already lists.
  • Codex security review: one P2 finding, which is open for a decision: unauthenticated dynamic client registration has only a per-IP budget. It checks out against main. The fix belongs in the app, in its own PR. Whether it gates lighting acceptance is Jeremy's call.

Still held as a draft. The remaining prerequisites are 1 to 3.

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 1ff574c443

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

* origin/main:
  Sweep dynamic registrations nobody authorized after a day, not thirty
@jeremy
jeremy requested a balanced review from Copilot October 29, 2026 05:18
@jeremy

jeremy commented Oct 29, 2026

Copy link
Copy Markdown
Member Author

@codex review
Merged main at 53b3a0c, bringing in #3255 (never-authorized dynamically registered clients pruned after 24 hours, cleanup_oauth_clients hourly). This PR's diff against main is unchanged. Please run the code and security reviews on this head.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Production activation awaits operator confirmation of deployment, DNS/TLS, staff gating, and client-secret provisioning, while host-local MCP exchange and API calls require validation because the upstream implementation was unavailable for review.

No open issues 🎉

Review effort: Balanced

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: 53b3a0cb6d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 53b3a0cb6d

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants