gpt-6.1-sol, security and reliability fixes, working deploys

[hugbubby]

Switches investigations to gpt-6.1-sol, fixes the security and correctness bugs found in a full review, makes the extension work on the live sites again, and repairs the deploy pipeline (broken since March).

Commits are by package, so intermediate commits don't build on their own. The final tree passes typecheck, lint/knip/prettier, 680 unit, 99 integration, migration drift, build, extension e2e 21/21 and frontend e2e 13/13 (both headless). The API and frontend images run as non-root with both migrations. A live smoke test against gpt-6.1-sol passed: pnpm --filter @openerrata/api smoke:openai.

What changes

• Model: gpt-6.1-sol is set in one request config, api/src/lib/investigators/openai-request-config.ts.
  • Investigation.model records the model that actually ran, written at completion. It used to be a hardcoded OPENAI_GPT_5.
  • The audit trail has one record per OpenAI request.
  • The worker refuses to start if OpenAI rejects the request shape.
• API security:
  • Post URLs and authors come from server-verified data, which closes a javascript: link injection on the public site.
  • User OpenAI keys are verified first, can't take over server-paid runs, and are dropped rather than marking the post FAILED.
  • Image fetches and fetch_url are SSRF-safe.
  • Re-queuing no longer overwrites audit history.
• Queue:
  • At most SELECTOR_DAILY_BUDGET selector admissions per UTC day, matching SPEC; it was per 5-minute run.
  • Lease ⇔ PROCESSING is enforced by the database, and losing the lease aborts the run.
  • The selector uses update lineage.
• Extension 0.4.0:
  • Fixes posts stuck on "investigating", duplicate injection, and highlight markup leaking into submitted HTML.
  • Fixes extraction on logged-out X and the Substack free-unlock paywall.
  • Excludes boilerplate on every Wikipedia language and Substack page chrome, and <br> now separates words.
• Interim carry-forward: a changed post keeps showing the earlier corrections whose exact text is still on the page, on every platform. SPEC §2.8/§2.9 are updated.
• Website:
  • Responses are validated against the shared schemas.
  • Real 404/502 pages.
  • Only http(s) links, no remote images, and a CSP.
• Deploy:
  • GitHub OIDC (AWS_DEPLOY_ROLE_ARN), Tailscale (tag:ci) and aws eks update-kubeconfig replace the static AWS keys and KUBE_CONFIG_DATA.
  • Private RDS (ingress 10.0.0.0/16, over the existing VPC peering).
  • No public bucket policy, non-root pods, and pods wait for migrations.
  • Deploys are serialized per stack without cancellation.
• Local dev: Postgres 17, and Versity S3 Gateway in place of MinIO, which can no longer be pulled.

Migrations

• 0024: Investigation.model becomes the recorded provider id, and the audit moves to InvestigationAttemptRequest rows. Existing audits become one LEGACY_COMBINED request each. It aborts if any COMPLETE investigation lacks exactly one SUCCEEDED attempt.
• 0025: adds investigation admission origin/time, deferred lease-invariant triggers and input snapshot fields. It repairs non-https post URLs and drops unused columns.

Already done outside the repo

• IAM: role openerrata-github-actions-deploy, its policy, and an EKS access entry mapping it to group openerrata-ci.
• Cluster: group RBAC applied, and the old ServiceAccount token revoked.
• Repo variables: AWS_DEPLOY_ROLE_ARN and PULUMI_MANAGED_DATABASE_{PUBLICLY_ACCESSIBLE=false, INGRESS_CIDRS=10.0.0.0/16, ENGINE_VERSION=17}. 17.9 is the current default minor, so this causes no restart.
• Secrets: TS_OAUTH_CLIENT_ID and TS_AUDIENCE are set.

Rollout

1. Merge into staging. This deploys staging and is the first test of the new deploy path.
2. Upload extension 0.4.0 to the stores with publishing deferred.
3. Once the store approves it, promote staging → main and publish 0.4.0 at the same time. The API's minimum supported extension version becomes 0.4.0, and the new extension needs the new API.
4. After main deploys cleanly, delete the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION and KUBE_CONFIG_DATA secrets and the openerrata-ci IAM user. The commands are in the README.

Behavior changes

• Fewer automatic investigations: at most 100 per day.
• The extraction fixes give many already-investigated posts a new content version. Carry-forward keeps their still-applicable corrections visible until the selector re-checks them.
• Paid Substack posts are skipped even for paying subscribers.
• Laptops lose direct database access, except via the tailnet.
• Staging's deactivated image-upload key is reactivated by the first deploy.

🤖 Generated with Claude Code