docs(factories): document measurement and improvement - #522
Open
hongyi-chen wants to merge 22 commits into
Open
Conversation
- Add new top-level Factories sidebar topic with 6 stub pages (Overview, Quickstart, How Factories work, Configure your Factory, Connect your Factory, Infrastructure & security) per HYC's IA doc - Relabel the Oz tab to Automation Platform and reorganize its 10 subsections into 6 groups (Cloud Agents, Environments, Integrations, Orchestration, Self-hosting, API/SDK); all page slugs unchanged - Add vars.ts entries: API_SDK_NAME (rename-sensitive) and FACTORY_WEB_APP/FACTORY_WEB_APP_URL (net-new, for platform.warp.dev) - Extend style_lint.py RENAME_SENSITIVE_VAR_STRINGS to catch bare "Oz" and "Oz Platform", with longest-match dedup so specific matches (Oz CLI, Oz Platform, etc.) aren't double-flagged by the general bare "Oz" entry - Add Automation Platform + Warp Factories terminology.md entries - Migrate platform/index.mdx prose to the vars system - Add redirect for /platform/software-factory -> /factories/ (content migration to the new tab flagged for HYC, not deleted) Co-Authored-By: Warp Agent <agent@warp.dev>
…etting-started
Replace hardcoded rename-sensitive Oz-branded strings (Oz, Oz CLI, Oz web
app, Oz API, Oz API & SDK, Oz Platform, Oz run) with {VARS.*} references
in body prose (and {{TOKEN}} in frontmatter) across:
- src/content/docs/agents/**
- src/content/docs/terminal/comparisons and terminal/input/classic-input
- src/content/docs/getting-started/migrate-to-warp-from-claude-code
- src/content/docs/index.mdx and quickstart.mdx
Left unmigrated (intentional, per scope): image alt/figcaption text that
transcribes actual on-screen branding of existing screenshots/graphics,
and Oz mentions inside fenced code blocks (example prompts).
Co-Authored-By: Warp Agent <agent@warp.dev>
…ion banner
Migrate hardcoded Oz product-name strings (Oz CLI, Oz web app, oz.warp.dev, Oz dashboard, Oz run, Oz API & SDK, Oz Platform, bare Oz) to the vars system across src/content/docs/reference/cli/** and src/content/docs/reference/api-and-sdk/**, using {VARS.KEY} in MDX body prose and {{TOKEN}} in frontmatter. Add a shared deprecation callout to the top of every reference/cli/* page pointing to the Warp Agent CLI as the replacement for the Oz CLI. Relabel the Reference tab's CLI sidebar entry to 'Oz CLI (legacy)' in src/sidebar.ts.
Co-Authored-By: Warp Agent <agent@warp.dev>
…and-community, enterprise, changelog
Replace hardcoded rename-sensitive Oz-branded strings (Oz CLI, Oz web app,
oz.warp.dev, Oz dashboard, Oz run, Oz API & SDK, Oz Platform, bare Oz) with
src/data/vars.ts references ({VARS.KEY} in body prose, {{TOKEN}} in
frontmatter) across all guides/, support-and-community/, enterprise/, and
changelog/ files containing product-name mentions.
Left unmigrated (intentional, per style_lint OZ_TERMS_TO_AVOID / screenshot
alt-text exceptions):
- changelog/2026.mdx:501 'Oz agents' (wording issue, not vars issue)
- changelog/2026.mdx:629 'Oz cloud agent' (wording issue)
- changelog/2026.mdx:787 'the Oz agent' (wording issue)
- changelog/2026.mdx:1362 image alt text describing the 2026.02.10 launch screenshot
Co-Authored-By: Warp Agent <agent@warp.dev>
Replace hardcoded rename-sensitive strings (Oz, Oz Platform, Oz CLI,
Oz web app, Oz dashboard, Oz run, Oz API & SDK) with {VARS.*} in body
prose and {{TOKEN}} in frontmatter across src/content/docs/platform/,
excluding index.mdx and software-factory.mdx which are handled
separately.
Co-Authored-By: Warp Agent <agent@warp.dev>
…-vars-reference' into rrenk/factory-launch-restructure
…-vars-guides-support' into rrenk/factory-launch-restructure
…-vars-platform' into rrenk/factory-launch-restructure
- Add back platform/oz-web-app to the Automation Platform sidebar (accidentally dropped during the reorg, broke the build) - Give the Automation Platform topic an explicit id and associate the orphaned platform/software-factory page with it via topic frontmatter, since it's no longer listed in any sidebar but still needs a topic to resolve (content stays as source material for HYC's Factories migration) - Fix style_lint.py's check_hardcoded_vars to match on word boundaries instead of plain substring search, so short literals like bare "Oz" don't false-positive inside unrelated tokens (e.g. a YouTube share link's si=OzvuInMl8DoNR97R query param) - Fix style_lint.py's video-title check to recognize JSX expression titles, not just quoted string literals Co-Authored-By: Warp Agent <agent@warp.dev>
MDX doesn't evaluate {} expressions inside plain markdown link
destination syntax [text](url) -- only inside JSX. 12 links across 8
reference/ pages used [label]({VARS.WEB_APP_URL}/path) syntax, which
the internal link checker correctly flagged as broken (the literal
string "{VARS.WEB_APP_URL}/path" isn't a real path).
Converted all 12 to the <a href={...}>label</a> JSX pattern already
used elsewhere in the codebase, using a template literal when a path
suffix is appended to the base URL.
Verified with:
- python3 .agents/skills/check_for_broken_links/check_links.py --internal-only (0 broken links, was 12)
- npm run build (370 pages, clean)
Co-Authored-By: Warp Agent <agent@warp.dev>
…oute limit
Vercel deployment was failing with errorCode "too_many_routes":
"Maximum number of routes (rewrites, redirects, etc) exceeded. Max is
2048, received 2049." The repo's vercel.json is already right at that
ceiling (~2047 processed routes on the base branch), and my two
separate redirect entries for /platform/software-factory (trailing
and non-trailing slash) each counted as a route, pushing the total
over by one.
Merged the two entries into a single rule using an optional trailing
slash pattern ("/platform/software-factory/?"), matching both URL
forms with one redirect instead of two.
Verified with npm run build (370 pages, clean).
Note for follow-up: the docs repo is now sitting almost exactly at
Vercel's 2048-route hard limit. Any future redirect addition (by
anyone) risks tripping this same failure. Worth a separate pass to
audit and consolidate redundant slash-variant redirect pairs across
vercel.json to create headroom.
Co-Authored-By: Warp Agent <agent@warp.dev>
The previous fix used /platform/software-factory/? as the source pattern to match both trailing-slash variants in one entry, but Vercel rejected it with errorCode invalid-route-source-pattern (? is not valid syntax in this position for Vercel redirect sources). Reverted to a single plain entry using only the trailing-slash canonical form (matching the simple, proven syntax used by the adjacent /agent-platform/ entry), still keeping the net route count at +1 instead of +2 to stay under the 2048 route cap. Verified with npm run build (370 pages, clean) and a JSON validity check on vercel.json. Co-Authored-By: Warp Agent <agent@warp.dev>
Automated (oz-for-oss) review comments:
- Revert incorrect variableization of the "Oz by Warp GitHub App"
proper noun in reference/cli/integration-setup.mdx,
reference/cli/api-keys.mdx, and enterprise/team-management/
admin-panel.mdx. This is a literal, external GitHub App display
name, not a docs-rename-sensitive string -- flipping
WARP_AUTOMATION_PLATFORM would otherwise make these pages document
a GitHub App name that doesn't actually exist.
- Fix sidebar.ts's hardcoded "Oz API & SDK reference" cross-link
label to use {VARS.API_SDK_NAME} via a template literal, so it
participates in the rename. Also imported VARS in sidebar.ts and
applied the same fix to the "Oz CLI (legacy)" label I introduced
earlier in this PR.
- (The flagged broken-markdown-link issue in reference/cli/api-keys.mdx
was already fixed in a prior commit on this branch.)
Human review feedback (rachaelrenk):
- Fixed several places where "{VARS.WARP_AUTOMATION_PLATFORM} platform"
would read as a duplicated word once the var's value changes to
something like "Automation Platform" (e.g. "Automation Platform
platform"). Removed the redundant literal "platform" suffix in 5
MDX body occurrences and 2 frontmatter occurrences, since the var's
value already conveys "platform" on its own.
- Wrapped the 6 Factories stub pages in an explicit sidebar group
(matching how every other topic in sidebar.ts organizes its items)
instead of leaving them as bare top-level entries, which is what
was likely causing them to render as headings instead of pages
under one collapsible "Factories" section.
Verified with npm run build (370 pages, clean), the internal link
checker (0 broken links), and style_lint.py --all (confirmed the
"Oz by Warp" reversions and "platform platform" fixes landed as
expected, no regressions).
Co-Authored-By: Warp Agent <agent@warp.dev>
Starlight's "On this page" TOC and heading-anchor-id generation extract
heading text from the raw MDX AST before JSX expressions are evaluated,
so a heading like `### {VARS.WARP_AGENT_CLI}` doesn't render "Oz CLI"
in the TOC/anchor -- it renders the literal, unevaluated expression
source ("VARS.WARP_AGENT_CLI"), producing anchor ids like
"#varswarp_agent_cli" and a "On this page" entry reading
"VARS.WARP_AGENT_CLI" verbatim. The heading text in the page body
renders correctly (full MDX/JSX evaluation), so this only surfaces in
the TOC sidebar and in the URL fragment for that heading.
Found and fixed 40 occurrences across 21 files. In each case, reworded
the heading to avoid embedding a {VARS.X} expression, while keeping
{VARS.X} usable in the body prose beneath it (unaffected by this bug):
- Headings that were only "{VARS.X}" (e.g. "### {VARS.WEB_APP}") became
static, descriptive text ("### Web app", "### CLI", "### API", etc.)
- Headings mixing static text + a var were reworded to drop the var
while preserving meaning (e.g. "## How runners fit into the
{VARS.WARP_AUTOMATION_PLATFORM}" -> "## How runners fit into cloud
agent runs")
- The one historical changelog heading ("Introducing Oz: orchestration
for cloud agents") was set to literal "Oz" text instead of the var,
since it's a dated historical announcement and shouldn't silently
reword itself when the var flips
Also fixed a related, unrelated heading collision on
agents/capabilities/computer-use/index.mdx: its own "## Overview"
heading collided with Starlight's synthetic top-of-page "Overview" TOC
entry (auto-generated for intro content before the first heading),
producing two adjacent "Overview" rows in the TOC. Renamed to
"## Capabilities" to match its actual content.
Per review feedback, also normalized the Computer Use API section to
consistently use {VARS.API_SDK_NAME} (matching the page's own later,
correct usage) instead of "{VARS.WARP_AUTOMATION_PLATFORM} API", since
both phrases link to the same /reference/api-and-sdk/ page and the
future name for that page is "Warp API & SDK", not "Automation
Platform API".
Verified with:
- npm run build (370 pages, clean)
- Inspected the built HTML for agents/capabilities/computer-use/index.html:
TOC and anchor ids now read "Capabilities", "CLI", "API", "Web app"
instead of "VARS.WARP_AGENT_CLI" / duplicate "Overview" / broken ids
- python3 .agents/skills/check_for_broken_links/check_links.py --internal-only (0 broken links)
- style_lint.py --all (no new hardcoded-var regressions)
Co-Authored-By: Warp Agent <agent@warp.dev>
…emory page
Per review feedback: "Agent Memory gives agents in Automation Platform
persistent memory..." reads awkwardly once the WARP_AUTOMATION_PLATFORM
var flips. Since "Warp" is already a locked, stable product name (not
rename-sensitive), it's the safer and more natural choice here.
Applied the same reasoning consistently across the rest of the page,
not just the flagged frontmatter line, since the same "lives on
Automation Platform" / "runs on Automation Platform" pattern recurs in
several other sentences and would have reintroduced the same
awkwardness once the var value changes:
- Frontmatter description
- "lives on Warp" / "part of Warp" / "run on Warp"
- "Warp extracts/saves/searches/creates..." (subject of a sentence)
Left one occurrence alone ("self-hosted {VARS.WARP_AUTOMATION_PLATFORM}")
since that specifically references the self-hostable orchestration
platform as a technical concept, not a generic "in Warp" phrasing.
Also normalized the "Programmatic API access" bullet to use
{VARS.API_SDK_NAME} instead of "{VARS.WARP_AUTOMATION_PLATFORM} API",
consistent with the same fix applied elsewhere in this PR (both link
to /reference/api-and-sdk/, and the correct future name is "Warp API
& SDK", not "Automation Platform API").
Verified with npm run build (370 pages, clean).
Co-Authored-By: Warp Agent <agent@warp.dev>
…t-flip audit, round 1)
Following up on the "Automation Platform API" bug caught in review: audited
every {VARS.WARP_AUTOMATION_PLATFORM} usage across the docs for the same
class of issue and found it recurring in 14 more files (~25 occurrences).
Root cause: whenever prose names the actual API/SDK product surface (usually
as "the [X] API" linking to /reference/api-and-sdk/, or an "[X] API and SDK"
phrase), it must use {VARS.API_SDK_NAME} (future value "Warp API & SDK"),
not {VARS.WARP_AUTOMATION_PLATFORM} + a literal " API"/" SDK" suffix (which
would render the invented, non-canonical name "Warp Automation Platform
API" once the platform var flips). This is a wrong-var bug, not just a
grammar/redundancy issue -- the two vars name different things.
Fixed across: platform/overview.mdx, platform/quickstart.mdx,
reference/api-and-sdk/{index,quickstart,demo-sentry-monitoring-with-sdk}.mdx,
reference/api-and-sdk/troubleshooting/{index,errors/index}.mdx,
reference/cli/{index,artifacts}.mdx, reference/index.mdx,
platform/team-access-billing-and-identity.mdx,
support-and-community/plans-and-billing/{credits,platform-credits}.mdx,
guides/agent-workflows/how-to-run-multiple-ai-coding-agents.mdx.
Where a sentence names the API/SDK as a generic noun without further
qualification (not the proper name of the product), simplified to plain
"API"/"the API" instead of introducing an unnecessary var, e.g. "calls the
API (directly or via an SDK)" and "The Python SDK ... calls the API from
Python services".
Also applied the same generic-scaffolding-to-plain-"Warp" simplification
established on the Agent Memory page fix earlier in this PR, for sentences
like "another agent CLI outside of {var}" and "uploads to {var} --
screenshots...", since "Warp" is the stable, locked product name and reads
correctly regardless of how WARP_AUTOMATION_PLATFORM resolves.
One grammar fix: "An {VARS.WARP_AUTOMATION_PLATFORM} cloud environment"
(reference/api-and-sdk/quickstart.mdx) used "An" for agreement with "Oz"
today, but "Warp Automation Platform" needs "A" -- reworded to "A cloud
environment" to avoid the var/article dependency entirely.
Broader audit scope covered in this pass (per user direction, judging
DASHBOARD/PLATFORM_RUN under the same-shape assumption where their future
value is still TBD): swept the whole docs tree for the API_SDK_NAME
mismatch class, redundant trailing-word duplication (e.g. "platform
platform"), and "an {VAR}" article-agreement breaks across all six
rename-sensitive vars (WARP_AUTOMATION_PLATFORM, WARP_AGENT_CLI, WEB_APP,
DASHBOARD, PLATFORM_RUN, API_SDK_NAME). No further redundant-word or
article-agreement issues were found beyond what's fixed here.
Not yet exhaustively reviewed: the full ~300-occurrence long tail of plain
{VARS.WARP_AUTOMATION_PLATFORM} usages for subtler phrasing/tone issues
(the kind fixed on the Agent Memory page) -- flagging as a follow-up since
that requires reading full sentence context per occurrence rather than
pattern-matching, and is lower risk than the two bug classes fixed here.
Verified with:
- npm run build (370 pages, clean)
- check_for_broken_links.py --internal-only (0 broken links, 3495 checked)
Co-Authored-By: Warp Agent <agent@warp.dev>
Continues the systematic sweep of {VARS.X} usages for issues that would
only surface after the Oz -> Warp Automation Platform rename ships.
Fixes applied:
- "platform...platform" duplicate wording (7 more instances across
agents/index.mdx, enterprise/index.mdx, enterprise/getting-started/*,
and root index.mdx), matching the earlier computer-use/index.mdx fix.
- Leftover un-migrated literal "Oz agents" / "@oz" mentions in
reference/cli/integration-setup.mdx (missed in the original migration).
- Leftover un-migrated VideoEmbed title in platform/oz-web-app.mdx.
- Reworded platform/team-access-billing-and-identity.mdx:160, which
incorrectly described personal-token auth as "authenticating as
{platform}" instead of authenticating as the triggering user.
Policy decision (confirmed with user): changelog/2026.mdx entries are
dated historical records of what shipped under the Oz brand at the time.
Reverted all ~58 rename-sensitive var usages in that file
(WARP_AUTOMATION_PLATFORM, WARP_AGENT_CLI, PLATFORM_RUN, DASHBOARD,
WEB_APP, WEB_APP_URL) back to their current literal Oz-branded text, so
historical entries don't get silently rewritten when the rename ships.
This is consistent with the earlier "Introducing Oz" launch-heading fix
in the same file. Removed the now-unused VARS import from that file.
Verified: clean build (370 pages), 0 broken internal links, and no new
hardcoded-var lint regressions (all 71 remaining findings are pre-existing
exceptions in files untouched by this round).
Co-Authored-By: Warp Agent <agent@warp.dev>
Starlight's built-in icon set has no literal factory glyph, so a true factory-with-smokestack icon would need an icon-library plugin (e.g. Phosphor or Material Symbols via starlight-plugin-icons) plus a Sidebar component override -- out of scope for now. Using 'setting' (gear) as a placeholder since it's the closest built-in match to automated machinery, and it's visually distinct from the icons on the Terminal (laptop) and Automation Platform (cloud-download) tabs. Co-Authored-By: Warp Agent <agent@warp.dev>
The horizontal "API" tab duplicated a link already present in the Reference tab's sidebar (API & SDK > API Reference > /api). Both pointed to the same standalone Scalar-rendered page. The original reason for splitting /api into its own top-level topic was that Scalar is a different rendering framework than the rest of the Astro/Starlight site, so /api can't have a normal per-topic sidebar tree (hence it was a link-only topic with no `items`). That constraint only explains why /api itself can't be a full Starlight topic -- it doesn't require /api to also have its own top-level nav pill. The destination remains fully reachable: - From the Reference tab's existing "API Reference" sidebar link - From WarpTopbar.astro's own breadcrumb/nav on the /api page itself, which already links back to "API & SDK" and "Quickstart" Removing the topic entry needed no other changes: WarpTopicNav.astro just renders whatever topics exist, and the search breadcrumb label map in CustomSidebar.astro is keyed off URL segments, not the topic list. Also fixed a related stale reference in WarpTopicNav.astro: the custom cloud-icon override still checked `topic.label === 'Oz'`, left over from before the Automation Platform rename. Updated to `'Automation Platform'` so the custom SVG actually renders again instead of silently falling back to Starlight's generic cloud-download icon. Removed the now-dead `</>`-bracket icon branch for the deleted API topic. Verified: clean build (370 pages, /api page itself untouched) and 0 broken internal links. Co-Authored-By: Warp Agent <agent@warp.dev>
… link
Per HYC/Rachael's Slack discussion following the top-level API tab
removal: HYC flagged that dropping the tab could make the standalone
Scalar-rendered /api page harder to discover. Two changes address that
without re-adding the redundant top-level tab:
1. Renamed the "Reference" tab to "API & Reference" so the tab label
itself signals that API docs live there.
2. Promoted the "API Reference" link from 3 levels deep (Reference >
API & SDK > 3rd item) to a new "Technical Reference" group at the very
top of the tab's sidebar, as the first item -- immediately visible
without expanding anything. The group's second item is a renamed
self-link ("Overview") to the former bare "Technical reference" page
link, following the same self-link-as-first-child pattern already used
by other groups in this file (e.g. Self-hosting > Overview).
Also updated two lower-priority, non-interactive spots that mirrored the
old "Reference" label for consistency: the llms.txt generator's
customSets list (astro.config.mjs) and the search-result breadcrumb label
map (CustomSidebar.astro).
Verified: clean build (370 pages) and 0 broken internal links.
Co-Authored-By: Warp Agent <agent@warp.dev>
Order flip only, per feedback on the previous commit. Co-Authored-By: Warp Agent <agent@warp.dev>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
hongyi-chen
marked this pull request as ready for review
August 13, 2026 19:30
Contributor
|
I'm starting a first review of this pull request. You can view the conversation on Warp. I completed the review and no human review was requested for this pull request. Comment Powered by Oz |
![oz-for-oss[bot]](https://avatars.githubusercontent.com/in/3450139?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Overview
This PR adds the Factories measurement and improvement documentation, including metrics, scorers, benchmark suites, autofix, and an improvement-loop workflow. The content is generally aligned with the docs style guide, and I did not find security concerns or spec-drift issues from the provided context.
Concerns
- The new page links readers to Factories pages that are not present in the attached diff, which risks publishing broken internal links.
Verdict
Found: 1 critical, 0 important, 0 suggestions
Request changes
Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).
Powered by Oz
Comment on lines
+108
to
+109
| * [**Factory agents**](./factory-agents) - Choose responsibilities, models, harnesses, skills, and tools for specialized roles. | ||
| * [**Factory definitions as code**](./factory-as-code) - Review changes to agent prompts, skills, and factory configuration. |
Contributor
There was a problem hiding this comment.
🚨 [CRITICAL] These related-page links point to ./factory-agents and ./factory-as-code, but neither target page appears in this diff; add those pages before publishing or link to existing Factories pages so the docs link check does not fail.
Co-Authored-By: Warp Agent <agent@warp.dev>
hongyi-chen
force-pushed
the
hyc/factories-measure
branch
from
August 13, 2026 20:31
3acee02 to
020e7cc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Organizes dashboard metrics, scorers, benchmarks, and autofix around the questions they answer and their limitations. Manual/periodic scoring and a six-step evidence-driven improvement loop form the operational path.
Final size: 883 prose words. Across the section, the senior editorial pass reduced prose from about 14,600 to 7,649 words while preserving verified behavior and security caveats.
Dependency
Depends on #513, which provides the Factories section scaffold and targets
hyc/factory-launch. Until #513 merges, its shared commits appear in this PR; afterward the diff reduces to this feature's content. Branch-local CI can report missing sibling-page links (and #519 can report #513's removed hub slug) until the dependent content and shared IA branch land; the full nine-page integration build is green.Validation
npm run typecheck: passednpm run build: 372 pages built successfullyScreenshots
Not included. The revision uses sourced tables, Mermaid diagrams, and verified code/config examples; no safe approved Factory UI assets exist yet.
Unverified claims
None — all UI labels, defaults, eligibility claims, diagrams, and configuration details were verified against source or deliberately omitted.
Follow-ups
A dependent shared IA PR will add the grouped nine-page sidebar, desktop/mobile Early Access badge, and legacy generic-page cleanup after the content pages land.
Conversation: https://staging.warp.dev/conversation/5ff89820-2d80-4518-981e-178845029de1
Plan: https://staging.warp.dev/drive/notebook/7ZPKWz7hM5I59o4Gg2ptYi
Co-Authored-By: Warp Agent agent@warp.dev