feat(client-generator): agent-friendly generators — python, go, php, cli + neutral authoring toolkit and eject workflow - #3016
feat(client-generator): agent-friendly generators — python, go, php, cli + neutral authoring toolkit and eject workflow#3016Marshevskyy wants to merge 132 commits into
Conversation
🦋 Changeset detectedLatest commit: e8df6a4 The changes in this PR will be included in the next version bump. This PR includes changesets to release 4 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Performance Benchmark (Lower is Faster)
|
|
📦 A new experimental 🧪 version v0.0.0-snapshot.1785856098 of Redocly CLI has been published for testing. Install with NPM: npm install @redocly/cli@0.0.0-snapshot.1785856098 |
Feedback from Rebilly (Replay Admin + Rebilly Core)Tried
Overall: generation succeeds for both; TypeScript still works when co-selected with 1. Core list one-shot vs
|
Follow-up: Python generator (same snapshot)Tried
Overall: generation succeeds; sync + async clients work; offline smokes pass for Admin cursor pagination and Core offset pagination + Python is in good shape relative to PHP on several points we flagged earlier — calling those out as already-good, then listing Python-specific asks. Already better than PHP (positive)
Python-specific asks1. Output filename is not a normal import pathEmitted names follow the TS stem:
Neither is a valid/idiomatic Python module name ( Ask: emit an importable name by default, e.g. 2. Cross-language auth config key mismatchPython resolves API keys from Ask: document the per-language auth dict shape next to each other (or normalize to one key with a documented alias). Easy footgun when someone copies a TS/PHP snippet into Python. 3. Reserved-word fields (
|
Clarification on single-file outputOne more product note on the earlier “large single-file” asks (PHP ~4.7MB / Python ~4.4MB for Rebilly Core): We like the single-file default and would keep it. It’s a great “Download client” artifact from Redoc/API docs — one Please treat the size/split comments as optional escape hatches for huge descriptions, not a request to change the default:
The other asks still stand, especially for Python: importable filename ( |
Follow-up: Go generator (same snapshot)Tried
Overall: generation succeeds; Same single-file preference as before: keep the one-file default for Redoc/API docs downloads. Core Go is notably smaller than Core PHP (~4.7MB) / Python (~4.4MB), which helps that story. Already in good shape (positive)
Go-specific asks1. Emit
|
Follow-up:
|
|
|
||
| Added agent-friendly client generation: built-in `python`, `go`, `php`, and `cli` generators, a language-neutral authoring toolkit with a per-generator `AGENTS.md` skill, an `eject-generator` command, `x-codeSamples` output, and verification against large real-world descriptions — with every generator now emitting through source-text templates. | ||
|
|
||
| **Note:** the AST exports (`ts`, `printStatements`, `schemaToTypeNode`, …) were removed from `@redocly/client-generator/generate` in favor of the text toolkit (`tsType`, `tsJsdoc`, `codeLiteral`). |
| - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 | ||
| with: | ||
| python-version: '3.12' | ||
| - name: Install httpx (the large-descriptions Python import bar needs it) | ||
| run: pip install httpx | ||
| - name: Cache the pinned GitHub REST description | ||
| uses: actions/cache@668228422ae6a00e4ad889ee87cd7109ec5666a7 # v5.0.4 | ||
| with: | ||
| path: tests/e2e/generate-client/.cache | ||
| key: large-descriptions-${{ hashFiles('tests/e2e/generate-client/large-descriptions.test.ts') }} |
There was a problem hiding this comment.
I think we need a separate test suite for generator.
I think it will be very big and slow eventually when we start testing compiled languages.
There was a problem hiding this comment.
Agreed, and it's already the biggest suite. Splitting generator tests (client-generator unit +
tests/e2e/generate-client, including the large-description compile bars) into their own vitest
suite and CI job so compiled-language testing can grow without slowing the main e2e job.
| Your agent (or you) edits the generator, `redocly generate-client` rebuilds the client, and next week's spec change regenerates with the customization intact. | ||
|
|
||
| Ejectable generators: `python`, `go`, `php` — the language generators built on the language-neutral authoring toolkit. | ||
| The TypeScript `sdk` and its satellite generators are customized through `client.setup`, middleware, and configuration instead; running `eject-generator sdk` prints that guidance. |
There was a problem hiding this comment.
I think typescript and satellites should be ejectable too as separate generators
There was a problem hiding this comment.
Agreed. Docs now say every built-in is ejectable. Mechanically the language generators are one
self-contained file, while the TypeScript ones are thin entries over shared emitters, so eject
will bundle each generator with the emitters it uses into a single .mjs, keeping the
@redocly/client-generator imports external. sdk.mjs will be large, but it's one file you own,
which is the point.
| | ---------- | ------- | ---------------------------------------------------------------------------------------------------- | | ||
| | generator | string | Built-in generator to eject: `python`, `go`, or `php`. | | ||
| | `--dir` | string | Directory to eject into. Default `./generators`. | | ||
| | `--update` | boolean | Three-way merge a newer generator version into your customized copy; conflicts get standard markers. | |
| Ejecting writes four things: | ||
|
|
||
| - `<dir>/<name>.mjs` — the generator, the exact code the built-in runs, readable plain ESM. | ||
| - `<dir>/.pristine/<name>.mjs` — a pristine snapshot (commit it); `--update` uses it as the merge base. |
There was a problem hiding this comment.
I think we don't need it. can update compare with the latest upstream version?
There was a problem hiding this comment.
Dropping .pristine/ entirely. --update now uses the version recorded in the ejected file's
header as the merge base, so nothing extra is committed and there's nothing to keep in sync.
| ### Code samples for docs | ||
|
|
||
| A generator that knows how to call an operation can also document it: implement the optional `sample(operation, ctx)` hook to return one idiomatic snippet (`{ lang, label, source }`) per operation. | ||
| With `codeSamples: true` in the `client` block, generation collects every selected generator's samples into `<output stem>.code-samples.yaml` — an [OpenAPI Overlay](https://spec.openapis.org/overlay/latest.html) adding `x-codeSamples` per operation, ready for docs tooling to apply. |
There was a problem hiding this comment.
I think we don't have support for Overlays. We need to either add support for overlays in bundle or figure out some other format
|
|
||
| The `cli` generator emits `<stem>.cli.ts` — a zero-dependency, bin-ready command-line interface over the generated client. | ||
| Path params are positional, query params become typed `--kebab-name` flags (enums list their choices in `--help`, array params repeat the flag), and JSON request bodies arrive via `--json '<json>'`, `--json @file.json`, or `--json @-` (stdin). | ||
| When `zod` is co-selected, requests are validated before they are sent. |
There was a problem hiding this comment.
I think it shoud just do it by default without the need to coselect zod
There was a problem hiding this comment.
Done. cli declares requires: ['sdk', 'zod'] and the resolver now pulls prerequisites in, so
--generator cli alone emits all three files and validates (exit 3) with nothing extra to
select. Same for the wrappers: --generator tanstack-query brings the sdk it wraps. The
trade-off is a zod runtime dependency for the CLI
| npx tsx src/client.cli.ts orders listOrders --status open --limit 10 | ||
| npx tsx src/client.cli.ts orders createOrder --json @order.json | ||
| npx tsx src/client.cli.ts orders listOrders --page-all # one JSON page per line | ||
| npx tsx src/client.cli.ts schema createOrder # request/response schemas |
| `tanstack-query`, `swr`, and `cli` wrap the throw-mode `sdk` client, so they require `--error-mode throw`; `transformers` requires `--date-type Date`. | ||
| See the [`zod`](https://github.com/Redocly/redocly-cli/tree/main/tests/e2e/generate-client/examples/zod), [`tanstack-query`](https://github.com/Redocly/redocly-cli/tree/main/tests/e2e/generate-client/examples/tanstack-query), and [`mock`](https://github.com/Redocly/redocly-cli/tree/main/tests/e2e/generate-client/examples/mock) examples. | ||
|
|
||
| ### Generated CLI |
There was a problem hiding this comment.
I think it should also be able to generate documentation for the CLI (markdown file)
But the same applies for other clients so it can be the next stage.
There was a problem hiding this comment.
It may need some documentation templates that can be ejected then too.
There was a problem hiding this comment.
Built it — there's now a cli-docs generator that writes .cli.md beside the generated CLI: usage, global flags, credential environment variables, the exit-code table, and one section per command with its positional arguments and flags (type, required, enum choices, description). --generator cli-docs is the whole selection, since it pulls in the CLI it documents.
On ejectable templates: the renderer is the template. Every built-in generator is ejectable now, so redocly eject-generator cli-docs hands you the page layout as code you own — same language and toolkit as everything else, no template syntax and no extra dependency to learn. Light customization stays declarative through the generator's own options (title, frontmatter) under client.options.cli-docs; anything structural is an eject. We deliberately didn't add a template engine, because that would be a second customization mechanism sitting next to eject.
One property worth calling out: the page renders from the same command table the CLI dispatches on, and the same functions the runtime uses to address groups and name credential variables — so the docs can't drift from the tool. An e2e walks the generated CLI's own --help at both levels and fails if any command it lists is missing a section.
| The `python` generator emits a self-contained `<stem>.py` next to the configured output — a full Python SDK over [httpx](https://www.python-httpx.org/) (`pip install httpx`, Python ≥ 3.9): | ||
| typed dataclass models (allOf flattened, enums, discriminated unions decoded by their discriminator), a `Client` and an `AsyncClient` with one method per operation, auth, retries with `Retry-After` and jittered backoff, timeouts, idempotency keys, middleware hooks, pagination iterators (`<op>_pages()` / `<op>_items()`, `async for` variants), SSE streaming, multipart bodies, `<op>_with_headers()` envelope variants for operations that declare response headers, and a `Servers` class for templated server URLs. |
There was a problem hiding this comment.
Why we need all of this details. It should be same as typescript one
There was a problem hiding this comment.
it doesn't support any of options we support for typescript? why?
There was a problem hiding this comment.
You were right on both counts, and one of them was a bug: serverUrl was silently ignored by the
language generators (fixed, 795f58ca9). dateType: Date is now implemented for all three
(ea3795e47). Options a language genuinely can't apply no longer vanish — they warn with the
reason, or fail fast where the output would be wrong (d6cf2ba4e). The docs no longer enumerate
per-language caveats: the language sections state that these are the TypeScript client in another
language, with one table for the differences the language forces (error idiom, date type, header
envelope, auth shape, reserved-word suffix, file layout).
| A generator adds artifacts _next to_ the client — it doesn't change the generated client's behavior; for that, use [publisher defaults](#publisher-defaults) or let the consumer compose [middleware](./use-generated-client.md#middleware). | ||
|
|
||
| A generator is `{ name, run }` (plus optional compatibility metadata); author it with `defineGenerator` from the package root, and build real TypeScript with the emit toolkit from `@redocly/client-generator/generate` — the same `ts.factory` + printer the built-in generators use, so the schema→type mapping matches the sdk's exactly: | ||
| A generator is `{ name, run }` (plus optional compatibility metadata); author it with `defineGenerator` from the package root. |
There was a problem hiding this comment.
maybe it should also export additional options this generator may support? (e.g. as json schema)
We can support them via config only for example.
There was a problem hiding this comment.
Yes, a generator declares options as a JSON Schema, validated before run, and publishers set them under client.options.<generator> in config.
…, discriminatorCases, nullability, enums)
… from the root and /generate
…oaded built-in generators
… hooks, sdk as reference implementation
…d test, and guide rework
…lkit-import names
… with build-time embed
… table with reflective hydration
…mples and a dogfooding guard
… helpers — slice-2 lessons landed, python items typed
…tdlib-only, vetted, embedded
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Marshevskyy
added
snapshot
|
📦 A new experimental 🧪 version v0.0.0-snapshot.1786088185 of Redocly CLI has been published for testing. Install with NPM: npm install @redocly/cli@0.0.0-snapshot.1786088185 |
An ejected asset stamped only name, run, sample, and requiresGenerator, so an ejected cli or cli-docs stopped pulling its prerequisites, enforcing throw-mode, and validating its options. The eject script now stamps the contract from BUILTIN_META itself, and passes cli-docs its options schema. Prerequisites were also expanded before import specifiers were loaded, so a path generator's requires was never read. resolveGenerators now loads every selected entry first, which also stops a built-in prerequisite from colliding with an ejected generator of the same name.
cli-docs was missing from the allowlist, so its runs were counted as custom and an ejected cli-docs provenance header was ignored. The test claimed to cover every built-in while checking seven of fourteen; it now compares the allowlist against the ejectable set plus the framework variants.
The build read the built-in table with @redocly/openapi-core left external, so the generated module resolved into packages/core/lib. That directory does not exist yet on prepare, which runs before compile, and a clean install failed. meta.ts only touches openapi-core inside functions this build never calls, so the import is stubbed out instead.
An error no branch accounts for — an unreadable asset, a failed write — left eject_generator_outcome unset, so the event could not be told apart from a bug in the collector. The outcome now starts at unexpected-error and every path that finishes overwrites it. Ejecting also set success before wiring the dependency and the config entry, so a failure there was reported as a successful eject; it is set last now.
| npm run cli -- lint openapi.yaml | ||
| ``` | ||
|
|
||
| ## Architecture |
There was a problem hiding this comment.
Why this block was removed?
There was a problem hiding this comment.
It shouldn't have been — restored in 771b9dd03.
The intent was to keep AGENTS.md to guidance an agent can't derive by reading the repo, and the package map looked derivable. It isn't: knowing which package a change belongs in is exactly the thing that's expensive to work out from the tree, and getting it wrong is costly.
It now lives in .claude/rules/architecture.md with all four packages and their key directories intact, and AGENTS.md points at it. Same content, one hop away, so AGENTS.md stays short without losing the map.
| The full testing and QA rules are in | ||
| [`.claude/rules/testing.md`](./.claude/rules/testing.md). | ||
|
|
||
| The rule test pattern looks like this: |
There was a problem hiding this comment.
Why do you remove rule test pattern ? Was it not working properly?
There was a problem hiding this comment.
Nothing wrong with it — the pattern is correct and still the one to copy. Removing it was the same over-trim, and it's restored in 771b9dd03.
The full parse → lintDocument → toMatchInlineSnapshot example is now in .claude/rules/testing.md, which AGENTS.md links to as "the rule test pattern to copy".
…ect an unusable --token
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit e8df6a4. Configure here.
| ' --output <path> Write the response body to a file (required for binary)', | ||
| ...(kinds.has('bearer') ? [' --token <token> Bearer token'] : []), | ||
| ` --json <json|@file|@-> Request body`, | ||
| ...(credentials.length > 0 ? ['', 'Environment:', ` ${credentials.join(', ')}`] : []), |
There was a problem hiding this comment.
CLI docs still advertise --token
Medium Severity
renderHelp now omits --token and bearer env vars when the API has no bearer scheme, but renderCliDocs still always lists --token under Global flags. That breaks the promise that cli-docs cannot drift from the CLI — generated Markdown can document a flag the tool rejects with exit 4.
Reviewed by Cursor Bugbot for commit e8df6a4. Configure here.
Marshevskyy
added
snapshot
|
📦 A new experimental 🧪 version v0.0.0-snapshot.1786199735 of Redocly CLI has been published for testing. Install with NPM: npm install @redocly/cli@0.0.0-snapshot.1786199735 |
6 participants
What this adds
Client generation for languages beyond TypeScript, built so that both we and our users extend it with AI agents driven by per-generator design skills.
New built-in generators
python— self-contained client over httpx: typed dataclasses, sync + async clients, pagination iterators, SSE, multipart, retries, discriminated-union decode via aDISCRIMINATORSregistry.go— stdlib-only single file: structs with json tags, typed-const enums,(T, error)methods,context.Context, range-over-func pagination, SSE.goPackagesets the package clause.php— PHP ≥ 8.1 over the curl extension, zero Composer dependencies: promoted-constructor models, native enums, named-argument methods,Generator-based pagination, SSE over a curl_multi pump.cli— a bin-ready command-line interface over the sdk. It validates requests by default: selectingclipulls in the generators it needs (sdkandzod), so nothing extra has to be listed.binNamenames the command.cli-docs— the Markdown reference for that CLI (<stem>.cli.md): usage, global flags, credential variables, exit codes, and every command with its arguments and flags. It renders from the same command table the CLI dispatches on, so the page cannot drift from the tool.All three SDK languages have parity on auth (bearer/basic/apiKey with token providers), retries with
Retry-After+ jittered backoff, timeouts, idempotency keys, middleware, pagination, SSE, multipart, binary downloads, templated-server helpers (Serversclass /<Name>URLfunctions), and response-header envelopes (<op>WithHeadersvariants mirroring the TS{ envelope: true }option from #3002).The authoring model
Printer(indentation-aware source-text builder), naming (casing,identifierFor,RESERVED_WORDS), schema semantics (flattenAllOf,discriminatorCases, nullability,enumValues,headerCoerceType,schemaAtPointer),paginationRuleFor, andNotSupportedErrorfor rejecting an option the generator can't honor. A dogfooding guard test pins that the built-in language generators use nothing else.typescriptdependency in the authoring path. The TypeScript emitters render source text like everything else; thets.factoryexports are gone.typescriptremains an optional peer needed only to bake a--setupmodule, the one place we parse TypeScript.run(unknown key, wrong type, value outside anenum, missing required key) with defaults applied. Publishers set them underclient.options.<generator>.Eject workflow
redocly eject-generator <name>vendors any built-in generator into the repo as an editable.mjs. A language generator ships as its own source; a TypeScript generator ships bundled with the emitters it uses (unminified, one comment per source module), sosdk,zod,mock,swr,tanstack-query,transformers,cli, andcli-docsare ejectable too. An ejected-unmodified generator produces byte-identical output, proven in e2e.Eject also:
.claude/skills/<name>-generator/SKILL.md) plus the shared authoring skill, where agents auto-load them, and leaves a short pointer beside the code;@redocly/client-generatorindevDependenciesand adds the entry toclient.generators, editing the config text so comments survive;--updatethree-way merges with no committed snapshot: the merge base is the version recorded in the ejected file's own header, fetched from the registry when it differs from the installed one.Compatibility
Generator compatibility is the package version under semver, not an invented number: a generator declares
requiresGenerator(^1.2.0,~1.2.0,>=1.2.0, or an exact version), and a CLI outside that range says which version it ships, which the generator needs, and how to reconcile them. Ejected generators record it automatically. Custom generators also run behind a validated contract: load-time shape validation, output-path containment (no writes outside--output), andrun()result validation. An IR-shape snapshot test forces the "additive or breaking?" question on any model change. Failures are attributable (Generator "<name>" failed: …) and categorized in telemetry along with eject/update outcomes (coarse categories only — never file contents, paths, or user-chosen names).Verification
tsc/py_compile+import /go build+go vet+gofmt/php -l+require(tests/e2e/generate-client/large-descriptions.test.ts).npm run generatorsruns the client-generator unit tests plustests/e2e/generate-client, sharded two ways, so a growing set of compiled-language bars cannot slow the job everything else shares.npm run e2ecovers everything else.tests/e2e/generate-client/examples/.Docs
New command page (
eject-generator), a customize-client-generation guide, per-language usage notes with one table for the differences each language forces, and configuration reference updates (goPackage,binName,options,codeSamples).Note for reviewers
The per-operation pagination extension is now
x-redoclyPagination(camelCase, like every other Redocly extension). A description that still declaresx-redocly-paginationsilently loses its pagination rule — called out in the changeset.Check yourself
Security
Note
High Risk
Large expansion of the experimental client-generator surface (new languages, eject workflow, emission model change) plus a breaking pagination extension rename that can silently drop pagination rules on existing API descriptions.
Overview
Adds built-in client generators for Python, Go, PHP, a bin-ready CLI (
cli), and CLI Markdown docs (cli-docs), alongside the existing TypeScript stack. Language SDKs are single-file, self-contained outputs with parity on auth, retries, pagination, SSE, multipart, and header envelopes; new CLI flags--go-packageand--bin-nameconfigure Go package names and CLI command naming.Introduces a language-neutral authoring toolkit (
Printer, schema helpers, per-generator JSON Schema options,requiresGeneratorcompatibility) and moves generator authoring off the removed AST/typescriptemit path—TypeScript is now optional peer for--setupbaking only.Adds
redocly eject-generatorto vendor any built-in generator into./generators/*.mjs, drop agent skills, wireclient.generatorsand devDependencies, and--updatevia three-way merge (header-recorded version +git merge-file). CLI build copieseject-assetsinto the published bundle.CI/docs: dedicated
npm run generatorsjob (Python/httpx, sharded e2e generate-client bars);x-redocly-paginationis renamed tox-redoclyPagination(old key no longer applies). Telemetry adds coarsegenerate-clientandeject-generatorcategories without user paths or custom generator names.Reviewed by Cursor Bugbot for commit e8df6a4. Bugbot is set up for automated code reviews on this repo. Configure here.