feat(registry): capability registry for Test Management - #441
Open
sarve-shreyas wants to merge 1 commit into
Open
sarve-shreyas wants to merge 1 commit into
sarve-shreyas wants to merge 1 commit into
Conversation
A generic five-tool MCP surface — listProducts, describeEntity, searchCapability, describeCapability, invokeCapability — over a prebuilt per-product index, replacing one hand-written tool per endpoint. tm ships 244 capabilities, of which 198 are published. The other 46 are withheld behind a `disabled` flag: everything that did not pass live probing, plus the destructive tier. A disabled capability is filtered out when the registry loads, so search, describe and the entity listings cannot surface one and no reader downstream has to remember to check. Capability names carry no version suffixes. Where dropping `_v1`/`_v2` collided, the variants differ by ID FORM rather than API version, so the prefixed-identifier variant keeps the plain name and the integer one takes `_by_integer_id`. tests/live/CAPABILITY-RENAME-MAP.csv maps all 244 old names to new. The index was verified against the live API rather than against the spec it came from: 244 capabilities probed on preprod and two production accounts, with the run records, payloads and findings under tests/live/. Squashed onto main as a single commit. The development history is not carried over — it contained real user email addresses in probe fixtures and captured responses, and the working tree is clean of them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Central YAML (base), Organization UI (inherited), Workspace UI (inherited) Review profile: ASSERTIVE Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| return; | ||
| } | ||
|
|
||
| console.log(`seeding ${ENV} as ${USERNAME}\n`); |
| mkdirSync(`${ROOT}tests/live`, { recursive: true }); | ||
| writeFileSync(POOL, JSON.stringify(pool, null, 2) + "\n"); | ||
| console.log(`\npool -> tests/live/.id-pool.json`); | ||
| console.log(JSON.stringify(pool, null, 2)); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces one hand-written MCP tool per endpoint with a generic five-tool surface over a prebuilt, per-product index.
listProducts·describeEntity·searchCapability·describeCapability·invokeCapabilityWhat ships
198 of 244 tm capabilities. The other 46 are withheld behind a
disabledflag while the product team works through them: everything that did not pass live probing, plus the destructive tier.The flag is enforced structurally, not by convention. Disabled capabilities are filtered out when the registry loads, so
searchCapability,describeCapabilityand the entity listings cannot surface one, and no reader downstream has to remember to check. There are four readers of the capabilities array plus the entity docs, and the cost of missing one is a capability hidden from search and still invokable — precisely the failure the flag exists to prevent.invokeCapabilityis the one exception: an exact-handle invoke answerscapability_disabledrather thanunknown_capability, so a caller that already had the name learns the absence is deliberate instead of hunting for a synonym.No destructive capability is published at all — deletes are withheld from the surface rather than refused after the fact, so they never appear in a search result.
Verified against the live API, not against the spec
Every capability was probed against preprod and two production accounts. That is the only check that catches the index lying — promising a body shape the API rejects, or a response field it never returns. It found capabilities that had shipped broken and unusable since the index was created, which no structural test could have seen.
scripts/check-contract.pyruns in CI (npm run check:contract) and fails the build when a capability'sreturnsdisagrees with its resolved 2xx schema. It compares againstscripts/contract-baseline/tm.json, so it breaks on new disagreements rather than on the known set.Naming
No version suffixes. Where dropping
_v1/_v2collided, the variants differ by ID form rather than API version — one takes an integerproject_id, the other a prefixedPR-NNN— so the prefixed variant keeps the plain name and the integer one takes_by_integer_id, matching the convention already in the index.Five renames go further and correct a name that described the wrong behaviour. The worst promised an undo and performed a recursive delete.
Reviewing this
92% of the diff is two generated JSON files. The reviewable surface is ~7,000 lines:
capability/— the generated indextests/fixtures/— test datasrc/— registry codetests/— test codescripts/— toolingtests/fixtures/capability/tm.capability-index.json(41,894 lines) is a frozen copy the tests pin against — safe to skip.tests/fixtures/capability-multi/holds a synthetic second product: a made-up widget catalogue with dummy capabilities, never published. The cross-product routing gate is shipped code, so it needs two products under test even though only tm ships. Its README records the two non-obvious constraints the tests place on it.Notes
main.🤖 Generated with Claude Code