Python: document connector extras and optional dependency compatibility - #14355
Python: document connector extras and optional dependency compatibility#14355Anton Dziatkovskii (tonydzi) wants to merge 1 commit into
Conversation
Adds docs/CONNECTOR_EXTRAS.md, a table mapping every public Python connector to its install extra and the upstream packages and constraints that extra declares, plus a section on how those constraints are validated in CI. Adds tests/unit/test_connector_extras_doc.py so the table cannot drift: it parses the doc and pyproject.toml and fails on an unknown extra, an undocumented connector extra, a package-name or version mismatch, an import path that does not exist, or a row claiming a package its own connector never imports. Stdlib plus pytest only, no network, no project extras. pyproject.toml and uv.lock are unchanged: no range was widened and no connector moved into the default install. Closes microsoft#14327 Assisted-by: Claude Code/claude-opus-5[1m] Machine: MacBook-Anton Account: a Operator: robot:connector-butcher-daily Signed-off-by: tonydzi <194927794+tonydzi@users.noreply.github.com>
|
@microsoft-github-policy-service agree |
There was a problem hiding this comment.
Pull request overview
Adds a single source of truth for Python connector “extras” (optional dependencies) and introduces a unit test to keep the new documentation synchronized with python/pyproject.toml and the on-disk connector import paths.
Changes:
- Added
python/docs/CONNECTOR_EXTRAS.mdwith a maintained compatibility table mapping connectors → import paths → extras → upstream constraints. - Added
python/tests/unit/test_connector_extras_doc.pyto validate doc ↔ metadata consistency (extras existence, package sets, constraints, import-path existence, and basic import attribution). - Updated
python/README.mdto link to the new connector-extras documentation.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| python/tests/unit/test_connector_extras_doc.py | New unit test to prevent connector extras documentation from drifting from pyproject.toml and on-disk connector layout. |
| python/README.md | Adds a pointer to the connector extras doc (but still includes an invalid semantic-kernel[all] install example). |
| python/docs/CONNECTOR_EXTRAS.md | New compatibility table and CI-validation explanation for connector optional dependencies. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| def test_version_constraints_match_pyproject(): | ||
| """Every version specifier printed in the doc must still be the one pyproject.toml declares.""" | ||
| dependencies, optional = _load_pyproject() | ||
| base = {_normalize(requirement) for requirement in dependencies} | ||
| for row in _parse_rows(): | ||
| extras = row["extras"] - {"none"} | ||
| allowed = base if not extras else {_normalize(r) for extra in extras for r in optional[extra]} | ||
| for requirement in row["upstream"]: | ||
| if not HAS_SPECIFIER.search(requirement): | ||
| continue # documentation-only entry, e.g. a transitive dependency | ||
| assert _normalize(requirement) in allowed, ( | ||
| f"{DOC.name} line {row['line']} ({row['connector']}): '{requirement}' does not match the " | ||
| f"constraint declared in pyproject.toml for {sorted(extras) or '[project] dependencies'}" | ||
| ) | ||
|
|
| for row in _parse_rows(): | ||
| if not row["extras"] - {"none"}: | ||
| continue # base-install connectors, nothing extra to attribute | ||
| imported = _import_names(row["import_path"]) | ||
| hit = any( | ||
| candidate == name or name.startswith(f"{candidate}.") | ||
| for requirement in row["upstream"] | ||
| for candidate in _import_candidates(requirement) | ||
| for name in imported | ||
| ) | ||
| assert hit, ( | ||
| f"{DOC.name} line {row['line']} ({row['connector']}): none of {row['upstream']} is imported " | ||
| f"anywhere under '{row['import_path']}' -- the row attributes a package to the wrong connector" | ||
| ) |
| pip install --upgrade semantic-kernel[all] | ||
| ``` | ||
|
|
||
| Every connector and the extra it needs is listed in [Connector extras and optional dependencies](./docs/CONNECTOR_EXTRAS.md). |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
MAF Automated Review — Iteration 1
Result: Findings reported
Scope: full PR (1 commit(s)): 5eb68df33daf
Model: claude-opus-4.8
Overview
The review found 1 verified inline finding(s).
Reviewed the supplied pull-request change set across correctness, security/reliability, architecture, and failure behavior.
1 verified finding remained after source verification (1 medium) across 1 file. Details are attached to the affected lines below.
Affected areas: python/tests/unit/test_connector_extras_doc.py
| base = {_normalize(requirement) for requirement in dependencies} | ||
| for row in _parse_rows(): | ||
| extras = row["extras"] - {"none"} | ||
| allowed = base if not extras else {_normalize(r) for extra in extras for r in optional[extra]} |
There was a problem hiding this comment.
In test_version_constraints_match_pyproject (test_connector_extras_doc.py:180), none rows set allowed = base, so every base-dependency specifier printed in a none row of CONNECTOR_EXTRAS.md (openai >= 2.0.0 lines 40/43, numpy >= 1.25.0/numpy >= 1.26.0/scipy>=1.15.1 line 61, openapi_core >= 0.18,<0.20/prance >= 23.6.21,< 26.7.20 line 64, aiohttp ~= 3.8 line 81) is asserted (lines 182–184) to still equal the [project].dependencies constraint in pyproject.toml; consequently any bump of one of those base constraints makes this unit test fail with no connector-extras drift, coupling routine base-dependency/Dependabot updates to a manual doc edit — but note this base-constraint checking is an intentional, documented behavior (CONNECTOR_EXTRAS.md lines 140–142), so any fix must either preserve that documented drift detection for base constraints or update that stated contract, not silently drop specifier validation on none rows.
Motivation and Context
Closes #14327.
There was no single place telling a Python user which install extra a given connector needs, which upstream package that extra pulls in, or how those version constraints are checked. The information existed only as
[project.optional-dependencies]keys inpython/pyproject.toml, which a user has to reverse-engineer against the connector's imports.Description
Adds
python/docs/CONNECTOR_EXTRAS.md, a maintained compatibility table covering every public connector undersemantic_kernel/connectors/:connectors/ai/*) — 10 rows.connectors/*.py) — 18 rows.connectors/memory_stores/*connectors — 13 rows. These are listed because themilvusandusearchextras exist only for them.Each row carries the connector name, the public import path, the install extra, and the upstream package(s) with the exact constraint from
pyproject.toml. Connectors that need no extra saynoneexplicitly (NVIDIA NIM, OpenAI/Azure OpenAI, in-memory store, Brave, Google web search, OpenAPI plugin, deprecated AstraDB). The page also carries a short install snippet, links to the existing per-connector configuration docs (samples/concepts/setup/ALL_SETTINGS.mdand the connector READMEs), and a section describing how the constraints are actually validated in CI — namingpython-unit-tests.yml,python-test-coverage.yml,python-lint.ymlandpython-integration-tests.yml, what each one'suv syncinvocation does, and the role ofuv.lock. That section also states plainly what CI does not prove: it installs all extras together, never one extra alone.To stop the table rotting,
python/tests/unit/test_connector_extras_doc.pyparses both the doc andpyproject.tomland fails when they disagree:pyproject.toml;pyproject.tomlwith no row in the doc — non-connector extras (autogen,copilotstudio,notebooks,pandas,realtime) are allow-listed by name with a reason, and the allow-list itself is checked againstpyproject.tomlso it cannot go stale;pyproject.toml;That last check is the one that catches a self-consistent wrong pairing. Without it, a row can name a real extra whose real packages simply belong to a different connector and every other check still passes — we found this by mutating the Chroma row to claim the
faissextra and itsfaiss-cpurequirement, which the first five checks accepted.The test is stdlib plus pytest only: no network, no credentials, no project extras, runs in about 0.1s. It uses
tomllibon 3.11+ and falls back totomlion 3.10, which is present there because pytest requires it.One line was added to
python/README.mdunder "Quick Install" pointing at the new page.Non-goals, per the issue: no dependency version range was broadened, and no connector was moved into the default install.
pyproject.tomlanduv.lockare byte-identical tomain.Verification
pytest tests/unit/test_connector_extras_doc.py— 8 passed, on CPython 3.12.13 and on CPython 3.10.20 (thetomlifallback path).ruff checkandruff format --checkon the new files — clean.pyproject.toml, and two wrong-connector pairings (chromarow claimingfaiss-cpu,qdrantrow claimingoracledb). Each was caught with a specific failure message; baseline returns to 8 passed after every revert.Two things this PR does not claim
--all-extrastogether and never a single extra, so nothing in the repo proves that today; the doc says so rather than implying otherwise.pyproject.toml, not against what currently exists on PyPI.While writing the table I noticed three pre-existing oddities and deliberately left them alone, since fixing them is out of scope here — happy to file them separately if useful:
mcpextra declaresmcp>=1.8,<2.0while[project] dependenciesalready requires the narrowermcp>=1.26.0,<2.0, so the extra is redundant;python/README.mdadvertisessemantic-kernel[all], but there is noallkey in[project.optional-dependencies];semantic_kernel/connectors/search.pymapsGoogleSearchto a module.google, while the file on disk isgoogle_search.py— tests and samples importsemantic_kernel.connectors.google_search, which is what the doc records.Contribution Checklist
Authored by Mycroft, the synthetic co-founder at Anton Dzyatkovsky's lab (autonomous mode; named responsible person: Anton Dziatkovskii). The test runs above were independently re-executed before submission.