docs: give each docstring claim one canonical home

The extension docstrings and the FFI guide had grown the same six claims 3-5
times each, in wording that had already started to drift. `with_extensions` was
123 lines — the longest docstring in the package by 30 — and most of it argued
a design rather than stating a contract.

Routes each recurring claim to one home and leaves a one-line pointer
elsewhere. Session sharing and context lifetime move onto the `SessionContext`
class docstring, since they are properties of the type and that is exactly why
five methods had each reworded them. The `None`-vs-`fallback` contract moves
onto `SessionPlannerExportable`, since it is a return-value contract of one
method. The two-phase rationale, the objects-not-capsules argument, and the
codec-order argument stay in the guide, which owns the "why".

The duck-type-the-session rule moves from `QueryPlannerExportable` to
`LogicalExtensionCodecExportable`, which is already the designated `session`
reference for that family and where the codec protocols were pointing for it
anyway. It also gains the fact the old text was missing: across the protocol
this argument is not always a session, so duck-typing it is not a style
preference.

Not every docstring shrank. `__datafusion_query_planner__` was 4 lines with no
example while `set_query_planner` told callers to capture the fallback through
it, so it grew to 25. Several others grew by gaining the `Args`/`Returns`/
`Raises` sections they were missing, and by trading `+SKIP` examples for
runnable ones — a `SessionContext` satisfies the capsule-getter protocols, so
its own exported capsule stands in for a library's without a build step. The
total across these sixteen docstrings is roughly flat, at 577 lines before and
629 after; what changed is that the rationale left and the contract arrived.

The 21 lines of `dataclasses.fields` rationale on
`SessionExtensionComponents.__post_init__` become comments in the method body.
That is the fix rather than touching `autoapi_options`: `__post_init__` is a
*special* member, so dropping `private-members` would not hide it, and dropping
`special-members` would delete the entire `__datafusion_*__` reference surface.
`conf.py` now records why that setting is deliberately left alone, and
de-duplicates the four re-exported `extensions` classes the way it already did
for `DataFrame` and `SessionContext`.

Adds `python/tests/test_docstrings.py`, which is what would have caught the
123-liner: a 95-line ceiling with no waiver list, a doctest-presence check over
the extension protocol, and a check that a docstring naming the guide actually
links it. The last two each found a real defect on first run.

Also fixes what this branch made newly load-bearing in the serialization
surface: `plan.py`'s single-backtick RST rendered `LogicalExtensionCodec` and
friends as italics rather than links into the new API, and the `global_ctx()`
fallback in `Expr.from_bytes` now silently yields a context with no extension
codecs, which before this branch lost only registrations.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: timsaucer

AGENTS.md

@@ -133,6 +133,47 @@ Every Python function must include a docstring with usage examples.
   `array_sort`) only need a one-line description and a `See Also` reference to the
   primary function. They do not need their own examples.
 
+### One canonical home per claim
+
+A claim lives where the reader already is when they need it — **exactly once**.
+
+- A property of one callable's arguments, return value, or errors → that
+  callable's docstring.
+- A property of a *type* that several callables share → the class docstring.
+  (Session sharing and context lifetime live on `SessionContext`, not on each
+  of the five `with_*` methods.)
+- **Why** the API is shaped this way, a multi-library recipe, Rust-side code, a
+  trade-off, or a limitation with an upstream issue → the narrative guide under
+  `docs/source/`.
+
+Everywhere else: one sentence plus one Sphinx role. A docstring may *state* a
+claim the guide also makes; it must not *argue* it — no "because", no "the
+reason is", no counter-argument.
+
+**The test:** if a paragraph would survive being moved into the guide unchanged,
+move it. `python/tests/test_docstrings.py` enforces a length ceiling, which is
+the symptom this rule treats.
+
+When you point at the guide, use a real `:ref:` to a specific label. Prose
+saying "see the extensions guide" with no role is a dead end in the rendered
+HTML.
+
+### Examples that need a compiled extension
+
+Some APIs cannot be demonstrated without a built FFI extension library, which
+this package does not ship. The convention is:
+
+1. A **runnable** example block first, using only the wheel. A `SessionContext`
+   satisfies the capsule-getter protocols, so its own exported capsule stands in
+   for a real library's in a doctest.
+2. Then a `# doctest: +SKIP` block showing real usage, at most a few lines, with
+   one line of prose naming the test that runs it for real.
+3. Every `+SKIP` block needs that mirror. See
+   `test_with_extensions_docstring_example_still_runs` in
+   `examples/datafusion-ffi-query-planner-example`, which parses the live
+   docstring, drops the skip, and executes it — so a renamed method or a wrong
+   expected output fails there.
+
 ## Aggregate and Window Function Documentation
 
 When adding or updating an aggregate or window function, ensure the corresponding

docs/source/conf.py

@@ -103,6 +103,11 @@
 # This pattern also affects html_static_path and html_extra_path.
 exclude_patterns = []
 
+# `autoapi_options` is deliberately left at the default. In particular
+# `special-members` is load-bearing: it is what publishes the
+# `__datafusion_*__` capsule getters, which are the public protocol an
+# extension library implements against. Dropping it would delete that
+# reference surface along with `DataFrame.__init__` and the Arrow dunders.
 autoapi_dirs = ["../../python"]
 autoapi_ignore = ["*tests*"]
 autoapi_member_order = "groupwise"
@@ -116,6 +121,10 @@ def autoapi_skip_member_fn(app, what, name, obj, skip, options) -> bool:  # noqa
         # Re-exports
         ("class", "datafusion.DataFrame"),
         ("class", "datafusion.SessionContext"),
+        ("class", "datafusion.QueryPlannerExportable"),
+        ("class", "datafusion.SessionExtensionComponents"),
+        ("class", "datafusion.SessionExtensionExportable"),
+        ("class", "datafusion.SessionPlannerExportable"),
         ("module", "datafusion.common"),
         # Duplicate modules (skip module-level docs to avoid duplication)
         ("module", "datafusion.col"),

docs/source/extension-guide/bundles.md

@@ -70,6 +70,59 @@ off the supplied context, wrapping its codecs in `BundledLogicalCodec` /
 `BundledPhysicalCodec` so they carry declared ids, and constructing a Python
 {py:class}`~datafusion.SessionExtensionComponents`.
 
+## A bundle you can run
+
+Before wiring up a cdylib, it is worth seeing the protocol work end to end in
+pure Python. This runs against the plain wheel — the codec here re-exports the
+host session's own capsule, where a real one would return its library's:
+
+```python
+from datafusion import SessionContext, SessionExtensionComponents
+
+
+class Codec:
+    """Wraps a capsule so it carries an id. A Rust library ships this shape."""
+
+    def __init__(self, codec_id, capsule):
+        self.__datafusion_codec_id__ = codec_id
+        self._capsule = capsule
+
+    def __datafusion_logical_extension_codec__(self, session=None):
+        return self._capsule
+
+
+class Bundle:
+    def __init__(self, codec_id):
+        self.codec_id = codec_id
+
+    def __datafusion_session_extension__(self, ctx):
+        # Fresh components on every call, bound to the `ctx` handed in.
+        # Never cache these, and never retain `ctx`.
+        return SessionExtensionComponents(
+            logical_extension_codecs=(
+                Codec(self.codec_id, ctx.__datafusion_logical_extension_codec__()),
+            )
+        )
+
+
+ctx = SessionContext().with_extensions(Bundle("tables.v1"), Bundle("engine.v1"))
+ctx.logical_extension_codec_ids()
+# ['tables.v1', 'engine.v1']
+```
+
+Three things this makes observable, each pinned by a test in
+`python/tests/test_context.py`:
+
+- **Ids accumulate in bundle order.** Decoding does not depend on that order;
+  only encoding does. See {ref}`extension_codec_order`.
+- **Two bundles claiming one id are refused**, with a `ValueError` naming the
+  id — not resolved by position, since a positional id would break stored plans
+  the first time a bundle reordered what it returns.
+- **A hook that raises leaves the receiving session untouched.** Add a bundle
+  whose hook raises and the source context's
+  {py:meth}`~datafusion.SessionContext.logical_extension_codec_ids` is still
+  empty afterwards.
+
 (extension_bundles_two_phases)=
 
 ## Two phases, because codecs and planners compose differently

examples/datafusion-ffi-query-planner-example/python/tests/_test_three_library_query_planner.py

@@ -1386,17 +1386,24 @@ def test_with_extensions_docstring_example_still_runs():
     statements are parsed out of the live docstring, the skip is dropped, and
     each one is executed and its output compared.
 
+    Only the ``+SKIP`` statements are taken. The docstring also carries a
+    runnable example above them, which the main suite already executes under
+    ``--doctest-modules``; running it again here would need its own namespace
+    and prove nothing.
+
     Only names are redirected: ``my_extension`` resolves to the bundle above,
     and ``SessionContext`` supplies the config this library's planner reads.
     A renamed method, a changed signature, or a wrong expected output in the
     docstring fails here.
     """
-    examples = doctest.DocTestParser().get_examples(
-        inspect.getdoc(SessionContext.with_extensions)
-    )
-    assert examples, "with_extensions docstring has no examples to check"
-    for example in examples:
-        example.options.pop(doctest.SKIP, None)
+    examples = [
+        example
+        for example in doctest.DocTestParser().get_examples(
+            inspect.getdoc(SessionContext.with_extensions)
+        )
+        if example.options.pop(doctest.SKIP, False)
+    ]
+    assert examples, "with_extensions docstring has no skipped examples to check"
 
     module = types.ModuleType("my_extension")
     module.DistributedEngineExtension = _DocstringExampleExtension