docs: add a user-facing extensions page and split distributing-work

The user guide had no page for someone who installs an extension library
and wants to run queries with it. `with_extensions` was described only in
a 123-line docstring and in the contributor-guide FFI page, which opens by
explaining that Rust has no stable ABI — the wrong altitude for a reader
who just wants their queries to run somewhere else.

Adds `user-guide/extensions.md`: what an extension library is, which kinds
register directly versus needing `with_extensions`, the two failure modes
that actually bite (a collected context, a version mismatch), and how to
check what a session was taught. It names no capsule, no ABI, and no codec
id; the only occurrences of `FFI` and `TaskContextProvider` are inside the
error string a reader would be searching for.

Splits `distributing-work.md` into a directory. The page was entirely about
pickling expressions to worker pools and treated query-level distribution
as two stubs at the end, so nothing on the site connected `with_extensions`
to distribution at all — which is the road a data scientist is actually
looking for. The index now asks who owns the partitioning decision and
routes accordingly; `query-engines.md` carries the missing bridge and
absorbs the two upstream work-in-progress sections.

Wires `sphinx-reredirects`, which has been a declared dependency since
#1578 without ever being enabled, so the old `distributing-work.html` URL
keeps working. Also fixes three pointers that went stale in the MyST
migration and still named `.rst` files, a malformed `ref:` role in
udf-and-udfa.md that rendered as literal text, and a doubled "the" in
data-sources.md.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: timsaucer

crates/core/src/codec.rs

@@ -660,7 +660,7 @@ impl PythonLogicalCodec {
     /// `cloudpickle.loads` on the inline `DFPY*` payload. It does
     /// **not** make `pickle.loads(untrusted_bytes)` safe; treat every
     /// `pickle.loads` on untrusted input as unsafe regardless of this
-    /// setting. See `docs/source/user-guide/io/distributing_work.rst`
+    /// setting. See `docs/source/user-guide/distributing-work/expressions.md`
     /// (Security section) for the full threat model, and Python's
     /// [pickle module security warning][1] for why `pickle.loads` is
     /// unsafe in general.

docs/source/conf.py

@@ -54,8 +54,20 @@
     # raises an extension conflict.
     "myst_nb",
     "autoapi.extension",
+    # Emits a meta-refresh stub at each old docname listed in `redirects`
+    # below, so inbound links to pages that have moved keep working.
+    "sphinx_reredirects",
 ]
 
+# Old page URLs that have moved. The site is single-version (each release
+# overwrites asf-site wholesale), so these exist purely for inbound external
+# links — issue comments, release blog posts, and README links that pinned a
+# heading anchor. Keys are docnames without a suffix; the source file must be
+# gone, or Sphinx builds the real page and the stub is never written.
+redirects = {
+    "user-guide/distributing-work": "distributing-work/index.html",
+}
+
 # NOTE: .rst stays alongside .md because sphinx-autoapi generates RST
 # under autoapi/ and Sphinx needs the suffix to parse it. The human-
 # authored docs are all MyST .md now. ".md" is routed through myst-nb so

docs/source/user-guide/common-operations/udf-and-udfa.md

@@ -391,7 +391,8 @@ df.select("a", exp_smooth(col("a")).alias("smooth_a")).show()
 User Defined Table Functions are slightly different than the other functions
 described here. These functions take any number of `Expr` arguments, but only
 literal expressions are supported. Table functions must return a Table
-Provider as described in the ref:`_io_custom_table_provider` page.
+Provider as described in the {ref}`Custom Table Provider <io_custom_table_provider>`
+page.
 
 Once you have a table function, you can register it with the session context
 by using {py:func}`datafusion.context.SessionContext.register_udtf`.

docs/source/user-guide/data-sources.md

@@ -214,7 +214,7 @@ Features that are available in PyIceberg but not yet in Iceberg Rust will not be
 ## Custom Table Provider
 
 You can implement a custom Data Provider in Rust and expose it to DataFusion through the
-the interface as describe in the {ref}`Custom Table Provider <io_custom_table_provider>`
+interface described in the {ref}`Custom Table Provider <io_custom_table_provider>`
 section. This is an advanced topic, but a
 [user example](https://github.com/apache/datafusion-python/tree/main/examples/datafusion-ffi-example)
 is provided in the DataFusion repository.

…s/source/user-guide/distributing-work.md …r-guide/distributing-work/expressions.mddocs/source/user-guide/distributing-work.md renamed to docs/source/user-guide/distributing-work/expressions.md docs/source/user-guide/distributing-work.md renamed to docs/source/user-guide/distributing-work/expressions.md

@@ -17,24 +17,19 @@
   under the License.
 -->
 
-# Distributing work
+(distributed_expressions)=
+
+# Shipping expressions to workers
 
 DataFusion supports splitting work across processes by shipping
 serialized expressions to workers: the driver builds an
 {py:class}`~datafusion.Expr`, each worker evaluates it against its
 own slice of data. This pattern suits embarrassingly-parallel
-workloads where the driver decides partitioning up front.
-
-Query-level distribution — where the runtime partitions a single
-logical or physical plan across worker nodes — is in progress
-upstream via [datafusion-distributed](https://github.com/apache/datafusion-distributed) and [Apache
-Ballista](https://github.com/apache/datafusion-ballista). Both
-have short sections at the end of this page; integration details
-will land as those projects become usable from datafusion-python.
-
-## Expression-level distribution
+workloads where the driver decides partitioning up front. If you
+would rather have a library partition your query plan for you, see
+{ref}`distributed_query_engines` instead.
 
-DataFusion expressions support distribution directly: pass one to a
+Expressions support this directly: pass one to a
 worker process and Python's standard
 [pickle](https://docs.python.org/3/library/pickle.html) machinery
 serializes it transparently — the same machinery
@@ -43,7 +38,7 @@ similar libraries already use to ship function arguments. Python UDFs
 — scalar, aggregate, and window — travel inside the serialized
 expression; the receiver does not need to pre-register them.
 
-### Basic worker-pool example
+## Basic worker-pool example
 
 Define a worker function that takes the expression plus a batch and
 returns the evaluated result:
@@ -92,7 +87,7 @@ see [Safe importing of main module](https://docs.python.org/3/library/multiproce
 in the Python docs.
 :::
 
-### What travels with the expression
+## What travels with the expression
 
 - **Built-in functions** (`abs`, `length`, arithmetic, comparisons,
   etc.) — fully portable. Worker needs nothing pre-registered.
@@ -113,7 +108,7 @@ in the Python docs.
 
 (distributed_udf_portability)=
 
-### Portability requirements for inline Python UDFs
+## Portability requirements for inline Python UDFs
 
 Inline Python UDFs ride on [cloudpickle](https://github.com/cloudpipe/cloudpickle), which imposes two
 requirements on the worker environment:
@@ -133,7 +128,7 @@ requirements on the worker environment:
   Self-contained UDFs (no imports beyond what the worker already has,
   e.g. `pyarrow`) avoid this entirely.
 
-### Registering shared UDFs on workers
+## Registering shared UDFs on workers
 
 When an expression references an FFI capsule UDF (or any UDF the
 worker must resolve from its registered functions), set up the
@@ -164,7 +159,7 @@ fine for expressions that only reference built-ins and Python UDFs,
 but FFI-capsule-backed registrations must be installed on the global
 context to resolve.
 
-### Python 3.14 default change
+## Python 3.14 default change
 
 Python 3.14 changed the Linux default start method for
 {py:mod}`multiprocessing` from `fork` to `forkserver` (macOS has
@@ -174,7 +169,7 @@ workers via copy-on-write; with `forkserver` and `spawn` it is
 not. The {py:func}`~datafusion.ipc.set_worker_ctx` pattern works on
 every start method — prefer it over relying on inherited state.
 
-### Practical considerations
+## Practical considerations
 
 - **Serialized size scales with what travels inline.** A serialized
   expression of just built-ins is small (tens of bytes). An
@@ -189,7 +184,7 @@ every start method — prefer it over relying on inherited state.
   the captured state is large, mutable, or not portable to the
   worker's environment. See {ref}`Portability requirements for inline Python UDFs <distributed_udf_portability>` for the Python-version and imported-module rules.
 
-### Disabling Python UDF inlining
+## Disabling Python UDF inlining
 
 For a stricter wire format, call
 {py:meth}`SessionContext.with_python_udf_inlining(enabled=False)
@@ -241,7 +236,7 @@ threat model.
 
 (distributed_expr_security)=
 
-### Security
+## Security
 
 :::{warning}
 Reconstructing an expression containing a Python UDF executes
@@ -255,7 +250,9 @@ functions and pre-registered Rust-side UDFs, and avoid
 {py:func}`pickle.loads` on externally supplied bytes entirely.
 :::
 
-### Reference: session context slots
+(session_context_slots)=
+
+## Reference: session context slots
 
 There is only one type — {py:class}`SessionContext`. It can occupy
 up to four *slots* in a running program:
@@ -297,32 +294,10 @@ Sharp edges:
 - The inlining toggle is per-context state, not a global switch.
   Two contexts with different toggles can coexist in one process.
 
-## Query-level distribution via datafusion-distributed
-
-🚧 *Work in progress upstream — not yet usable from datafusion-python.*
-
-[datafusion-distributed](https://github.com/apache/datafusion-distributed)
-splits a single physical plan into stages and runs each stage on a
-different worker node. The driver writes a SQL or DataFrame query
-once; the runtime handles partitioning, shuffles, and reassembly.
-
-A datafusion-python integration is in development. This section will
-document the integration once it lands. In the meantime, the
-expression-level approach above covers most use cases that do not
-require automatic plan partitioning.
-
-## Query-level distribution via Apache Ballista
-
-🚧 *Work in progress upstream — not yet usable from datafusion-python.*
-
-[Apache Ballista](https://github.com/apache/datafusion-ballista)
-provides distributed query execution on top of DataFusion with a
-scheduler / executor model better suited to long-lived cluster
-deployments. A datafusion-python integration is on the roadmap; this
-section will fill in once the integration is usable.
-
 ## See also
 
+- {ref}`distributed_query_engines` — the other road: let a library
+  partition the plan for you.
 - {py:mod}`datafusion.ipc` — worker context API.
 - `examples/multiprocessing_pickle_expr.py` — runnable
   `multiprocessing.Pool` example that ships a different parametric

docs/source/user-guide/distributing-work/index.md

@@ -0,0 +1,81 @@
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+(distributing_work)=
+
+# Distributing work
+
+A single {py:class}`~datafusion.SessionContext` already uses every
+core on the machine — DataFusion partitions and parallelizes within
+a process without being asked. See
+{doc}`../configuration` for tuning that.
+
+This section is about the step after that: getting work onto more
+than one process or more than one machine. There are two roads, and
+they suit different problems.
+
+## Pick your road
+
+**You decide the partitioning → ship expressions.**
+
+You already know how the data divides — one file per worker, one
+customer per worker, one parameter setting per worker. You build an
+{py:class}`~datafusion.Expr` in the driver and hand a copy to each
+worker along with its slice. Standard Python `pickle` moves it, so
+{py:mod}`multiprocessing`, Ray, and anything else that ships function
+arguments works with no extra machinery.
+
+Best for embarrassingly-parallel work: parameter sweeps, per-file
+transforms, scoring batches. Available today.
+
+→ {ref}`distributed_expressions`
+
+**A library decides the partitioning → install a query engine.**
+
+You write one ordinary SQL or DataFrame query against a table too
+large for one machine, and an engine library splits the plan into
+stages, runs them on its workers, and reassembles the result. You do
+not partition anything and your queries do not change.
+
+Best for large-scale analytical queries — joins and aggregations over
+data that does not fit on one node. Being built upstream; not yet
+usable from datafusion-python.
+
+→ {ref}`distributed_query_engines`
+
+## Choosing between them
+
+The two are not competing implementations of one feature. The
+question is who owns the partitioning decision.
+
+If you can state the partitioning in one line — "one worker per input
+file" — the expression road is simpler, has no cluster to operate,
+and works now. If stating it requires knowing how a join will
+shuffle, that decision belongs to a query planner, which is what an
+engine library provides.
+
+They also compose. An engine handles the query; expressions handle
+whatever you want to fan out around it.
+
+```{toctree}
+:maxdepth: 2
+
+expressions
+query-engines
+```

docs/source/user-guide/distributing-work/query-engines.md

@@ -0,0 +1,100 @@
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+(distributed_query_engines)=
+
+# Distributed query engines
+
+A distributed query engine takes the query you already wrote and runs
+it across several machines for you. You do not partition anything by
+hand and you do not change your queries — the engine installs itself
+on your session, rewrites the plan into stages, and executes those
+stages on its own workers.
+
+This is the counterpart to {ref}`distributed_expressions`, where
+*you* decide the partitioning and ship one expression per slice of
+data. Choose an engine when you want a single query spread across a
+cluster; choose expressions when you already know how to split the
+work and only need parallelism.
+
+## How an engine attaches to your session
+
+An engine library ships an object you hand to
+{py:meth}`~datafusion.SessionContext.with_extensions`:
+
+```python
+from datafusion import SessionContext
+import my_engine
+
+ctx = SessionContext().with_extensions(my_engine.Extension("scheduler:50050"))
+
+ctx.register_table("events", my_engine.TableProvider("s3://bucket/events"))
+ctx.sql("SELECT country, count(*) FROM events GROUP BY country").show()
+```
+
+The `sql` call is unchanged from a single-process program. What
+changed is underneath it: installing the engine gave the session a
+query planner of the engine's own, and that planner is what turns
+your plan into distributed stages.
+
+Two consequences worth knowing:
+
+- **The engine must be installed before you run the query**, not
+  before you register tables. Registration order does not matter;
+  `with_extensions` is a session-level setup step.
+- **The engine has to be able to reach your tables and functions.**
+  It ships the plan to its workers, so anything the plan references
+  has to be reconstructible there. Table providers from the engine's
+  own library always are. A Python UDF may or may not be — the engine
+  library documents what it supports, and
+  {ref}`distributed_udf_portability` describes the constraints that
+  apply to Python callables crossing a process boundary in general.
+
+If you install more than one library, pass them in one
+`with_extensions` call so they can see each other. See
+{ref}`user_guide_extensions` for the details of installing extension
+libraries, and {ref}`ffi` if you want to write an engine yourself.
+
+## Available engines
+
+Query-level distribution is being built upstream. Neither project
+below is usable from datafusion-python yet; both sections will fill
+in as the integrations land.
+
+### datafusion-distributed
+
+🚧 *Work in progress upstream — not yet usable from datafusion-python.*
+
+[datafusion-distributed](https://github.com/apache/datafusion-distributed)
+splits a single physical plan into stages and runs each stage on a
+different worker node. The driver writes a SQL or DataFrame query
+once; the runtime handles partitioning, shuffles, and reassembly.
+
+A datafusion-python integration is in development. In the meantime,
+{ref}`distributed_expressions` covers most use cases that do not
+require automatic plan partitioning.
+
+### Apache Ballista
+
+🚧 *Work in progress upstream — not yet usable from datafusion-python.*
+
+[Apache Ballista](https://github.com/apache/datafusion-ballista)
+provides distributed query execution on top of DataFusion with a
+scheduler / executor model better suited to long-lived cluster
+deployments. A datafusion-python integration is on the roadmap.