diff --git a/docs/source/extension-guide/checklist.md b/docs/source/extension-guide/checklist.md index 6e204b3f5..6868a11ff 100644 --- a/docs/source/extension-guide/checklist.md +++ b/docs/source/extension-guide/checklist.md @@ -91,8 +91,9 @@ publish. Each links to the page that explains it. derived from it is in use. This is the rule most likely to arrive as a bug report against your library. → {ref}`extension_sessions` - [ ] **Your production codec serializes durable metadata**, not a - process-local token. The examples in this repository use tokens to make - ownership observable; that is a demonstration, not a pattern. + process-local token. While one arm of one codec in this repository uses a + token to make ownership observable, that is a marked workaround for an + upstream defect, not a pattern. → {ref}`extension_codec_durable_metadata` - [ ] **You have integration tests across a real FFI boundary.** The two example crates in this repository are the pattern: build the cdylib, diff --git a/docs/source/extension-guide/codecs.md b/docs/source/extension-guide/codecs.md index 160439927..c134b35ab 100644 --- a/docs/source/extension-guide/codecs.md +++ b/docs/source/extension-guide/codecs.md @@ -58,15 +58,24 @@ Your payload has to be enough to rebuild the object somewhere your process is not. Write the metadata a fresh instance can be constructed from — a path, a connection string, a schema, the options the object was created with. -The example codecs in this repository do not do this, and it is worth knowing -before copying them. They keep a process-local `HashMap` of live providers and -encode an integer token into it: encoding inserts, decoding removes. That makes -Rust type identity observable across three separately loaded libraries in one -test, which is what the examples exist to show. It also means a decode consumes -its token, so the same bytes cannot be decoded twice, one encoded plan cannot -fan out to several readers, and a plan that never reaches a decoder keeps its -provider alive for the life of the process. A real codec has none of those -properties because it does not park the object anywhere. +For working examples, see the logical codec in [`datafusion-ffi-example`] and +the codecs in `examples/distributed/storage-library`. + +You may see a codec that uses a process-local `HashMap` of live objects and +encodes an integer token into it: encoding inserts, decoding removes. Do not copy +this pattern. It means a decode consumes its token, so the same bytes cannot be +decoded twice, one encoded plan cannot fan out to several readers, and a plan +that never reaches a decoder keeps its object alive for the life of the process. +A real codec has none of those properties because it does not park the object +anywhere. + +A token registry is a consequence, not a choice. A codec that downcasts to its +own concrete types is never handed something it cannot describe. The only place +this repository uses one is the `ForeignExecutionPlan` arm of the physical codec +in [`datafusion-ffi-example`]. It is a marked workaround for an upstream defect +([apache/datafusion#25152](https://github.com/apache/datafusion/issues/25152)) +and will be deleted once that is fixed. For why a codec would ever claim a foreign +node it does not own in the first place, see {ref}`extension_codec_order`. (extension_codec_ids)= @@ -151,7 +160,9 @@ after it. The query still succeeds. What changes is which library wrote the bytes — so a plan that has to decode in another process now needs whichever library happened to win, not the one whose node it is. `MyPhysicalExtensionCodec` in [`datafusion-ffi-example`] claims this way, and -the query-planner example's test suite pins the consequence. +the query-planner example's test suite pins the consequence. See +{ref}`extension_codec_durable_metadata` for why that arm exists and when it will +be removed. Two rules of thumb: