Implement stemmers

Cargo.toml

@@ -3,7 +3,7 @@ resolver = "3"
 members = ["boldi-vigna", "postgres", "tinql", "tokenizer"]
 
 [workspace.package]
-version = "1.0.3"
+version = "1.0.4"
 edition = "2024"
 authors = ["PlanetScale"]
 license = "AGPL-3.0-or-later"

README.md

@@ -24,7 +24,9 @@ Then run `CREATE EXTENSION tin` in the database. Lead loads on demand and does n
 
 ## Compatibility boundary
 
-Lead provides the `tin` access method, the `==>` operator, TINQL parsing, tokenizer and index reloptions, and the scoring functions `tin.score`, `tin.full_score`, `tin.max_score`, and `tin.score_inspect`, plus explicit and implicitly bound `tin.highlight` and `tin.highlight_ansi`. Postgres 17 and 18 are build targets. Search results are exact because the access method returns whole-page candidates and Postgres evaluates `==>` against each visible heap tuple, including expression and partial-index rechecks.
+Lead provides the `tin` access method, the `==>` operator, TINQL parsing, tokenizer and index reloptions (including the Snowball `stemmer` option), and the scoring functions `tin.score`, `tin.full_score`, `tin.max_score`, and `tin.score_inspect`, plus explicit and implicitly bound `tin.highlight` and `tin.highlight_ansi`. Postgres 17 and 18 are build targets. Search results are exact because the access method returns whole-page candidates and Postgres evaluates `==>` against each visible heap tuple, including expression and partial-index rechecks.
+
+The planner binds each `==>` and each `tin.highlight` or `tin.highlight_ansi` call to the analysis of the tin index that covers its column, so a stemmed index matches and highlights inflected words in queries over that column, including joins, partitions, and cached plans. That includes `body ==> ANY(...)`, whose array elements are bound to the index analysis too; `tin.highlight` marks only the implicit queries of plain `==>` quals, as TIN does, though an `ANY(...)` search binds the analysis of an explicit `query`. `tin.tokenize`, `tin.ql_parse`, `tin.highlight`, and `tin.highlight_ansi` take a trailing `stemmer` argument as in TIN 1.0.4; their 1.0.3 signatures remain as `tin.tokenize_v1_0_3`, `tin.ql_parse_v1_0_3`, `tin.highlight_v1_0_3`, and `tin.highlight_ansi_v1_0_3`. As in TIN, changing `stemmer` on an existing index requires `REINDEX`.
 
 Scoring deliberately rescans and retokenizes the visible indexed column or expression, once per statement and search, under that statement's snapshot. A score call must be in the same query level as the matching `==>` predicate, and a partial tin index binds only when the query's quals imply its `WHERE` condition, as in tin. Implicit highlighting binds the same way and, like tin, returns the text unmarked when no index binds a search; passing its `query` argument explicitly works without a bound predicate.