Skip to content

Seed value search misses non-ASCII matches on SQLite: case folding is ASCII only #3664

Open

Description

Description

get_seed_examples_async documents value_search as matching stored values that contain the literal text, ignoring case, but on SQLite the filter compiles to lower(value) LIKE lower(pattern) and SQLite's lower() only folds ASCII letters. A seed stored as CAFÉ au lait tactics is found when searching CAFÉ but not when searching café, while the same data and query on Azure SQL (SQL Server LOWER) matches both.

The memory layer already established Unicode-aware folding as the contract for its filters: UnicodeLower exists precisely because SQLite's ASCII-only folding was considered wrong for the JSON harm category and attack class filters, and those paths use it on both backends. The plain ilike in the seed value search bypasses that machinery.

Steps to reproduce

  1. Add a text seed whose value contains a non-ASCII uppercase letter, for example CAFÉ au lait tactics.
  2. Search the dataset with value_search="café" through get_seed_examples_async on SQLite.
  3. The example is not returned; searching CAFÉ returns it.

Expected behavior

The case-insensitive value search should fold non-ASCII letters the same way on both backends, matching the documented contract and the harm category filters next to it.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions