Skip to content

fix: convert a character_not_in_repertoire error to an invalid-class error - #866

Merged
zachdaniel merged 1 commit into
ash-project:mainfrom
grempe:fix/nul-byte-conversion
Sep 24, 2026
Merged

zachdaniel merged 1 commit into
ash-project:mainfrom
grempe:fix/nul-byte-conversion

Conversation

@grempe

@grempe grempe commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Closes #854.

@g3nke11 claimed this on 2026-09-17. I linked this branch on the issue on 2026-09-22 and offered to open it if they had moved on; there has been no reply since, so I am opening it now. Happy to close it if they would rather finish their own.

PostgreSQL rejects text that is not valid UTF-8 or contains a NUL byte with 22021 character_not_in_repertoire. Every Postgrex.Error clause in handle_raised_error/4 ends in Ecto.Adapters.Postgres.Connection.to_constraints/2, which finds nothing for that code, so a create, update or filter with such a value returns Ash.Error.Unknown, and the API extensions answer 500. Only the data layer knows this storage has that limit; ETS and Mnesia store the value.

This adds a clause matching %Postgrex.Error{postgres: %{code: :character_not_in_repertoire}}. It matches on the error code, not on message text. It sits before the existing Postgrex.Error clauses for the {:bulk_create, _} and {:ecto_changeset, _, _} contexts, which would otherwise capture it first.

  • In a query it becomes Ash.Error.Query.InvalidFilterValue.
  • In an update it becomes Ash.Error.Changes.InvalidAttribute for each change whose value fails String.valid?/1 or contains <<0>>. The error names no column, so the attribute comes from the changes.
  • In a create it becomes Ash.Error.Changes.InvalidChanges. The rescue there only has the placeholder changeset that bulk_create/3 builds for it, with no changes, so the attribute cannot be named.

The message is Postgres's own (invalid byte sequence for encoding "UTF8": 0x00). One thing I would change once the no_value? option you suggested on #863 exists: the query case has no value to put on InvalidFilterValue, so it currently renders as Invalid filter value `nil`: ..., the same as #863. Happy to adjust it here along with that.

Tests: test/repertoire_error_test.exs covers a filter, an update with a NUL byte and with invalid UTF-8 (asserting the attribute), a create, and an ordinary-text control. On main the four error cases return Ash.Error.Unknown; with the change they pass. Full suite 1011 passed on main, where the same three JoinSubquerySortTest / UniqAggregateSortTest cases from #858 fail with and without this change, and 983 passed with the patch on v2.13.1. mix format --check-formatted, mix credo --strict, mix sobelow and mix dialyzer clean.

The branch is based on current main, and it also merges cleanly with #863 in either order; I checked main plus both together, and the full suite passes on that with only the three #858 failures.

Found by an AI agent working with a human fuzz-testing their own application; the reproduction outside this repo is https://github.com/grempe/ash-fuzz-repros/blob/main/test/ash_postgres/nul_byte_in_text_test.exs, where all eight cases pass against this branch, across Ash, AshJsonApi, AshGraphql and ash_lua.

Contributor checklist

Leave anything that you believe does not apply unchecked.

  • I accept the AI Policy, or AI was not used in the creation of this PR.
  • Bug fixes include regression tests
  • Chores
  • Documentation changes
  • Features include unit/acceptance tests
  • Refactoring
  • Update dependencies

…error

PostgreSQL rejects text that is not valid UTF-8 or contains a NUL byte
with `22021 character_not_in_repertoire`. Every `Postgrex.Error` clause in
`handle_raised_error/4` ends in a constraint lookup, which finds nothing
for that code, so a create, update or filter with such a value returned
`Ash.Error.Unknown` and the API extensions answered 500. Only the data
layer knows this storage has that limit; ETS and Mnesia store the value.

The error names no column, so for an update the attribute is found by
checking the changeset's changes for a value that is not valid UTF-8 or
contains a NUL byte and each becomes `InvalidAttribute`. A create's rescue
only has the placeholder changeset `bulk_create/3` builds for it, so it
becomes `InvalidChanges`. A query becomes `InvalidFilterValue`.

Closes ash-project#854
@zachdaniel
zachdaniel merged commit 945073e into ash-project:main Sep 24, 2026
111 of 126 checks passed
@zachdaniel

Copy link
Copy Markdown
Contributor

🚀 Thank you for your contribution! 🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A NUL byte in text is an unconverted Postgrex.Error

2 participants