Skip to content

fix: report regex compile failures as DataFusion errors, consistently across the regexp family - #25352

Merged
adriangb merged 8 commits into
apache:mainfrom
pydantic:regex-compile-diagnosis
Sep 16, 2026
Merged

adriangb merged 8 commits into
apache:mainfrom
pydantic:regex-compile-diagnosis

Conversation

@adriangb

@adriangb adriangb commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

A regular expression that comes from user SQL and does not compile is reported in four different ways today. Two of them are Arrow error: Compute error, which is the same shape the engine uses for internal faults, so a caller cannot tell an invalid query from an engine problem. For regexp_count and regexp_instr the reason is dropped, and the user only sees the pattern.

On main:

CREATE TABLE t (s VARCHAR, p VARCHAR);
INSERT INTO t VALUES ('abc', 'a(b');

SELECT regexp_like(s, 'a(b') FROM t;   -- Optimizer rule 'simplify_expressions' failed / Invalid regex / External error: ...
SELECT regexp_like(s, p) FROM t;       -- Arrow error: Compute error: Regular expression did not compile: Syntax( ... )
SELECT regexp_match(s, 'a(b') FROM t;  -- Arrow error: Compute error: Regular expression did not compile: Syntax( ... )
SELECT regexp_count(s, 'a(b') FROM t;  -- Arrow error: Compute error: Regular expression did not compile: a(b
SELECT regexp_replace(s, 'a(b','x') FROM t; -- External error: regex parse error: ...

The ~, ~*, !~ and !~* operators, and every two-argument regexp_like, are affected as well. RegexpLikeFunc::simplify rewrites all of them to an operator, and the operators are evaluated in datafusion/physical-expr/src/expressions/binary/kernels.rs, which calls the arrow kernels directly. That is the most common spelling, and it produced the ComputeError shape.

What changes are included in this PR?

Three commits.

  1. The regex compile helpers move to datafusion-physical-expr-common. compile_regex and compile_and_cache_regex return a DataFusionError that carries the diagnosis from the regex crate, and they take the name of the calling function so the "global" flag message names the function the user called. datafusion_functions::regex re-exports both, so the old path still resolves. regexp_count and regexp_instr propagate the new type.
  2. A failed compile in an arrow kernel becomes a DataFusion error. A new helper inspects the patterns the kernel received and reports the first one that does not compile. It runs only after the kernel returns an error, so a pattern that compiles costs exactly what it costs on main: the same number of Regex::new calls, none added. It is used by regexp_match, regexp_like, regexp_replace and by the ~ / ~* / !~ / !~* kernels in physical-expr.
  3. A literal pattern that does not compile is a plan error. simplify_regex_expr returns DataFusionError::Plan with the diagnosis, instead of Context("Invalid regex", External(..)).

The result is one wording for the whole family. The variant is Plan when a literal pattern is caught at planning, and Execution otherwise:

invalid pattern (literal) invalid pattern (column) invalid flag pattern too large g flag
regexp_like Plan Execution Execution Execution Plan, regexp_like()
~ ~* !~ !~* Plan Execution Execution n/a
regexp_match Execution Execution Execution Execution Plan, regexp_match()
regexp_count Execution Execution Execution Execution Plan, regexp_count()
regexp_instr Execution Execution Execution Execution Plan, regexp_instr()
regexp_replace Execution Execution Execution Execution supported
Execution error: Regular expression did not compile: regex parse error:
    a(b
     ^
error: unclosed group

Two related defects are fixed along the way:

  • regexp_match rejected only the exact flags string g, so gi reached the kernel and failed with unrecognized flag. The check is now contains('g'), which is what regexp_like already did.
  • Three blocks in regexp_count.slt asserted nothing. They were written as a bare statement error with the expected message on the following line, so the message was parsed as part of the SQL, the SQL failed to parse, and the assertion passed. One of them expected a message that main does not produce.

What is the testing strategy for this PR?

New and updated cases in datafusion/sqllogictest/test_files/regexp/*.slt cover, for every function in the family, an invalid pattern as a literal and as a column, an invalid flag, a pattern over the size limit, and the g flag, plus the NULL cases (regexp_match(NULL, 'a(b'), a NULL pattern) which must stay NULL rather than raise.

Run locally on this branch: cargo fmt --all -- --check; cargo clippy --all-targets --all-features for datafusion-functions, datafusion-physical-expr, datafusion-physical-expr-common and datafusion-optimizer; the tests of those four crates; the full sqllogictest suite (520 files); cargo check -p datafusion-examples; and cargo build --locked -p datafusion.

Are there any user-facing changes?

Yes, in error messages and in one public API.

  • An invalid pattern or flag now produces a DataFusion Plan or Execution error with the reason from the regex crate, instead of Arrow error: Compute error, a bare External error, or a message that names only the pattern.
  • A pattern over the size limit now reads Compiled regex exceeds size limit of 10485760 bytes. instead of CompiledTooBig(10485760).
  • Breaking: compile_regex and compile_and_cache_regex change their error type from ArrowError to DataFusionError and take the calling function's name. They also move to datafusion-physical-expr-common, with a re-export from datafusion_functions::regex so existing import paths keep working. Code that matches on ArrowError from these functions needs updating.

Results do not change, and no plans change.

🤖 Generated with Claude Code

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels Sep 15, 2026
@codecov-commenter

codecov-commenter commented Sep 15, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 82.53425% with 51 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.95%. Comparing base (140c7c5) to head (270299c).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/functions/src/regex/regexpcount.rs 56.25% 29 Missing and 6 partials ⚠️
datafusion/physical-expr-common/src/regex.rs 89.02% 9 Missing ⚠️
...on/physical-expr/src/expressions/binary/kernels.rs 76.47% 4 Missing ⚠️
datafusion/functions/src/regex/regexpinstr.rs 97.14% 1 Missing ⚠️
datafusion/functions/src/regex/regexpmatch.rs 97.56% 0 Missing and 1 partial ⚠️
datafusion/functions/src/regex/regexpreplace.rs 91.66% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #25352      +/-   ##
==========================================
+ Coverage   81.93%   81.95%   +0.01%     
==========================================
  Files        1136     1137       +1     
  Lines      429152   429242      +90     
  Branches   429152   429242      +90     
==========================================
+ Hits       351633   351773     +140     
+ Misses      56475    56443      -32     
+ Partials    21044    21026      -18     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@adriangb adriangb changed the title fix: keep the regex diagnosis in regexp_count and regexp_instr errors fix: report regex compile failures as DataFusion errors, consistently across the regexp family Sep 16, 2026
@github-actions

github-actions Bot commented Sep 16, 2026

Copy link
Copy Markdown

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details
     Cloning apache/main
    Building datafusion-functions v55.1.0 (current)
       Built [  32.062s] (current)
     Parsing datafusion-functions v55.1.0 (current)
      Parsed [   0.091s] (current)
    Building datafusion-functions v55.1.0 (baseline)
       Built [  31.351s] (baseline)
     Parsing datafusion-functions v55.1.0 (baseline)
      Parsed [   0.092s] (baseline)
    Checking datafusion-functions v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.594s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure function_missing: pub fn removed or renamed ---

Description:
A publicly-visible function cannot be imported by its prior path. A `pub use` may have been removed, or the function itself may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/function_missing.ron

Failed in:
  function datafusion_functions::regex::compile_regex, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/284f2e243733b904663594d4e15787cf92963ff9/datafusion/functions/src/regex/mod.rs:176
  function datafusion_functions::regex::compile_and_cache_regex, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/284f2e243733b904663594d4e15787cf92963ff9/datafusion/functions/src/regex/mod.rs:142

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  65.834s] datafusion-functions
    Building datafusion-optimizer v55.1.0 (current)
       Built [  27.475s] (current)
     Parsing datafusion-optimizer v55.1.0 (current)
      Parsed [   0.036s] (current)
    Building datafusion-optimizer v55.1.0 (baseline)
       Built [  27.582s] (baseline)
     Parsing datafusion-optimizer v55.1.0 (baseline)
      Parsed [   0.037s] (baseline)
    Checking datafusion-optimizer v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.218s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  56.339s] datafusion-optimizer
    Building datafusion-physical-expr v55.1.0 (current)
       Built [  29.220s] (current)
     Parsing datafusion-physical-expr v55.1.0 (current)
      Parsed [   0.053s] (current)
    Building datafusion-physical-expr v55.1.0 (baseline)
       Built [  29.338s] (baseline)
     Parsing datafusion-physical-expr v55.1.0 (baseline)
      Parsed [   0.055s] (baseline)
    Checking datafusion-physical-expr v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.494s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  60.235s] datafusion-physical-expr
    Building datafusion-physical-expr-common v55.1.0 (current)
       Built [  24.581s] (current)
     Parsing datafusion-physical-expr-common v55.1.0 (current)
      Parsed [   0.023s] (current)
    Building datafusion-physical-expr-common v55.1.0 (baseline)
       Built [  24.610s] (baseline)
     Parsing datafusion-physical-expr-common v55.1.0 (baseline)
      Parsed [   0.025s] (baseline)
    Checking datafusion-physical-expr-common v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.291s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  50.381s] datafusion-physical-expr-common
    Building datafusion-sqllogictest v55.1.0 (current)
       Built [ 100.660s] (current)
     Parsing datafusion-sqllogictest v55.1.0 (current)
      Parsed [   0.023s] (current)
    Building datafusion-sqllogictest v55.1.0 (baseline)
       Built [ 105.528s] (baseline)
     Parsing datafusion-sqllogictest v55.1.0 (baseline)
      Parsed [   0.026s] (baseline)
    Checking datafusion-sqllogictest v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.124s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 209.558s] datafusion-sqllogictest

@github-actions github-actions Bot added the auto detected api change Auto detected API change label Sep 16, 2026
@adriangb
adriangb requested a balanced review from Copilot September 16, 2026 14:54

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Pull request overview

This PR standardizes how DataFusion reports regex compilation failures across regexp_* functions and the ~ / ~* / !~ / !~* operators, ensuring invalid user patterns/flags surface as DataFusion Plan/Execution errors with diagnostics from the regex parser.

Changes:

  • Introduces shared regex compilation/error-explanation helpers in datafusion-physical-expr-common, and re-exports them from datafusion_functions::regex.
  • Wraps Arrow regexp kernel failures to produce consistent DataFusion errors (while only doing extra work on the error path).
  • Updates optimizer planning-time errors and expands/adjusts sqllogictest coverage and example assertions.

Reviewed changes

Copilot reviewed 17 out of 18 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
datafusion/sqllogictest/test_files/regexp/regexp_replace.slt Adds expectations for consistent execution errors on invalid patterns/flags and validates g support.
datafusion/sqllogictest/test_files/regexp/regexp_match.slt Updates “too big” message match and adds invalid pattern/flag + NULL behavior cases.
datafusion/sqllogictest/test_files/regexp/regexp_like.slt Adds planning-time vs execution-time error expectations and operator (~ family) coverage.
datafusion/sqllogictest/test_files/regexp/regexp_instr.slt Fixes error-shape assertions and adds invalid pattern/flag coverage.
datafusion/sqllogictest/test_files/regexp/regexp_count.slt Fixes previously non-asserting blocks and adds invalid pattern/flag coverage.
datafusion/physical-expr/src/expressions/binary/kernels.rs Converts Arrow regexp kernel errors for the ~ operator family into DataFusion errors with diagnostics.
datafusion/physical-expr-common/src/regex.rs Adds shared regex compilation + Arrow-kernel error explanation helper.
datafusion/physical-expr-common/src/lib.rs Exposes the new regex module.
datafusion/physical-expr-common/Cargo.toml Adds regex dependency for the shared module.
datafusion/optimizer/src/simplify_expressions/regex.rs Converts invalid literal-regex detection into a Plan error with parser diagnosis.
datafusion/functions/src/regex/regexpreplace.rs Routes compilation through shared helper and updates unit test expectation.
datafusion/functions/src/regex/regexpmatch.rs Uses shared kernel error explanation and improves global-flag rejection (contains('g')).
datafusion/functions/src/regex/regexplike.rs Explains Arrow-kernel failures consistently and uses shared compile helper in scalar path.
datafusion/functions/src/regex/regexpinstr.rs Switches to DataFusion errors and shared compile helper for cached regex compilation.
datafusion/functions/src/regex/regexpcount.rs Switches to DataFusion errors and shared compile/cache helper; improves error consistency.
datafusion/functions/src/regex/mod.rs Re-exports shared helpers from datafusion-physical-expr-common and removes old local implementations.
datafusion-examples/examples/builtin_functions/regexp.rs Updates example assertion for new “too big” message wording.
Suppressed comments (1)

datafusion/physical-expr-common/src/regex.rs:1

  • explain_regexp_kernel_error allocates Vec<Option<&str>> for the full patterns (and flags) arrays on every kernel error via string_values(...).iter().collect(). On large batches, a single invalid regex can therefore trigger a large allocation and extra copying on the error path, which is avoidable and can have operational impact (memory spikes / potential DoS via invalid regex on large inputs). Consider iterating the underlying Arrow string arrays directly (without collecting) and using type-specific accessors (e.g., value(i) / is_null(i)), or introducing a lightweight accessor enum that provides len() + get(row) with scalar-broadcast support, so the error explanation remains O(1) additional memory.
// Licensed to the Apache Software Foundation (ASF) under one

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread datafusion/physical-expr-common/src/regex.rs
@github-actions github-actions Bot added physical-expr Changes to the physical-expr crates optimizer Optimizer rules labels Sep 16, 2026
adriangb and others added 5 commits September 16, 2026 14:37
`compile_regex` discarded the `regex::Error` and reported only the pattern, so
a user could not see why a pattern or a flag was invalid. Report the diagnosis
from the regex crate instead. It contains the pattern, so nothing is lost.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`compile_regex` and `compile_and_cache_regex` move to a new `regex` module in
datafusion-physical-expr-common, so that the physical expressions can use them
too. They now return a `DataFusionError` instead of an `ArrowError`, and they
take the name of the SQL function of the caller, so that an unsupported flag
names the function that the user called instead of a fixed pair of names.
`datafusion_functions::regex` re-exports both, so the paths that callers use
still resolve.

regexp_count and regexp_instr propagate the new error type. Their tests are
updated, including three that asserted nothing because the expected message
was parsed as part of the SQL statement.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
regexp_match, regexp_like, regexp_replace and the `~` family of operators hand
the pattern to an arrow kernel, which compiles it and reports a failure as an
opaque `ArrowError::ComputeError`. A user saw an internal error instead of the
reason the pattern was rejected.

The kernel keeps compiling the pattern. Only when it fails does
`explain_regexp_kernel_error` compile the patterns again, to report the first
one that does not compile with the diagnosis of the regex crate. A query that
succeeds compiles the pattern exactly as many times as before.

The "global" flag check in regexp_match now tests every flags string that
contains 'g', so "gi" no longer reaches the kernel.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`simplify_regex_expr` compiles a literal pattern to rewrite it, and reported a
pattern that does not compile as `Invalid regex`, wrapping the diagnosis in an
`External` error. A literal pattern that does not compile is an error in the
query text, known before execution, so report it as a plan error carrying the
diagnosis of the regex_syntax crate.

Every two argument regexp_like is simplified to the `~` operator, so this is
the error that the most common spelling produces. Its wording now matches the
one that the same pattern produces at execution time.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`explain_regexp_kernel_error` collected the whole `patterns` array (and
`flags`) into a `Vec<Option<&str>>` before looking for the pattern that did
not compile. The collection is proportional to the length of the arrays, so
a single invalid pattern in a large batch allocated and copied once per row
on the error path.

Borrow the arrays instead, through an accessor that holds the typed array
and reads a row on demand. Explaining an error now allocates nothing beyond
the pattern that `compile_regex` builds, whatever the length of the batch.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9fJcUW5cbNf72vbWazRhh
@adriangb
adriangb force-pushed the regex-compile-diagnosis branch from 8b6e72b to 807ca6a Compare September 16, 2026 19:37
Comment on lines +34 to +36
/// Compiles `regex` with [`compile_regex`], keeping the compiled pattern in
/// `regex_cache` under the key `(regex, flags)`.
pub fn compile_and_cache_regex<'strings, 'cache>(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: these were just moved

/// A pattern that does not compile is reported as
/// [`DataFusionError::Execution`] carrying the diagnosis of the `regex` crate,
/// which names the position and the reason the pattern was rejected.
pub fn compile_regex(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: these were just moved

Comment thread datafusion/physical-expr-common/src/regex.rs
`explain_regexp_kernel_error` is `pub` because `datafusion-physical-expr`
and `datafusion-functions` call it from their own crates, not because it is
meant for callers outside the workspace. Mark it `#[doc(hidden)]`, as the
rest of the workspace marks the items that are public only to cross a crate
boundary.

`compile_regex` and `compile_and_cache_regex` keep their documentation: they
were already public API before this branch moved them here.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H9fJcUW5cbNf72vbWazRhh
@adriangb

Copy link
Copy Markdown
Contributor Author

@neilconway could I ask for your review here? The goal is just to make errors more uniform for upstream systems. We attempt to determine if errors are our bug or a user error, and currently it's impossible to do with many of these regex cases (requires parsing error strings, and there are many variations of them).

@adriangb

Copy link
Copy Markdown
Contributor Author

Thanks @martin-g !

@neilconway neilconway left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor nit but otherwise lgtm!

Comment on lines +123 to +132
for row in 0..rows {
// A NULL pattern or NULL flags produce a NULL result, not an error.
let Some(pattern) = patterns.broadcast_value(row) else {
continue;
};
let flags = flags.as_ref().and_then(|flags| flags.broadcast_value(row));
if let Err(error) = compile_regex(function_name, pattern, flags) {
return error;
}
}

@neilconway neilconway Sep 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think to match Arrow semantics, we should skip NULL rows. Otherwise, given two rows with invalid patterns, one NULL and the other non-NULL, we might report an error about the wrong pattern.

In general it's a bit fragile to couple the logic here to Arrow's logic, but I suppose there's no way around that...

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed in 3f8ef4c and 270299c

The arrow regexp kernels produce NULL for a row whose value is NULL and
never compile that row's pattern. `explain_regexp_kernel_error` compiled
every pattern, so with two invalid patterns on two rows, one of them a
NULL row, it reported the pattern the kernel had skipped rather than the
one that actually made it fail.

Pass the values array to the explanation on the call sites whose kernel
compiles a pattern per row, and skip a row whose value is NULL. The
kernels that compile one pattern up front, before reading any value,
pass `None` and keep explaining that pattern whatever the values are.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MiunrdKMjyE9ZRe5KoVtHf
Comment on lines +470 to +471
query error DataFusion error: Execution error: Regular expression did not compile: regex parse error[\s\S]*unclosed character class
SELECT regexp_like(str, pattern, 'm') FROM t_null_value;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a control:

SELECT str ~ pattern FROM t_null_value WHERE str IS NULL;

Worth also testing when pattern is null?

Add the control that the rows the kernel skips give NULL on their own, so
that the error of the other row is what the surrounding tests assert, and
cover a NULL pattern beside a NULL value: the kernel skips a row of
either kind, and neither row's pattern may be reported as the one that
failed to compile.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MiunrdKMjyE9ZRe5KoVtHf
@adriangb
adriangb enabled auto-merge September 16, 2026 22:59
@adriangb
adriangb added this pull request to the merge queue Sep 16, 2026
Merged via the queue into apache:main with commit 28c0dec Sep 16, 2026
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto detected api change Auto detected API change functions Changes to functions implementation optimizer Optimizer rules physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants