fix: report regex compile failures as DataFusion errors, consistently across the regexp family - #25352
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #25352 +/- ##
==========================================
+ Coverage 81.93% 81.95% +0.01%
==========================================
Files 1136 1137 +1
Lines 429152 429242 +90
Branches 429152 429242 +90
==========================================
+ Hits 351633 351773 +140
+ Misses 56475 56443 -32
+ Partials 21044 21026 -18 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Thank you for opening this pull request! Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details |
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Pull request overview
This PR standardizes how DataFusion reports regex compilation failures across regexp_* functions and the ~ / ~* / !~ / !~* operators, ensuring invalid user patterns/flags surface as DataFusion Plan/Execution errors with diagnostics from the regex parser.
Changes:
- Introduces shared regex compilation/error-explanation helpers in
datafusion-physical-expr-common, and re-exports them fromdatafusion_functions::regex. - Wraps Arrow regexp kernel failures to produce consistent DataFusion errors (while only doing extra work on the error path).
- Updates optimizer planning-time errors and expands/adjusts sqllogictest coverage and example assertions.
Reviewed changes
Copilot reviewed 17 out of 18 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| datafusion/sqllogictest/test_files/regexp/regexp_replace.slt | Adds expectations for consistent execution errors on invalid patterns/flags and validates g support. |
| datafusion/sqllogictest/test_files/regexp/regexp_match.slt | Updates “too big” message match and adds invalid pattern/flag + NULL behavior cases. |
| datafusion/sqllogictest/test_files/regexp/regexp_like.slt | Adds planning-time vs execution-time error expectations and operator (~ family) coverage. |
| datafusion/sqllogictest/test_files/regexp/regexp_instr.slt | Fixes error-shape assertions and adds invalid pattern/flag coverage. |
| datafusion/sqllogictest/test_files/regexp/regexp_count.slt | Fixes previously non-asserting blocks and adds invalid pattern/flag coverage. |
| datafusion/physical-expr/src/expressions/binary/kernels.rs | Converts Arrow regexp kernel errors for the ~ operator family into DataFusion errors with diagnostics. |
| datafusion/physical-expr-common/src/regex.rs | Adds shared regex compilation + Arrow-kernel error explanation helper. |
| datafusion/physical-expr-common/src/lib.rs | Exposes the new regex module. |
| datafusion/physical-expr-common/Cargo.toml | Adds regex dependency for the shared module. |
| datafusion/optimizer/src/simplify_expressions/regex.rs | Converts invalid literal-regex detection into a Plan error with parser diagnosis. |
| datafusion/functions/src/regex/regexpreplace.rs | Routes compilation through shared helper and updates unit test expectation. |
| datafusion/functions/src/regex/regexpmatch.rs | Uses shared kernel error explanation and improves global-flag rejection (contains('g')). |
| datafusion/functions/src/regex/regexplike.rs | Explains Arrow-kernel failures consistently and uses shared compile helper in scalar path. |
| datafusion/functions/src/regex/regexpinstr.rs | Switches to DataFusion errors and shared compile helper for cached regex compilation. |
| datafusion/functions/src/regex/regexpcount.rs | Switches to DataFusion errors and shared compile/cache helper; improves error consistency. |
| datafusion/functions/src/regex/mod.rs | Re-exports shared helpers from datafusion-physical-expr-common and removes old local implementations. |
| datafusion-examples/examples/builtin_functions/regexp.rs | Updates example assertion for new “too big” message wording. |
Suppressed comments (1)
datafusion/physical-expr-common/src/regex.rs:1
explain_regexp_kernel_errorallocatesVec<Option<&str>>for the fullpatterns(andflags) arrays on every kernel error viastring_values(...).iter().collect(). On large batches, a single invalid regex can therefore trigger a large allocation and extra copying on the error path, which is avoidable and can have operational impact (memory spikes / potential DoS via invalid regex on large inputs). Consider iterating the underlying Arrow string arrays directly (without collecting) and using type-specific accessors (e.g.,value(i)/is_null(i)), or introducing a lightweight accessor enum that provideslen()+get(row)with scalar-broadcast support, so the error explanation remains O(1) additional memory.
// Licensed to the Apache Software Foundation (ASF) under one
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
`compile_regex` discarded the `regex::Error` and reported only the pattern, so a user could not see why a pattern or a flag was invalid. Report the diagnosis from the regex crate instead. It contains the pattern, so nothing is lost. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`compile_regex` and `compile_and_cache_regex` move to a new `regex` module in datafusion-physical-expr-common, so that the physical expressions can use them too. They now return a `DataFusionError` instead of an `ArrowError`, and they take the name of the SQL function of the caller, so that an unsupported flag names the function that the user called instead of a fixed pair of names. `datafusion_functions::regex` re-exports both, so the paths that callers use still resolve. regexp_count and regexp_instr propagate the new error type. Their tests are updated, including three that asserted nothing because the expected message was parsed as part of the SQL statement. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
regexp_match, regexp_like, regexp_replace and the `~` family of operators hand the pattern to an arrow kernel, which compiles it and reports a failure as an opaque `ArrowError::ComputeError`. A user saw an internal error instead of the reason the pattern was rejected. The kernel keeps compiling the pattern. Only when it fails does `explain_regexp_kernel_error` compile the patterns again, to report the first one that does not compile with the diagnosis of the regex crate. A query that succeeds compiles the pattern exactly as many times as before. The "global" flag check in regexp_match now tests every flags string that contains 'g', so "gi" no longer reaches the kernel. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`simplify_regex_expr` compiles a literal pattern to rewrite it, and reported a pattern that does not compile as `Invalid regex`, wrapping the diagnosis in an `External` error. A literal pattern that does not compile is an error in the query text, known before execution, so report it as a plan error carrying the diagnosis of the regex_syntax crate. Every two argument regexp_like is simplified to the `~` operator, so this is the error that the most common spelling produces. Its wording now matches the one that the same pattern produces at execution time. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`explain_regexp_kernel_error` collected the whole `patterns` array (and `flags`) into a `Vec<Option<&str>>` before looking for the pattern that did not compile. The collection is proportional to the length of the arrays, so a single invalid pattern in a large batch allocated and copied once per row on the error path. Borrow the arrays instead, through an accessor that holds the typed array and reads a row on demand. Explaining an error now allocates nothing beyond the pattern that `compile_regex` builds, whatever the length of the batch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H9fJcUW5cbNf72vbWazRhh
8b6e72b to
807ca6a
Compare
| /// Compiles `regex` with [`compile_regex`], keeping the compiled pattern in | ||
| /// `regex_cache` under the key `(regex, flags)`. | ||
| pub fn compile_and_cache_regex<'strings, 'cache>( |
There was a problem hiding this comment.
Note: these were just moved
| /// A pattern that does not compile is reported as | ||
| /// [`DataFusionError::Execution`] carrying the diagnosis of the `regex` crate, | ||
| /// which names the position and the reason the pattern was rejected. | ||
| pub fn compile_regex( |
There was a problem hiding this comment.
Note: these were just moved
`explain_regexp_kernel_error` is `pub` because `datafusion-physical-expr` and `datafusion-functions` call it from their own crates, not because it is meant for callers outside the workspace. Mark it `#[doc(hidden)]`, as the rest of the workspace marks the items that are public only to cross a crate boundary. `compile_regex` and `compile_and_cache_regex` keep their documentation: they were already public API before this branch moved them here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H9fJcUW5cbNf72vbWazRhh
|
@neilconway could I ask for your review here? The goal is just to make errors more uniform for upstream systems. We attempt to determine if errors are our bug or a user error, and currently it's impossible to do with many of these regex cases (requires parsing error strings, and there are many variations of them). |
|
Thanks @martin-g ! |
neilconway
left a comment
There was a problem hiding this comment.
Minor nit but otherwise lgtm!
| for row in 0..rows { | ||
| // A NULL pattern or NULL flags produce a NULL result, not an error. | ||
| let Some(pattern) = patterns.broadcast_value(row) else { | ||
| continue; | ||
| }; | ||
| let flags = flags.as_ref().and_then(|flags| flags.broadcast_value(row)); | ||
| if let Err(error) = compile_regex(function_name, pattern, flags) { | ||
| return error; | ||
| } | ||
| } |
There was a problem hiding this comment.
I think to match Arrow semantics, we should skip NULL rows. Otherwise, given two rows with invalid patterns, one NULL and the other non-NULL, we might report an error about the wrong pattern.
In general it's a bit fragile to couple the logic here to Arrow's logic, but I suppose there's no way around that...
The arrow regexp kernels produce NULL for a row whose value is NULL and never compile that row's pattern. `explain_regexp_kernel_error` compiled every pattern, so with two invalid patterns on two rows, one of them a NULL row, it reported the pattern the kernel had skipped rather than the one that actually made it fail. Pass the values array to the explanation on the call sites whose kernel compiles a pattern per row, and skip a row whose value is NULL. The kernels that compile one pattern up front, before reading any value, pass `None` and keep explaining that pattern whatever the values are. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MiunrdKMjyE9ZRe5KoVtHf
| query error DataFusion error: Execution error: Regular expression did not compile: regex parse error[\s\S]*unclosed character class | ||
| SELECT regexp_like(str, pattern, 'm') FROM t_null_value; |
There was a problem hiding this comment.
Can we add a control:
SELECT str ~ pattern FROM t_null_value WHERE str IS NULL;Worth also testing when pattern is null?
Add the control that the rows the kernel skips give NULL on their own, so that the error of the other row is what the surrounding tests assert, and cover a NULL pattern beside a NULL value: the kernel skips a row of either kind, and neither row's pattern may be reported as the one that failed to compile. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MiunrdKMjyE9ZRe5KoVtHf
Which issue does this PR close?
Rationale for this change
A regular expression that comes from user SQL and does not compile is reported in four different ways today. Two of them are
Arrow error: Compute error, which is the same shape the engine uses for internal faults, so a caller cannot tell an invalid query from an engine problem. Forregexp_countandregexp_instrthe reason is dropped, and the user only sees the pattern.On
main:The
~,~*,!~and!~*operators, and every two-argumentregexp_like, are affected as well.RegexpLikeFunc::simplifyrewrites all of them to an operator, and the operators are evaluated indatafusion/physical-expr/src/expressions/binary/kernels.rs, which calls the arrow kernels directly. That is the most common spelling, and it produced theComputeErrorshape.What changes are included in this PR?
Three commits.
datafusion-physical-expr-common.compile_regexandcompile_and_cache_regexreturn aDataFusionErrorthat carries the diagnosis from theregexcrate, and they take the name of the calling function so the "global" flag message names the function the user called.datafusion_functions::regexre-exports both, so the old path still resolves.regexp_countandregexp_instrpropagate the new type.main: the same number ofRegex::newcalls, none added. It is used byregexp_match,regexp_like,regexp_replaceand by the~/~*/!~/!~*kernels inphysical-expr.simplify_regex_exprreturnsDataFusionError::Planwith the diagnosis, instead ofContext("Invalid regex", External(..)).The result is one wording for the whole family. The variant is
Planwhen a literal pattern is caught at planning, andExecutionotherwise:gflagregexp_likeregexp_like()~~*!~!~*regexp_matchregexp_match()regexp_countregexp_count()regexp_instrregexp_instr()regexp_replaceTwo related defects are fixed along the way:
regexp_matchrejected only the exact flags stringg, sogireached the kernel and failed withunrecognized flag. The check is nowcontains('g'), which is whatregexp_likealready did.regexp_count.sltasserted nothing. They were written as a barestatement errorwith the expected message on the following line, so the message was parsed as part of the SQL, the SQL failed to parse, and the assertion passed. One of them expected a message thatmaindoes not produce.What is the testing strategy for this PR?
New and updated cases in
datafusion/sqllogictest/test_files/regexp/*.sltcover, for every function in the family, an invalid pattern as a literal and as a column, an invalid flag, a pattern over the size limit, and thegflag, plus the NULL cases (regexp_match(NULL, 'a(b'), a NULL pattern) which must stay NULL rather than raise.Run locally on this branch:
cargo fmt --all -- --check;cargo clippy --all-targets --all-featuresfordatafusion-functions,datafusion-physical-expr,datafusion-physical-expr-commonanddatafusion-optimizer; the tests of those four crates; the full sqllogictest suite (520 files);cargo check -p datafusion-examples; andcargo build --locked -p datafusion.Are there any user-facing changes?
Yes, in error messages and in one public API.
PlanorExecutionerror with the reason from theregexcrate, instead ofArrow error: Compute error, a bareExternalerror, or a message that names only the pattern.Compiled regex exceeds size limit of 10485760 bytes.instead ofCompiledTooBig(10485760).compile_regexandcompile_and_cache_regexchange their error type fromArrowErrortoDataFusionErrorand take the calling function's name. They also move todatafusion-physical-expr-common, with a re-export fromdatafusion_functions::regexso existing import paths keep working. Code that matches onArrowErrorfrom these functions needs updating.Results do not change, and no plans change.
🤖 Generated with Claude Code