Skip to content

Invalid CSV null_regex panics the query task instead of returning an error #25260

Description

@Developer1010x

Describe the bug

A malformed null_regex panics the query task instead of returning an error.
CsvFormat::infer_schema_from_stream compiles the pattern with

// datafusion/datasource-csv/src/file_format.rs
let regex = Regex::new(null_regex.as_str())
    .expect("Unable to parse CSV null regex.");

so any pattern the regex crate rejects aborts the task rather than surfacing
as a DataFusionError. It is reachable straight from SQL, from a CREATE EXTERNAL TABLE that does not spell out its columns, since that is what makes
schema inference run.

To Reproduce

CREATE EXTERNAL TABLE bad_regex
STORED AS CSV
LOCATION 'data.csv'
OPTIONS ('format.has_header' 'true', 'format.null_regex' '(');
task 9 panicked with message "Unable to parse CSV null regex.: Syntax(
regex parse error:
    (
    ^
error: unclosed group
)"

Reproduced on main (9082d6b) through the sqllogictest harness. Any invalid
pattern does it; ( is just the shortest.

Expected behavior

An invalid null_regex is a bad option value, so it should come back as an
error naming the offending pattern — the same way other malformed CSV options
are handled — and leave the session usable.

Additional context

Noticed while working on #25213 / #25254. That PR changes CsvSource::builder
to return Result for the same reason on the read side, so it does not add a
second panic; this one is on the inference side and is independent of it.

The regex is also recompiled for every chunk inside the inference loop, so
hoisting the compile out is both the fix and a small saving.

I have a fix ready and will open a PR shortly.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions