Skip to content

[FLINK-6417][connector/filesystem] Support glob paths in FileSource - #29219

Open
Cribbee wants to merge 1 commit into
apache:masterfrom
Cribbee:codex/flink-6417-filesource-glob
Open

Cribbee wants to merge 1 commit into
apache:masterfrom
Cribbee:codex/flink-6417-filesource-glob

Conversation

@Cribbee

@Cribbee Cribbee commented Sep 17, 2026

Copy link
Copy Markdown

What is the purpose of the change

This pull request resolves FLINK-6417 for FileSource by allowing its default file enumerators to accept glob patterns in input paths. This makes it possible to select partition directories and files without constructing every concrete path in user code.

The implementation is file-system independent. It traverses from the longest non-glob path prefix and reuses Flink's existing GlobFilePathFilter, rather than changing a specific file system implementation or the semantics of FileSystem#listStatus.

Brief change log

  • Add glob path handling to NonSplittingRecursiveEnumerator, which is also inherited by BlockSplittingRecursiveEnumerator.
  • Support *, ?, and character classes in any path segment.
  • Return no splits when a glob has no matches, including when its non-glob search root does not exist.
  • Document glob input paths in the FileSource JavaDocs and DataStream filesystem connector documentation.

Verifying this change

This change added tests covering:

  • wildcard matches across directory and file path segments;
  • recursive enumeration when a glob matches a directory;
  • ? and character class patterns;
  • glob paths with no matches and with a missing search root;
  • both non-splitting and block-splitting recursive enumerators.

The complete flink-connector-files test suite passes locally: 211 tests run, 0 failures, 0 errors, 6 skipped.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): yes (behavior and JavaDocs only; no signature changes)
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no (file enumeration only)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: yes (generic FileSource enumeration behavior)

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? docs and JavaDocs

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Codex (GPT-5)

@flinkbot

flinkbot commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@Cribbee

Cribbee commented Sep 17, 2026

Copy link
Copy Markdown
Author

Hi @snuyanzin, apologies for the unsolicited ping. I noticed that you recently reviewed filesystem connector path-handling work such as #27553. Would you mind taking a quick look at this approach and, if it seems appropriate, helping assign FLINK-6417?

Unlike the earlier #15436, this PR implements glob handling in the default FileSource enumerators, preserves non-glob behavior, and uses only the generic FileSystem API. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants