Skip to content

[BP-1.20][FLINK-40070][tests] Fix flaky DynamicParameterITCase reading rolled JobManager logs - #29215

Merged
MartijnVisser merged 1 commit into
apache:release-1.20from
MartijnVisser:FLINK-40070-1.20
Sep 17, 2026
Merged

MartijnVisser merged 1 commit into
apache:release-1.20from
MartijnVisser:FLINK-40070-1.20

Conversation

@MartijnVisser

Copy link
Copy Markdown
Contributor

What is the purpose of the change

Backport of #28636 (f9e0756fbfd) to release-1.20.

The distribution rolls its log file during JobManager startup, so the startup banner with the
program arguments frequently ends up in a rolled .log.N file that
FlinkDistribution.searchAllLogs skips. DynamicParameterITCase then waits for that banner in
an unbounded loop, and the class has no @Timeout, so the whole Java e2e leg runs until the CI
watchdog kills it at its time budget. On release-2.3 that took out E2E group 2 on the 2026-09-04,
09-11, 09-15 and 09-16 nightlies. The readiness loop, FlinkDistribution and the distribution
log4j configuration are identical on this branch.

Brief change log

  • FlinkDistribution.searchAllLogs gains an includeRolledLogs overload, so the banner is
    found when it has been rolled. The existing two-argument method delegates with false, so
    the other callers are unchanged.
  • DynamicParameterITCase waits with CommonTestUtils.waitUtil(..., 1 minute, 500 ms, msg)
    instead of while (!allProgramArgumentsLogged(dist)) Thread.sleep(500), so a banner that
    never arrives fails in a minute with a message rather than consuming the leg's budget.

Verifying this change

FlinkDistribution.java is byte-identical to the master commit. DynamicParameterITCase.java was
resolved by hand: release-1.20 passes the host and port positionally and asserts on
EntrypointClusterConfiguration#getHostname and #getRestPort, so the argument building and the
assertions are left alone. Master's loadConfiguration helper and its Configuration based
assertions come from a later change that is not on this branch, and are not introduced here. The
three helpers the fix adds or rewrites are byte-identical to master's.

DynamicParameterITCase green twice against a built 1.20 distribution, 6 tests in 29.5 s and
29.8 s. spotless and checkstyle green under JDK 11, since this branch pins google-java-format 1.7
rather than 1.24.0.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no

Was generative AI tooling used to co-author this PR?
  • Yes (Claude Opus 5, via Claude Code)

Generated-by: Claude Opus 5 (1M context)

…JobManager logs

The distribution log4j configuration rolls the log file on startup, so the JobManager startup banner frequently lands in a rolled .log.N file that FlinkDistribution.searchAllLogs skips; the test then either spins unboundedly waiting for the banner (multi-hour e2e_4 hang) or parses a half-written arguments block ("Missing required option: c"). Search rolled logs for the startup banner and bound the wait so a missing banner fails fast.

Generated-by: Claude Opus 4.8 (1M context)
(cherry picked from commit f9e0756)
@flinkbot

flinkbot commented Sep 16, 2026

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@spuru9 spuru9 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean BP. LGTM.

@github-actions github-actions Bot added target:release-1.20 community-reviewed PR has been reviewed by the community. labels Sep 16, 2026
@MartijnVisser
MartijnVisser merged commit 4048a89 into apache:release-1.20 Sep 17, 2026
@MartijnVisser
MartijnVisser deleted the FLINK-40070-1.20 branch September 17, 2026 12:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-reviewed PR has been reviewed by the community. target:release-1.20

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants