Skip to content

[ISSUE #11083] Release owned gRPC event loops and stabilize POP priority tests - #11084

Merged
lizhimins merged 2 commits into
apache:developfrom
qianye1001:codex/fix-grpc-event-loop-leak
Sep 10, 2026
Merged

[ISSUE #11083] Release owned gRPC event loops and stabilize POP priority tests#11084
lizhimins merged 2 commits into
apache:developfrom
qianye1001:codex/fix-grpc-event-loop-leak

Conversation

@qianye1001

@qianye1001 qianye1001 commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Which Issue(s) This PR Fixes

Fixes #11083

Brief Description

GrpcServerBuilder supplies explicitly created boss and worker event loop groups to gRPC, but GrpcServer.shutdown() never releases them. gRPC treats these groups as caller-owned, so their threads remain alive after the server shuts down.

Track internally created groups and shut them down when the server stops, including startup failure, shutdown timeout, and interruption. Create owned groups during build() and release them if building fails. Preserve existing constructor defaults and add overloads for explicit thread counts or caller-owned groups; supplied groups remain the caller's responsibility and can outlive an individual server. Declare the gRPC test's direct grpc-protobuf dependency for Bazel.

Stabilize PopPriorityIT by waiting for the expected normal and retry messages to reach their consume queues before asserting priority. Waiting a fixed invisibleTime + 3 seconds can start consumption before a retry becomes available. Replace fixed queue-lock sleeps with bounded readiness checks so sequential requests do not skip a priority queue whose previous completion callback has not released its lock. Use deterministic input priorities and retain the existing ordering, count, and distribution assertions; parameter names and histogram failures now identify the tested configuration.

Upload Bazel JUnit XML reports when a job fails. The failing CI run for this PR reported PopPriorityIT but omitted its 25 MB output, including the original assertion, because it exceeded Bazel's console limit.

How Did You Test This Change?

mvn -B -pl test -am -Dmaven.gitcommitid.skip=true \
  -Dtest=PopPriorityIT,GrpcServerTest \
  -Dsurefire.failIfNoSpecifiedTests=false test

On JDK 11, the build, Checkstyle, and SpotBugs passed:

  • PopPriorityIT: all 32 enabled tests passed across legacy/KV POP and ascending/descending priority. The existing four ignored queue-expansion cases remain ignored.
  • GrpcServerTest: six tests passed; the Epoll case was skipped on macOS. Coverage includes actual gRPC requests, owned-group termination, bind failures, shared caller-owned groups, shutdown timeout, and interruption.
  • Controlled scheduling experiment: delay legacy retry revival until eight seconds after the first POP response. The original test_priority_consume_retry_as_highest fails with expected:<1> but was:<0>; the updated test passes under the same delay. This independently reproduces the readiness race; the original CI assertion was unavailable in its truncated log.
  • Workflow YAML parsed successfully. Local Bazel remote execution could not connect to the remote executor; Bazel validation is pending in the updated branch's GitHub Actions run.

@RockteMQ-AI RockteMQ-AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

This PR correctly fixes a resource leak where GrpcServerBuilder-created boss/worker EventLoopGroups were never released on shutdown. The approach of tracking owned vs. borrowed groups via constructor parameters is clean, and the lifecycle handling covers startup failure, shutdown timeout, and interruption paths.

The test coverage is thorough — verifying owned group termination, borrowed group survival, bind-failure cleanup, timeout, and interrupt scenarios.

LGTM.


Automated review by github-manager-bot

@codecov-commenter

codecov-commenter commented Sep 9, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 80.00000% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 48.97%. Comparing base (4fd0e3b) to head (bd065d7).

Files with missing lines Patch % Lines
.../apache/rocketmq/proxy/grpc/GrpcServerBuilder.java 75.67% 8 Missing and 1 partial ⚠️
...ava/org/apache/rocketmq/proxy/grpc/GrpcServer.java 86.95% 3 Missing ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             develop   #11084      +/-   ##
=============================================
+ Coverage      48.93%   48.97%   +0.04%     
- Complexity     13846    13860      +14     
=============================================
  Files           1382     1382              
  Lines         101602   101640      +38     
  Branches       13213    13224      +11     
=============================================
+ Hits           49717    49782      +65     
+ Misses         45858    45807      -51     
- Partials        6027     6051      +24     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@qianye1001
qianye1001 force-pushed the codex/fix-grpc-event-loop-leak branch from ec1fe92 to 1ea4346 Compare September 9, 2026 08:56
@qianye1001 qianye1001 changed the title [ISSUE #11083] Release owned gRPC event loop groups on shutdown [ISSUE #11083] Release owned gRPC event loops and stabilize POP priority tests Sep 9, 2026

@RockteMQ-AI RockteMQ-AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

The new commits significantly improve test stability by replacing timing-dependent waits with proper polling helpers. The changes make the POP priority integration tests deterministic and resilient to CI timing issues.

Changes reviewed:

  • Commit 1ea4346: Released owned gRPC event loop groups on shutdown (resource leak fix)
  • Commit bd065d7: Stabilized POP priority tests with awaitStoredMessages/awaitRetryMessages helpers, deterministic queue distribution, and Bazel test result upload on failure

Excellent test stability improvements. The move from Random().nextInt() to deterministic i % writeQueueNum is particularly good for reproducibility.

LGTM


Automated review by RockteMQ-AI

@lizhimins
lizhimins merged commit eedd760 into apache:develop Sep 10, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Proxy gRPC server leaks boss and worker event loop threads on shutdown

5 participants