[rpc] Handle endpoint changes for cached server connections - #4263
Open
BackendArchitectX wants to merge 2 commits into
Open
[rpc] Handle endpoint changes for cached server connections#4263BackendArchitectX wants to merge 2 commits into
BackendArchitectX wants to merge 2 commits into
Conversation
Open
2 tasks
Author
|
@wuchong @loserwang1024 @swuferhong @luoyuxia, would appreciate your thoughts on the connection-handling approach in this PR. It overlaps with the same-UID endpoint handling discussed in #4216, but keeps physical connections keyed by UID + host + port instead of replacing the existing UID connection. Any feedback on the preferred direction would be very helpful. |
BackendArchitectX
force-pushed
the
fix-4256-stale-tabletserver-endpoint
branch
from
September 9, 2026 14:14
1d4bf34 to
00f43e1
Compare
Contributor
|
@BackendArchitectX Thanks! Using |
gyang94
reviewed
Sep 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: #4256
NettyClientcurrently caches connections only by server UID. During a rolling upgrade, the same TabletServer UID may be advertised with a different host or port while the previous endpoint remains reachable.In that case, even after metadata is refreshed with the new endpoint, connection lookup can continue returning the connection to the previous endpoint. Requests may therefore keep reaching the wrong TabletServer.
This change makes the physical RPC connection identity endpoint-aware by using server UID, host, and port.
This is related to #4216, which also handles same-UID endpoint changes but replaces the existing connection. This change keeps connections to different physical endpoints independently keyed so the RPC layer does not have to decide which reachable endpoint is newer.
Brief change log
ServerNode.disconnect(ServerNode)for disconnecting a specific endpoint.disconnect(serverUid)as a logical-server operation that closes all connections associated with that UID.Tests
.\mvnw.cmd test -Dtest=NettyClientTest -pl fluss-rpc.\mvnw.cmd verify -pl fluss-rpc -Dtest="!ApiErrorTest"Checkstyle passed with 0 violations.
Spotless passed.
git diff --checkpassed.A full
fluss-rpcverify on Windows also encounters the existingApiErrorTest#testStringifyExceptionline-ending assertion, which expects LF while the Windows JVM stack trace uses CRLF. This test is unrelated to this change.The regression coverage intentionally isolates the transport-level condition: two reachable TabletServers use the same UID with different endpoints, and requests using the updated
ServerNodereach the updated endpoint rather than the previously cached one.The tests also verify that readiness and endpoint-specific disconnect operate on the exact
serverUid + host + portconnection, while UID-level disconnect continues to close all physical connections associated with the logical server.API and Format
No user-facing API, RPC protocol, or storage format changes. The internal
RpcClientlifecycle API is aligned with endpoint-aware connection identity.Documentation
No documentation changes. This fixes existing RPC connection behavior when a server endpoint changes.
Generative AI disclosure
Yes - ChatGPT (GPT-5.6 Sol) was used for root-cause analysis, implementation review, and test planning.