fix(ilp): fix a leaked socket and native memory when an HTTP sender fails to start - #78
Open
puzpuzpuz wants to merge 4 commits into
Open
fix(ilp): fix a leaked socket and native memory when an HTTP sender fails to start#78puzpuzpuz wants to merge 4 commits into
puzpuzpuz wants to merge 4 commits into
Conversation
The ILP HTTP senders build their client through this fork's HttpClientFactory, whose constructors took a socket and native buffers with no rollback. A throw past the first acquisition left a half-built client the caller never receives, so close() never ran and everything taken so far leaked, the socket file descriptor included. HttpClient now takes the socket, both buffers and the response parser inside one try and frees what it took in reverse order, attaching a close failure to the primary exception rather than replacing it. ResponseHeaders does the same for the parser its super() call completed. HttpHeaderParser moves the direct UTF-8 sink out of its field initialiser, which runs before the constructor body, so the constructor's own catch can reach it. The three platform subclasses close the completed base client when their poller fails to open. HttpClientWindows also reads the select facade before it takes the FD set. Nothing between that acquisition and the end of the constructor can then throw, which removes a partial-FDSet cleanup branch that only a Windows host could ever have reached. HttpClientConstructorTest covers each rollback point with a close-counting socket, and HttpHeaderParserTest covers the sink. Both inject the fault deterministically and without a host dependency: a negative buffer size makes Unsafe.allocateMemory throw, and a throwing configuration getter fails each platform subclass before it touches platform natives. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
puzpuzpuz
added a commit
to questdb/questdb
that referenced
this pull request
Jul 30, 2026
The client-side port of the HTTP constructor rollback lives in questdb/java-questdb-client#78, which is not merged yet. Pinning the submodule at that branch commit would make this PR depend on an unmerged revision and, if merged first, would permanently fix the pointer at a commit that is not on the client's main branch. Point the submodule back at 2e84db07, the commit this PR already carried and the one main was on when the branch was cut. Once #78 merges, the pointer moves forward to a main commit that contains the fix. Chosen over the current main head, 9ccfadbe: that is 1.3.7-SNAPSHOT and would require bumping questdb.client.version in three parent poms, and it would drag in a version bump and a PEM CA feature unrelated to this PR. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
AbstractLineHttpSender's constructor assigned this.client - either the one createLineSender() handed in or one it built through HttpClientFactory - and then called newRequest(), with neither statement inside a try. newRequest() writes the POST line, the path, the User-Agent header and any auth header into the client's request buffer, and growBuffer() throws once they exceed the maximum. A throw there left a fully built client - a socket plus two native buffers plus the response parser - that no caller ever receives a reference to, so nothing closed it. A Java-heap OutOfMemoryError from the version lookup or the User-Agent concatenation lands in the same window. createLineSender() cannot clean this up. Its three Misc.free(cli) calls all sit before the switch, and the switch builds the sender inside a return expression that no try covers. Both entry shapes leak: an explicit protocol version makes the constructor build the client itself, and an auto-detected one hands in the client the version probe used. The constructor now takes the client, the version and the request inside one try and closes the client on the way out, attaching a close failure to the primary exception rather than replacing it. It frees a handed-in client too: close() already frees the client regardless of where it came from, so the sender owns it from the assignment on either way. LineHttpSenderConstructorTest covers both shapes. A configuration whose maximum request buffer is smaller than the preamble makes newRequest() throw deterministically, with no server and no new production seam. The self-built case asserts against the enclosing leak check; the handed-in case adds a close-counting socket, since the leak check cannot see a file descriptor. Both fail without the fix: the first on 65552 bytes leaked under NATIVE_DEFAULT, the second on the missing socket close. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… probe facadeCloseIsBoundedUnderRepeatedInterruptsDuringSenderCreation failed on windows-msvc-2022-x64 with "sender close left its creation wait before the interrupt storm landed twice; interrupts landed: 6" -- a false negative, not a product defect. Both pools' hasCreationWaiterForTesting() reported the wait region as lock.hasWaiters(creationFinished). An interrupt does not leave the waiter on the condition queue: AQS transfers the node to the lock's sync queue to reacquire before awaitNanos rethrows, so under the test's interrupt storm the probe reads false for a measurable slice of a wait close() never left. A harness replicating close()'s loop measures 5-8% of polls reading false with the 1s deadline honoured exactly. Count the threads inside the wait loop instead: the state no longer flickers, and the rising edge no longer depends on the closer having actually parked. awaitRepeatedInterrupts() also re-reads the interrupt count before failing, since the count it samples precedes the predicate read that fails on it. creationWaitStateDoesNotFlickerUnderInterruptStorm pins the contract: red on the old predicate (flickered after 46 polls, 2 interrupts landed), green now. Full suite green locally: 2766 tests, 0 failures. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
[PR Coverage check]😍 pass : 85 / 102 (83.33%) file detail
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while reviewing questdb/questdb#7434, which made the equivalent
constructors exception-safe in the core repo. That fix does not reach production:
AbstractLineHttpSenderbuilds its client through this repo'sHttpClientFactory, and the core one has no caller outside tests. So the ILPHTTP senders were still running the pre-fix construction sequence.
The branch carries two fixes. Both leak the same resources for the same reason -
a constructor that acquires and then throws hands no reference back, so nothing
can close what it took - but they sit at different levels of the call stack and
were found separately.
It also carries one change that has nothing to do with either: a CI deflake, in
its own commit, described at the bottom.
Fix 1: the HTTP client constructors
The defect
HttpClient's constructor takes a socket and two native buffers, then builds aResponseHeaderswhose own parser allocates again. None of it was guarded. Athrow past the first acquisition leaves a half-built client that the caller never
receives, so
close()never runs and everything taken so far leaks - the socketfile descriptor included.
HttpHeaderParserhad the same problem in the oppositedirection: its
DirectUtf8Sinkwas a field initialiser, which Java runs beforethe constructor body, so no
tryin the body could ever have covered it.Reachable through
Unsafe.mallocfailing under memory pressure, and through aSocketFactoryor TLS handshake failing.The fix
HttpClienttakes the socket, both buffers and the response parser inside onetryand frees what it took in reverse order. A close failure attaches to theprimary exception as suppressed rather than replacing it, so the caller still
sees why construction failed.
HttpClient.ResponseHeadersreleases the parser itssuper()call completed.HttpClienthas no reference to reach that parser through at the point the tworesponse objects are built, and Java does not run a superclass
close()when asubclass constructor fails.
HttpHeaderParsermoves the sink out of its field initialiser into the guardedregion, with a local the catch can reach.
HttpClientLinux,HttpClientOsxandHttpClientWindowsclose the completedbase client when their poller fails to open.
HttpClientWindowsalso reads the select facade before it takes the FD set.Nothing between that acquisition and the end of the constructor can then throw,
which removes a partial-
FDSetcleanup branch that only a Windows host couldever have reached -
FDSet's first statement touches the Windows-onlySelectAccessornatives, so a test on Linux or macOS cannot get past it to failanything later.
ResponseHeaders's rollback is the one block with no test: its two responseobjects allocate nothing native, so only a Java-heap
OutOfMemoryErrorreachesthe catch, and there is no seam that can inject one without adding test-only
production surface. The code comment says so rather than leaving the gap implicit.
Fix 2: the sender constructor, one level up
The defect
AbstractLineHttpSender's constructor assignedthis.client- either the onecreateLineSender()handed in or one it built throughHttpClientFactory- andthen called
newRequest(), with neither statement inside atry.newRequest()writes the POST line, the path, the
User-Agentheader and any auth header intothe client's request buffer, and
growBuffer()throws once they exceed themaximum. A throw there stranded a fully built client - socket, two native
buffers, response parser - that no caller ever receives a reference to.
createLineSender()cannot clean that up. Its threeMisc.free(cli)calls allsit before the
switch, and theswitchbuilds the sender inside areturnexpression that no
trycovers. Both entry shapes leak: an explicit protocolversion makes the constructor build the client itself, and an auto-detected one
hands in the client the version probe already used.
Confirmed by direct measurement before the fix: 65552 bytes under
NATIVE_DEFAULTplus the socket, on both shapes.How reachable it is depends on who builds the sender. Through
LineSenderBuilderthe
growBufferthrow is remote -maxBufferCapacityis floored at 64 KiB anddefaults to 100 MB, so the preamble would have to be absurd - which leaves a
Java-heap
OutOfMemoryErrorfrom the version lookup or theUser-Agentconcatenation as the realistic trigger there.
io.questdb.client.cutlass.line.httpis an exported package, though, and a third-party
HttpClientConfigurationsetsthe maximum to anything it likes, with no floor.
The fix
The constructor takes the client, the version and the request inside one
tryand closes the client on the way out, with the same suppressed-exception handling
as fix 1. It frees a handed-in client too:
close()already frees the clientregardless of where it came from, so the sender owns it from the assignment on
either way, and
createLineSender()has no way to free one for it.Tradeoffs
None on the happy path, for either fix - the guarded regions add no allocation and
no branch that was not already there. Moving
new DirectUtf8Sink(0)from a fieldinitialiser into the constructor body is runtime-identical. The two
field-assignment reorders are not observable for any configuration in this repo,
whose getters are plain field reads or
INSTANCEreturns.Fix 2 changes what a failed constructor does with a client the caller supplied:
it now closes it. A third party that passes its own client, catches the failure
and then closes that client itself will double-close - which is safe, since
HttpClient.close()guards onbufLo != 0,Misc.free(socket)is idempotent,and
HttpHeaderParser.close()guards onheaderPtr != 0. The alternative,leaving a handed-in client open, contradicts
close(), which frees itunconditionally on the success path.
Unrelated: the windows-msvc-2022-x64 deflake
Not part of either fix, and touching no file they touch. The branch's Azure run
failed on Windows in
QuestDBImplCloseLifecycleTest.facadeCloseIsBoundedUnderRepeatedInterruptsDuringSenderCreationwith
sender close left its creation wait before the interrupt storm landed twice; interrupts landed: 6. Every other leg was green, including both GitHubJDK jobs and the Linux and macOS Azure legs.
It is a false negative in the observation, not a defect in
close(). Both poolsexposed the close-time creation wait to tests as
lock.hasWaiters(creationFinished), which is not a valid "is it still waiting"probe under interrupts: when a thread parked in
awaitNanosis interrupted, AQStransferAfterCancelledWaitmoves its node off the condition queue and onto thelock's sync queue to reacquire before
awaitrethrows.hasWaiters()countscondition-queue nodes only, so it reads false mid-wait. A harness replicating
SenderPool.close()'s loop against the same interrupt storm measures 5-8% ofpolls reading false with the 1s deadline honoured exactly. The helper compounded
it by sampling the interrupt count before the probe it fails on - hence a
failure that reports 6 interrupts having landed while claiming fewer than 2 did.
SenderPoolandQueryClientPoolnow count the threads inside the wait loopunder the lock and report that. The state no longer flickers, and the rising edge
no longer depends on the closer having actually parked - which is what the
earlier
awaitCreationWaitercallers wanted anyway.awaitRepeatedInterrupts()re-reads the count before failing.Test plan
HttpClientConstructorTest: five tests, one per rollback point in fix 1,each with a close-counting
SocketFactoryand the enclosingassertMemoryLeakwatching the native blocks.
HttpHeaderParserTest.testConstructorFailureFreesNativeAllocationsfor thesink.
LineHttpSenderConstructorTest: two tests for fix 2, one per entry shape. Aconfiguration whose maximum request buffer is smaller than the preamble makes
newRequest()throw deterministically, with no server and no new productionseam. The self-built case asserts against the leak check; the handed-in case
adds a close-counting socket, since the leak check cannot see a file descriptor.
from the core ones but just as deterministically, and without a host dependency:
a negative buffer size makes
sun.misc.Unsafe.allocateMemorythrowIllegalArgumentExceptionon every supported JDK, and a throwing configurationgetter fails each platform subclass as a constructor argument - so
new Kqueue(...)andnew FDSet(...)are never invoked and neitherKqueueAccessornorSelectAccessoris ever initialised.all six fail: five on
the constructor must close the socket it took expected:<1> but was:<0>, the parser test onnative memory leaked or over-freed under tag NATIVE_DIRECT_UTF8_SINK expected:<0> but was:<32>. TheWindows one errors with
UnsatisfiedLinkError: SelectAccessor.arrayOffsetbefore the reorder, which is the untestability the reorder removes. With fix 2
reverted, both new sender tests fail: one on
native memory leaked or over-freed under tag NATIVE_DEFAULT expected:<0> but was:<65552>, the other onthe socket close count.
io.questdb.client.test.cutlass.**suite green: 1807 tests, 0 failures.QuestDBImplCloseLifecycleTest.creationWaitStateDoesNotFlickerUnderInterruptStormpins the contract the interrupt tests rely on, so nobody reverts the accessor to
a condition-queue probe. Verified red on the old predicate - it flickers after
46 polls with 2 interrupts landed, the same signature Windows reported - and
green on the new one. Plus 20/20 passes of all three interrupt tests under 40
busy-loop threads on a 24-core box.
this needs a locally cmake-built
libquestdb.so: the native binaries are nolonger committed, and a stale one left in
target/classesby an older checkoutfails ~340 tests with
UnsatisfiedLinkErroron symbols it predates.the real ILP path:
LineHttpSenderTestandServerMainHttpAuthTest, 83 testsrun, 1 skipped, 0 failures.
--release 8; nothing newer than Java 8 inproduction or test code.
🤖 Generated with Claude Code