fix(tls): preserve BUFFER_OVERFLOW when array-destination unwrap drains nothing

[detail-app]

Detail bug report: View on Detail

Change

NativeSslEngine.unwrap(ByteBuffer, ByteBuffer[], int, int) reported BUFFER_UNDERFLOW (0 consumed / 0 produced) while a complete, decryptable record was pending — a contract violation that stalls any caller using the standard array-destination SSLEngine.unwrap API with a first destination smaller than the pending record's plaintext.

Root cause (Java layer, in the array-destination unwrap drain loop): when the first destination is too small, the native engine returns BUFFER_OVERFLOW from a pre-check (record too large for the destination, nothing decrypted/queued). The Java loop then issued a zero-length-source "drain" call to retrieve queued plaintext — but in the pre-check case nothing was queued, so that drain correctly returned BUFFER_UNDERFLOW, and the loop overwrote the genuine BUFFER_OVERFLOW with it. A compliant caller then waits for more inbound bytes that can never help, retrying indefinitely.

The native pre-check path (in crates/bemo/src/tls/engine.rs) is valid engine behavior; the contract violation is in the Java loop masking it.

Fix (packages/netty/.../NativeSslEngine.java): the drain loop now distinguishes the two overflow causes:

• If the zero-length drain produced data (produced(drain) > 0), queued plaintext from a prior decryption is adopted — the existing, correct path is unchanged.
• Otherwise the overflow was the pre-check (nothing queued); the loop retries the same record with the full source against the next destination, delivering the plaintext into the first destination that has room.

When every destination is too small, the final status stays BUFFER_OVERFLOW (never BUFFER_UNDERFLOW). This also makes progress into later destinations instead of merely re-reporting the overflow.

Affected layers: JVM (NativeSslEngine, the public SSLEngine surface shipped to consumers). No ABI change, no native behavior change — the native pre-check overflow and empty-source underflow are part of the existing engine contract.

Closes #57

Validation

Bug introduced in d99f693; the drain loop has been byte-identical since, with no test coverage on the array-destination unwrap (the repo and Netty's SslHandler only exercise the 2-arg variant). Added regression coverage at both layers, then ran the full gate.

New regression tests (versioned):

• Rust tls::engine::tests::engine_pre_check_overflow_queues_no_plaintext_and_empty_drain_underflows (crates/bemo/src/tls/engine.rs) — pins the native contract the Java loop depends on: a too-small destination returns BufferOverflow (consumed 0, produced 0, destination untouched); an empty-source drain returns BufferUnderflow 0/0; the still-pending record then decrypts into a large destination with Ok (consumed = record length, produced = plaintext length).
• Java NativeSslEngineTest.arrayUnwrapOverflow (tests/transport/java/NativeSslEngineTest.java) — drives a real TLS 1.3 handshake (no sockets) and the unwrap(ByteBuffer, ByteBuffer[], int, int) public API: a single 100-byte destination correctly reports BUFFER_OVERFLOW 0/0 with the record left on the wire (matching the 2-arg unwrap); a {100-byte, 64 KB} destination pair returns OK, consuming the full record and delivering 10,000 plaintext bytes into the large destination, leaving the small one untouched.

Commands run (Linux x86_64):

• cargo nextest run -p bemo --lib --tests --locked — all tls::engine::tests::* pass, including the new test.
• cargo nextest run -p bemo --locked --test tls --test tls_abi --test engine — 23/23 pass.
• cargo clippy --workspace --all-targets --locked -- -D warnings and cargo fmt --package bemo --check — clean.
• make build-jvm — production classes compile (-Xlint:all -Werror --release 22).
• make check Java path (tools/check_java.py) — Error Prone + NullAway (JSpecify mode) + Checker Framework Regex/Formatter pass.
• google-java-format --dry-run --set-exit-if-changed on both Java files — clean.
• make test-jvm — NativeSslEngineTest (incl. the new contract), NativeSslPolicyTest, and JSSE-interop cases all pass. NativeSslInteropTest's OpenSSL CLI cases fail in this sandbox because the JDK cannot posix_spawn subprocesses (EACCES, error 13); re-running that contract with -Djdk.lang.Process.launchMechanism=FORK (the JDK's legacy fork mechanism) makes all its cases pass, including both OpenSSL s_client/s_server interop — i.e. the failure is environmental, not the fix. /usr/bin/openssl itself runs fine from the shell.
• make test-native-image — passes after the same JDK posix_spawn workaround is applied to native-image (it spawns gcc/cc for toolchain checks and linking). CapiTlsChannelTest (which calls NativeSslEngineTest.verify + NativeSslPolicyTest.verify + NativeSslInteropTest.verify, including the new arrayUnwrapOverflow test) builds ahead-of-time and passes, along with the OpenSSL interop cases.
• make test-asan and make test-tsan — all TLS tests pass under both sanitizers (the retry path that re-enters the native engine with the full source shows no ASAN or TSAN finding).
• make test-miri — passes (the new Rust test is correctly #[cfg_attr(miri, ignore)] since it drives a rustls handshake, matching the existing engine-test convention).
• make bench-smoke and make bench-tls-records — TLS correctness and record-encryption throughput pass; the bemo provider is on par with the openssl/tcnative reference, so no throughput regression.

Two unrelated tests fail in the workspace-wide, ASAN, and TSAN runs (persistent_keepalive_reuses_receive_registration_and_fixed_send_state, arm_receive_failure_leaves_socket_dormant); both are pre-existing on the clean baseline (verified by re-running on the unmodified tree) and live in driver/http io_uring code, untouched by this change.

---

Automatic Fixes PRs can be configured here.

[codecov]

Codecov Report

❌ Patch coverage is 88.88889% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.91%. Comparing base (26fb123) to head (c99e998).
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
.../dev/elide/bemo/transport/tls/NativeSslEngine.java	60.00%	3 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main      #64      +/-   ##
============================================
+ Coverage     86.82%   86.91%   +0.09%     
- Complexity      616      622       +6     
============================================
  Files            60       60              
  Lines         16979    17013      +34     
  Branches        490      491       +1     
============================================
+ Hits          14742    14787      +45     
+ Misses         1913     1898      -15     
- Partials        324      328       +4     

Flag	Coverage Δ
jvm	57.96% <60.00%> (+0.45%)	⬆️
rust	93.04% <100.00%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

[codspeed]

Merging this PR will degrade performance by 10.46%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 4 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

• response[0]
• retained-response[0]
• retained-response[65536]
• allocate-drop[1048576]

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
❌ 4 regressed benchmarks
✅ 112 untouched benchmarks
⏩ 65 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	retain-drop[1048576]	300.8 ns	354.9 ns	-15.26%
❌	Simulation	retain-drop[4096]	300.8 ns	354.9 ns	-15.26%
❌	Simulation	retain-drop[64]	300.8 ns	354.9 ns	-15.26%
❌	Simulation	retain-drop[65536]	300.8 ns	354.9 ns	-15.26%
⚡	Simulation	overwrite[64]	520.2 ns	466.1 ns	+11.62%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing detail/bug-fix/fix-tls-preserve-buffer-overflow-when-array-destin-32d64e (c99e998) with main (26fb123)}

Footnotes

1. 65 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩