Skip to content

Fix stalled TLS handshake when the server's first flight is too large - #1070

Open
lhellebr wants to merge 1 commit into
apache:mainfrom
lhellebr:bz-flight-stall
Open

lhellebr wants to merge 1 commit into
apache:mainfrom
lhellebr:bz-flight-stall

Conversation

@lhellebr

Copy link
Copy Markdown

Fixes bug 70236.

OpenSSL writes a complete handshake flight to the network BIO in a single operation. The BIO pair used by the OpenSSL based SSLEngine implementations is created with the default 17408 byte buffer so, when the flight is larger than that - which in practice requires a large certificate chain - OpenSSL retains the remainder internally. wrap() copies whatever the BIO holds into the destination buffer and returns NEED_UNWRAP without giving OpenSSL any opportunity to write the rest.

The remainder is currently only written as a side effect of the priming SSL_read() performed by a later call to unwrap(). That hides the problem whenever the client has already sent something - a TLSv1.3 client normally sends a middlebox compatibility change cipher spec record - but a client that is waiting for the server to complete its flight sends nothing, so unwrap() is never reached and the connection stalls until it times out. With a TLSv1.2 client the failure is deterministic.

After the network BIO has been drained by wrap(), and while a handshake is still in progress, drive OpenSSL again so it can write out whatever did not previously fit. When there is nothing left to write the call is a no-op. A dedicated method is used rather than handshake() because handshake() re-snapshots the handshake counter that is used to detect completion, which must not happen in the middle of a flight. Neither implementation binds SSL_get_error() so it is not currently possible to test for SSL_ERROR_WANT_WRITE and make the call conditional.

OpenSSL writes a complete handshake flight to the network BIO in a single
operation. The BIO pair used by the OpenSSL based SSLEngine implementations
is created with the default 17408 byte buffer so, when the flight is larger
than that - which in practice requires a large certificate chain - OpenSSL
retains the remainder internally. wrap() copies whatever the BIO holds into
the destination buffer and returns NEED_UNWRAP without giving OpenSSL any
opportunity to write the rest.

The remainder is currently only written as a side effect of the priming
SSL_read() performed by a later call to unwrap(). That hides the problem
whenever the client has already sent something - a TLSv1.3 client normally
sends a middlebox compatibility change cipher spec record - but a client
that is waiting for the server to complete its flight sends nothing, so
unwrap() is never reached and the connection stalls until it times out.
With a TLSv1.2 client the failure is deterministic.

After the network BIO has been drained by wrap(), and while a handshake is
still in progress, drive OpenSSL again so it can write out whatever did not
previously fit. When there is nothing left to write the call is a no-op.
A dedicated method is used rather than handshake() because handshake()
re-snapshots the handshake counter that is used to detect completion, which
must not happen in the middle of a flight. Neither implementation binds
SSL_get_error() so it is not currently possible to test for
SSL_ERROR_WANT_WRITE and make the call conditional.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant