Conversation
Close packets have been best-effort: one the socket is not ready to accept gets parked on a when_writeable callback and then thrown away when the endpoint destroys its socket. Until now the only way to give them a chance to go out was to hold an Endpoint alive by hand, or to use Network, which reaches into the private _close_conns. close_conns(d, wait) closes as before and then blocks the caller, up to `wait`, until every close packet it produced has been handed to the socket. Expiry needs no handling: what has not flushed by then is dropped, which is what happened unconditionally before. Completion is tracked by the lifetime of a shared close_flush rather than by counting: a copy rides along with each close packet's send completion callback, and the waiter is released when the last one dies. That covers the paths where the callback is never invoked at all -- a dropped socket, or a cancelled job queue -- so an indefinite wait cannot hang, and it needs no guard count to avoid firing in the gap between one connection's close completing and the next being queued.
CMake option LIBQUIC_BUILD_PYTHON (off by default) builds a pybind11 extension, and a scikit-build-core pyproject drives the same CMake for `pip install .`. Nothing is bound yet beyond the version. The package is a thin compiled `seshquic._core` under a pure-Python `seshquic`, so that the convenience layer -- keyword arguments, context managers, futures -- can change without touching C++.
Address is exposed as-is; the flexible input forms a script wants (a "host:port" string, a (host, port) tuple) belong in the Python layer that will call this, not here. RemoteAddress is deliberately not bound: the remote pubkey reads better as a keyword argument to connect() than as a distinct address type. Credentials covers the three GNUTLSCreds factories plus the unencrypted credentials, keeping the latter's deliberately unmissable naming. Key material is taken as any bytes-like object, and str is rejected rather than guessed at: the caller picks the encoding, as they do for a socket. Also extends format.sh to cover python/, and adds the GIL and lifetime helpers everything else will be built on -- see common.hpp for why every call into libquic has to drop the GIL first.
An Endpoint owns its own event loop thread unless handed one to share, so a script holds one object rather than a Network and an Endpoint. Ownership is the part worth reading twice. Python holds the endpoint strongly and everything below it weakly: libquic's endpoint owns its connections, which own their streams, and a strong reference from Python lets one outlive the owner whose destructor it then reaches into. The first cut did hold them strongly, and a connection kept past endpoint.close() was a use-after-free at interpreter shutdown. Observing rather than owning keeps the C++ ownership as designed and turns a stale handle into a RuntimeError; see `observed` in common.hpp. The other recurring hazard is the GIL. libquic's accessors mostly block on the loop thread, so every call into it drops the GIL first, and every callback out of it takes the GIL -- including when merely *destroying* the held callable, which happens on the loop thread. A callback that raises is reported through sys.unraisablehook rather than escaping into ngtcp2's C callback stack, where it would terminate the process. During interpreter finalization the callback reference is leaked deliberately instead: CPython parks non-main threads that ask for the GIL, so trying to release it there hangs the process at exit. Keyword arguments drive libquic's variadic option interfaces through their std::optional overloads, which no-op when empty, so no template machinery is needed to make the options optional. The Pythonic half -- blocking connect, context managers, iterating a stream -- is added onto the bound classes from Python rather than wrapping them, so an object handed to a callback has it too.
pyproject.toml moves into python/, pointing at the libquic tree above it with scikit-build-core's cmake.source-dir, so `pip install ./python` builds the extension without a Python project file sitting in the root of a C++ one. Black formatting (settings borrowed from session-pysogs) runs from utils/format.sh alongside clang-format. From the earlier pyoxquic attempt: - py::keep_alive on connect() and open_stream(), which fixes a real footgun: a connection is only a weak handle on something the endpoint owns, so `Endpoint(...).connect(...)` had the endpoint collected, and the connection closed, as the expression finished. - Holding the Python callable through a shared_ptr with a GIL-acquiring deleter. libquic copies the std::function these end up inside, and that copy can happen off the loop thread without the GIL, which a bare py::object member would not survive; a shared_ptr copy is a refcount bump, and the GIL is needed only in the deleter. - value_option/flag_option to go with duration_option, which between them cover every keyword argument fed to libquic's variadic options. - enable_logging(), which makes libquic's own logging reachable from a script. The original called logger_config, which lives in the test suite rather than the library, so this goes through oxen-logging's public add_sink/reset_level instead, with flush_logs() alongside because the sinks buffer. Credentials.from_ed_keys now takes the 64-byte combined seed its docstring promises: libquic's make_from_ed_keys documents that it accepts one but hands it to gnutls whole, which rejects it, so the truncation happens here. A combined seed carries the pubkey, so a mismatch against the pubkey argument is now an error rather than a handshake that fails for no visible reason. Test keys come from PyNaCl rather than cryptography: two lines instead of six, and it is what the rest of the Session stack already uses.
The docstring has promised since it was written that the seed may be a combined seed+pk libsodium value, of which only the first 32 bytes are used. Nothing ever truncated it: make_from_ed_seckey does, but make_from_ed_keys handed the value straight to the constructor, which embeds it in a DER template whose lengths are fixed at 32 (04 20 for the seed's OCTET STRING, 03 21 00 for the pubkey's BIT STRING). A 64-byte seed therefore produced DER declaring a length it did not have, and every such call failed with "gnutls import of raw Ed keys failed". make_from_ed_keys now truncates, as documented. Since the combined value carries a pubkey of its own, one that disagrees with the pubkey argument is rejected rather than silently ignored -- no caller can be relying on the old behaviour there, because the old behaviour was to throw regardless. The constructor also checks both sizes up front, so any wrong-sized key reaching it says which one is wrong and how long it should be instead of surfacing as a generic gnutls import failure.
The binding truncated the combined seed itself and checked it against the pubkey, because make_from_ed_keys did neither. It does both now, so this just passes the keys through and lets the std::invalid_argument come back as a ValueError.
A bt-request stream is per-stream and symmetric: `open_bt_stream()` for one you open, `queue_incoming_bt_stream()` for one the other end will open, both available on either side of any connection. Handlers are registered per stream, so the two directions are independent. `request()` returns a concurrent.futures.Future rather than taking a callback the way the earlier pyoxquic attempt did. An error response becomes a RequestError and a timeout a RequestTimeout (which is also a builtin TimeoutError), so a script can write `fut.result(5)` and handle failures with except rather than by inspecting a message in a callback. The callback form is still there underneath as `_command`, and a Future is what asyncio.wrap_future takes if the async layer happens later. `command()` stays separate for the fire-and-forget case, which is how libquic distinguishes the two anyway (by whether a response callback was given). Which streams are bt-request streams is a protocol decision, not a client/server one -- the quic file server makes stream 0 a bt stream and leaves later streams as plain ones with their own framing -- so listen()/connect() now take on_stream_construct(connection, stream_id), returning BTRequestStream or None, with on_stream_open(stream) to configure what was built. Returning anything but BTRequestStream falls through to libquic's own default rather than building a bare Stream, because only libquic can wire that one up with the endpoint's stream callbacks. Two fixes fell out of testing this: - wrap() now recovers the subclass, since callbacks receive a Stream& and a bt-request stream was arriving at on_stream_open without any of the methods that make it one. - pybind11/stl.h moves into common.hpp. btstream.cpp used std::optional without it and instantiated a different caster for the same type; the resulting ODR violation broke argument conversion elsewhere, including in files that did include it -- Endpoint became unconstructible.
Datagrams are off unless asked for, and both ends have to ask, so `datagrams=True` on the Endpoint enables them, `datagram_splitting=True` lets one span two QUIC packets (roughly doubling what fits, with datagram_bufsize sizing the reassembly buffer), and datagram_queue_limit caps what a connection will hold before dropping. Passing any of those without datagrams=True is an error rather than silently doing nothing. Sending is `connection.send_datagram(data)` rather than going through the Datagrams channel object: the channel exists in C++ to share the IOChannel send machinery with streams, and has nothing else on it worth exposing. `max_datagram_size` is documented as worth re-reading before each send, since it grows as the path MTU is discovered. The receive callback is given the connection and the data as bytes. The data has to be copied there in any case: libquic warns that a datagram's buffer is not guaranteed to outlive the callback. opt::enable_datagrams reports a bad bufsize as std::out_of_range, which pybind11 renders as IndexError; a bad argument is a ValueError in Python, so that one is translated.
seshquic.aio wraps the threaded API in pure Python. It is a separate import: `import seshquic` does not pull it in, no event loop is needed without it, and nothing in the threaded layer changed to accommodate it. That request() already returns a concurrent.futures.Future is what makes this cheap -- asyncio.wrap_future takes one directly. Only what genuinely waits is a coroutine: connect, close, reading a stream, and a bt-request. Sending queues and returns, so it stays synchronous, which is also what asyncio does with StreamWriter.write against an awaited drain. Callbacks are the part with a real decision in it. Most are notifications and are handed to the asyncio loop, so they may be coroutine functions. Three cannot be: on_stream_construct has to return a type before libquic can build the stream, and on_connection and on_stream_open are where streams get queued and handlers registered -- data can arrive as soon as they return, so deferring them onto the asyncio loop loses whatever arrives in between. The first draft did defer them and the bt tests failed exactly that way, answering "endpoint not found" for a handler that had not been registered yet. Those three now run on libquic's thread and reject coroutine functions with an explanation. Because those setup callbacks run off the asyncio loop, the wrapper objects carry their loop rather than calling get_running_loop() when needed, so registering a handler from one still yields a handler that runs on the asyncio loop. Anything not wrapped is reachable via `.raw`.
pip install . was the only instruction, which on a distro that marks its Python externally managed fails outright, and which quietly installs a copy that goes stale the moment the C++ is rebuilt.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds libquic Python bindings via pybind11.