`bitreq`: Support multiple connections per origin

[tnull]

Client currently caches one AsyncConnection per ConnectionKey, so capacity limits cached origins rather than connections to one origin.

Consequently, concurrent requests to the same HTTP/1.1 server share one socket. Pipelining does not provide server-side concurrency with common servers such as Go's net/http, resulting in head-of-line blocking. This is causing serialized remote-storage reads in: lightningdevkit/orange-sdk#90

We should add support for a configurable max_connections_per_origin, with connections selected by availability/load. The existing cache capacity could remain a separate limit on cached origins.

This limitation and possible support for two or three connections was previously noted here: lightningdevkit/vss-client#56 (comment)

[TheBlueMatt]

Pipelining does not provide server-side concurrency with common servers such as Go's net/http, resulting in head-of-line blocking. This is causing serialized remote-storage reads in: lightningdevkit/orange-sdk#90

That shouldn't matter here, though. For VSS, the server-side processing of a single request should be a few milliseconds, so the pipelining should still help a lot.

[tnull]

That shouldn't matter here, though. For VSS, the server-side processing of a single request should be a few milliseconds, so the pipelining should still help a lot.

But out-of-scope for this issue here, but in VSS we can pipeline reads can't pipeline writes/removes due to idempotency concerns. So IMO using multiple connections would help a lot here to avoid head of line blocking, especially in cases where writes and reads are interleaved, as any write would be a serialization point/bottleneck.

[TheBlueMatt]

Yea I feel like we should treat non-pipelined requests when there's no available no-requests-pending connections as needing a fresh connection always.