Unix socket support

[dometto]

Resolves #3630

This PR adds support for Unix Domain Sockets...

1. as a protocol/backend in comms
2. as an optional way to serve the dashboard server

The rationale for this is twofold:

1. improved security
2. improved performance

Test this branch with:

1. the provided tests
2. or by creating a cluster with the new protocol:

    cluster = LocalCluster(
        n_workers=2,
        protocol="uds",
    )
   # do something with the cluster here

New UDS backend

This is the main part of this PR. We implement a new backend UDS which subclasses the standard TCP classes. When setting protocol on a cluster/worker/scheduler to uds://, this backend is used. When a worker or scheduler is given an absolute path, it will use that absolute path to create the unix socket. If instead any other value is passed in (e.g. the default localhost), the backend will create the socket at a new temporary path.

Benefits

Performance

@mrocklin tested this:

I just benchmarked this on
a macbook as a consideration for Frisky and got a 1.5x speedup (15 GB/s ->
25 GB/s) and 3x latency reduction (18us -> 6us) compared to TCP.  Maybe
still not worth doing there though just due to code complexity and I've yet
to find a workload that's bound by network on localhost there.  Still
pretty cool though.

Security

UDS support is desireable for usecases where:

• Multi-processing is preferable to multi-threading (so no inproc)
• ucx is not available (e.g. not on Linux)
• We care about other users (e.g. on a shared VM) not being able to access our data and processes (so no tcp).
  • within a single node, enabling TLS is not always feasible (in dask-labextension for instance), or desired (more overhead).

Dashboard listening on UDS

When the dashboard_url is set to an absolute path, listen on UDS instead of TCP. This can help prevent new CVE's of the form GHSA-c336-7962-wfj2

Todo

• Tests added / passed
• Passes pixi run lint
• Make unix socket directory configurable
• Fold tests into general test_comms.py?
• Use unix:// instead of uds://?

[dometto]

@jacobtomlinson could you let me know if you would be interested in merging this PR (after feedback/refactoring of course)? Thanks!

[jacobtomlinson]

This generally seems like a positive change. I agree that I like unix:// more. I would be apprehensive to make this the default without broader community testing, but I agree that it's a better default in the long run.

Could I ask you to make sure CI is happy (unrelated failures are fine) and add docs?

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

    40 files  ±  0      40 suites  ±0   14h 46m 35s ⏱️ + 14m 55s
 4 178 tests + 19   3 987 ✅ +  7    178 💤 ± 0   13 ❌ + 12 
81 317 runs  +378  76 933 ✅ +232  4 248 💤 +11  136 ❌ +135 

For more details on these failures, see this check.

Results for commit 34397b6. ± Comparison against base commit 40fcd99.

♻️ This comment has been updated with latest results.

[dometto]

Does this make sense or should we do a check on whether the server is bound to unix socket again in stop, and remove there?

[dometto]

I added some docs here and here.

It's pretty short, but the existing documentation on protocols is also pretty basic.

For the dashboard_address which now also supports UDS I couldn't find any docs apart from the function description for LocalCluster().

Let me know if there's anywhere else where I should add any more extensive documentation!

[dometto]

The remaining linting errors are pre-existing.