Unix socket support

[dometto]

Resolves #3630

This PR adds support for Unix Domain Sockets...

1. as a protocol/backend in comms
2. as an optional way to serve the dashboard server

The rationale for this is twofold:

1. improved security
2. improved performance

Test this branch with:

1. the provided tests
2. or by creating a cluster with the new protocol:

    cluster = LocalCluster(
        n_workers=2,
        protocol="uds",
    )
   # do something with the cluster here

New UDS backend

This is the main part of this PR. We implement a new backend UDS which subclasses the standard TCP classes. When setting protocol on a cluster/worker/scheduler to uds://, this backend is used. When a worker or scheduler is given an absolute path, it will use that absolute path to create the unix socket. If instead any other value is passed in (e.g. the default localhost), the backend will create the socket at a new temporary path.

Benefits

Performance

@mrocklin tested this:

I just benchmarked this on
a macbook as a consideration for Frisky and got a 1.5x speedup (15 GB/s ->
25 GB/s) and 3x latency reduction (18us -> 6us) compared to TCP.  Maybe
still not worth doing there though just due to code complexity and I've yet
to find a workload that's bound by network on localhost there.  Still
pretty cool though.

Security

UDS support is desireable for usecases where:

• Multi-processing is preferable to multi-threading (so no inproc)
• ucx is not available (e.g. not on Linux)
• We care about other users (e.g. on a shared VM) not being able to access our data and processes (so no tcp).
  • within a single node, enabling TLS is not always feasible (in dask-labextension for instance), or desired (more overhead).

Dashboard listening on UDS

When the dashboard_url is set to an absolute path, listen on UDS instead of TCP. This can help prevent new CVE's of the form GHSA-c336-7962-wfj2

Todo

• Tests added / passed
• Passes pixi run lint
• Make unix socket directory configurable
• Fold tests into general test_comms.py?
• Use unix:// instead of uds://?

[dometto]

@jacobtomlinson could you let me know if you would be interested in merging this PR (after feedback/refactoring of course)? Thanks!

[jacobtomlinson]

This generally seems like a positive change. I agree that I like unix:// more. I would be apprehensive to make this the default without broader community testing, but I agree that it's a better default in the long run.

Could I ask you to make sure CI is happy (unrelated failures are fine) and add docs?

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

    20 files   -     20      20 suites   - 20   7h 22m 5s ⏱️ - 7h 13m 11s
 4 121 tests  -     38   3 958 ✅  -     22    157 💤  -    21   6 ❌ + 5 
41 216 runs   - 39 720  39 184 ✅  - 37 516  2 014 💤  - 2 221  18 ❌ +17 

For more details on these failures, see this check.

Results for commit 34d977e. ± Comparison against base commit 40fcd99.

This pull request removes 42 and adds 4 tests. Note that renamed tests count towards both.

distributed.cli.tests.test_dask_scheduler ‑ test_signal_handling[Signals.SIGINT]
distributed.cli.tests.test_dask_scheduler ‑ test_signal_handling[Signals.SIGTERM]
distributed.cli.tests.test_dask_spec ‑ test_signal_handling_scheduler[Signals.SIGINT]
distributed.cli.tests.test_dask_spec ‑ test_signal_handling_scheduler[Signals.SIGTERM]
distributed.cli.tests.test_dask_spec ‑ test_signal_handling_worker[Signals.SIGINT-Nanny]
distributed.cli.tests.test_dask_spec ‑ test_signal_handling_worker[Signals.SIGINT-Worker]
distributed.cli.tests.test_dask_spec ‑ test_signal_handling_worker[Signals.SIGTERM-Nanny]
distributed.cli.tests.test_dask_spec ‑ test_signal_handling_worker[Signals.SIGTERM-Worker]
distributed.cli.tests.test_dask_ssh
distributed.cli.tests.test_dask_worker ‑ test_signal_handling[Signals.SIGINT---nanny]
…

distributed.cli.tests.test_dask_scheduler ‑ test_dashboard_unix_socket
distributed.comm.tests.test_comms ‑ test_uds_specific[tornado]
distributed.comm.tests.test_uds ‑ test_parse_uds_address
distributed.comm.tests.test_uds ‑ test_registered

This pull request skips 4 tests.

distributed.deploy.tests.test_ssh ‑ test_defer_to_old
distributed.shuffle.tests.test_shuffle ‑ test_handle_null_partitions_2
distributed.tests.test_client ‑ test_computation_object_code_not_available
distributed.tests.test_scheduler ‑ test_decide_worker_rootish_while_last_worker_is_retiring

♻️ This comment has been updated with latest results.