[Feature]: Balance and fail over in-run server access across server replicas

[peterschmidt85]

Problem

PR #4023 exposes the dstack server inside a run through /run/dstack/server.sock.

With multiple dstack server replicas, the current implementation keeps one stable owner for this socket. Other replicas probe and adopt the existing socket instead of replacing it.

This avoids ownership churn, but leaves two limitations:

1. Load can be distributed unevenly. The first replica that establishes the socket for a job handles all server requests from that job. Depending on job processing and server startup timing, one replica may own the sockets for most or all jobs.
2. Failover is delayed. If the owning replica stops or its tunnel fails, requests from the job fail until another replica detects the unhealthy socket and replaces it. Detection is currently throttled by the 30-second connection health-check interval and can be delayed further by job processing and locking.

Solution

Allow every server replica to expose its own Unix socket in the job, and place a load-balancing/failover layer in front of those sockets at /run/dstack/server.sock.

The gateway implementation provides a similar model: each server replica creates a uniquely named socket, while HTTPMultiClient selects among available sockets and retries alternatives. The runner could provide equivalent routing inside the job.

The implementation should:

• distribute requests across healthy server replicas;
• retry another replica when the selected replica is unavailable;
• add and remove replica sockets without interrupting the stable /run/dstack/server.sock endpoint;
• clean up stale replica sockets;
• preserve support for multiple server processes sharing PostgreSQL.