HDDS-16360. Add liveness and readiness health endpoints to S3 Gateway

[yandrey321]

What changes were proposed in this pull request?

Adds two unauthenticated HTTP health endpoints to the S3 Gateway, served on the
web admin server (default port 19878), separate from the S3 data listener
(9878):

• GET /health/live → 200 OK while the gateway process is up and its web
  admin server can serve requests. It does not check OM reachability.
• GET /health/ready → 200 READY only when the gateway can reach OM, and
  503 NOT READY during startup or while OM is unreachable.

Readiness is backed by S3GatewayReadinessProbe, a single background thread
that periodically probes OM (getServiceInfo) and stores the result in a
volatile flag. The servlet only reads that flag, so /health/ready always
responds immediately — no OM RPC on the request path — and a load balancer
polling it can never be blocked by a slow or unreachable OM. The probe runs its
OM call on a separate single-thread executor with an enforced deadline, so a
hung call flips the gateway to "not ready" rather than wedging the scheduler. It
owns a dedicated OzoneClient (created via OzoneClientCache.createClient) so
it exercises the same OM transport the gateway serves with, with S3 auth
disabled and a bounded OM RPC timeout.

The endpoints are registered with addInternalServlet, so no authentication
filter is mapped to them and they remain reachable by a load balancer without
Kerberos/SPNEGO, including in secure mode. Because they live on the admin port,
they cannot collide with S3 bucket names or require SigV4 signing.

Wiring updated to use the new endpoints:

• Kubernetes S3G statefulsets: livenessProbe now points at /health/live
  (was /) and a readinessProbe on /health/ready is added.
• HAProxy examples (compose/common and compose/ozonesecure-ha) health-check
  the readiness endpoint (GET /health/ready, expect 200) with
  inter 2s rise 1 so backends are routed to promptly once ready.
• compose/testlib.sh gains wait_for_s3g_ready(), invoked from
  start_docker_env, so acceptance tests don't start issuing S3 requests while
  HAProxy still has all backends DOWN (which would return 503). It only
  waits when the multi-instance HAProxy setup (s3g1/s3g2/s3g3) is present.

Configuration

New ozone.s3g.health-check.* config group (S3GatewayHealthCheckConfig):

Key	Default	Description
ozone.s3g.health-check.enabled	true	Expose the /health/live and /health/ready endpoints.
ozone.s3g.health-check.probe.interval	10s	How often the readiness endpoint refreshes its cached OM reachability.
ozone.s3g.health-check.probe.timeout	10s	Upper bound on a single background readiness probe.

Why are the changes needed?

When S3 Gateway is horizontally scaled behind a load balancer, the balancer
needs an HTTP health check to route S3 traffic only to gateways that can
actually serve it. The existing admin server had no dedicated liveness/readiness
endpoints; the k8s liveness probe hit / and there was no readiness signal at
all, so traffic could be sent to a gateway that is up but cannot reach OM.

Generated-by: Claude Code (Claude Opus 4.8)

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16360

How was this patch tested?

• New unit tests:
  • TestS3GatewayLivenessServlet — liveness returns 200 OK.
  • TestS3GatewayHealthCheck — readiness reflects the probe flag (200 READY
    vs 503 NOT READY) and endpoints are gated by health-check.enabled.
• New acceptance test: smoketest/s3/health.robot exercises both endpoints;
  compose/ozone/test.sh runs it.
• Verified the endpoints are served without authentication in secure mode
  (registered via addInternalServlet).

[yandrey321]

@jojochuang could you please take a look?

[Copilot]

🟡 Changes recommended

One or more issues must be addressed before approval.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds unauthenticated S3 Gateway liveness/readiness endpoints backed by asynchronous OM reachability checks, with deployment, testing, HAProxy, and documentation updates.

Changes:

• Adds configurable health servlets and readiness probing.
• Updates Kubernetes, HAProxy, Compose, and acceptance-test wiring.
• Adds unit/integration tests and S3 health documentation.

File summaries

File	Description
hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/TestS3GatewayLivenessServlet.java	Updated as part of this pull request.
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/S3GatewayWebAdminServer.java	Updated as part of this pull request.
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/S3GatewayReadinessProbe.java	Updated as part of this pull request.
hadoop-ozone/s3gateway/pom.xml	Updated as part of this pull request.
hadoop-ozone/integration-test-s3/src/test/java/org/apache/hadoop/ozone/s3/TestS3GatewayHealthCheck.java	Updated as part of this pull request.
hadoop-ozone/dist/src/main/smoketest/s3/health.robot	Updated as part of this pull request.
hadoop-ozone/dist/src/main/k8s/examples/ozone/s3g-statefulset.yaml	Updated as part of this pull request.
hadoop-ozone/dist/src/main/k8s/examples/ozone-ha/s3g-statefulset.yaml	Updated as part of this pull request.
hadoop-ozone/dist/src/main/k8s/examples/ozone-dev/s3g-statefulset.yaml	Updated as part of this pull request.
hadoop-ozone/dist/src/main/k8s/examples/minikube/s3g-statefulset.yaml	Updated as part of this pull request.
hadoop-ozone/dist/src/main/k8s/examples/getting-started/s3g-statefulset.yaml	Updated as part of this pull request.
hadoop-ozone/dist/src/main/k8s/definitions/ozone/s3g-ss.yaml	Updated as part of this pull request.
hadoop-ozone/dist/src/main/compose/testlib.sh	Updated as part of this pull request.
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/s3-haproxy.cfg	Updated as part of this pull request.
hadoop-ozone/dist/src/main/compose/ozone/test.sh	Updated as part of this pull request.
hadoop-ozone/dist/src/main/compose/common/s3-haproxy.cfg	Updated as part of this pull request.
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/conf/S3GatewayHealthCheckConfig.java	Updated as part of this pull request.
hadoop-hdds/docs/content/interface/S3.md	Updated as part of this pull request.

Review details

Suppressed comments (2)

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/S3GatewayReadinessProbe.java:105

• When OM is unavailable, OzoneClientCache.createClient can throw during RpcClient construction (the client performs getServiceInfo before construction returns), so execution never reaches client = c. Each interval then retries construction with a fresh transport/channel, but the failed partial client is not closed on this path; a prolonged OM outage can therefore leak RPC resources. Reuse a reconnectable client or add cleanup/backoff for failed construction.

        c = OzoneClientCache.createClient(probeClientConf());
        client = c;
      }

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/S3GatewayWebAdminServer.java:98

• BaseHttpServer.start() is a no-op when the web-admin server is disabled, but this branch still starts the readiness scheduler and its OM client. A deployment that intentionally disables the web-admin listener will therefore create background probe threads and OM traffic even though neither health endpoint can be served. Gate probe startup on the effective web-admin server state.

    if (readinessProbe != null) {
      readinessProbe.start();
    }

• Files reviewed: 7/18 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[jojochuang]

Overall this looks good to me—the admin-port endpoints, cached readiness probe, and compose/k8s wiring are the right shape for HDDS-16360. A few notes before merge:

Readiness semantics
/health/ready reflects OM reachability via getServiceInfo() on a dedicated probe client. It does not verify that SCM is reachable or that the cluster has left safe mode. That is reasonable for “can this S3 Gateway talk to OM for metadata?”, but operators should treat it as an S3G↔OM signal, not full cluster health. Worth calling out explicitly in S3.md (one sentence) so LB/K8s users do not assume readiness implies SCM or safe-mode status.

Compose: wait_for_s3g_ready
Nice fix for HAProxy gating. It only waits on s3g1 and only when the s3g1 service exists, which is enough for the bundled multi-S3G compose layouts; just noting that it does not wait for s3g2/s3g3 individually—fine for smoke tests if all gateways share the same OM readiness behavior.

Tests
TestS3GatewayLivenessServlet also covers readiness servlet behavior and the health-check config toggle. Consider renaming to something like TestS3GatewayHealthServlet (or split classes) so the name matches scope.

Release note
Please add a short release-note entry, e.g.:

S3 Gateway exposes unauthenticated /health/live and /health/ready on the web admin port (default 19878). Readiness indicates OM reachability only. Configurable via ozone.s3g.health-check.* (default enabled). Kubernetes examples and HAProxy compose configs use these endpoints for probes.

Security / ops (FYI)
Unauthenticated health on the admin port is intentional (addInternalServlet); ensure deployments keep 19878 on an operator/LB network, not the public S3 client path.

Happy to approve once the readiness scope is documented (and release note added if your process requires it).

[yandrey321]

Fixed review comments