Distinguish metastore retry attempts from final errors

[Mallets]

Summary

Tower gRPC request metrics are emitted when the request future is dropped. At that point, a failed attempt is recorded as status="error", even if the retry policy will retry it. Consequently, transient failures appear as hard errors in metric queries.

This PR introduces GrpcRetryPolicy, a gRPC-aware wrapper around the generic retry policy. It observes retry decisions and identifies failed attempts that will be retried. These attempts are recorded with status="retry", while preserving the original gRPC status code in the code label. Final failures continue to use status="error".

The metastore client is updated to use this policy and to place the gRPC metrics layer outside the retry loop. As a result:

• retryable failures that lead to another attempt are recorded as status="retry"
• status="error" is reserved for the final client-visible failure, including exhausted retries
• status="success" represents the final successful outcome, including success after retries
• in-flight and duration metrics describe the complete logical RPC rather than each individual attempt

Scope

Metastore is the only service behaviorally impacted by this change because it is the only service that combines GrpcMetricsLayer with Quickwit's RetryPolicy. Control plane, indexing, ingest, and cluster also use GrpcMetricsLayer, but they do not use it with this retry policy, so their metric semantics remain unchanged.

The shared gRPC metastore metrics and retry status semantics are also documented.

Contributions

Integrates changes proposed in https://github.com/quickwit-oss/quickwit/compare/guilload/retry-metrics

Validation

• cargo +nightly fmt --all
• cargo test -p quickwit-common test_grpc_metrics
• cargo clippy -p quickwit-common --all-features --tests
• cargo clippy -p quickwit-serve --all-features --tests