feat(client): retry transient failures, optionally indefinitely

changelog/+transient-retry.added.md

@@ -0,0 +1 @@
+With `retry_on_failure` enabled, the client now retries every transient failure, not only connection errors: dropped or timed-out connections, HTTP `500`/`502`/`503`/`504` responses, and GraphQL errors the server flags with one of those statuses, on every request path. Retries use exponential backoff with jitter (`retry_delay` up to the new `retry_max_delay`), `retry_status_codes` tunes what counts as transient, and `max_retry_duration=0` retries indefinitely.

docs/docs/python-sdk/guides/client.mdx

@@ -432,6 +432,52 @@ export INFRAHUB_PROXY_MOUNTS_HTTPS=http://https-proxy.example.com:8080
 The `proxy` and `proxy_mounts` configurations are mutually exclusive and cannot be used together. Specifying both will cause a `ValueError` to be raised when the client is initialized.
 :::
 
+### Retry transient failures
+
+Long-running scripts and generators can fail halfway through when Infrahub is temporarily unavailable: a database restart, an overloaded backend or a short network outage. With `retry_on_failure` enabled, the client retries a request when it fails with a transient error and raises immediately for every other error. A failure is transient when it is one of:
+
+- A connection error, or a connection dropped before the response arrived, such as when a load balancer restarts.
+- A timeout, while connecting, sending the request or reading the response.
+- An HTTP response with a status listed in `retry_status_codes` (`500`, `502`, `503` and `504` by default).
+- A GraphQL response in which every error is flagged by the server with one of those statuses.
+
+Retries use exponential backoff with jitter, starting at `retry_delay` seconds and capped at `retry_max_delay`. They stop after `max_retry_duration` seconds (5 minutes by default). Set `max_retry_duration` to `0` to keep retrying until the request succeeds, which lets a generator survive an outage instead of aborting.
+
+<Tabs groupId="env-config">
+  <TabItem value="Code" default>
+
+  ```python
+  from infrahub_sdk import Config, InfrahubClient
+  config = Config(retry_on_failure=True, max_retry_duration=0)
+  client = InfrahubClient(config=config)
+  ```
+
+  </TabItem>
+  <TabItem value="Environment">
+
+  ```bash
+  export INFRAHUB_RETRY_ON_FAILURE=true
+  export INFRAHUB_MAX_RETRY_DURATION=0
+  ```
+
+  ```python
+  from infrahub_sdk import InfrahubClient
+  client = InfrahubClient()  # retry settings are read from the environment variables
+  ```
+
+  </TabItem>
+</Tabs>
+
+Every retry is logged at `WARNING` level with the attempt number and the time spent so far. After 5 minutes of retrying the log level switches to `ERROR`, so an operation that keeps retrying stays visible.
+
+:::warning Retrying mutations
+A mutation that timed out may have been applied by the server before the retry. Saving with `allow_upsert=True` is safe to retry. A plain create that had already succeeded fails on retry with a non-transient error, which is raised.
+:::
+
+:::note Unclassified server errors
+Infrahub reports some transient database failures as HTTP `500` without further classification, so `500` is part of the default `retry_status_codes`. The same status also covers genuine bugs, which are therefore retried until `max_retry_duration` expires. Remove `500` from `retry_status_codes` to fail fast on them instead.
+:::
+
 ## Next steps
 
 Now that you have a fully configured Infrahub client, you're ready to start working with your infrastructure data. Here's what you can explore next:

docs/docs/python-sdk/reference/config.mdx

@@ -144,21 +144,39 @@ The following settings can be defined in the `Config` class
 ## retry_delay
 
 <!-- vale on -->
-**Description**: Number of seconds to wait until attempting a retry.<br />
+**Description**: Base delay in seconds before retrying a request that failed with a transient error. The delay doubles after every attempt, with jitter, up to the maximum retry delay.<br />
 **Type**: `integer`<br />
 **Default value**: 5<br />
 **Environment variable**: `INFRAHUB_RETRY_DELAY`<br />
 <!-- vale off -->
 
+## retry_max_delay
+
+<!-- vale on -->
+**Description**: Maximum delay in seconds between two retries of a request that failed with a transient error.<br />
+**Type**: `integer`<br />
+**Default value**: 60<br />
+**Environment variable**: `INFRAHUB_RETRY_MAX_DELAY`<br />
+<!-- vale off -->
+
 ## retry_on_failure
 
 <!-- vale on -->
-**Description**: Retry operation in case of failure<br />
+**Description**: Retry requests that fail with a transient error: connection error, timeout, an HTTP status listed in the retry status codes, or a GraphQL error the server flags with one of those statuses. Other errors are never retried. The maximum retry duration controls how long to keep retrying.<br />
 **Type**: `boolean`<br />
 **Default value**: False<br />
 **Environment variable**: `INFRAHUB_RETRY_ON_FAILURE`<br />
 <!-- vale off -->
 
+## retry_status_codes
+
+<!-- vale on -->
+**Description**: HTTP status codes treated as transient when retrying on failure is enabled. Also matched against the HTTP status reported in GraphQL error extensions. 500 is included because Infrahub reports some transient database errors without further classification; remove it to fail fast on them.<br />
+**Type**: `array`<br />
+**Default value**: [500, 502, 503, 504]<br />
+**Environment variable**: `INFRAHUB_RETRY_STATUS_CODES`<br />
+<!-- vale off -->
+
 ## rate_limit_retry_enabled
 
 <!-- vale on -->
@@ -198,7 +216,7 @@ The following settings can be defined in the `Config` class
 ## max_retry_duration
 
 <!-- vale on -->
-**Description**: Maximum duration until we stop attempting to retry if enabled.<br />
+**Description**: Maximum number of seconds to keep retrying a request that fails with transient errors when retrying on failure is enabled. Set to 0 to retry indefinitely.<br />
 **Type**: `integer`<br />
 **Default value**: 300<br />
 **Environment variable**: `INFRAHUB_MAX_RETRY_DURATION`<br />

docs/docs/python-sdk/sdk_ref/infrahub_sdk/client.mdx

@@ -344,7 +344,9 @@ execute_graphql(self, query: str, variables: dict | None = None, branch_name: st
 
 Execute a GraphQL query (or mutation).
 
-If retry_on_failure is True, the query will retry until the server becomes reachable.
+If retry_on_failure is True, transient failures (connection errors, timeouts, transient HTTP statuses and
+GraphQL errors the server flags as transient) are retried until max_retry_duration is exhausted, or
+indefinitely when max_retry_duration is 0.
 
 **Args:**
 
@@ -364,11 +366,12 @@ client-wide default for this request only. When None, the client default (if any
 
 **Raises:**
 
-- `GraphQLError`: When the GraphQL response contains errors.
-- `ServerNotReachableError`: If the server is not reachable after exhausting retries.
+- `GraphQLError`: When the GraphQL response contains errors that are not transient, or transient ones
+once the retry budget is exhausted.
+- `ServerNotReachableError`: If the server is not reachable, after exhausting retries when enabled.
+- `ServerNotResponsiveError`: If the server does not answer before the timeout, after exhausting retries.
 - `AuthenticationError`: If the server returns a 401 or 403 response.
 - `URLNotFoundError`: If the server returns a 404 response.
-- `Error`: If the response is unexpectedly missing.
 
 #### `refresh_login`
 
@@ -661,7 +664,9 @@ execute_graphql(self, query: str, variables: dict | None = None, branch_name: st
 
 Execute a GraphQL query (or mutation).
 
-If retry_on_failure is True, the query will retry until the server becomes reachable.
+If retry_on_failure is True, transient failures (connection errors, timeouts, transient HTTP statuses and
+GraphQL errors the server flags as transient) are retried until max_retry_duration is exhausted, or
+indefinitely when max_retry_duration is 0.
 
 **Args:**
 
@@ -681,11 +686,12 @@ client-wide default for this request only. When None, the client default (if any
 
 **Raises:**
 
-- `GraphQLError`: When the GraphQL response contains errors.
-- `ServerNotReachableError`: If the server is not reachable after exhausting retries.
+- `GraphQLError`: When the GraphQL response contains errors that are not transient, or transient ones
+once the retry budget is exhausted.
+- `ServerNotReachableError`: If the server is not reachable, after exhausting retries when enabled.
+- `ServerNotResponsiveError`: If the server does not answer before the timeout, after exhausting retries.
 - `AuthenticationError`: If the server returns a 401 or 403 response.
 - `URLNotFoundError`: If the server returns a 404 response.
-- `Error`: If the response is unexpectedly missing.
 
 #### `count`
 
@@ -1076,6 +1082,34 @@ Base class for InfrahubClient and InfrahubClientSync.
 
 **Methods:**
 
+#### `retry_on_failure`
+
+```python
+retry_on_failure(self) -> bool
+```
+
+Whether transient failures are retried. Can be toggled at runtime, e.g. by a long-running generator.
+
+#### `retry_on_failure`
+
+```python
+retry_on_failure(self, value: bool) -> None
+```
+
+#### `retry_delay`
+
+```python
+retry_delay(self) -> float
+```
+
+Base delay in seconds between retries of a transient failure; doubles per attempt up to retry_max_delay.
+
+#### `retry_delay`
+
+```python
+retry_delay(self, value: float) -> None
+```
+
 #### `request_context`
 
 ```python