fix(rest-api): Retry HTTP/2 INTERNAL_ERROR streams over HTTP/1.1

[kfelternv]

This is a client-side mitigation for #4673, not a fix for the unknown server-side reset. When the shared nicocli client used by the command and TUI receives the exact HTTP/2 INTERNAL_ERROR on a GET, it retries once over HTTP/1.1 with the same request path, query, authorization, and timeout; mutations and unrelated errors are not retried.

Related issues

• nicocli: instance list returns HTTP/2 INTERNAL_ERROR and TUI machine list should show IP addresses #4673

Type of Change

• Add - New feature or capability
• Change - Changes in existing functionality
• Fix - Bug fixes
• Remove - Removed features or deprecated functionality
• Internal - Internal changes (refactoring, tests, docs, etc.)

Breaking Changes

• This PR contains breaking changes

Testing

• Unit tests added/updated

• Integration tests added/updated

• Manual testing performed

• No testing required (docs, internal refactor, etc.)

• go test -race ./cli/pkg -count=1

• go vet, golangci-lint, staticcheck, errcheck, ineffassign, unused, and revive on ./cli/pkg

• Built nicocli and ran instance list --page-size 100 --output table against a local TLS server that reset HTTP/2 and returned the same request successfully over HTTP/1.1

• The broader go test -race ./cli/... -count=1 remains blocked by an existing TUI resource-policy test failure that also reproduces on clean main

Additional Notes

ytl-dev2, other deployed environments, and the interactive TUI were not exercised. This change only recovers the observed failure when the same GET succeeds over HTTP/1.1; it does not identify or repair the server-side cause of the reset.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}

[kfelternv]

What changed

PASS for a970bc8c58acefac66b5efe649fd8665d1b4345a. A nicocli GET that receives an HTTP/2 INTERNAL_ERROR while reading the response now retries once over HTTP/1.1. Mutations and non-HTTP/2 read errors are not retried.

Scenario and setup

The regression uses a local TLS server that accepts HTTP/2 and HTTP/1.1. It commits an HTTP 200 response and resets the HTTP/2 stream, then returns valid JSON for the HTTP/1.1 retry. The built nicocli command was exercised with the same organization, bearer-header presence, path, and pageSize=100 on both requests.

ytl-dev2, other deployed environments, and the interactive TUI were not exercised, per Kyle's direction. This verifies client recovery from the reported protocol error and does not claim a server-side root cause.

Verification

Step 1: Reproduce the reported failure before the retry

Why this step exists: It proves the focused regression reaches the same response-read failure reported in #4673.

Runnable command:

go test ./cli/pkg -run '^TestClient_Do$' -count=1

Observed result on base 5c086124b4e82fcfa4977b57ae76f3e4cac52797 with only the regression added:

reading response: stream error: stream ID 1; INTERNAL_ERROR; received from peer
FAIL github.com/NVIDIA/infra-controller/rest-api/cli/pkg

Why this proves the behavior: The failure is returned by the same Client.Do path used by both noninteractive instance list and the TUI.

Step 2: Verify the retry and no-retry safeguards at the PR head

Why this step exists: It covers the successful HTTP/1.1 retry, one-attempt limit, mutation exclusion, unrelated read-error exclusion, and failed-body close.

Runnable command:

go test -race ./cli/pkg -count=1

Observed result:

ok github.com/NVIDIA/infra-controller/rest-api/cli/pkg 25.180s

Why this proves the behavior: The table asserts request protocols [HTTP/2, HTTP/1.1] for GET, [HTTP/2] for POST, one request for a non-HTTP/2 read error, and closure of the failed response body.

Step 3: Exercise the built nicocli command

Why this step exists: It verifies the compiled command through the user-facing noninteractive interface rather than only calling the client package directly.

Runnable command:

SSL_CERT_FILE=./server.crt NICO_BASE_URL=https://localhost:18473 NICO_ORG=test-org NICO_TOKEN='<redacted>' ./nicocli instance list --page-size 100 --output table

Observed result:

request protocol=HTTP/2 method=GET path=/v2/org/test-org/nico/instance pageSize=100 auth=true
request protocol=HTTP/1 method=GET path=/v2/org/test-org/nico/instance pageSize=100 auth=true
0 items
(no results)

Why this proves the behavior: The built CLI preserved the request path, query, and authorization header, retried once with HTTP/1.1 after the HTTP/2 reset, and exited successfully.