Skip to content

Intermittent ACR connection refusal during core image builds #5259

Open

Description

@JayDoubleu

Core image builds can fail when az acr login cannot connect to Azure Container Registry. In run #11117, the three image jobs reported connection refusals on attempt 1 and passed on attempt 2.

Image job Attempt 1 Attempt 2
Resource processor Failed during ACR login Passed
API Connection refused before cancellation Passed
Airlock processor Connection refused before cancellation Passed

The resource-processor job failed at 15:39:36 UTC on 14 October 2026:

HTTPSConnectionPool(host='<registry>.azurecr.io', port=443):
Max retries exceeded with url: /oauth2/exchange
Failed to establish a new connection: [Errno 111] Connection refused

The API and Airlock jobs also reported connection refusals to the registry's /v2/ endpoint. API logins had succeeded shortly before these errors. The failure prevented core deployment and smoke tests from running on attempt 1.

The resource-processor checkout logs confirm the same commit in both attempts: c0b0f7ae9075bd08ace9ca138ddcb7c994ad3529, the merge of PR #5255 head 9ad8d125c101736be193edac6ced6cb5125d06b6 into main c8203b84bef541ad24ad40b6a843761097ffde64. No code change was needed for the image builds to pass on rerun.

The underlying connection failure remains unexplained. The result shows recovery on rerun, but does not establish whether the registry or the runner network caused it.

The Makefile image build and push recipes call az acr login without an explicit retry loop. Investigate the connection failure and consider a bounded retry for transient connection errors, while preserving failures for invalid credentials or configuration.

Related issue #3778 was fixed by #3861, which corrected shell quoting in a separate cache-registry login step. That fix does not cover these Makefile calls.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingworkflowsGitHub workflows (CI/CD)

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions