[shoumikhin]

What is wrong today

The OpenVINO test-harness jobs have failed at setup on every run since September 9, on main and release/1.5. The setup script downloads a pinned nightly that OpenVINO has since deleted. The server answers the missing file with an HTML page and HTTP 200, so curl --fail does not catch it, and the job dies in tar:

gzip: stdin: not in gzip format

The YOLO26 test script and its README pin a deleted nightly wheel too. The openvino pip extra, the docs and the runtime error message still cap OpenVINO below 2026.0.0, although CI runs the backend on 2026 releases.

What this change does

• Both Linux OpenVINO jobs install one pinned release, 2026.4.1, the latest. Before, the harness used that nightly and test-openvino-linux used 2026.0.
• The download is checked against a pinned SHA-256, so a missing or replaced archive stops before tar.
• --nightly stays for quick tests only. It installs the newest nightly from OpenVINO's latest.txt, checked against its published .sha256. No CI job passes it.
• Pull requests that change setup-openvino.sh or test_backend.sh also run the harness.
• The YOLO26 script and README install 2026.4.1 from PyPI.
• The extra, docs and error message allow openvino>=2025.1.0,<2027.0.0. The extra still skips Python 3.15 (no build) and Linux aarch64 with Python 3.14 (builds need glibc 2.35, the ExecuTorch wheel 2.28).

Not in this change: nncf stays at 3.1.0, and release/1.5 still has the deleted pin.

resnet50, inception_v3 and wide_resnet50_2 fail the harness tolerance, not from this change: on bfloat16 CPUs OpenVINO runs float32 models in bfloat16 unless given the f32 hint. On the same inputs, 2026.0.0, 2026.1.0 and 2026.4.1 give identical results.

What was tested

On Linux x86_64 (AMD EPYC, bfloat16):

• The release and --nightly paths download, pass the check and extract; the deleted nightly URL and a swapped archive stop at the checksum.
• Harness models: 29 of 35 pass; the 6 failures are those three models, static and dynamic. Operator sub-tests: 778 of 798 pass, same failures as 2026.1.0.
• test-openvino-linux gives the same results with 2026.0.0 and 2026.4.1.

Also:

• A dry pip resolve of the extra picks 2026.4.1 on Linux x86_64, on Windows, and on Linux aarch64 with glibc 2.35 (Python 3.11 and 3.13).
• With 2026.4.1, YOLO26 exports to the OpenVINO delegate (fp32, on macOS arm64). CI does not run the YOLO26 script.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/23502

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit 4480ddc with merge base cf6d4f2 ():

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Test CoreML Backend / test-coreml / test-backend-macos (coreml, models) / macos-job (gh) (trunk failure)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[shoumikhin]

@suryasidd, could you take a look? This brings the OpenVINO harness jobs back. They had been failing since the 2026.1.0 nightly from #17730 was deleted from OpenVINO's storage.

The PR now pins the 2026.1.0 release, which contains that nightly. One thing to check: with the release, resnet50, inception_v3 and wide_resnet50_2 land just outside the model test tolerance (the largest resnet50 difference is 0.36 on logits up to 78). Is that an expected numeric change between the nightly and the release? Or would you rather pin a newer release?

cc @daniil-lyakhov

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[shoumikhin]

@suryasidd, @daniil-lyakhov, a correction to my question above. The three models are not a nightly vs. release difference. Since September these jobs run on AMD EPYC runners with bfloat16, and there OpenVINO runs the float32 models in bfloat16 by default. On a bfloat16 CPU, 2026.0.0 and 2026.1.0 give identical results, and with the f32 precision hint all three models pass. I've updated the description, so please ignore that question.

[shoumikhin]

@digantdesai, could you review this when you get a chance? The OpenVINO test-harness jobs have failed at setup since September 9, because the nightly they downloaded was deleted. This points them at the 2026.1.0 release and checks each download against a pinned SHA-256. It is one script.

Both harness jobs now reach the tests again. The two red jobs in CI fail the same way on main: the Samsung yolo26 compile error (#23481) and the CoreML job that hangs in pip and hits the time limit. Three models land just outside the test tolerance because of bfloat16 on the new runners, not because of this change. The description explains that.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[shoumikhin]

@claude review

[claude]

Claude finished @shoumikhin's task in 3m 19s —— View job

---

I found no blocking issues in #23502, which pins both Linux OpenVINO jobs to the 2026.4.1 release with a SHA-256 check, so I recommend approving. The three points below are minor, and the --nightly path could not be run here because the review had no network access.

Detailed review

PR Review: #23502

Summary

Both Linux OpenVINO CI jobs now install the 2026.4.1 release and check it against a pinned SHA-256 before extracting it. --nightly now resolves the newest nightly at run time instead of pinning one that gets deleted, and the harness also runs on pull requests that change the setup scripts. There are no blocking issues, only three minor ones.

Code Quality

• Parsing in the --nightly path is fragile (.ci/scripts/setup-openvino.sh:25). I could not check the real file format offline, and no CI job runs this path, so this is low confidence. sed -n 's/^ubuntu22_x86_64://p' expects exactly one ubuntu22_x86_64:<url> line, with no space after the colon and LF line endings. The :? guard only catches an empty result. It misses three cases:

  • a CRLF line ending leaves a trailing \r on the URL;
  • a space after the colon leaves a leading space;
  • two matching lines put a newline inside the URL.

  Each of these shows up later as a confusing .sha256 fetch or basename failure, not as the clear guard message. Adding | tr -d '\r[:space:]' | head -n1 or something similar would make it robust. Fix this →

• The sourced script now reads the caller's arguments (.ci/scripts/test_backend.sh:122). source without arguments keeps the caller's positional parameters. Before, --nightly replaced them. Now the script's for arg in "$@" loop scans test_backend.sh's own arguments: "$SUITE" "$FLOW" "$ARTIFACT_DIR", as passed in _test_backend.yml:88. None of those is --nightly today, so this does no harm. Still, whether nightly runs now depends on the caller's arguments. test_backend.sh already copies $1–$3 into variables at lines 13–15, so a set -- just before the source would cut this link safely. Fix this →

Infrastructure

• CI now tests a version outside the supported range. Both OpenVINO jobs test only 2026.4.1, but these three places still say users need openvino>=2025.1.0,<2026.0.0:

  • pyproject.toml:91
  • backends/openvino/runtime/OpenvinoBackend.cpp:215
  • docs/source/build-run-openvino.md:63

  No job covers 2025.x, which is what executorch[openvino] installs. nncf==3.1.0 has also not been checked against 2026.4. The PR says it defers this, and test-openvino-linux already sat outside that range at 2026.0, but this PR moves it further away. Please open a tracking issue so the follow-up is not lost.

Recommendation

Approve. Nothing blocks merging. The --nightly parsing and the inherited arguments only matter for manual runs today.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

[shoumikhin]

Thanks for the review.

• --nightly parsing: I checked it against the real latest.txt and .sha256 files. They have LF line endings, one ubuntu22_x86_64: line and no space after the colon. If that format ever changes, the download fails and the job still stops before tar. No CI job passes --nightly, so I kept it simple.
• Arguments of the sourced script: right, it now sees the arguments of test_backend.sh. Those are the suite, the flow and the artifact directory, so none of them can be --nightly. I did not add code for a case that cannot happen.
• Supported version range: agreed, it needs to move. As the description says, the pip extra, nncf, the docs and the error message are left for a separate change, so the OpenVINO maintainers can move them together. The description lists all four, so the follow-up has them in one place.

[shoumikhin]

@claude review

[claude]

Claude finished @shoumikhin's task in 2m 15s —— View job

---

I found no issues in #23502 and recommend approving it; the pinned SHA-256 and build string could not be checked here because the review had no network access.

Detailed review

PR Review: #23502

Summary

The PR moves both Linux OpenVINO CI jobs to the 2026.4.1 release and checks each download against a SHA-256 before tar runs. --nightly now fetches the newest nightly at run time, and the harness also runs on PRs that change setup-openvino.sh or test_backend.sh. Three independent reviewers covered all three changed files and found nothing that needs changing.

Two points raised in this round did not hold up when I checked them:

• Unrelated PRs would get a red OpenVINO job: one reviewer thought the new test_backend.sh trigger would do this because of the three known bf16 model failures. That is not the case. test_backend.sh:163 passes --exit-code to generate_markdown_summary_json, but generate_markdown (backends/test/suite/generate_markdown_summary_json.py:118) never exits with it. Failing tests only show up in the report and never fail the job. The trigger also matches how CoreML, Cortex-M and NXP already gate on that script.
• The supported version range is stale, and test_yolo26.sh pins a nightly: both are real but were already discussed and deferred in this thread, so I am not raising them again.

I did not check the pinned hash, the build string 2026.4.1.22982.07f9c262b05 or the latest.txt format against OpenVINO's storage, because the review had no network access. CI on this PR covers the pinned release path: test-openvino-linux runs on every PR.

Recommendation

Approve. Nothing blocks merging.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

[shoumikhin]

@claude review

[claude]

Claude finished @shoumikhin's task in 3m 3s —— View job

---

I found no blocking issues in #23502 and recommend approving it, with two minor follow-ups: the harness path filter still leaves out the files that setup-openvino.sh runs, and the OpenVINO 2026.4.1 pin is copied by hand into three places.

Detailed review

PR Review: #23502

Summary

The PR moves both Linux OpenVINO jobs to a pinned 2026.4.1 release and checks each download against a SHA-256 before tar. --nightly now installs the newest nightly from latest.txt, and the supported range becomes openvino>=2025.1.0,<2027.0.0. There are no blocking issues, only the minor points below.

I did not raise again the points already settled in this thread: how --nightly parses latest.txt, the sourced script seeing the caller's arguments, and leaving nncf at 3.1.0. I could not check the pinned hash, the build string or the PyPI wheel tags, because the review had no network access. test-openvino-linux runs on every PR and covers the release path.

Infrastructure

• The harness still skips PRs that change what it installs (.github/workflows/test-backend-openvino.yml:11-16). setup-openvino.sh installs backends/openvino/requirements.txt (the nncf pin) and runs backends/openvino/scripts/openvino_build.sh, but neither path is in the filter. Changes under backends/openvino/** are not in it either.

  • Effect: a PR that bumps nncf or edits the build script merges without a harness run, then breaks in the 02:00 cron run. That is the same late failure this PR fixes.
  • Context: OpenVINO is the only test-backend-*.yml with a paths: filter. arm, coreml, cortex-m, nxp, qnn, vulkan, webgpu and xnnpack all use a bare pull_request:.
  • Fix: add backends/openvino/** to the filter, or drop the filter so it matches the other backends.

  Fix this →

Code Quality

• The comment links two version pins that nothing keeps in step (.ci/scripts/test_yolo26.sh:79-80). The comment says "The same OpenVINO release that setup-openvino.sh installs". In fact 2026.4.1 is written by hand in three places:

  • setup-openvino.sh:30
  • test_yolo26.sh:80
  • examples/models/yolo26/README.md:21

  No CI job runs test_yolo26.sh, so when OPENVINO_VERSION is next bumped, this comment becomes false and nothing catches it. Either drop the claim, or leave a note at setup-openvino.sh:30 that the other two copies need the same bump.

• The install guide link is still for 2025 (docs/source/build-run-openvino.md:67). The paragraph above it now allows 2026, but the link still points to docs.openvino.ai/2025/.... This line is not in the diff.

Recommendation

Approve. Nothing blocks merging, and both points can go in a follow-up.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

[shoumikhin]

Thanks for the review. On the three points:

1. Path filter. I am leaving it as it is. test-openvino-linux in pull.yml runs on every pull request and runs setup-openvino.sh end to end, including the nncf install and the build script. So an nncf pin that does not install, or a build script that fails, already fails on its own pull request. Running the full harness on every OpenVINO backend change is a CI cost question for a separate change.
2. Version copies. Agreed, that comment could go stale without anyone noticing. I removed it, so nothing claims the two pins match.
3. 2025 install guide link. I am leaving it. The line is not part of this change, the link still works, and 2025 releases are still in the supported range.

[shoumikhin]

@claude review

[claude]

Claude finished @shoumikhin's task in 2m 30s —— View job

---

I found no blocking issues in #23502 and recommend approving it; the two minor points below are about the wider <2027.0.0 range and one vague sentence in the YOLO26 README.

Detailed review

PR Review: #23502

Summary

The PR moves both Linux OpenVINO jobs to a pinned 2026.4.1 release checked against a SHA-256, keeps --nightly for manual use only, and widens the supported range to openvino>=2025.1.0,<2027.0.0. There are no blocking issues. Three independent reviewers read all 11 changed files.

I did not raise again the points already settled in this thread: how latest.txt is parsed, the sourced script seeing the caller's arguments, the paths: filter, nncf staying at 3.1.0, the hand-copied version pins, CI not running test_yolo26.sh, and the 2025 install guide link. I could not check the pinned hash, the build string or the PyPI wheels, because the review had no network access.

Infrastructure

• The ABI comment still says the pyproject pin protects it, but the pin now covers 2026.x and CI tests only 2026.4.1 (backends/openvino/runtime/OpenvinoApi.h:27, affected by pyproject.toml:92). The delegate dlopens libopenvino_c and uses its own copies of ov_shape_t, ov_available_devices_t and a hard-coded numeric ov_element_type_e (lines 28–64). The comment says "These are stable C ABI — pinned via version constraint in pyproject.toml". A layout or enum mismatch would give wrong tensor types or shapes with no load error.
  • Not tested: the new range lets in 2026.0–2026.3 and any later 2026.x, and no job runs them. Once a newer 2026.x is published, .ci/scripts/wheel/test_windows.py:95-121 will pick it up with no pin.
  • Confidence: low. The OpenVINO C API enums have historically been stable, and I could not compare ov_common.h across 2026 releases offline.
  • Suggested fix: check that the copied types match the 2026 headers, or reword the comment so it no longer suggests the pin limits the ABI to tested releases.

Code Quality

• The YOLO26 note now says almost nothing (examples/models/yolo26/README.md:19). "Please install the OpenVINO version that supports the YOLO26 model" used to mean "a nightly". Now that the command below it is a plain release, the sentence does not tell a user whether the 2025.x runtime that executorch[openvino] can install is enough. If 2026.4.1 (or some earlier 2026.x) is the minimum, say so.

Recommendation

Approve. Nothing blocks merging. Both points are minor and fine to handle in a follow-up.

[shoumikhin]

Thanks. Both points checked, no code change:

1. Copied C types and the wider range. I compared the types the delegate copies with OpenVINO's own C headers at 2025.1.0, 2026.0.0, 2026.2.0 and 2026.4.1. The element type values, the success status value, and the shape and device list structs are the same in all four. The only change in 2026 is that the UNDEFINED name was removed. It shared value 0 with DYNAMIC, which is still there. So the comment still holds for the 2026 releases so far.
2. The YOLO26 note. OpenVINO 2025.4.1 is not enough. YOLO26 exports with aten.remainder.Scalar left outside the delegate. With 2026.4.1 the partitioner reports no unsupported op. That is why the command right under the note pins 2026.4.1.

[shoumikhin]

@pytorchbot cherry-pick --onto release/1.6 -c testci

[pytorch-bot]

❌ 🤖 pytorchbot command failed:

@pytorchbot cherry-pick: error: argument -c/--classification: invalid choice: 'testci' (choose from 'regression', 'critical', 'fixnewfeature', 'docs', 'release')

usage: @pytorchbot cherry-pick --onto ONTO [--fixes FIXES] -c
                               {regression,critical,fixnewfeature,docs,release}

Try @pytorchbot --help for more info.

[shoumikhin]

@pytorchbot cherry-pick --onto release/1.6 -c release

[pytorchbot]

Cherry picking #23502

The cherry pick PR is at #24212

Details for Dev Infra team

Raised by workflow job