Deselect VGF tests when model-converter is not installed (#19282) by psiddh · Pull Request #19282 · pytorch/executorch

psiddh · 2026-05-04T17:57:16Z

Summary:

VGF tests require the Arm MLSDK model-converter binary which is not provisioned in CI when _ENABLE_VGF = False (the current default in targets.bzl). These tests correctly skip via SkipIfNoModelConverter, but TestInfra reports skipped/disabled tests as FAILURE in the OMH dashboard, creating ~1,025 phantom failures for the ai_infra_mobile_platform oncall.

This change uses the existing pytest_collection_modifyitems hook in conftest.py to deselect (not collect) tests that carry the SkipIfNoModelConverter marker (detected via reason string on skipif markers). This is more precise than name-based filtering because:

It catches parametrized variants like test_roundtrip[VGF] that use marks=SkipIfNoModelConverter
It does NOT deselect tests like test_compile_spec_vgf_no_quant that have "vgf" in the name but do not need model-converter

Deselected tests are invisible to TestInfra, eliminating false failure reports. When _ENABLE_VGF is later flipped to True and model-converter is provisioned, VGF tests will automatically be collected and run again.

Differential Revision: D103644035

pytorch-bot · 2026-05-04T17:57:20Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/19282

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

Ubuntu services are down

⏳ 122 Pending, 1 Unrelated Failure

As of commit b15b795 with merge base 0a113f8 ():

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / android / run-emulator (gh) (trunk failure)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

meta-codesync · 2026-05-04T17:57:34Z

@psiddh has exported this pull request. If you are a Meta employee, you can view the originating Diff in D103644035.

github-actions · 2026-05-04T17:58:24Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Copilot

Pull request overview

This PR is mainly aimed at reducing false CI noise in the Arm backend by changing how VGF tests are handled when the Arm MLSDK model-converter binary is unavailable. It also includes two unrelated test-only changes outside the Arm VGF collection logic.

Changes:

Add Arm pytest_collection_modifyitems logic to deselect VGF tests when model-converter is not installed.
Broaden an EXIR partitioner test's expected AttributeError regex for cross-version message compatibility.
Add a new skip to the Vulkan Hardshrink delegate test.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
`backends/arm/test/conftest.py`	Adds collection-time deselection of VGF tests based on `model_converter_installed()`.
`exir/backend/test/test_partitioner.py`	Relaxes a test assertion to accept multiple setter error message variants.
`backends/vulkan/test/test_vulkan_delegate.py`	Unconditionally skips the Vulkan Hardshrink backend test.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Summary: VGF tests require the Arm MLSDK model-converter binary which is not provisioned in CI when `_ENABLE_VGF = False` (the current default in targets.bzl). These tests correctly skip via `SkipIfNoModelConverter`, but TestInfra reports skipped/disabled tests as FAILURE in the OMH dashboard, creating ~1,025 phantom failures for the ai_infra_mobile_platform oncall. This change uses the existing `pytest_collection_modifyitems` hook in conftest.py to deselect (not collect) tests that carry the `SkipIfNoModelConverter` marker (detected via `reason` string on `skipif` markers). This is more precise than name-based filtering because: - It catches parametrized variants like `test_roundtrip[VGF]` that use `marks=SkipIfNoModelConverter` - It does NOT deselect tests like `test_compile_spec_vgf_no_quant` that have "vgf" in the name but do not need model-converter Deselected tests are invisible to TestInfra, eliminating false failure reports. When `_ENABLE_VGF` is later flipped to True and model-converter is provisioned, VGF tests will automatically be collected and run again. Differential Revision: D103644035

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    from executorch.backends.arm.test.runner_utils import model_converter_installed
+
+    if not model_converter_installed():
+        deselected = []
+        remaining = []
+        for item in items:
+            for marker in item.iter_markers("skipif"):
+                if "model-converter" in marker.kwargs.get("reason", ""):


+        vk_icd = os.environ.get("VK_ICD_FILENAMES", "")
+        if "swiftshader" in vk_icd.lower():
+            return True
+        RTLD_NOLOAD = 4
+        for lib_name in ("libvk_swiftshader.so", "libvk_swiftshader_fbcode.so"):
+            try:
+                ctypes.CDLL(lib_name, mode=RTLD_NOLOAD)
+                return True


Summary: VGF tests require the Arm MLSDK model-converter binary which is not provisioned in CI when `_ENABLE_VGF = False` (the current default in targets.bzl). These tests correctly skip via `SkipIfNoModelConverter`, but TestInfra reports skipped/disabled tests as FAILURE in the OMH dashboard, creating ~1,025 phantom failures for the ai_infra_mobile_platform oncall. This change uses the existing `pytest_collection_modifyitems` hook in conftest.py to deselect (not collect) tests that carry the `SkipIfNoModelConverter` marker (detected via `reason` string on `skipif` markers). This is more precise than name-based filtering because: - It catches parametrized variants like `test_roundtrip[VGF]` that use `marks=SkipIfNoModelConverter` - It does NOT deselect tests like `test_compile_spec_vgf_no_quant` that have "vgf" in the name but do not need model-converter Deselected tests are invisible to TestInfra, eliminating false failure reports. When `_ENABLE_VGF` is later flipped to True and model-converter is provisioned, VGF tests will automatically be collected and run again. Differential Revision: D103644035

… skip (pytorch#19281) Summary: Fix two test failures for the ai_infra_mobile_platform oncall: 1. test_partitioner_with_spec: Python 3.12 changed the AttributeError message for property setters from "can't set attribute 'spec'" to "property 'spec' of '...' object has no setter". Updated the assertRaisesRegex pattern to match both versions. 2. test_vulkan_backend_hardshrink (and 7 other swiftshader-incompatible tests): Replaced unconditional `unittest.skip` with a conditional `skip_if_swiftshader` decorator that detects swiftshader via env var (`ETVK_USING_SWIFTSHADER`), VK_ICD_FILENAMES, or loaded library check. Tests will now run on real Vulkan drivers (non-swiftshader environments) while still skipping in CI where swiftshader is used. Differential Revision: D103628836

Summary: VGF tests require the Arm MLSDK model-converter binary which is not provisioned in CI when `_ENABLE_VGF = False` (the current default in targets.bzl). These tests correctly skip via `SkipIfNoModelConverter`, but TestInfra reports skipped/disabled tests as FAILURE in the OMH dashboard, creating ~1,025 phantom failures for the ai_infra_mobile_platform oncall. This change uses the existing `pytest_collection_modifyitems` hook in conftest.py to deselect (not collect) tests that carry the `SkipIfNoModelConverter` marker (detected via `reason` string on `skipif` markers). This is more precise than name-based filtering because: - It catches parametrized variants like `test_roundtrip[VGF]` that use `marks=SkipIfNoModelConverter` - It does NOT deselect tests like `test_compile_spec_vgf_no_quant` that have "vgf" in the name but do not need model-converter Deselected tests are invisible to TestInfra, eliminating false failure reports. When `_ENABLE_VGF` is later flipped to True and model-converter is provisioned, VGF tests will automatically be collected and run again. Differential Revision: D103644035

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    try:
+        if os.environ.get("ETVK_USING_SWIFTSHADER", "0") in ("1", "True"):
+            return True
+        vk_icd = os.environ.get("VK_ICD_FILENAMES", "")
+        if "swiftshader" in vk_icd.lower():
+            return True
+        RTLD_NOLOAD = 4
+        for lib_name in ("libvk_swiftshader.so", "libvk_swiftshader_fbcode.so"):
+            try:
+                ctypes.CDLL(lib_name, mode=RTLD_NOLOAD)
+                return True
+            except Exception:
+                continue
+        return False
+    except Exception:
+        return True


 def pytest_collection_modifyitems(config, items):
-    pass
+    from executorch.backends.arm.test import common
+    from executorch.backends.arm.test.runner_utils import model_converter_installed


Copilot AI review requested due to automatic review settings May 4, 2026 17:57

psiddh requested review from JacobSzwejbka, SS-JIA, digantdesai and larryliu0820 as code owners May 4, 2026 17:57

meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label May 4, 2026

github-actions Bot added ciflow/trunk module: arm Issues related to arm backend and removed CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. labels May 4, 2026

meta-codesync Bot added fb-exported meta-exported labels May 4, 2026

Copilot started reviewing on behalf of psiddh May 4, 2026 17:57 View session

Copilot AI reviewed May 4, 2026

View reviewed changes

Comment thread backends/arm/test/conftest.py Outdated

Comment thread backends/arm/test/conftest.py Outdated

Comment thread backends/vulkan/test/test_vulkan_delegate.py Outdated

meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label May 4, 2026

psiddh force-pushed the export-D103644035 branch from e266765 to 924bb25 Compare May 4, 2026 21:51

meta-codesync Bot changed the title ~~Deselect VGF tests when model-converter is not installed~~ Deselect VGF tests when model-converter is not installed (#19282) May 5, 2026

Copilot AI review requested due to automatic review settings May 5, 2026 03:38

psiddh force-pushed the export-D103644035 branch from 924bb25 to 2b55de5 Compare May 5, 2026 03:38

Copilot started reviewing on behalf of psiddh May 5, 2026 03:39 View session

Copilot AI reviewed May 5, 2026

View reviewed changes

psiddh force-pushed the export-D103644035 branch from 2b55de5 to a850260 Compare May 5, 2026 03:48

psiddh added 2 commits May 4, 2026 22:45

Copilot AI review requested due to automatic review settings May 5, 2026 05:46

psiddh force-pushed the export-D103644035 branch from a850260 to b15b795 Compare May 5, 2026 05:46

Copilot started reviewing on behalf of psiddh May 5, 2026 05:46 View session

Copilot AI reviewed May 5, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deselect VGF tests when model-converter is not installed (#19282)#19282

Deselect VGF tests when model-converter is not installed (#19282)#19282
psiddh wants to merge 2 commits intopytorch:mainfrom
psiddh:export-D103644035

psiddh commented May 4, 2026 •

edited by meta-codesync Bot

Loading

Uh oh!

pytorch-bot Bot commented May 4, 2026 •

edited

Loading

Uh oh!

meta-codesync Bot commented May 4, 2026

Uh oh!

github-actions Bot commented May 4, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

psiddh commented May 4, 2026 • edited by meta-codesync Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot Bot commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/19282

❗ 1 Active SEVs

⏳ 122 Pending, 1 Unrelated Failure

Uh oh!

meta-codesync Bot commented May 4, 2026

Uh oh!

github-actions Bot commented May 4, 2026

This PR needs a release notes: label

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

psiddh commented May 4, 2026 •

edited by meta-codesync Bot

Loading

pytorch-bot Bot commented May 4, 2026 •

edited

Loading

This PR needs a `release notes:` label