feat(image-tests): initial pytest-based static image validation#16947
Open
reubeno wants to merge 1 commit intomicrosoft:tomls/base/mainfrom
Open
feat(image-tests): initial pytest-based static image validation#16947reubeno wants to merge 1 commit intomicrosoft:tomls/base/mainfrom
reubeno wants to merge 1 commit intomicrosoft:tomls/base/mainfrom
Conversation
Introduce a pytest harness for offline validation of built Azure Linux images, wired into azldev via the new [test-suites] TOML config. A small, representative set of tests is included to establish the extension pattern: shared cases live under base/images/tests/cases/ and image-specific cases under cases/<image-name>/ (auto-restricted to that image by the pytest plugin). Image-type plumbing — VM mount via guestmount, container extract via skopeo+umoci — and shared fixtures (rootfs, os_release, installed_packages, partition_table) live under utils/, with native-tool dependencies declared next to their use and validated up-front by a preflight check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
❌ Rendered specs are out of date🚧🚧🚧🚧🚧 Warning Disregard this comment. Spec rendering is still under development and checked-in specs 🚧🚧🚧🚧🚧 FIX: — run this and commit the result: azldev component render -a --clean-staleOr download the fix patch and apply it: gh run download 25133149787 -R microsoft/azurelinux -n rendered-specs-patch
git apply rendered-specs.patch
Content diffs`specs/3/389-ds-base/389-ds-base.spec`--- committed/specs/3/389-ds-base/389-ds-base.spec
+++ rendered/specs/3/389-ds-base/389-ds-base.spec
@@ -1,5 +1,5 @@
## START: Set by rpmautospec
-## (rpmautospec version 0.8.4)
+## (rpmautospec version 0.8.3)
## RPMAUTOSPEC: autorelease, autochangelog
%define autorelease(e:s:pb:n) %{?-p:0.}%{lua:
release_number = 8;
@@ -1123,7 +1123,7 @@
%changelog
## START: Generated by rpmautospec
-* Wed Apr 22 2026 azldev <azurelinux@microsoft.com> - 3.1.4-8
+* Wed Apr 29 2026 azldev <azurelinux@microsoft.com> - 3.1.4-8
- Latest state for 389-ds-base
* Fri Feb 13 2026 Viktor Ashirov <vashirov@redhat.com> - 3.1.4-7
`specs/7/7zip/7zip.spec`--- committed/specs/7/7zip/7zip.spec
+++ rendered/specs/7/7zip/7zip.spec
@@ -1,5 +1,5 @@
## START: Set by rpmautospec
-## (rpmautospec version 0.8.4)
+## (rpmautospec version 0.8.3)
## RPMAUTOSPEC: autorelease, autochangelog
%define autorelease(e:s:pb:n) %{?-p:0.}%{lua:
release_number = 2;
@@ -138,7 +138,7 @@
%changelog
## START: Generated by rpmautospec
-* Wed Apr 22 2026 azldev <azurelinux@microsoft.com> - 25.01-2
+* Wed Apr 29 2026 azldev <azurelinux@microsoft.com> - 25.01-2
- Latest state for 7zip
* Wed Nov 26 2025 Michel Lind <salimma@fedoraproject.org> - 25.01-1
`specs/a/AMF/AMF.spec`--- committed/specs/a/AMF/AMF.spec
+++ rendered/specs/a/AMF/AMF.spec
@@ -1,5 +1,5 @@
## START: Set by rpmautospec
-## (rpmautospec version 0.8.4)
+## (rpmautospec version 0.8.3)
## RPMAUTOSPEC: autorelease, autochangelog
%define autorelease(e:s:pb:n) %{?-p:0.}%{lua:
release_number = 2;
@@ -87,7 +87,7 @@
%changelog
## START: Generated by rpmautospec
-* Wed Apr 22 2026 azldev <azurelinux@microsoft.com> - 1:1.5.0-2
+* Wed Apr 29 2026 azldev <azurelinux@microsoft.com> - 1:1.5.0-2
- Latest state for AMF
* Fri Oct 31 2025 Simone Caronni <negativo17@gmail.com> - 1:1.5.0-1
`specs/a/aardvark-dns/aardvark-dns.spec`--- committed/specs/a/aardvark-dns/aardvark-dns.spec
+++ rendered/specs/a/aardvark-dns/aardvark-dns.spec
@@ -1,5 +1,5 @@
## START: Set by rpmautospec
-## (rpmautospec version 0.8.4)
+## (rpmautospec version 0.8.3)
## RPMAUTOSPEC: autorelease, autochangelog
%define autorelease(e:s:pb:n) %{?-p:0.}%{lua:
release_number = 2;
@@ -130,7 +130,7 @@
%changelog
## START: Generated by rpmautospec
-* Wed Apr 22 2026 azldev <azurelinux@microsoft.com> - 2:1.17.0-2
+* Wed Apr 29 2026 azldev <azurelinux@microsoft.com> - 2:1.17.0-2
- Latest state for aardvark-dns
* Wed Nov 12 2025 Packit <hello@packit.dev> - 2:1.17.0-1
`specs/a/abseil-cpp/abseil-cpp.spec`--- committed/specs/a/abseil-cpp/abseil-cpp.spec
+++ rendered/specs/a/abseil-cpp/abseil-cpp.spec
@@ -1,5 +1,5 @@
## START: Set by rpmautospec
-## (rpmautospec version 0.8.4)
+## (rpmautospec version 0.8.3)
## RPMAUTOSPEC: autorelease, autochangelog
%define autorelease(e:s:pb:n) %{?-p:0.}%{lua:
release_number = 2;
@@ -267,7 +267,7 @@
%changelog
## START: Generated by rpmautospec
-* Wed Apr 22 2026 azldev <azurelinux@microsoft.com> - 20250814.1-2
+* Wed Apr 29 2026 azldev <azurelinux@microsoft.com> - 20250814.1-2
- Latest state for abseil-cpp
* Tue Sep 23 2025 Benjamin A. Beasley <code@musicinmybrain.net> - 20250814.1-1
`specs/a/accountsservice/accountsservice.spec`--- committed/specs/a/accountsservice/accountsservice.spec
+++ rendered/specs/a/accountsservice/accountsservice.spec
@@ -1,5 +1,5 @@
## START: Set by rpmautospec
-## (rpmautospec version 0.8.4)
+## (rpmautospec version 0.8.3)
## RPMAUTOSPEC: autorelease, autochangelog
%define autorelease(e:s:pb:n) %{?-p:0.}%{lua:
release_number = 15;
@@ -130,7 +130,7 @@
%changelog
## START: Generated by rpmautospec
-* Wed Apr 22 2026 azldev <azurelinux@microsoft.com> - 23.13.9-15
+* Wed Apr 29 2026 azldev <azurelinux@microsoft.com> - 23.13.9-15
- Latest state for accountsservice
* Wed Jul 23 2025 Fedora Release Engineering <releng@fedoraproject.org> - 23.13.9-9
`specs/a/adobe-source-code-pro-fonts/adobe-source-code-pro-fonts.spec`--- committed/specs/a/adobe-source-code-pro-fonts/adobe-source-code-pro-fonts.spec
+++ rendered/specs/a/adobe-source-code-pro-fonts/adobe-source-code-pro-fonts.spec
@@ -1,5 +1,5 @@
## START: Set by rpmautospec
-## (rpmautospec version 0.8.4)
+## (rpmautospec version 0.8.3)
## RPMAUTOSPEC: autorelease, autochangelog
%define autorelease(e:s:pb:n) %{?-p:0.}%{lua:
release_number = 8;
@@ -71,7 +71,7 @@
%changelog
## START: Generated by rpmautospec
-* Wed Apr 22 2026 azldev <azurelinux@microsoft.com> - 2.042.1.062.1.026-8
+* Wed Apr 29 2026 azldev <azurelinux@microsoft.com> - 2.042.1.062.1.026-8
- Latest state for adobe-source-code-pro-fonts
* Wed Jul 23 2025 Fedora Release Engineering <releng@fedoraproject.org> - 2.042.1.062.1.026-7
`specs/a/adwaita-icon-theme-legacy/adwaita-icon-theme-legacy.spec`--- committed/specs/a/adwaita-icon-theme-legacy/adwaita-icon-theme-legacy.spec
+++ rendered/specs/a/adwaita-icon-theme-legacy/adwaita-icon-theme-legacy.spec
@@ -1,5 +1,5 @@
## START: Set by rpmautospec
-## (rpmautospec version 0.8.4)
+## (rpmautospec version 0.8.3)
## RPMAUTOSPEC: autorelease, autochangelog
%define autorelease(e:s:pb:n) %{?-p:0.}%{lua:
release_number = 6;
@@ -87,7 +87,7 @@
%changelog
## START: Generated by rpmautospec
-* Wed Apr 22 2026 azldev <azurelinux@microsoft.com> - 46.2-6
+* Wed Apr 29 2026 azldev <azurelinux@microsoft.com> - 46.2-6
- Latest state for adwaita-icon-theme-legacy
* Wed Jul 23 2025 Fedora Release Engineering <releng@fedoraproject.org> - 46.2-4
`specs/a/adwaita-icon-theme/adwaita-icon-theme.spec`--- committed/specs/a/adwaita-icon-theme/adwaita-icon-theme.spec
+++ rendered/specs/a/adwaita-icon-theme/adwaita-icon-theme.spec
@@ -1,5 +1,5 @@
## START: Set by rpmautospec
-## (rpmautospec version 0.8.4)
+## (rpmautospec version 0.8.3)
## RPMAUTOSPEC: autorelease, autochangelog
%define autorelease(e:s:pb:n) %{?-p:0.}%{lua:
release_number = 4;
@@ -93,7 +93,7 @@
%changelog
## START: Generated by rpmautospec
-* Wed Apr 22 2026 azldev <azurelinux@microsoft.com> - 49.0-4
+* Wed Apr 29 2026 azldev <azurelinux@microsoft.com> - 49.0-4
- Latest state for adwaita-icon-theme
* Mon Sep 15 2025 Michael Catanzaro <mcatanzaro@gnome.org> - 49.0-1
`specs/a/afflib/afflib.spec`--- committed/specs/a/afflib/afflib.spec
+++ rendered/specs/a/afflib/afflib.spec
@@ -1,5 +1,5 @@
## START: Set by rpmautospec
-## (rpmautospec version 0.8.4)
+## (rpmautospec version 0.8.3)
## RPMAUTOSPEC: autorelease, autochangelog
%define autorelease(e:s:pb:n) %{?-p:0.}%{lua:
release_number = 16;
@@ -223,7 +223,7 @@
%changelog
## START: Generated by rpmautospec
-* Wed Apr 22 2026 azldev <azurelinux@microsoft.com> - 3.7.20-16
+* Wed Apr 29 2026 azldev <azurelinux@microsoft.com> - 3.7.20-16
- Latest state for afflib
* Wed Jul 23 2025 Fedora Release Engineering <releng@fedoraproject.org> - 3.7.20-15
… and 3998 more file(s). Run the remediation command above to see all changes. Files to addThese files are produced by
… and 219 more file(s). Files to removeThese files are in your branch but are not produced by render. Remove them.
… and 262 more file(s). |
reubeno
changed the title
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an initial pytest-based framework for offline/static validation of built Azure Linux VM and container images, and wires it into azldev via a new static-image-checks test suite defined in base/images/images.toml.
Changes:
- Introduces a standalone pytest project under
base/images/tests/(pyproject + plugin) with custom CLI options, marker logic, and native-tool preflight checks. - Adds utilities for mounting/extracting images (VM via libguestfs, containers via skopeo+umoci) and parsing common metadata (os-release, rpm package list, disk partitions).
- Adds a small starter set of shared and image-specific pytest cases, and attaches the suite to
vm-baseandcontainer-baseimages.
Reviewed changes
Copilot reviewed 17 out of 20 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| base/images/tests/utils/types.py | Adds dataclasses for disk/partition metadata shared across fixtures/tests. |
| base/images/tests/utils/tools.py | Adds native-tool registry + PATH preflight helpers and a CLI status command. |
| base/images/tests/utils/pytest_plugin.py | Implements pytest plugin: CLI options, markers, tool preflight, and auto image-dir test routing. |
| base/images/tests/utils/parsers.py | Adds parsers for /etc/os-release and rpm package queries via rpm --root. |
| base/images/tests/utils/extract.py | Implements VM mount (guestmount) and container rootfs extraction (skopeo+umoci) plus cleanup helpers. |
| base/images/tests/utils/disk.py | Adds VM disk inspection via virt-inspector and XML parsing into dataclasses. |
| base/images/tests/utils/init.py | Declares utils as a package for the installed pytest plugin/utilities. |
| base/images/tests/pyproject.toml | Defines the pytest project, dependencies, and registers the pytest11 entry point. |
| base/images/tests/conftest.py | Provides session-scoped fixtures for image inputs, rootfs access, and parsed metadata. |
| base/images/tests/cases/vm-base/test_partitions.py | VM-specific partition layout checks. |
| base/images/tests/cases/vm-base/test_kernel.py | VM-specific kernel modules presence check. |
| base/images/tests/cases/vm-base/init.py | Package marker for vm-base test subdir. |
| base/images/tests/cases/test_packages.py | Shared package presence/blocklist tests gated by capability. |
| base/images/tests/cases/test_os_release.py | Shared /etc/os-release validation tests. |
| base/images/tests/cases/container-base/test_container.py | Container-specific kernel-module exclusion check. |
| base/images/tests/cases/container-base/init.py | Package marker for container-base test subdir. |
| base/images/tests/cases/init.py | Package marker for shared cases directory. |
| base/images/tests/README.md | Documents invocation via azldev/manual runs, layout, fixtures, and tool prerequisites. |
| base/images/tests/.gitignore | Ignores per-suite virtualenv/cache/work artifacts and uv.lock. |
| base/images/images.toml | Wires static-image-checks into vm-base/container-base and defines the pytest suite config. |
Comment on lines
+118
to
+124
| rc = subprocess.run( | ||
| ["buildah", "unshare", "rm", "-rf", str(p)], | ||
| check=False, | ||
| capture_output=True, | ||
| text=True, | ||
| ) | ||
| if rc.returncode != 0: |
There was a problem hiding this comment.
subprocess.run(["buildah", ...]) will raise FileNotFoundError if buildah isn't installed (even with check=False), which would error the entire session during teardown. Since this cleanup path runs for all images (including VM-only runs where buildah isn't preflight-checked), wrap the call in try/except FileNotFoundError and fall back to shutil.rmtree, or gate the buildah path behind an availability check (e.g., shutil.which).
Suggested change
| rc = subprocess.run( | |
| ["buildah", "unshare", "rm", "-rf", str(p)], | |
| check=False, | |
| capture_output=True, | |
| text=True, | |
| ) | |
| if rc.returncode != 0: | |
| use_fallback_cleanup = True | |
| if shutil.which("buildah") is not None: | |
| rc = subprocess.run( | |
| ["buildah", "unshare", "rm", "-rf", str(p)], | |
| check=False, | |
| capture_output=True, | |
| text=True, | |
| ) | |
| use_fallback_cleanup = rc.returncode != 0 | |
| if use_fallback_cleanup: |
|
|
||
| Test selection follows standard pytest positional arguments. Tests | ||
| under `cases/<image-name>/` auto-skip when `--image-name` doesn't match | ||
| (via the `image` marker, applied in each subdirectory's `conftest.py`). |
There was a problem hiding this comment.
The README says image-specific tests are restricted via an image marker "applied in each subdirectory's conftest.py", but the implementation actually auto-applies the marker in utils.pytest_plugin.pytest_collection_modifyitems (and there are no per-subdir conftest files). Update this description to match the actual mechanism to avoid confusing contributors.
Suggested change
| (via the `image` marker, applied in each subdirectory's `conftest.py`). | |
| (via the `image` marker, auto-applied during collection by | |
| `utils.pytest_plugin.pytest_collection_modifyitems`). |
Comment on lines
+168
to
+171
| from pathlib import Path | ||
|
|
||
| for item in items: | ||
| parts = Path(str(item.fspath)).parts |
There was a problem hiding this comment.
item.fspath is the legacy py.path API and is deprecated in newer pytest versions (the modern attribute is item.path, a pathlib.Path). Using item.path here will avoid deprecation warnings and future breakage while keeping the same behavior.
Suggested change
| from pathlib import Path | |
| for item in items: | |
| parts = Path(str(item.fspath)).parts | |
| for item in items: | |
| parts = item.path.parts |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
NOTE: I'm still reviewing some of the copilot-generated implementation details; posting for easier commenting, iteration, and early skim of general structure.
Introduce a pytest harness for offline validation of built Azure Linux images, wired into azldev via the new [test-suites] TOML config.
A small, representative set of tests is included to establish the extension pattern: shared cases live under base/images/tests/cases/ and image-specific cases under cases// (auto-restricted to that image by the pytest plugin). Image-type plumbing — VM mount via guestmount, container extract via skopeo+umoci — and shared fixtures (rootfs, os_release, installed_packages, partition_table) live under utils/, with native-tool dependencies declared next to their use and validated up-front by a preflight check.