Skip to content

Check netlist diffs as part of unit tests#478

Merged
ducky64 merged 7 commits intomasterfrom
ci-netlist-diff
Apr 20, 2026
Merged

Check netlist diffs as part of unit tests#478
ducky64 merged 7 commits intomasterfrom
ci-netlist-diff

Conversation

@ducky64
Copy link
Copy Markdown
Collaborator

@ducky64 ducky64 commented Apr 20, 2026

No longer just rely on the diffs showing up, it'll now cause a CI failure if the generated netlists change without the reference files being updated.

Resolves #426

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds netlist “golden file” checking to the examples unit tests so CI fails when generated netlists change without updating committed references.

Changes:

  • Added examples/util.py::run_test_board() to compile a design and compare the generated .net against a committed .net.ref.
  • Updated many examples/test_*.py tests to use run_test_board() instead of only compiling.
  • Added multiple *.net.ref reference netlists and updated .gitignore to ignore generated examples/*/*.net.

Reviewed changes

Copilot reviewed 36 out of 89 changed files in this pull request and generated 36 comments.

Show a summary per file
File Description
examples/util.py Adds helper to compile boards and assert generated netlists match committed references.
examples/test_usb_uart.py Uses run_test_board() for netlist regression checking.
examples/test_usb_source_measure.py Uses run_test_board() for netlist regression checking.
examples/test_usb_key.py Uses run_test_board() for netlist regression checking.
examples/test_usb_fpga_programmer.py Uses run_test_board() for netlist regression checking.
examples/test_tofarray.py Uses run_test_board() for netlist regression checking.
examples/test_swd_debugger.py Uses run_test_board() for netlist regression checking (including PicoProbe).
examples/test_simon.py Uses run_test_board() for netlist regression checking.
examples/test_seven_segment.py Uses run_test_board() for netlist regression checking.
examples/test_robotowl.py Uses run_test_board() for netlist regression checking.
examples/test_robotdriver.py Uses run_test_board() for netlist regression checking.
examples/test_robotcrawler.py Uses run_test_board() for netlist regression checking.
examples/test_protected_charger.py Uses run_test_board() for netlist regression checking.
examples/test_pcbbot.py Uses run_test_board() for netlist regression checking.
examples/test_multimeter.py Uses run_test_board() for netlist regression checking.
examples/test_lora.py Uses run_test_board() for netlist regression checking.
examples/test_ledmatrix.py Uses run_test_board() for netlist regression checking.
examples/test_keyboard.py Uses run_test_board() for netlist regression checking.
examples/test_jd_keyswitch.py Uses run_test_board() for netlist regression checking.
examples/test_iot_thermal_camera.py Uses run_test_board() for netlist regression checking.
examples/test_iot_led_driver.py Uses run_test_board() for netlist regression checking.
examples/test_iot_knob.py Uses run_test_board() for netlist regression checking.
examples/test_iot_iron.py Uses run_test_board() for netlist regression checking.
examples/test_iot_fan.py Uses run_test_board() for netlist regression checking.
examples/test_iot_display.py Uses run_test_board() for netlist regression checking.
examples/test_iot_blinds.py Uses run_test_board() for netlist regression checking (both designs).
examples/test_high_switch.py Uses run_test_board() for netlist regression checking.
examples/test_fcml.py Uses run_test_board() for netlist regression checking.
examples/test_esp_programmer.py Uses run_test_board() for netlist regression checking.
examples/test_deskcontroller.py Uses run_test_board() for netlist regression checking.
examples/test_datalogger.py Uses run_test_board() for netlist regression checking.
examples/test_can_adapter.py Uses run_test_board() for netlist regression checking.
examples/test_blinky.py Uses run_test_board() for multiple blinky-related designs.
examples/test_ble_joystick.py Uses run_test_board() for netlist regression checking.
examples/test_bldc_controller.py Uses run_test_board() for netlist regression checking.
examples/test_basickeyboard.py Uses run_test_board() for netlist regression checking.
examples/UsbUart/UsbUart.net.ref Adds golden reference netlist for regression comparison.
examples/UsbKey/UsbKey.net.ref Adds golden reference netlist for regression comparison.
examples/TestLed/TestLed.net.ref Adds golden reference netlist for regression comparison.
examples/TestBlinkyPacked/TestBlinkyPacked.net.ref Adds golden reference netlist for regression comparison.
examples/TestBlinkyMicro/TestBlinkyMicro.net.ref Adds golden reference netlist for regression comparison.
examples/TestBlinkyImplicit/TestBlinkyImplicit.net.ref Adds golden reference netlist for regression comparison.
examples/TestBlinkyExpanded/TestBlinkyExpanded.net.ref Adds golden reference netlist for regression comparison.
examples/TestBlinkyEmpty/TestBlinkyEmpty.net.ref Adds golden reference netlist for regression comparison.
examples/TestBlinkyComplete/TestBlinkyComplete.net.ref Adds golden reference netlist for regression comparison.
examples/TestBlinkyBasicBattery/TestBlinkyBasicBattery.net.ref Adds golden reference netlist for regression comparison.
examples/TestBlinkyBasic/TestBlinkyBasic.net.ref Adds golden reference netlist for regression comparison.
examples/TestBlinkyArray/TestBlinkyArray.net.ref Adds golden reference netlist for regression comparison.
examples/ProtectedCharger/ProtectedCharger.net.ref Adds golden reference netlist for regression comparison.
examples/EspProgrammer/EspProgrammer.net.ref Adds golden reference netlist for regression comparison.
examples/BasicKeyboard/BasicKeyboard.net.ref Adds golden reference netlist for regression comparison.
.gitignore Ignores generated examples/*/*.net outputs while keeping *.net.ref committed.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

from typing_extensions import override

from edg import *
from .util import run_test_board
Copy link

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This relative import will fail when the test module is executed as a top-level module (no parent package), e.g. the CI job that does cd examples && python -m unittest test_blinky / python test_blinky.py. Use an import that works in both contexts (eg try importing from .util and fall back to util), or adjust the test runner to import examples.test_* from the repo root.

Copilot uses AI. Check for mistakes.
from typing_extensions import override

from edg import *
from .util import run_test_board
Copy link

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This relative import will fail when the test module is executed as a top-level module (no parent package), e.g. the CI job that does cd examples && python -m unittest test_blinky / python test_blinky.py. Use an import that works in both contexts (eg try importing from .util and fall back to util), or adjust the test runner to import examples.test_* from the repo root.

Copilot uses AI. Check for mistakes.
from typing_extensions import override

from edg import *
from .util import run_test_board
Copy link

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This relative import will fail when the test module is executed as a top-level module (no parent package), e.g. the CI job that does cd examples && python -m unittest test_blinky / python test_blinky.py. Use an import that works in both contexts (eg try importing from .util and fall back to util), or adjust the test runner to import examples.test_* from the repo root.

Copilot uses AI. Check for mistakes.
from typing_extensions import override

from edg import *
from .util import run_test_board
Copy link

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This relative import will fail when the test module is executed as a top-level module (no parent package), e.g. the CI job that does cd examples && python -m unittest test_blinky / python test_blinky.py. Use an import that works in both contexts (eg try importing from .util and fall back to util), or adjust the test runner to import examples.test_* from the repo root.

Copilot uses AI. Check for mistakes.
from typing_extensions import override

from edg import *
from .util import run_test_board
Copy link

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This relative import will fail when the test module is executed as a top-level module (no parent package), e.g. the CI job that does cd examples && python -m unittest test_blinky / python test_blinky.py. Use an import that works in both contexts (eg try importing from .util and fall back to util), or adjust the test runner to import examples.test_* from the repo root.

Copilot uses AI. Check for mistakes.
Comment thread examples/test_iot_iron.py
from typing_extensions import override

from edg import *
from .util import run_test_board
Copy link

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This relative import will fail when the test module is executed as a top-level module (no parent package), e.g. the CI job that does cd examples && python -m unittest test_blinky / python test_blinky.py. Use an import that works in both contexts (eg try importing from .util and fall back to util), or adjust the test runner to import examples.test_* from the repo root.

Copilot uses AI. Check for mistakes.
from typing_extensions import override

from edg import *
from .util import run_test_board
Copy link

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This relative import will fail when the test module is executed as a top-level module (no parent package), e.g. the CI job that does cd examples && python -m unittest test_blinky / python test_blinky.py. Use an import that works in both contexts (eg try importing from .util and fall back to util), or adjust the test runner to import examples.test_* from the repo root.

Copilot uses AI. Check for mistakes.
from typing_extensions import override

from edg import *
from .util import run_test_board
Copy link

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This relative import will fail when the test module is executed as a top-level module (no parent package), e.g. the CI job that does cd examples && python -m unittest test_blinky / python test_blinky.py. Use an import that works in both contexts (eg try importing from .util and fall back to util), or adjust the test runner to import examples.test_* from the repo root.

Copilot uses AI. Check for mistakes.
from typing_extensions import override

from edg import *
from .util import run_test_board
Copy link

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This relative import will fail when the test module is executed as a top-level module (no parent package), e.g. the CI job that does cd examples && python -m unittest test_blinky / python test_blinky.py. Use an import that works in both contexts (eg try importing from .util and fall back to util), or adjust the test runner to import examples.test_* from the repo root.

Copilot uses AI. Check for mistakes.
from typing_extensions import override

from edg import *
from .util import run_test_board
Copy link

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This relative import will fail when the test module is executed as a top-level module (no parent package), e.g. the CI job that does cd examples && python -m unittest test_blinky / python test_blinky.py. Use an import that works in both contexts (eg try importing from .util and fall back to util), or adjust the test runner to import examples.test_* from the repo root.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 37 out of 90 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread examples/util.py
Comment on lines +21 to +23
assert (
generated_netlist == reference_netlist
), f"netlist differs from reference for {design.__name__}, if this is expected you may need to update the reference"
Copy link

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

run_test_board uses a bare assert for the test check. Python can skip asserts when run with -O, which would make these tests silently stop validating netlist diffs. Prefer raising AssertionError explicitly (or accepting a unittest.TestCase and calling self.assertEqual).

Copilot uses AI. Check for mistakes.
Comment thread .github/workflows/pr-python.yml Outdated
Comment on lines +80 to +83
- name: unittest
run: cd examples && python -m unittest test_blinky
run: python -m unittest examples.test_blinky
- name: toptest
run: cd examples && python test_blinky.py
run: python blinky_skeleton.py
Copy link

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this packaging smoke-test job, running python -m unittest examples.test_blinky and python blinky_skeleton.py from the repo root means import edg can be satisfied by the in-tree ./edg package (because the working directory is on sys.path), which weakens the goal of validating the installed distribution. Consider running the import/smoke test from a directory outside the checkout (eg cd /tmp) or otherwise ensuring the checkout isn’t on sys.path for these steps.

Copilot uses AI. Check for mistakes.
Comment thread examples/util.py
Comment on lines +12 to +14
designfile = inspect.getfile(design)
with open(os.path.join(os.path.dirname(designfile), design.__name__, design.__name__ + ".net"), newline=None) as f:
generated_netlist = f.read()
Copy link

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This new file is not Black-formatted: the long open(os.path.join(... line will be reformatted by black --check and cause CI failure. Please run Black (or wrap that path construction) so black --check . passes.

Copilot uses AI. Check for mistakes.
@ducky64 ducky64 merged commit 7ba5a03 into master Apr 20, 2026
12 checks passed
@ducky64 ducky64 deleted the ci-netlist-diff branch April 20, 2026 04:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enforce netlist diff as part of CI

2 participants