Check netlist diffs as part of unit tests#478
Conversation
There was a problem hiding this comment.
Pull request overview
Adds netlist “golden file” checking to the examples unit tests so CI fails when generated netlists change without updating committed references.
Changes:
- Added
examples/util.py::run_test_board()to compile a design and compare the generated.netagainst a committed.net.ref. - Updated many
examples/test_*.pytests to userun_test_board()instead of only compiling. - Added multiple
*.net.refreference netlists and updated.gitignoreto ignore generatedexamples/*/*.net.
Reviewed changes
Copilot reviewed 36 out of 89 changed files in this pull request and generated 36 comments.
Show a summary per file
| File | Description |
|---|---|
| examples/util.py | Adds helper to compile boards and assert generated netlists match committed references. |
| examples/test_usb_uart.py | Uses run_test_board() for netlist regression checking. |
| examples/test_usb_source_measure.py | Uses run_test_board() for netlist regression checking. |
| examples/test_usb_key.py | Uses run_test_board() for netlist regression checking. |
| examples/test_usb_fpga_programmer.py | Uses run_test_board() for netlist regression checking. |
| examples/test_tofarray.py | Uses run_test_board() for netlist regression checking. |
| examples/test_swd_debugger.py | Uses run_test_board() for netlist regression checking (including PicoProbe). |
| examples/test_simon.py | Uses run_test_board() for netlist regression checking. |
| examples/test_seven_segment.py | Uses run_test_board() for netlist regression checking. |
| examples/test_robotowl.py | Uses run_test_board() for netlist regression checking. |
| examples/test_robotdriver.py | Uses run_test_board() for netlist regression checking. |
| examples/test_robotcrawler.py | Uses run_test_board() for netlist regression checking. |
| examples/test_protected_charger.py | Uses run_test_board() for netlist regression checking. |
| examples/test_pcbbot.py | Uses run_test_board() for netlist regression checking. |
| examples/test_multimeter.py | Uses run_test_board() for netlist regression checking. |
| examples/test_lora.py | Uses run_test_board() for netlist regression checking. |
| examples/test_ledmatrix.py | Uses run_test_board() for netlist regression checking. |
| examples/test_keyboard.py | Uses run_test_board() for netlist regression checking. |
| examples/test_jd_keyswitch.py | Uses run_test_board() for netlist regression checking. |
| examples/test_iot_thermal_camera.py | Uses run_test_board() for netlist regression checking. |
| examples/test_iot_led_driver.py | Uses run_test_board() for netlist regression checking. |
| examples/test_iot_knob.py | Uses run_test_board() for netlist regression checking. |
| examples/test_iot_iron.py | Uses run_test_board() for netlist regression checking. |
| examples/test_iot_fan.py | Uses run_test_board() for netlist regression checking. |
| examples/test_iot_display.py | Uses run_test_board() for netlist regression checking. |
| examples/test_iot_blinds.py | Uses run_test_board() for netlist regression checking (both designs). |
| examples/test_high_switch.py | Uses run_test_board() for netlist regression checking. |
| examples/test_fcml.py | Uses run_test_board() for netlist regression checking. |
| examples/test_esp_programmer.py | Uses run_test_board() for netlist regression checking. |
| examples/test_deskcontroller.py | Uses run_test_board() for netlist regression checking. |
| examples/test_datalogger.py | Uses run_test_board() for netlist regression checking. |
| examples/test_can_adapter.py | Uses run_test_board() for netlist regression checking. |
| examples/test_blinky.py | Uses run_test_board() for multiple blinky-related designs. |
| examples/test_ble_joystick.py | Uses run_test_board() for netlist regression checking. |
| examples/test_bldc_controller.py | Uses run_test_board() for netlist regression checking. |
| examples/test_basickeyboard.py | Uses run_test_board() for netlist regression checking. |
| examples/UsbUart/UsbUart.net.ref | Adds golden reference netlist for regression comparison. |
| examples/UsbKey/UsbKey.net.ref | Adds golden reference netlist for regression comparison. |
| examples/TestLed/TestLed.net.ref | Adds golden reference netlist for regression comparison. |
| examples/TestBlinkyPacked/TestBlinkyPacked.net.ref | Adds golden reference netlist for regression comparison. |
| examples/TestBlinkyMicro/TestBlinkyMicro.net.ref | Adds golden reference netlist for regression comparison. |
| examples/TestBlinkyImplicit/TestBlinkyImplicit.net.ref | Adds golden reference netlist for regression comparison. |
| examples/TestBlinkyExpanded/TestBlinkyExpanded.net.ref | Adds golden reference netlist for regression comparison. |
| examples/TestBlinkyEmpty/TestBlinkyEmpty.net.ref | Adds golden reference netlist for regression comparison. |
| examples/TestBlinkyComplete/TestBlinkyComplete.net.ref | Adds golden reference netlist for regression comparison. |
| examples/TestBlinkyBasicBattery/TestBlinkyBasicBattery.net.ref | Adds golden reference netlist for regression comparison. |
| examples/TestBlinkyBasic/TestBlinkyBasic.net.ref | Adds golden reference netlist for regression comparison. |
| examples/TestBlinkyArray/TestBlinkyArray.net.ref | Adds golden reference netlist for regression comparison. |
| examples/ProtectedCharger/ProtectedCharger.net.ref | Adds golden reference netlist for regression comparison. |
| examples/EspProgrammer/EspProgrammer.net.ref | Adds golden reference netlist for regression comparison. |
| examples/BasicKeyboard/BasicKeyboard.net.ref | Adds golden reference netlist for regression comparison. |
| .gitignore | Ignores generated examples/*/*.net outputs while keeping *.net.ref committed. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| from typing_extensions import override | ||
|
|
||
| from edg import * | ||
| from .util import run_test_board |
There was a problem hiding this comment.
This relative import will fail when the test module is executed as a top-level module (no parent package), e.g. the CI job that does cd examples && python -m unittest test_blinky / python test_blinky.py. Use an import that works in both contexts (eg try importing from .util and fall back to util), or adjust the test runner to import examples.test_* from the repo root.
| from typing_extensions import override | ||
|
|
||
| from edg import * | ||
| from .util import run_test_board |
There was a problem hiding this comment.
This relative import will fail when the test module is executed as a top-level module (no parent package), e.g. the CI job that does cd examples && python -m unittest test_blinky / python test_blinky.py. Use an import that works in both contexts (eg try importing from .util and fall back to util), or adjust the test runner to import examples.test_* from the repo root.
| from typing_extensions import override | ||
|
|
||
| from edg import * | ||
| from .util import run_test_board |
There was a problem hiding this comment.
This relative import will fail when the test module is executed as a top-level module (no parent package), e.g. the CI job that does cd examples && python -m unittest test_blinky / python test_blinky.py. Use an import that works in both contexts (eg try importing from .util and fall back to util), or adjust the test runner to import examples.test_* from the repo root.
| from typing_extensions import override | ||
|
|
||
| from edg import * | ||
| from .util import run_test_board |
There was a problem hiding this comment.
This relative import will fail when the test module is executed as a top-level module (no parent package), e.g. the CI job that does cd examples && python -m unittest test_blinky / python test_blinky.py. Use an import that works in both contexts (eg try importing from .util and fall back to util), or adjust the test runner to import examples.test_* from the repo root.
| from typing_extensions import override | ||
|
|
||
| from edg import * | ||
| from .util import run_test_board |
There was a problem hiding this comment.
This relative import will fail when the test module is executed as a top-level module (no parent package), e.g. the CI job that does cd examples && python -m unittest test_blinky / python test_blinky.py. Use an import that works in both contexts (eg try importing from .util and fall back to util), or adjust the test runner to import examples.test_* from the repo root.
| from typing_extensions import override | ||
|
|
||
| from edg import * | ||
| from .util import run_test_board |
There was a problem hiding this comment.
This relative import will fail when the test module is executed as a top-level module (no parent package), e.g. the CI job that does cd examples && python -m unittest test_blinky / python test_blinky.py. Use an import that works in both contexts (eg try importing from .util and fall back to util), or adjust the test runner to import examples.test_* from the repo root.
| from typing_extensions import override | ||
|
|
||
| from edg import * | ||
| from .util import run_test_board |
There was a problem hiding this comment.
This relative import will fail when the test module is executed as a top-level module (no parent package), e.g. the CI job that does cd examples && python -m unittest test_blinky / python test_blinky.py. Use an import that works in both contexts (eg try importing from .util and fall back to util), or adjust the test runner to import examples.test_* from the repo root.
| from typing_extensions import override | ||
|
|
||
| from edg import * | ||
| from .util import run_test_board |
There was a problem hiding this comment.
This relative import will fail when the test module is executed as a top-level module (no parent package), e.g. the CI job that does cd examples && python -m unittest test_blinky / python test_blinky.py. Use an import that works in both contexts (eg try importing from .util and fall back to util), or adjust the test runner to import examples.test_* from the repo root.
| from typing_extensions import override | ||
|
|
||
| from edg import * | ||
| from .util import run_test_board |
There was a problem hiding this comment.
This relative import will fail when the test module is executed as a top-level module (no parent package), e.g. the CI job that does cd examples && python -m unittest test_blinky / python test_blinky.py. Use an import that works in both contexts (eg try importing from .util and fall back to util), or adjust the test runner to import examples.test_* from the repo root.
| from typing_extensions import override | ||
|
|
||
| from edg import * | ||
| from .util import run_test_board |
There was a problem hiding this comment.
This relative import will fail when the test module is executed as a top-level module (no parent package), e.g. the CI job that does cd examples && python -m unittest test_blinky / python test_blinky.py. Use an import that works in both contexts (eg try importing from .util and fall back to util), or adjust the test runner to import examples.test_* from the repo root.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 37 out of 90 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| assert ( | ||
| generated_netlist == reference_netlist | ||
| ), f"netlist differs from reference for {design.__name__}, if this is expected you may need to update the reference" |
There was a problem hiding this comment.
run_test_board uses a bare assert for the test check. Python can skip asserts when run with -O, which would make these tests silently stop validating netlist diffs. Prefer raising AssertionError explicitly (or accepting a unittest.TestCase and calling self.assertEqual).
| - name: unittest | ||
| run: cd examples && python -m unittest test_blinky | ||
| run: python -m unittest examples.test_blinky | ||
| - name: toptest | ||
| run: cd examples && python test_blinky.py | ||
| run: python blinky_skeleton.py |
There was a problem hiding this comment.
In this packaging smoke-test job, running python -m unittest examples.test_blinky and python blinky_skeleton.py from the repo root means import edg can be satisfied by the in-tree ./edg package (because the working directory is on sys.path), which weakens the goal of validating the installed distribution. Consider running the import/smoke test from a directory outside the checkout (eg cd /tmp) or otherwise ensuring the checkout isn’t on sys.path for these steps.
| designfile = inspect.getfile(design) | ||
| with open(os.path.join(os.path.dirname(designfile), design.__name__, design.__name__ + ".net"), newline=None) as f: | ||
| generated_netlist = f.read() |
There was a problem hiding this comment.
This new file is not Black-formatted: the long open(os.path.join(... line will be reformatted by black --check and cause CI failure. Please run Black (or wrap that path construction) so black --check . passes.
No longer just rely on the diffs showing up, it'll now cause a CI failure if the generated netlists change without the reference files being updated.
Resolves #426