CLI flags/args added via plugin not behaving as expected

[dbarbuzzi]

I believe there may be an issue with extending command-line args via pytest_load_initial_conftests in a plugin, or perhaps I am lacking some understanding of how different things work and I should be adding different flags with a different approach.

I currently have a plugin our team uses which dynamically generates the flag for JUnit report files. It works by defining the pytest_load_initial_conftests hook and using the args[:] = [*args, *new_args] pattern (where new_args is a list[str] containing the dynamically generated flags. In this trivial example, the list looks something like ['--junit-xml=test-results/junit-report-1739919926.824418.xml'].

I have also added some features to this plugin to add some additional CLI flags and, with everything enabled, new_args looks like this:

[
  '--junit-xml=test-results/junit-report-1739919926.824418.xml',
  '-o',
  'junit_logging=all',
  '-o',
  'junit_log_passing_tests=True',
  '--cov=MODULE_NAME',
  '--cov-append',
  '--cov-report=html:coverage-html',
  '--cov-report=json:coverage.json'
]

However, none of the other flags result in their anticipated effects:

• the JUnit report does not have all the expected output captured
• nothing is being captured for passing tests
• no coverage is generated despite having coverage and pytest-cov installed).

I have confirmed the contents via debugging the plugin when used from an editable install in a separate project. In this separate project, I also have a test to debug which uses the request fixture and prints out request.config.option (minimal example test below). In this Namespace, I can see what appears to be the various options configured as expected, though they do not "work":

...
cov_source=['MODULE_NAME'],
cov_report={'html': 'coverage-html', 'json': 'coverage.json'},
cov_append=True,
...
override_ini=['junit_logging=all', 'junit_log_passing_tests=True'],
...

For reference, the plugin is open-source here: https://github.com/neuralmagic/pytest-nm-releng

A simple set of test files I am using locally to test it (the unit tests all work to the best of my capability with regards to testing the plugin) – will not include any of the pytest-cov flags by design:

# test file: test_example.py
# requirements: pytest, pytest-nm-releng
import pytest

def test_example(request: pytest.FixtureRequest):
    print(request.config.option)

# running with flags added via plugin - report naming works, rest doesn't
NMRE_JUNIT_BASE="test-results" NMRE_JUNIT_PREFIX="junit-report-" NMRE_JUNIT_FULL="1" pytest test_example.py

# running with flags added via PYTEST_ADDOPTS - work as expected
PYTEST_ADDOPTS='--junit-xml test-results/report-pytest_addopts.xml -o junit_logging=all -o junit_log_passing_tests=True' pytest test_example.py

# running with flags passed directly - works as expected
pytest test_example.py --junit-xml test-results/report-direct.xml -o junit_logging=all -o junit_log_passing_tests=True

[portyu9]

Hi @dbarbuzzi

The issue is when those arguments are being added.

pytest_load_initial_conftests can modify args, but not every option can safely be introduced there. In particular:

• -o junit_logging=... / -o junit_log_passing_tests=... are too late because pytest has already processed the ini overrides before this hook finishes.
• pytest-cov also uses pytest_load_initial_conftests and declares it with tryfirst=True, so adding --cov... from another implementation of the same hook can happen after pytest-cov has already inspected the early options.

That explains why the values can appear later in request.config.option even though the corresponding plugin behavior was never initialized from them.

I would not solve this with tryfirst=True: that might change ordering relative to pytest-cov, but it still does not fix the -o options.

For dynamically generated defaults that must behave exactly like real command-line options, generate them before pytest starts configuration, for example with a small wrapper entry point:

import sys

import pytest


def main():
    generated_args = [
        "--junit-xml=test-results/report.xml",
        "-o",
        "junit_logging=all",
        "-o",
        "junit_log_passing_tests=True",
        "--cov=MODULE_NAME",
    ]

    raise SystemExit(pytest.main([*generated_args, *sys.argv[1:]]))

Another valid option is to generate PYTEST_ADDOPTS before launching pytest. Static values can of course go directly into the pytest configuration file.

So the argument mutation itself is not generally broken; these particular options depend on startup work that has already happened by the time your hook adds them.

Please mark this answer as accepted if it helped, thank you.

[dbarbuzzi]

Thanks for this answer, I had forgotten about this and ultimately went with generating flags prior to pytest being invoked.