WorkflowOperator can't run tests that execute generated Python, and spawns an interpreter per operator

[kz930]

Task Summary

The amber job runs the WorkflowOperator module's tests and installs no Python packages. A test that executes a generated operator template needs pandas and plotly, fails its dependency probe, and calls cancel(...) — which is neither a pass nor a failure, so the suite still reports "All tests passed".

PR #7149 hits this: of its ten new tests, the only one that actually executes the generated guard is the one that cancels.

Commit	Tests run	Canceled	Verdict
main (ba63cedf)	2016	0	All tests passed
#7149 (e9ac8bad)	2025	1	All tests passed

Fix — mirror the split amber already uses. amber/build.sbt reads AMBER_TEST_FILTER: the amber job sets it to skip-integration and excludes @IntegrationTest specs, while amber-integration sets it to integration-only, runs just those, and installs amber/requirements.txt and amber/operator-requirements.txt (pandas 2.2.3, plotly 5.24.1). WorkflowOperator has none of this wiring, so a Python-executing test there has nowhere to run. Three steps:

1. Add an IntegrationTest tag annotation under common/workflow-operator/src/test. amber's tag can't be reused — it lives in amber/src/test/integration, and amber depends on WorkflowOperator, not the reverse.
2. Add a Test / testOptions filter to common/workflow-operator/build.sbt reading the same AMBER_TEST_FILTER — -l <tag> on skip-integration, -n <tag> on integration-only. The amber job already sets that variable in the step that invokes WorkflowOperator/jacoco, so no workflow change is needed there. Tests opt in per-case with taggedAs, keeping a spec's stdlib-only assertions in the unit job.
3. Add "WorkflowOperator/test" to the amber-integration sbt invocation, which already runs integration-only with the Python dependencies installed.

Since the selection logic would then exist in two modules, it can live in project/ — the build already keeps shared sbt logic there (AddMetaInfLicenseFiles.scala, JdkOptions.scala) — with each module passing its own env var and tag.

Net effect: a Python-executing test stops cancelling in the unit job and starts executing in the integration job.

Second problem: one spawn per operator

Even where a Python-executing test can run, testing operators one at a time does not scale. The py_compile check in PythonCodeRawInvalidTextSpec spawns python -I -S -B -m py_compile once per PythonOperatorDescriptor — 117 of them today, serially — where the interpreter boot is the entire cost and the compile itself is under a millisecond. It grows linearly with the operator count, and the tests this module is about to gain are far more expensive: a spawn that imports pandas and plotly costs 260-310 ms, roughly 96 % of a job whose real work is ~4 ms.

Fix — a long-lived worker and a bounded pool. A worker process pays the boot and the imports once at startup, then serves many jobs over a line-delimited JSON protocol, so N spawns become one. Jobs go to at most a fixed number of workers at a time, matching the -P bound ScalaTest is given on the integration side so the two concurrency limits agree rather than multiply. A hard worker crash falls back to the one-shot spawn, so behavior is never worse than today, and an env var selects the old path outright for debugging.

This mechanism is not new — it is described in #6975, where it was built for a suite that runs every operator's generated code. What this issue asks for is making it available to this module instead of each test reinventing a driver, a protocol and a timeout of its own.

On sequencing: the wiring can land on its own. #7207 carries it together with the pool and the first tagged case — an assertion that pandas and plotly import in the resolved interpreter, which both exercises the routing and turns a missing install in amber-integration into a failure instead of a cancellation. The runtime test in #7149, the case cancelling today, then only needs the tag.

Task Type

• DevOps / Deployment / CI
• Testing / QA