perf(qc): compress chopper output with bgzip (8x faster QC stage) + wait on every pipeline stage

[sanjaynagi-eit]

Problem

chopper() builds the chain gunzip -c reads.gz | chopper --threads N | gzip. Every stage is parallel-aware except the last one, and gzip dominated the stage completely.

Measured on a real 300 MB ONT fastq, 8 threads:

chain	time	output
gunzip | chopper (no compression)	5.4 s	-
... | gzip (current)	47.7 s	293.4 MiB
... | bgzip -@ 8	5.9 s	276.7 MiB

So ~42 of the 47.7 seconds were serial gzip. On the same isolate the whole plassembler long run is 499 s, making this ~8% of total wall clock for free.

Change

Use bgzip -@ threads as the compressor.

• No new dependency: bgzip ships with htslib/samtools, which plassembler already requires and already checks for in check_dependencies.
• No format change: BGZF is a valid gzip stream. flye, minimap2, chopper, gunzip and python's gzip module all read it unchanged. chopper_long_reads.fastq.gz is in any case a purely internal intermediate that remove_intermediate_files deletes.
• Falls back to plain gzip if bgzip is somehow absent.
• Here it also happens to be smaller (277 vs 293 MiB).

Verified content-identical: with chopper --threads 1 the decompressed output of the old and new chains is byte-for-byte equal. (Above 1 thread chopper does not preserve read order, so only the single-threaded comparison is meaningful — that is pre-existing chopper behaviour, not something this PR changes.)

gzip_file(), used by --skip_qc, went through python's gzip module — slower still — and now uses the same compressor with a python fallback.

Bug fix in the same function

Only gzip_proc was ever waited on. source_proc and chopper_proc were never wait()ed and the parent never closed its copies of their pipe read ends, so:

• a failing gunzip or chopper was silently swallowed (the bare except: never saw it, because the exception never happened — the process just exited non-zero), leaving an empty or truncated read file that the rest of the run happily consumed;
• zombie processes accumulated.

Every stage is now waited on and a non-zero exit is reported with the stage name and the log path. test_chopper_reports_a_failing_stage covers this: feeding plain fastq with gzip_flag=True used to pass silently and now raises.

The uncompressed-input branch also no longer spawns a cat process just to copy the file into a pipe — the file is handed to chopper's stdin directly.

Tests

New tests/test_qc.py: compressor selection and fallback, gzip_file round-trip through python's gzip reader, chopper output well-formedness for both gzipped and plain input, and the failing-stage regression above. All existing tests still pass.

[sanjaynagi-eit]

Behaviour change worth calling out explicitly, found while testing this against real data.

The process-handling fix in this PR is not only cosmetic. I had a staged ONT FASTQ that turned out to be a truncated gzip file (gzip -t exits 1). Running it through both versions:

	exit code	outcome
before	0	silently assembled a 2,781,657 bp "chromosome" and a 20,403 bp "plasmid" from a partial read set, and wrote a normal-looking plassembler_summary.tsv
after	1	ERROR - Error with chopper: gunzip (return code 1), chopper (return code 101)

gunzip exits 1 on the truncated stream and chopper exits 101 — but since only the final gzip was ever wait()ed, neither was noticed, and the run continued on whatever had been decompressed before the file ran out.

So this changes corrupt input from "quietly produces a plausible-looking wrong answer" to "fails". I think that is clearly correct, but it is a real behaviour change: batches containing corrupt reads that previously appeared to succeed will now report failures.

[sanjaynagi-eit]

Follow-up: this PR surfaced a pre-existing fault in the repository's own test data.

Running the full suite (including slow) against this branch, exactly one test failed:

tests/test_end_to_end.py::test_plassembler_case_depth_filter_some
  ERROR - Error with chopper: chopper (return code 101)

tests/test_data/end_to_end/input_half.fastq.gz is 778 lines — 194.5 FASTQ records. It ends mid-record: a header and a sequence line, with no + or quality line. chopper rejects it (IncompleteRecord) and exits 101 after emitting 193 complete reads.

That has presumably been true since the fixture was committed. The test passed only because the chopper failure was swallowed — which is exactly the bug this PR fixes.

I have repaired the fixture by dropping the two orphan lines. chopper then exits 0 and emits the same 193 records, so nothing the test exercises changes, and that test asserts only that the command exits 0 — no expectation depends on the fixture's exact contents.

Full suite: 169 passed / 1 failed before, 170 passed after.

If you would rather keep the fixture as-is and handle malformed input differently (warn-and-continue rather than fail), that is a reasonable alternative policy — but it should be a deliberate choice, not the accident of an unwaited-on process. Happy to switch it if you prefer.