perf(mapping): pipe minimap2 straight into samtools sort (removes ~1.3 GiB of disk traffic per mapping)

[sanjaynagi-eit]

Problem

Every depth calculation does:

minimap_long_reads(reads, fasta, sam_file, ...)   # minimap2 -a ... > combined_long.sam
sam_to_sorted_bam(sam_file, sorted_bam, ...)      # samtools sort combined_long.sam -o ...

The SAM has exactly one consumer. It is written to disk in full, uncompressed, purely so that the next process can read it straight back.

On one real ONT isolate that is 0.64 GiB written and 0.64 GiB re-read per mapping, and plassembler run does it twice (long and short reads). Nextflow traces from a 90-isolate plate show PLASSEMBLER_LONG doing 2.2-2.7 GB of writes and 3.4-4.2 GB of reads per sample; a large part of that is this pattern.

Change

Add ExternalTool.run_piped, a shell-free pipeline runner that:

• connects each stage's stdout to the next stage's stdin;
• closes the parent's copy of every upstream read end, so downstream stages actually see EOF;
• waits on every stage, downstream-first, so an upstream process blocked on a dead pipe gets its SIGPIPE;
• raises CalledProcessError for the earliest failing stage, which is the informative one;
• supports a last stage that writes its own file (samtools sort -o) as well as one whose stdout is captured.

On top of it, minimap_long_reads_to_sorted_bam / minimap_short_reads_to_sorted_bam, used at the six map-then-sort sites in plass_class.py.

Also factors the duplicated --pacbio_model -> minimap2 preset mapping into minimap2_model_for().

Equivalence

Comparing the old two-step and the new piped form on the same reads and reference:

• samtools view alignment records: byte-for-byte identical
• samtools depth output: same md5
• the only difference is the @PG header line, which records the command line that produced the file

Wall clock

On a fast local NVMe with a warm page cache, 20.4 s -> 20.0 s. The change is about I/O volume and peak disk footprint rather than CPU: the benefit shows up when many isolates run concurrently (as in a Nextflow plate) and the page cache is contended, or on network-backed storage.

Scope

minimap_long_reads / minimap_short_reads and sam_to_sorted_bam are unchanged and still used: the hybrid path's long_read.sam has two consumers, so it needs a different treatment and is out of scope here.

Tests

New tests/test_piped_tools.py covers stdout capture, a last stage writing its own file, failure propagation, and that the earliest failing stage is the one reported. All existing tests pass.