Skip to content

perf(long): pipe minimap2 straight into the plasmid FASTQ extraction (removes the last SAM in long mode) - #8

Open
sanjaynagi-eit wants to merge 1 commit into
perf/pipe-minimap-to-sortfrom
perf/pipe-long-read-extraction
Open

perf(long): pipe minimap2 straight into the plasmid FASTQ extraction (removes the last SAM in long mode)#8
sanjaynagi-eit wants to merge 1 commit into
perf/pipe-minimap-to-sortfrom
perf/pipe-long-read-extraction

Conversation

@sanjaynagi-eit

Copy link
Copy Markdown
Owner

Stacked on #3 — it uses ExternalTool.run_piped from that PR, so it is based on perf/pipe-minimap-to-sort rather than main. Review #3 first.

Problem

#3 removed the intermediate SAM from the depth mappings. This is the other one.

In long-only mode:

samfile = Path(outdir) / "long_read.sam"
minimap_long_reads(input_long_reads, fasta, samfile, ...)   # writes the whole SAM
...
extract_long_fastqs_fast(samfile, plasmidfastqs, threads)   # samtools view | awk

long_read.sam has exactly one consumer — the samtools view | awk filter that pulls out plasmid-mapped and unmapped reads. minimap2's full uncompressed SAM is written to disk purely so samtools can read it straight back: 0.64 GiB written and re-read for a real ONT isolate, on top of the 0.64 GiB #3 already avoids in the depth mapping.

Change

map_and_extract_long_fastqs runs minimap2 | samtools view | awk as one pipeline through ExternalTool.run_piped, so nothing lands on disk between the stages.

two-step (SAM to disk) 19.7 s
piped 18.7 s
intermediate SAM avoided 0.64 GiB written + re-read

The plasmid FASTQ is byte-identical to the two-step form.

Scope

Hybrid mode is untouched. There long_read.sam is also read by sam_to_bam for split_bams, so it genuinely has two consumers and still has to exist. Only the long-only call site changes.

The awk program moves into a PLASMID_READ_AWK constant shared by the piped and the existing file-based paths. The command extract_long_fastqs_fast builds is unchanged — verified by string comparison against the original literal, since it is easy to shift a backslash while refactoring an embedded awk script.

Tests

New tests/test_long_extraction.py: the piped output hashes equal to the two-step output on real reads, no .sam is left on disk, the output is well-formed FASTQ with matching sequence and quality lengths, and the awk filter still selects flags 0/16 on plas contigs plus flag 4. All existing tests pass (77 non-slow).

@sanjaynagi-eit
sanjaynagi-eit force-pushed the perf/pipe-minimap-to-sort branch from bdc474e to d76ab2c Compare August 13, 2026 22:18
In long-only mode long_read.sam has exactly one consumer: the
samtools view | awk filter that pulls out plasmid-mapped and unmapped reads.
minimap2's full uncompressed SAM was written to disk purely so that samtools
could read it straight back - 0.64 GiB written and re-read for a real ONT
isolate, on top of the 0.64 GiB already avoided in the depth mapping.

map_and_extract_long_fastqs runs minimap2 | samtools view | awk as one
pipeline via ExternalTool.run_piped, so nothing lands on disk between them.

  two-step (SAM to disk)   19.7s
  piped                    18.7s
  intermediate SAM         0.64 GiB written + re-read, now avoided

The plasmid fastq is byte-identical.

The awk program moves to a PLASMID_READ_AWK constant shared by both the piped
and the existing file-based paths; the command extract_long_fastqs_fast builds
is unchanged, verified by string comparison against the original literal.

Hybrid mode is untouched: there long_read.sam is also read by sam_to_bam, so
it still has two consumers and still needs to exist.
@sanjaynagi-eit
sanjaynagi-eit force-pushed the perf/pipe-long-read-extraction branch from e39f3fb to fdd2f2f Compare August 13, 2026 22:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant