perf(long): pipe minimap2 straight into the plasmid FASTQ extraction (removes the last SAM in long mode) - #8
Open
sanjaynagi-eit wants to merge 1 commit into
Conversation
sanjaynagi-eit
force-pushed
the
perf/pipe-minimap-to-sort
branch
from
August 13, 2026 22:18
bdc474e to
d76ab2c
Compare
In long-only mode long_read.sam has exactly one consumer: the samtools view | awk filter that pulls out plasmid-mapped and unmapped reads. minimap2's full uncompressed SAM was written to disk purely so that samtools could read it straight back - 0.64 GiB written and re-read for a real ONT isolate, on top of the 0.64 GiB already avoided in the depth mapping. map_and_extract_long_fastqs runs minimap2 | samtools view | awk as one pipeline via ExternalTool.run_piped, so nothing lands on disk between them. two-step (SAM to disk) 19.7s piped 18.7s intermediate SAM 0.64 GiB written + re-read, now avoided The plasmid fastq is byte-identical. The awk program moves to a PLASMID_READ_AWK constant shared by both the piped and the existing file-based paths; the command extract_long_fastqs_fast builds is unchanged, verified by string comparison against the original literal. Hybrid mode is untouched: there long_read.sam is also read by sam_to_bam, so it still has two consumers and still needs to exist.
sanjaynagi-eit
force-pushed
the
perf/pipe-long-read-extraction
branch
from
August 13, 2026 22:21
e39f3fb to
fdd2f2f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #3 — it uses
ExternalTool.run_pipedfrom that PR, so it is based onperf/pipe-minimap-to-sortrather thanmain. Review #3 first.Problem
#3 removed the intermediate SAM from the depth mappings. This is the other one.
In long-only mode:
long_read.samhas exactly one consumer — thesamtools view | awkfilter that pulls out plasmid-mapped and unmapped reads. minimap2's full uncompressed SAM is written to disk purely so samtools can read it straight back: 0.64 GiB written and re-read for a real ONT isolate, on top of the 0.64 GiB #3 already avoids in the depth mapping.Change
map_and_extract_long_fastqsrunsminimap2 | samtools view | awkas one pipeline throughExternalTool.run_piped, so nothing lands on disk between the stages.The plasmid FASTQ is byte-identical to the two-step form.
Scope
Hybrid mode is untouched. There
long_read.samis also read bysam_to_bamforsplit_bams, so it genuinely has two consumers and still has to exist. Only the long-only call site changes.The awk program moves into a
PLASMID_READ_AWKconstant shared by the piped and the existing file-based paths. The commandextract_long_fastqs_fastbuilds is unchanged — verified by string comparison against the original literal, since it is easy to shift a backslash while refactoring an embedded awk script.Tests
New
tests/test_long_extraction.py: the piped output hashes equal to the two-step output on real reads, no.samis left on disk, the output is well-formed FASTQ with matching sequence and quality lengths, and the awk filter still selects flags 0/16 onplascontigs plus flag 4. All existing tests pass (77 non-slow).