perf(sam_to_fastq): take FASTQ fields from the raw SAM record (7.2x faster) - #7
Open
sanjaynagi-eit wants to merge 1 commit into
Open
perf(sam_to_fastq): take FASTQ fields from the raw SAM record (7.2x faster)#7sanjaynagi-eit wants to merge 1 commit into
sanjaynagi-eit wants to merge 1 commit into
Conversation
…ter) extract_long_fastqs_slow_keep_fastqs - the --keep_fastqs path, named for how it performs - built a python list holding one string per *alignment* just to count read names, then made two more full passes, and encoded every quality score with a python-level loop: ''.join(chr(q + 33) for q in quality). Profiling a real minimap2 sam showed the bookkeeping was not the problem. Parsing the whole sam takes 0.04s and either tallying scheme 0.05s; 2.08s of the 2.24s total went on read.query_sequence / read.query_qualities, which make pysam decode each record into python objects - for ONT reads a 60,000-element array of ints per read, immediately re-encoded back to phred+33. The SAM line already holds SEQ and QUAL as strings. Splitting them out of read.to_string() skips the decode entirely: seconds 2.23 -> 0.31 (7.2x) peak RSS 154 MB -> 151 MB All three output fastqs are byte-identical on a real 0.06 GiB sam. The read-name list is also gone: reads are tallied into a dict of small bitmasks, since the classification only ever asks 'more than one alignment?' and 'hit a plasmid/chromosome at all?', never for the counts. Every value is below 16, so CPython's small-int cache means those values are free. The singles-then-multimapped write order is preserved deliberately: writing in a single interleaved pass would be fewer passes but would reorder reads, and the resulting fastqs feed an assembler.
sanjaynagi-eit
force-pushed
the
perf/single-pass-sam-to-fastq
branch
from
August 13, 2026 21:46
90d9519 to
159b524
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
extract_long_fastqs_slow_keep_fastqs— the--keep_fastqspath, named for how it performs.What it did
"".join(chr(q + 33) for q in quality).What actually costs the time
Profiling a real minimap2 SAM (10,666 alignments of ONT reads, 0.06 GiB) says the bookkeeping is not the problem:
Nearly all of it is
read.query_sequence/read.query_qualities, which make pysam decode each record into python objects — for ONT reads that is a 60,000-element array of ints per read, immediately re-encoded back to phred+33 character by character. (Swapping the join forpysam.qualities_to_qualitystringdoes not help: 2.16 s. The array construction is the cost, not the encoding.)Change
The SAM line already holds SEQ and QUAL as strings, so split them out of
read.to_string()and skip the decode:All three output FASTQs are byte-identical.
The read-name list is gone too: reads are tallied into a dict of small bitmasks, because the classification only ever asks "more than one alignment?" and "hit a plasmid / a chromosome at all?", never for the counts themselves. Every value is below 16, so CPython's small-int cache makes the values free and the dict costs no more than its keys.
Write order preserved deliberately
Singles are still written before multimapped reads. Doing it in one interleaved pass would be fewer passes, but it reorders records within each FASTQ, and these files feed an assembler — I checked, and that reordering was the only difference in an earlier version of this patch.
Tests
New
tests/test_sam_to_fastq.pybuilds SAMs with pysam and covers: routing of singly-mapped and unmapped reads, a read hitting both a plasmid and the chromosome (written once, primary only), a read multimapping within one replicon, exact sequence/quality round-tripping, a reverse-strand record, and an empty SAM. All existing tests pass (75 non-slow).