Skip to content

perf(sam_to_fastq): take FASTQ fields from the raw SAM record (7.2x faster) - #7

Open
sanjaynagi-eit wants to merge 1 commit into
mainfrom
perf/single-pass-sam-to-fastq
Open

perf(sam_to_fastq): take FASTQ fields from the raw SAM record (7.2x faster)#7
sanjaynagi-eit wants to merge 1 commit into
mainfrom
perf/single-pass-sam-to-fastq

Conversation

@sanjaynagi-eit

Copy link
Copy Markdown
Owner

extract_long_fastqs_slow_keep_fastqs — the --keep_fastqs path, named for how it performs.

What it did

  • Built a python list holding one string per alignment purely to count read names, then counted it into a dict, then derived two sets from that.
  • Made three full passes over the SAM.
  • Encoded every quality score with a python-level loop: "".join(chr(q + 33) for q in quality).

What actually costs the time

Profiling a real minimap2 SAM (10,666 alignments of ONT reads, 0.06 GiB) says the bookkeeping is not the problem:

time
parse the whole SAM, touch nothing 0.04 s
build the read-name list (old pass 1) 0.05 s
build the tally dict (new pass 1) 0.05 s
a write pass 2.08 s

Nearly all of it is read.query_sequence / read.query_qualities, which make pysam decode each record into python objects — for ONT reads that is a 60,000-element array of ints per read, immediately re-encoded back to phred+33 character by character. (Swapping the join for pysam.qualities_to_qualitystring does not help: 2.16 s. The array construction is the cost, not the encoding.)

Change

The SAM line already holds SEQ and QUAL as strings, so split them out of read.to_string() and skip the decode:

before after
time 2.23 s 0.31 s 7.2x
peak RSS 154 MB 151 MB

All three output FASTQs are byte-identical.

The read-name list is gone too: reads are tallied into a dict of small bitmasks, because the classification only ever asks "more than one alignment?" and "hit a plasmid / a chromosome at all?", never for the counts themselves. Every value is below 16, so CPython's small-int cache makes the values free and the dict costs no more than its keys.

Write order preserved deliberately

Singles are still written before multimapped reads. Doing it in one interleaved pass would be fewer passes, but it reorders records within each FASTQ, and these files feed an assembler — I checked, and that reordering was the only difference in an earlier version of this patch.

Tests

New tests/test_sam_to_fastq.py builds SAMs with pysam and covers: routing of singly-mapped and unmapped reads, a read hitting both a plasmid and the chromosome (written once, primary only), a read multimapping within one replicon, exact sequence/quality round-tripping, a reverse-strand record, and an empty SAM. All existing tests pass (75 non-slow).

…ter)

extract_long_fastqs_slow_keep_fastqs - the --keep_fastqs path, named for how
it performs - built a python list holding one string per *alignment* just to
count read names, then made two more full passes, and encoded every quality
score with a python-level loop: ''.join(chr(q + 33) for q in quality).

Profiling a real minimap2 sam showed the bookkeeping was not the problem.
Parsing the whole sam takes 0.04s and either tallying scheme 0.05s; 2.08s of
the 2.24s total went on read.query_sequence / read.query_qualities, which make
pysam decode each record into python objects - for ONT reads a 60,000-element
array of ints per read, immediately re-encoded back to phred+33.

The SAM line already holds SEQ and QUAL as strings. Splitting them out of
read.to_string() skips the decode entirely:

  seconds       2.23 -> 0.31   (7.2x)
  peak RSS    154 MB -> 151 MB

All three output fastqs are byte-identical on a real 0.06 GiB sam.

The read-name list is also gone: reads are tallied into a dict of small
bitmasks, since the classification only ever asks 'more than one alignment?'
and 'hit a plasmid/chromosome at all?', never for the counts. Every value is
below 16, so CPython's small-int cache means those values are free.

The singles-then-multimapped write order is preserved deliberately: writing
in a single interleaved pass would be fewer passes but would reorder reads,
and the resulting fastqs feed an assembler.
@sanjaynagi-eit
sanjaynagi-eit force-pushed the perf/single-pass-sam-to-fastq branch from 90d9519 to 159b524 Compare August 13, 2026 21:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant