IOUtils rework

[max-leuthaeuser]

A few clean-ups. No public API changes.

readEntireFile: single-pass reading

• Replaced the read loop (which allocated a new 1 KiB char array on every iteration and copied data through reader → buffer → StringBuilder → String) with Files.readAllBytes followed by a single new String(bytes, offset, length, UTF_8) decode.
• One bulk read, one decode pass, no intermediate char copies, no per-iteration allocations.
• The String constructor is documented to always replace malformed and unmappable input with the default replacement character, matching the previous explicit CharsetDecoder configuration. This also removes one use of a per-call CharsetDecoder (decoders are not thread-safe).

readLinesInFile: simpler line splitting

• Replaced the BufferedReader.lines() (Java Stream) → asScala iterator adapter → toSeq
  pipeline with a plain readLine() loop into a list builder. Same semantics.

skipBOMIfPresent: cheaper and more correct BOM skipping

• No longer allocates a one-element char array and no longer does a boxed Set[Char] lookup;
  a single read() compared against U+FEFF is sufficient.
• Dropped the '\uefbb' and '\ufffe' entries from the BOM set. With a UTF-8 decoder these can never result from an actual BOM (a UTF-16 file decoded as UTF-8 yields U+FFFD replacement chars, not these code points). They could only match genuinely encoded characters, in which case the old code silently stripped a legitimate first character. This is a bug fix.

Docs/cleanup

• Removed the stale "removes unpaired surrogates" scaladoc line (that behavior was dropped in
  Do not longer remove surrogates #1804) and an unused import.

Behavior changes

• A leading U+EFBB/U+FFFE that is genuinely encoded in the file is now preserved instead of being stripped. Real BOMs (which decode to U+FEFF, e.g. the UTF-8 BOM EF BB BF) are still skipped, now consistently in both methods.

Benchmark results
Corpus: 74,502 real source files, 412 MB (C 3,121 / Kotlin 46 / JS 43,549 / Swift 27,786), page-cache warm, 3 measured passes after warmup, alternating order.

Reading of all files produced byte-identical output between old and new implementation.

=== readEntireFile ===
  pass 1: old   7,30s (  4,7 GB alloc) | new   6,45s (  1,1 GB alloc)
  pass 2: old   7,31s (  4,7 GB alloc) | new   7,47s (  1,1 GB alloc)
  pass 3: old   7,37s (  4,7 GB alloc) | new   7,25s (  1,1 GB alloc)
  old: avg 7,33s, best 7,30s, avg alloc 4,7 GB
  new: avg 7,06s, best 6,45s, avg alloc 1,1 GB
  speedup: 1,04x (avg), 1,13x (best), allocation reduction: 4,4x

=== readLinesInFile ===
  pass 1: old   8,37s (  3,0 GB alloc) | new   8,04s (  3,0 GB alloc)
  pass 2: old   7,53s (  3,0 GB alloc) | new   7,54s (  3,0 GB alloc)
  pass 3: old   8,11s (  3,0 GB alloc) | new   7,66s (  3,0 GB alloc)
  old: avg 8,00s, best 7,53s, avg alloc 3,0 GB
  new: avg 7,74s, best 7,54s, avg alloc 3,0 GB
  speedup: 1,03x (avg), 1,00x (best), allocation reduction: 1,0x

The readEntireFile rewrite is an allocation win: 4.4x less garbage (no per-1KiB char arrays, no StringBuilder double-copy), but only ~4-13% faster. Pays off in the real use case: ForkJoinParallelCpgPass with N threads doing readEntireFile concurrently. 4.4x less allocation rate means proportionally less GC pressure and allocation contention during parallel AST creation.

readLinesInFile is performance-neutral. That change is simplicity only, not speed.

[maltek]

if you are motivated, you could also try these ideas (if not just merge):

• contentFromBufferedSource
  • use scala.collection.immutable.ArraySeq.newBuilder instead of List.newBuilder
  • replace with Files.readAllLines(path, charset).asScala and change the return type to scala.collection.Seq - not sure how invasive this is
• readEntireFile: Files.readString(path, charset).stripPrefix("\UFEFF") - should avoid keeping the whole file in memory twice during decoding

[max-leuthaeuser]

Good ideas, however:

1. ArraySeq.newBuilder — semantically identical, performance-neutral (within noise, same allocation). Not worth changing; List.newBuilder is fine.
2. Files.readAllLines — throws on malformed input, keeps the BOM, allocates more (3.0 vs 2.2 GB on the corpus from above), and would widen the public return type to scala.collection.Seq. Ruled out on semantics.
3. Files.readString — throws on malformed input. The memory claim is real: 0.7 vs 1.1 GB per pass, because the JDK can adopt the byte array without copying for LATIN1-decodable content. But speed is identical, and getting lenient decoding back would require a manually configured decoder — i.e., what we already do. So no real benefits.