feat: native character encoding with Effect streams - #8133
joepjoosten wants to merge 12 commits into
Conversation
🦋 Changeset detectedLatest commit: d9ed84e The changes in this PR will be included in the next version bump. This PR includes changesets to release 30 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Bundle Size AnalysisGenerated from PR build output; treat the content below as untrusted.
|
|
Implemented in 4ad6253: explicit codec entry points, isolated registries, and an opt-in All registry. How it worksimport * as CharacterEncoding from "effect/CharacterEncoding"
import * as Utf8 from "effect/encoding/Utf8"
import * as Windows1251 from "effect/encoding/Windows1251"
const encoded = CharacterEncoding.encode("Привет", Windows1251.encoding)
const converted = source.pipe(
CharacterEncoding.transcodeStream(Windows1251.encoding, Utf8.encoding)
)
const registry = CharacterEncoding.makeRegistry([
Utf8.encoding,
Windows1251.encoding
])
registry.resolveUnsafe("cp1251") // Windows1251.encoding
registry.encodingExists("cp932") // false
// registry.resolve(label) provides a typed Effect lookup instead.The core operators import no mapping tables. Each codec module loads only its own mapping data. Typed lookup arrays and decoding tries are still initialized lazily on first use and cached; each conversion has independent incremental state. For applications needing all runtime labels, explicitly import Verified bundle boundariesMinified browser ESM; sizes are bytes, with gzip in parentheses. These standalone decoder entry points include retained Effect infrastructure.
The restricted registry retains Effect infrastructure for its typed lookup method; it still imports only one mapping table. These cases expose different APIs, so sizes are not interchangeable workload comparisons. The checked-in bundle harness asserts the imported mapping files and executes output bundles from both bundlers. Throughput benchmarks were also rerun; Effect rates remain roughly similar, with baseline variability called out in the report. Verification: 110 tests pass on Node/Bun/Deno, type tests pass on TS 5.9.3/6.0.3, 7,556 differential checks pass, and regeneration is reproducible. Source check and lint pass. Full local checks still report the previously documented unchanged Schedule/Logger issues. |
Multibyte codec optimization: what, why, and benchmark resultsFollowing the runtime experiments, we retained only the runtime-independent multibyte optimization. Status: committed and pushed in What changed
There are no runtime checks, new dependencies, or public API changes. Why retain this optimization?The simpler encoder gives substantial gains on Node, Bun, and Deno. It reduces work and over-allocation within the codec, without relying on runtime-specific APIs or changing Effect Streams. The other two experiments are not included in the committed implementation: the Buffer UTF-16LE path regressed on Deno, and direct CP1251-to-UTF8 transcoding regressed on Bun. Neither is part of these measurements. Fresh benchmark resultsMedian throughput in MiB/s, baseline → candidate. Percentages are medians of paired changes, so they need not equal the ratio of the displayed, rounded medians.
Method: macOS ARM64; Node 24.20.0, Bun 1.4.0, Deno 2.9.6. Baseline and candidate run in the same process, with five alternating 150 ms measurement rounds after 250 ms warmups per implementation. Runtime processes run sequentially. Inputs are approximately 64 KiB UTF-8-equivalent text; streaming uses 4,093-byte chunks. String-input throughput uses UTF-8 byte length; byte-input throughput uses actual input length. Output equality is checked before timing; timed one-shot cases consume output lengths, and streams consume chunks without collecting them. No I/O or cold-start costs are included. The clearest reproducible gain is CP932 encoding: approximately 42–47% across all three runtimes. Decoder and streaming benefits vary. Small changes may be noise; these microbenchmarks are not application-wide speedup claims. Correctness and validation
The benchmark report, reproduction commands, and links to raw samples are now committed alongside the benchmark harness and differential checks. |
|
I think this will need some redesign before being considered. It feels too complex right now. |
UTF-16LE Buffer encoding enabled — rerun and Deno fixCommitted and pushed in What changedUTF-16LE encoding now uses the available global UTF-16 decoding and UTF-16BE encoding are unchanged. No new dependency or runtime/version guard was added. Deno issue and solutionThe original Deno regression led to denoland/deno#36803 and the proposed fix denoland/deno#36804. The fix replaces the per-code-unit JavaScript conversion plus temporary-array copy with V8's native UTF-16 write into aligned destination buffers, preserving raw code units and safely handling unaligned destinations. That Deno PR is still open and unmerged. Released Deno 2.9.6 still regresses with this Effect change enabled. The fixed-build results below require the proposed Deno patch; they do not describe an already released fix. Fresh resultsEffect baseline: Median MiB/s, baseline → candidate. Percentages are medians of paired changes.
The matched local Deno builds make the difference clear: on the fixed runtime, the Buffer path improves raw encoding throughput about 22× over the JavaScript loop, the full-byte-checksum workload by 47%, and streaming throughput by 96%. On the unpatched local runtime, the same Effect change still regresses. Earlier results, for comparisonThe earlier confirmation run used Effect baseline
The rerun remains broadly consistent with the previous Node/Bun gains and the old Deno regression. The patched Deno results demonstrate that the bottleneck was in Deno's Buffer conversion path, not a fundamental Effect Streams limitation. Method and validation
|
I think it got more complicated because of the registry. But it's there because when you want to limit the bundle size, you can decide to register the encodings you want to bundle yourself. Otherwise it would inflate the bundle to 500kb. This means for example that if you know which encodings you need to support you can supply functions your own registry, which they can use to check if the encoding is supported, and use it, otherwise fail. But the registry makes it more complicated indeed. |
Redesign the CharacterEncoding API from descriptor-based to function-first, dramatically simplifying the user experience while maintaining all technical benefits (bundle size, tree-shaking, performance). Changes: - All 94 encoding modules now export direct functions (encode, decode, encodeUnsafe, decodeUnsafe, encodeStream, decodeStream) - Eliminated need to import central CharacterEncoding module for basic usage - Kept encoding descriptors for optional registry use (advanced cases) - Updated generator to produce new API structure - Updated benchmarks to use simplified API Before: import * as CharacterEncoding from "effect/CharacterEncoding" import * as Utf8 from "effect/encoding/Utf8" const bytes = CharacterEncoding.encodeUnsafe(text, Utf8.encoding) After: import * as Utf8 from "effect/encoding/Utf8" const bytes = Utf8.encodeUnsafe(text) Benefits: - 50% reduction in user code (1 import vs 2) - 75% reduction in concepts (1 vs 4) - Smaller bundles (no need to import central module) - Identical tree-shaking behavior - Same performance characteristics - Better ergonomics for database drivers via dependency injection Benchmarks verified: All encodings perform identically to original PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Mapping encoding labels to codecs is an application concern. Drop Registry, makeRegistry, the "resolve" error operation, the aliases field on Encoding, the generated encoding/All module and the label normalizer. Regenerate the codec modules without alias data. Also fix the generated stream operators' type parameters so upstream error and requirement types infer correctly, fix relative resolution in the Rolldown bundle benchmark and refresh its measurements, remove the unused tsx dev dependency and a stray scratch file, and update tests, typetests, benchmarks, docs and the changeset accordingly. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FFwCwys4zXvefgTs4keFRE
Move the encode/decode/stream operators duplicated across the generated codec modules into internal/characterEncoding/operators.ts, bound to a codec by its first argument. The generated modules become thin wrappers around it. The shared module imports nothing the codec modules did not already import, so single-codec bundle sizes are unchanged. Stream operators now wrap write and end failures with the CharacterEncodingError shape instead of casting raw errors. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FFwCwys4zXvefgTs4keFRE
The mapping tables and codec modules are stable, so commit them as ordinary source instead of generated output. Drop the generator script and the "do not edit" headers; keep the iconv-lite provenance line and attribution in each data file. iconv-lite stays a development dependency as the reference implementation for tests and benchmarks. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FFwCwys4zXvefgTs4keFRE
|
Agreed, it was indeed too complex. I got distracted by #8129, where I was looking for a way to prevent large bundles, and carried that concern over into this PR. That should not be a concern for character encoding: it simply needs to be properly tree shakeable. I have simplified the branch accordingly:
A single codec entry point now bundles to roughly 3 KB minified, and the core |
Keep internal/characterEncoding for codec machinery only and place the 89 mapping data modules under internal/data. Update the codec module imports, the multibyte tables' Mapping type import, and the bundle benchmark's table detection. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FFwCwys4zXvefgTs4keFRE
Comparison with iconv-lite and node-iconvMedian MiB/s, Node 24 on Apple M2 Max, ~64 KiB input and 4 KiB stream chunks, all providers measured in the same run with output-equality checks. The last column is the median of paired throughput changes against iconv-lite. Measured at 53900c0; the cleanup since then only moved code around and did not touch the conversion loops.
* UTF-16LE decoding goes through In short: encoding is on par with or faster than iconv-lite everywhere, UTF-8 and single-byte decoding are faster, and CP932/GB18030 decoding plus most streaming workloads are still 5-25% behind iconv-lite. Against node-iconv, Effect is faster on every encode workload and on UTF-8, UTF-16LE and CP1251 decoding; node-iconv stays ahead on CP932/GB18030 decoding and CP932 streaming. This is not an across-the-board win, and it has no native dependency. Full methodology, the alpha comparison, raw samples and bundle measurements: packages/effect/benchmark/CharacterEncoding.md. |
2 participants
Summary
Draft a native character-set conversion module with synchronous incremental codecs, typed Effect failures, and backpressured Effect Stream encode/decode/transcode operators.
2472166ea5a4825ca091b9550713c403852a566b(local 1.0.0-alpha.2 checkout), with full MIT attribution and a regeneration script.Draft decisions / scope
The requested source location is
packages/effect/src, so this currently exportseffect/CharacterEncoding, not a separately published@effect/encodingpackage. Core versus standalone packaging needs review. Existingeffect/Encodingremains unchanged.This is not a compatibility-complete replacement. UTF-7, CESU-8, automatic UTF-16/32 endianness, pseudo-codecs, transliteration and iconv's wider stateful codec set are outside this initial draft. UTF-16 decoding follows TextDecoder / iconv-lite alpha replacement semantics rather than 0.7.x raw Buffer semantics. Codecs now have explicit entry points, with an optional complete registry. Further mapping compression, broader malformed-input corpora and multibyte performance need work. No MSSQL dependency changes are included.
Explicit codecs and selective bundles
Conversion takes codec values, not globally resolved strings:
makeRegistryaccepts only explicit codecs, supports normalized aliases, and offers typedresolveplus synchronousresolveUnsafe. Unknown labels do not trigger implicit loading.effect/encoding/Allis an explicit opt-in to every codec and alias.esbuild and Rolldown bundle checks verify zero mapping modules for UTF-8, exactly one for CP1251 (also for a restricted UTF-8/CP1251 registry), and 89 for All. They execute the generated bundles too. The report includes exact byte/gzip sizes and reproduction.
Benchmarks
Node 24.20.0, macOS ARM64; approximately 64 KiB source text, 4,093-byte stream chunks, 100 ms warmup, five rotating-order rounds of 150 ms per provider. Full output equality is checked before timing. Synchronous rows use Unsafe APIs; streaming rows include each framework's pipeline overhead.
Throughput was rerun after the explicit-codec refactor, with label resolution outside timing. Effect throughput remains roughly in line with the first draft. Results are still mixed: single-byte conversion is promising; multibyte conversion and most streaming workloads trail iconv-lite. Published 0.7.3 has a substantial UTF-16 speed advantage through raw Buffer paths. The alpha CP1251 streaming baseline was unusually slow in this rerun; the report flags this variability rather than attributing an apparent win to the refactor.
Complete stable/alpha tables, all raw samples, limitations and reproduction commands:
Verification
pnpm lint-fix,pnpm --filter effect check, andgit diff --checkpass.pnpm checkreports TS6133 in unchangedpackages/effect/typetest/Schedule.tst.ts:80(metadata).pnpm jsdocs --checkreports two issues in unchangedLogger.ts(tag ordering and unresolvedconsolePrettylink); none in the new module.Keeping this draft for API/placement review, broader compatibility testing and performance work.