[Tokenizers] BpeOptions.ByteLevel=true silently drops spaces and newlines when PreTokenizer is not set

[zpczpc]

Describe the bug

When a BpeTokenizer is created with BpeOptions.ByteLevel = true but without setting BpeOptions.PreTokenizer, the tokenizer silently drops spaces and newlines.

No exception is thrown and no warning is logged. Decode cannot recover the original text, and the resulting token IDs are structurally valid but semantically wrong — which makes the problem very hard to notice in a downstream LLM pipeline.

To Reproduce

// vocab  : Dictionary<string,int> parsed from tokenizer.json -> model.vocab
// merges : IEnumerable<string>   parsed from tokenizer.json -> model.merges
var options = new BpeOptions(vocab)
{
    Merges            = merges,
    ByteLevel         = true,      // enables the byte <-> unicode mapping
    UnknownToken      = "<unk>",
    FuseUnknownTokens = true,
    // NOTE: PreTokenizer is intentionally NOT set here
};

var tokenizer = BpeTokenizer.Create(options);

string[] samples =
{
    "Hello, world!",
    "Tokenizer 是一个开源的中文分词器。",
    "<|im_start|>system\nYou are a helpful assistant.<|im_end|>",
};

foreach (string s in samples)
{
    IReadOnlyList<int> ids = tokenizer.EncodeToIds(s);
    string? back = tokenizer.Decode(ids);
    Console.WriteLine($"in : {s}");
    Console.WriteLine($"out: {back}");
    Console.WriteLine(s == back ? "OK" : "LOST DATA");
}

Actual result — spaces and newlines are silently gone:

in : Hello, world!
out: Hello,world!                                              <- LOST DATA (space)
in : Tokenizer 是一个开源的中文分词器。
out: Tokenizer是一个开源的中文分词器。                             <- LOST DATA (space)
in : <|im_start|>system\nYou are a helpful assistant.<|im_end|>
out: <|im_start|>systemYou are a helpful assistant.<|im_end|>    <- LOST DATA (newline)

Lossless round-trip rate with this configuration: 1 / 5.

Expected behavior

Enabling ByteLevel alone should not produce a tokenizer that corrupts its input. One of the following would be enough:

1. BpeTokenizer.Create throws (or logs a warning) when ByteLevel == true and PreTokenizer == null, or
2. the documentation for BpeOptions.ByteLevel states explicitly that a PreTokenizer must also be configured.

Setting a PreTokenizer fixes it completely

With RegexPreTokenizer built from the GPT-2 pattern, every sample above round-trips losslessly:

options.ByteLevel = true;
options.PreTokenizer = new RegexPreTokenizer(
    new Regex(@"'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+"),
    specialTokens);   // second argument: the added_tokens map

Result with PreTokenizer set: 6 / 6 samples match the HuggingFace reference token-for-token, 5 / 5 lossless round-trip.

Measured configuration matrix

Reference implementation: HuggingFace tokenizers (Rust), loading the same tokenizer.json.

ByteLevel	PreTokenizer	Match HF (6 samples)	Lossless round-trip
true	(not set)	3 / 6	1 / 5
true	RobertaPreTokenizer	5 / 6	—
true	RegexPreTokenizer (GPT-2 pattern)	6 / 6	5 / 5
false	RobertaPreTokenizer	0 / 6	—
false	RegexPreTokenizer (GPT-2 pattern)	0 / 6	—

Additional context — two documentation points

1. The current docs for BpeOptions.ByteLevel only describe the byte <-> unicode mapping (e.g. Space -> 'Ġ'). They do not mention that a PreTokenizer is required for correct whitespace handling.

2. The BPE sample in https://learn.microsoft.com/dotnet/ai/how-to/use-tokenizers contains this note:

   "BpeTokenizer might not always decode IDs to the exact original text as it can remove spaces during tokenization depending on the model configuration."

   Based on the measurements above, the behaviour is not dependent on the model configuration. It depends solely on whether a PreTokenizer is configured. Once a proper pre-tokenizer is set, decoding is exact for every input tested.

Why this matters for HuggingFace models

tokenizer.json carries the full pipeline configuration — pre_tokenizer, added_tokens, normalizer, post_processor, decoder. The BpeTokenizer.Create(vocab, merges) entry point consumes only the vocabulary data and cannot see any of it. Users following the current guidance (download vocab.json + merges.txt from HuggingFace) therefore end up with a tokenizer that is missing its pipeline configuration, and the failure mode is silent.

Environment

• Microsoft.ML.Tokenizers 2.0.0 (stable)
• .NET 10
• OS: Windows

[github-actions]

🔍 No similar issues found with high confidence.

---

AI-powered similar issue detection