Issues

• Streaming memory issues and allocation hotspots (loadChunk quadratic growth, GetSnippet bug, Stream.String allocations, token retention)

  bzick/tokenizer#47 · Filter by author bzick opened on Jul 13, 2026

• Trailing Whitespace

  bzick/tokenizer#41 · Filter by author GeenDutchman opened on Jul 2, 2026

• String token matching is happening after normal token matching

  bzick/tokenizer#40 · Filter by author Angus-Warman opened on May 16, 2026

• Unicode codepoint/grapheme boundaries

  enhancement
  bzick/tokenizer#34 · Filter by author tassa-yoniso-manasi-karoto opened on Jan 19, 2025
  7

  

• Specify case insensitive matching of token upon definition

  enhancement
  bzick/tokenizer#12 · Filter by author ktsivkov opened on Mar 7, 2024
  1