XlmRobertaTokenizer class

XLM-RoBERTa-family SentencePiece/Unigram tokenizer, e.g. for multilingual-e5-small.

Composes a from-scratch CharsmapTrie normalizer (the one piece of SentencePiece's pipeline this project could not find a working Dart implementation of anywhere — see CharsmapTrie's own doc comment) with dart_sentencepiece_tokenizer's public API for everything else: vocab loading, Unigram Viterbi decoding, and BOS/EOS post-processing.

Why this class exists instead of using dart_sentencepiece_tokenizer

directly

dart_sentencepiece_tokenizer parses HuggingFace tokenizer.json's Precompiled normalizer type (the charsmap trie bytes) but never applies it — HuggingFaceTokenizerLoader's JSON-loading path silently drops the charsmap, making its own SpNormalizer an unconditional (charsmap-less) pass-through for models like multilingual-e5-small. See this package's README.md ("Why not dart_sentencepiece_tokenizer alone") for the full write-up of this and a second, independent defect this class also works around.

SentencePieceTokenizer's real constructor is private and every public factory routes through internal, non-injectable construction — there is no way to subclass or swap in a corrected normalizer. Composition (running our own charsmap normalization before handing text to the library's public encode()) is therefore the only viable integration seam, and is what encode does.

Pipeline

encode applies, in order:

1. CharsmapTrie.normalize — the charsmap substitution described above.
2. Whitespace-run collapse (two-or-more plain spaces → one).
3. Metaspace escaping (prepend a leading space if absent; replace every space with ▁, U+2581).
4. dart_sentencepiece_tokenizer's own SentencePieceTokenizer.encode().

Why steps 2–3 are done manually here rather than trusted to the library's own SpNormalizer: dart_sentencepiece_tokenizer's HuggingFace-JSON metadata parser derives its addDummyPrefix/removeExtraWhitespaces/escapeWhitespaces flags by pattern-matching the normalizer section of tokenizer.json, but multilingual-e5-small's tokenizer.json puts this configuration entirely in pre_tokenizer ({"type": "Metaspace", "replacement": "▁", "add_prefix_space": true}) instead — a section the parser never reads. All three flags therefore come back false, and the library's own whitespace/prefix handling silently no-ops for this file. This is a second, independent defect from the already-known charsmap-drop one (confirmed empirically: feeding the library literal pre-escaped text like "▁Hello" produces the correct single-token id, while plain "Hello" or " Hello" does not) — so this class must own both normalization concerns, not just the charsmap.

Implements ModelTokenizer so OnnxEmbeddingModel can select this tokenizer at runtime alongside BertTokenizer, both sharing the same TokenizerOutput return shape.