normalizeForTokenization static method

  1. @visibleForTesting
String normalizeForTokenization(
  1. CharsmapTrie trie,
  2. String text
)

Applies the charsmap substitution, whitespace-run collapse, and Metaspace escaping steps of encode's pipeline (steps 1–3), without requiring a loaded SentencePieceTokenizer/vocabulary.

Exposed (rather than kept private) purely so these offline, vocab-independent steps can be unit-tested directly against the small committed charsmap fixture, without needing the full ~17 MB tokenizer.json — see test/xlmr_tokenizer_test.dart. Not intended for use outside this package or its tests.

Implementation

@visibleForTesting
static String normalizeForTokenization(CharsmapTrie trie, String text) {
  final substituted = trie.normalize(text);
  final collapsed = _collapseSpaceRuns(substituted);
  return _metaspace(collapsed);
}