load static method

Future<XlmRobertaTokenizer> load(
  1. String tokenizerJsonPath, {
  2. int maxLength = 512,
})

Loads a tokenizer from a HuggingFace tokenizer.json file at tokenizerJsonPath.

Extracts the Precompiled normalizer's precompiled_charsmap bytes (normalizer.normalizers[], type: "Precompiled") to build a CharsmapTrie, and separately passes the full JSON to HuggingFaceTokenizerLoader.fromJsonString to build the underlying SentencePieceTokenizer (vocabulary, Unigram model, BOS/EOS configuration).

maxLength is the maximum sequence length, including the <s>/</s> sentinel tokens. Defaults to 512, matching both BertTokenizer's own default and multilingual-e5-small's published max_seq_length (the model's tokenizer.json itself declares no truncation/padding section, so there is no stronger signal to defer to).

Throws FormatException if tokenizerJsonPath's JSON has no Precompiled normalizer entry, or if its precompiled_charsmap bytes are malformed (see CharsmapTrie.parse).

Implementation

static Future<XlmRobertaTokenizer> load(
  String tokenizerJsonPath, {
  int maxLength = 512,
}) async {
  // coverage:ignore-start
  // Requires the real ~17 MB multilingual-e5-small tokenizer.json (250k-entry
  // vocab) to exercise meaningfully — covered by the network-gated
  // integration test in integration_test_app/, not make coverage. See this
  // package's README for why that fixture isn't committed.
  final raw = await File(tokenizerJsonPath).readAsString();
  final tokenizerJson = jsonDecode(raw) as Map<String, dynamic>;
  final charsmapTrie = _extractCharsmapTrie(tokenizerJson);
  final tokenizer = HuggingFaceTokenizerLoader.fromJsonString(raw);
  return XlmRobertaTokenizer._(charsmapTrie, tokenizer, maxLength);
  // coverage:ignore-end
}