load static method

Future<BertTokenizer> load(
  1. String vocabPath, {
  2. int maxLength = 512,
  3. Tokenizer? tokenizer,
})

Loads the vocabulary from vocabPath and returns a BertTokenizer.

vocabPath must point to a vocab.txt file where each line is a vocabulary token and the line index (0-based) is the token ID. The BGE Small En v1.5 vocabulary has 30,522 entries.

maxLength is the maximum sequence length including the [CLS] and [SEP] sentinel tokens (default 512 per the BERT specification).

tokenizer controls word segmentation before WordPiece splitting. Defaults to RegExpTokenizer. Supply IcuTokenizer() from package:betto_icu for improved Unicode coverage.

Implementation

static Future<BertTokenizer> load(
  String vocabPath, {
  int maxLength = 512,
  Tokenizer? tokenizer,
}) async {
  final lines = await File(vocabPath).readAsLines();
  final vocab = <String, int>{};
  for (var i = 0; i < lines.length; i++) {
    vocab[lines[i].trim()] = i;
  }
  return BertTokenizer._(vocab, maxLength, tokenizer ?? RegExpTokenizer());
}