BertTokenizer class

A BERT WordPiece tokenizer backed by a vocab.txt file.

Converts arbitrary text into BERT token IDs suitable for feeding into the BGE Small En v1.5 ONNX model via OnnxSession.run.

Pipeline

1. Normalise — lower-case and strip combining accent characters.
2. Word segmentation — delegate to the Tokenizer supplied at construction time. RegExpTokenizer is used by default; IcuTokenizer from package:betto_icu can be substituted as a drop-in replacement for superior Unicode coverage.
3. WordPiece — split each word into sub-word pieces and look up IDs in the vocabulary loaded from vocab.txt. Unknown pieces map to [UNK].
4. Assemble — prepend [CLS] (101), append [SEP] (102), and pad to maxLength with [PAD] (0).

Token ID space

BERT token IDs are entirely distinct from the stemmed token strings produced by the lexical search pipeline (FtsManager / BM25). They must not be interchanged.

IcuTokenizer

import 'package:betto_icu/betto_icu.dart';
final tokenizer = await BertTokenizer.load(vocabPath,
  tokenizer: IcuTokenizer());