Shared interface implemented by every tokenizer family OnnxEmbeddingModel can select at runtime.
BertTokenizer and XlmRobertaTokenizer both implement this interface
and both already return the concrete TokenizerOutput type verbatim —
there is no separate ModelInput wrapper type. OnnxEmbeddingModel.load
picks which concrete tokenizer to construct based on
ModelSpec.meta['tokenizerFamily'] ('bert' or 'xlmr'), so
OnnxEmbeddingModel.embed has exactly one _tokenizer.encode(text) call
site regardless of which model family is loaded — registering a third
tokenizer family later only requires a new ModelTokenizer implementation
and a new tokenizerFamily value, not a change to OnnxEmbeddingModel
itself.
- Implementers
Properties
- hashCode → int
-
The hash code for this object.
no setterinherited
- runtimeType → Type
-
A representation of the runtime type of the object.
no setterinherited
Methods
-
encode(
String text) → TokenizerOutput -
Encodes
textinto a TokenizerOutput ready for ONNX inference. -
noSuchMethod(
Invocation invocation) → dynamic -
Invoked when a nonexistent method or property is accessed.
inherited
-
toString(
) → String -
A string representation of this object.
inherited
Operators
-
operator ==(
Object other) → bool -
The equality operator.
inherited