TokenizerOutput class final
The output of BertTokenizer.encode: three parallel int64 arrays ready for ONNX Runtime inference.
All three arrays have exactly maxLength elements. Padding positions have
inputIds = 0, attentionMask = 0, tokenTypeIds = 0.
Constructors
- TokenizerOutput({required Int64List inputIds, required Int64List attentionMask, required Int64List tokenTypeIds, required bool truncated})
-
Creates a TokenizerOutput.
const
Properties
- attentionMask → Int64List
-
1 for real tokens (including
[CLS]and[SEP]), 0 for padding.final - hashCode → int
-
The hash code for this object.
no setterinherited
- inputIds → Int64List
-
BERT token IDs, starting with
[CLS](101) and ending with[SEP](102), then zero-padded toBertTokenizer.maxLength.final - runtimeType → Type
-
A representation of the runtime type of the object.
no setterinherited
- tokenTypeIds → Int64List
-
Segment IDs — all zeros for single-segment BERT input.
final
- truncated → bool
-
trueif the input text exceeded the usable token budget and was silently truncated before the[SEP]token.final
Methods
-
noSuchMethod(
Invocation invocation) → dynamic -
Invoked when a nonexistent method or property is accessed.
inherited
-
toString(
) → String -
A string representation of this object.
inherited
Operators
-
operator ==(
Object other) → bool -
The equality operator.
inherited