encode method

  1. @override
TokenizerOutput encode(
  1. String text
)
override

Encodes text into a TokenizerOutput ready for ONNX inference.

The output always starts with [CLS] (101) and ends with [SEP] (102). If text contains more WordPiece tokens than maxLength - 2 (510 usable tokens), the excess is silently discarded and TokenizerOutput.truncated is true.

An empty or whitespace-only text produces a two-token sequence [CLS][SEP] with all remaining positions padded — TokenizerOutput.truncated is false.

All three output arrays (TokenizerOutput.inputIds, TokenizerOutput.attentionMask, TokenizerOutput.tokenTypeIds) have exactly maxLength elements.

Implementation

@override
TokenizerOutput encode(String text) {
  final normalized = _normalize(text);
  final words = _tokenizer.tokenise(normalized);

  // Build the token ID list starting with [CLS].
  // Leave one slot for the closing [SEP] token.
  final tokenIds = <int>[clsId];
  var wasTruncated = false;

  outer:
  for (final word in words) {
    if (word.isEmpty) continue;
    for (final id in _wordPiece(word)) {
      // Reserve the last slot for [SEP].
      if (tokenIds.length >= _maxLength - 1) {
        wasTruncated = true;
        break outer;
      }
      tokenIds.add(id);
    }
  }
  tokenIds.add(sepId);

  // Build attention mask: 1 for real tokens, 0 for padding.
  final attentionMask = List<int>.filled(tokenIds.length, 1, growable: true);
  while (tokenIds.length < _maxLength) {
    tokenIds.add(padId);
    attentionMask.add(0);
  }

  return TokenizerOutput(
    inputIds: Int64List.fromList(tokenIds),
    attentionMask: Int64List.fromList(attentionMask),
    // BERT token_type_ids are all-zeros for single-segment input.
    tokenTypeIds: Int64List.fromList(List.filled(_maxLength, 0)),
    truncated: wasTruncated,
  );
}