encode method

  1. @override
TokenizerOutput encode(
  1. String text
)
override

Encodes text into a TokenizerOutput ready for ONNX inference.

Reuses the same TokenizerOutput type BertTokenizer.encode returns (not a parallel type), so both tokenizers already share an identical concrete output shape.

  • TokenizerOutput.tokenTypeIds is all-zeros: XLM-RoBERTa/RoBERTa models don't use segment ids, but the field is required by TokenizerOutput — this is a direct widen of Encoding.typeIds, which is already all-zeros for single-segment input in dart_sentencepiece_tokenizer, not invented data.
  • Padding uses the loaded vocabulary's own pad id (1 for multilingual-e5-small's <pad>), sourced automatically by SentencePieceTokenizer — not the BERT-specific padId = 0 BertTokenizer uses.
  • TokenizerOutput.truncated is derived via two encode passes (see below), because Encoding exposes no overflow signal and a padded output is always exactly maxLength tokens long regardless of whether truncation actually occurred.

All three output arrays have exactly the maxLength passed to load elements.

Implementation

@override
TokenizerOutput encode(String text) {
  final normalizedText = normalizeForTokenization(_charsmapTrie, text);

  // coverage:ignore-start
  // Requires the real multilingual-e5-small vocabulary to exercise
  // meaningfully — covered by the network-gated integration test in
  // integration_test_app/, not make coverage.
  //
  // `truncated` derivation is a two-pass process. `enablePadding`/
  // `enableTruncation` mutate *persistent* instance state on the shared
  // `SentencePieceTokenizer` (they are not per-call arguments), so an
  // explicit noPadding()/noTruncation() reset is mandatory on *every* call
  // — without it, the "unbounded" first pass below would still be bounded
  // by whatever config a *previous* encode() call left in place, and
  // `truncated` would silently stick `false` after the first call.
  _tokenizer.noPadding();
  _tokenizer.noTruncation();

  // Pass 1 (unbounded): get the true token count with no truncation, to
  // detect whether pass 2's truncation will actually cut anything.
  final rawLength = _tokenizer
      .encode(normalizedText, addSpecialTokens: true)
      .ids
      .length;
  final truncated = rawLength > _maxLength;

  // Pass 2 (bounded): the real, padded/truncated result. Both passes use
  // identical addSpecialTokens: true so rawLength (which includes the
  // <s>/</s> sentinels) is compared on equal terms against the maxLength
  // the bounded pass truncates to.
  _tokenizer
    ..enablePadding(direction: SpPaddingDirection.right, length: _maxLength)
    ..enableTruncation(
      maxLength: _maxLength,
      direction: SpTruncationDirection.right,
    );
  final encoding = _tokenizer.encode(normalizedText, addSpecialTokens: true);

  return TokenizerOutput(
    inputIds: Int64List.fromList(encoding.ids),
    attentionMask: Int64List.fromList(encoding.attentionMask),
    tokenTypeIds: Int64List.fromList(encoding.typeIds),
    truncated: truncated,
  );
  // coverage:ignore-end
}