encode abstract method

TokenizerOutput encode(
  1. String text
)

Encodes text into a TokenizerOutput ready for ONNX inference.

See the implementing class (BertTokenizer or XlmRobertaTokenizer) for the exact special-token framing, padding id, and truncation behaviour — these differ between tokenizer families, but the shape of the returned TokenizerOutput does not.

Implementation

TokenizerOutput encode(String text);