tokeniseSpans abstract method

List<TokenSpan> tokeniseSpans(
  1. String text
)

Segments text into word tokens with their character offsets.

Equivalent to tokenise but additionally reporting each token's (start, end) span. An empty text must return an empty list without error.

Implementation

List<TokenSpan> tokeniseSpans(String text);