tokeniseSpans method

  1. @override
List<TokenSpan> tokeniseSpans(
  1. String text
)
override

Segments text into word tokens with their character offsets.

Equivalent to tokenise but additionally reporting each token's (start, end) span. An empty text must return an empty list without error.

Implementation

@override
List<TokenSpan> tokeniseSpans(String text) {
  if (text.isEmpty) return const [];
  return _wordPattern
      .allMatches(text)
      .map((m) => TokenSpan(m.group(0)!, m.start, m.end))
      .toList(growable: false);
}