tokenise method

  1. @override
List<String> tokenise(
  1. String text
)
override

Segments text into word tokens.

Returns only word-like spans (letters, numbers, mixed-case identifiers). Punctuation, whitespace, and other non-word spans are discarded.

An empty text must return an empty list without error.

Implementation

@override
List<String> tokenise(String text) {
  if (text.isEmpty) return const [];
  return _wordPattern
      .allMatches(text)
      .map((m) => m.group(0)!)
      .toList(growable: false);
}