tokenise method

  1. @override
List<String> tokenise(
  1. String text
)
override

Segments text into word tokens.

Returns only word-like spans (letters, numbers, mixed-case identifiers). Punctuation, whitespace, and other non-word spans are discarded.

An empty text must return an empty list without error.

Implementation

@override
List<String> tokenise(String text) =>
    throw UnsupportedError('IcuTokenizer is not available on web targets.');