tokenise abstract method

List<String> tokenise(
  1. String text
)

Segments text into word tokens.

Returns only word-like spans (letters, numbers, mixed-case identifiers). Punctuation, whitespace, and other non-word spans are discarded.

An empty text must return an empty list without error.

Implementation

List<String> tokenise(String text);