Tokenizer class abstract interface

Abstract interface for text segmentation.

Implementations segment a string into word-like tokens, discarding whitespace and punctuation boundaries. The pipeline then normalises and stems the returned tokens; this interface is responsible only for the segmentation step.

Implementations

This package provides two implementations:

• RegExpTokenizer — pure Dart, suitable for English-language prose and common technical identifiers. Zero FFI dependencies.
• IcuTokenizer — backed by the system ICU library via FFI. Conforms to UAX #29 Unicode Text Segmentation and handles non-Latin scripts (CJK, Thai, Arabic, etc.) correctly. Prefer this implementation for multi-language use cases.

The interface is intentionally narrow so the implementation can be swapped without touching the calling pipeline.

Unicode Text Segmentation

Conformant implementations should follow UAX #29 Unicode Text Segmentation rules for word boundaries. IcuTokenizer provides a full UAX #29 implementation via the system ICU library.

Example

final tokenizer = RegExpTokenizer();
final tokens = tokenizer.tokenise('Dr. Jekyll and Mr. Hyde');
// → ['Dr', 'Jekyll', 'and', 'Mr', 'Hyde']