betto_icu library
Unicode text tokenization for Dart.
Provides these exports:
- Tokenizer — the abstract segmentation interface.
- OffsetTokenizer — a Tokenizer that also reports each token's character offsets in the source text (TokenSpan). Implemented by IcuTokenizer and RegExpTokenizer.
- IcuTokenizer — UAX #29 word boundaries via the system ICU FFI library. Handles non-Latin scripts (CJK, Thai, Arabic, etc.). Preferred for multi-language use cases.
- RegExpTokenizer — pure-Dart, Latin/English fallback using RegExp. Zero FFI dependencies; suitable for English prose and common technical identifiers.
Classes
- BrowserTokenizer
- Stub BrowserTokenizer for native (non-web) targets.
- IcuTokenizer
-
Stub IcuTokenizer for web targets where
dart:ffiis unavailable. - OffsetTokenizer
- A Tokenizer that can also report each token's position in the source text.
- RegExpTokenizer
- The pure-Dart Tokenizer implementation, using RegExp for word boundary detection.
- Tokenizer
- Abstract interface for text segmentation.
- TokenSpan
- A tokenised word span together with its character offsets in the original text.