betto_lexical library
Classes
- BrowserTokenizer
- Stub BrowserTokenizer for native (non-web) targets.
- IcuTokenizer
-
Stub IcuTokenizer for web targets where
dart:ffiis unavailable. - OffsetTokenizer
- A Tokenizer that can also report each token's position in the source text.
- RegExpTokenizer
- The pure-Dart Tokenizer implementation, using RegExp for word boundary detection.
- Stemmer
- Reduces words to their base form (stem) using the Snowball algorithm.
- Tokenizer
- Abstract interface for text segmentation.
- TokenSpan
- A tokenised word span together with its character offsets in the original text.
Enums
- Stopwords
- Stop-word sets for 58 languages, keyed by IETF language code.
Functions
-
createDefaultTokenizer(
) → Tokenizer - Returns the default Tokenizer for native platforms.
-
getStopWords(
Locale locale) → Stopwords -
Returns the Stopwords set for the given
locale.