RegExpTokenizer class
The pure-Dart Tokenizer implementation, using RegExp for word boundary detection.
Scope — English and Latin scripts
This implementation is sufficient for English-language prose and common
technical identifiers (mTLS, 0x8004210B, etc.). It is not suitable
for non-Latin scripts (CJK, Thai, Arabic, etc.) where word boundaries do
not follow whitespace rules. For those use cases, prefer IcuTokenizer,
which uses the system ICU library and conforms to UAX #29 Unicode Text
Segmentation. The Tokenizer interface makes that swap transparent to the
calling pipeline.
Example
final tokenizer = RegExpTokenizer();
print(tokenizer.tokenise('Hello, world!')); // ['Hello', 'world']
print(tokenizer.tokenise('')); // []
print(tokenizer.tokenise('mTLS handshake')); // ['mTLS', 'handshake']
- Implemented types
Constructors
- RegExpTokenizer()
-
Creates a new RegExpTokenizer.
const
Properties
- hashCode → int
-
The hash code for this object.
no setterinherited
- runtimeType → Type
-
A representation of the runtime type of the object.
no setterinherited
Methods
-
noSuchMethod(
Invocation invocation) → dynamic -
Invoked when a nonexistent method or property is accessed.
inherited
-
tokenise(
String text) → List< String> -
Segments
textinto word tokens.override -
tokeniseSpans(
String text) → List< TokenSpan> -
Segments
textinto word tokens with their character offsets.override -
toString(
) → String -
A string representation of this object.
inherited
Operators
-
operator ==(
Object other) → bool -
The equality operator.
inherited