A Tokenizer that can also report each token's position in the source text.
Implemented by IcuTokenizer and RegExpTokenizer — position data is a
natural byproduct of both algorithms' underlying implementation (ICU's
break iterator and RegExp matches both already carry span boundaries).
Not implemented by BrowserTokenizer: Intl.Segmenter's JS result does
carry a comparable index field, but no consumer needs offsets from the
web tokenizer today — left as a documented future extension rather than
implemented speculatively.
- Implemented types
- Implementers
Properties
- hashCode → int
-
The hash code for this object.
no setterinherited
- runtimeType → Type
-
A representation of the runtime type of the object.
no setterinherited
Methods
-
noSuchMethod(
Invocation invocation) → dynamic -
Invoked when a nonexistent method or property is accessed.
inherited
-
tokenise(
String text) → List< String> -
Segments
textinto word tokens.inherited -
tokeniseSpans(
String text) → List< TokenSpan> -
Segments
textinto word tokens with their character offsets. -
toString(
) → String -
A string representation of this object.
inherited
Operators
-
operator ==(
Object other) → bool -
The equality operator.
inherited