1 betto_icu

Unicode text tokenization for Dart.

Four exports, one import:

1.1 Platform support

IcuTokenizer links against the system ICU library — no bundling required.

RegExpTokenizer works on every platform including web. BrowserTokenizer works on web only — it uses dart:js_interop to call the browser’s built-in Intl.Segmenter.

1.2 Getting started

dependencies:
  betto_icu: ^0.1.0

Requires Dart SDK ^3.12.0.

1.3 Usage

import 'package:betto_icu/betto_icu.dart';

void main() {
  // IcuTokenizer — UAX #29, handles any script
  final icu = IcuTokenizer();
  print(icu.tokenise('"The Strange Case of Dr. Jekyll and Mr. Hyde"'));
  // → [The, Strange, Case, of, Dr, Jekyll, and, Mr, Hyde]

  // RegExpTokenizer — pure Dart, English/Latin only
  final re = RegExpTokenizer();
  print(re.tokenise('mTLS handshake in 0x8004'));
  // → [mTLS, handshake, in, 0x8004]

  // Both implement Tokenizer, so they are interchangeable
  final Tokenizer t = IcuTokenizer();
  print(t.tokenise(''));  // → []
}

The tokenize command-line tool takes an input string and returns the token list for each or both tokenizer:

dart run bin/tokenize.dart "The Strange Case of Dr. Jekyll and Mr. Hyde"

1.3.1 Choosing an implementation

Use IcuTokenizer when your text may contain non-Latin scripts (CJK, Thai, Arabic, Devanagari, etc.) on a native platform — it delegates word-boundary detection to the OS-provided ICU library and conforms to UAX #29.

Use BrowserTokenizer on Flutter Web when your text may contain non-Latin scripts. It calls Intl.Segmenter in the browser’s own JavaScript engine, which is backed by the same ICU data, with no bundle overhead and no FFI. Requires Chrome 87+, Firefox 125+, or Safari 16.4+.

Use RegExpTokenizer when you only process English prose or technical identifiers and want zero FFI dependencies, or as a fallback on older browsers that don’t support Intl.Segmenter.

1.3.2 ubrk_getRuleStatus note (macOS / iOS)

Apple’s libicucore does not export UAX #29 rule-status tags in its compiled word break rules. IcuTokenizer uses character-class RegExp matching for span classification rather than ubrk_getRuleStatus(), making it portable across all supported platforms.

1.4 Testing

Run the test suite:

make test

Collect line coverage (requires the coverage pub global tool):

make coverage

1.4.1 Test structure

Tokenizer contract — a shared _tokenizerContractTests helper runs the same invariants (empty input, punctuation stripping, numbers, prose sentences) against both implementations to ensure they are interchangeable.

UAX #29 specifics — tests that verify ICU-only behaviour: CJK ideographs, Arabic, combining diacritics, emoji filtering, and hex/mixed-case identifiers such as mTLS and 0x8004210B.

Platform library loading — IcuTokenizer.forPlatform(String platform) lets tests drive each OS branch on any host machine. Each test is skipped on its native platform (where IcuTokenizer() already covers those lines) and runs on all other platforms, so coverage stays above 90% regardless of which CI runner executes the suite.

The android test is skipped on Linux because libicuuc.so is present there (installed by libicu-dev).

1.4.2 Android emulator

The integration_test_app/ directory contains a minimal Flutter app that runs the full contract and UAX #29 test suite on a real Android runtime via package:integration_test.

Prerequisites: Flutter SDK and a running emulator (or connected device).

make emulator_android_create
make android_test
make emulators_stop_android

1.4.3 iOS simulator

The same integration_test_app/ runs on an iOS simulator or physical device. ICU on iOS comes from Apple’s libicucore.dylib, which ships with the OS and exports symbols without version renaming.

Prerequisites: Xcode and a booted simulator (or connected device).

make emulator_ios_create
make ios_test
make emulators_stop_ios

1.4.4 Web browser

BrowserTokenizer tests live in the main package and run with dart test, which launches Chrome directly — no WebDriver server required.

make web_test

1.4.5 Linux container

The Containerfile can be used to test the package on Linux.

make container_test

1.5 License

Apache 2.0 — see LICENSE.