• Package: betto_lang_detector
• Version: 0.1.0-dev.2
• Dart SDK: ^3.12.0

1 Purpose and scope

A pure-Dart, zero-runtime-dependency language detector. It serves three narrow purposes for downstream consumers (originally kmdb’s vault search roadmap, WI-5): lexical analyzer selection, document metadata, and cheap script-based tokenizer routing. It is not designed for high-accuracy NLP-grade classification, and does not participate in semantic/embedding search pipelines (those are served by a shared multilingual embedding model elsewhere, which needs no language label).

2 Algorithm

Two independent stages, composed by CompositeBackend:

1. Script pre-filter. A generated, sorted Unicode codepoint-range table (from the Unicode Character Database’s Scripts.txt and PropertyValueAliases.txt) resolves dominantScript() — the most common Unicode script among a text’s letter codepoints — via binary search. Seven scripts in the target language set (Bengali, Gujarati, Armenian, Greek, Hebrew, Thai, Hangul) map to exactly one supported language each and short-circuit detection immediately. Han-script text is split into Japanese/Chinese via a kana-presence check (any Hiragana/Katakana codepoint present → Japanese; Han-only → Chinese) rather than by dominant script count alone, since Japanese text is often Han-majority by raw codepoint count.

2. Character n-gram model. For the remaining four scripts (Latin, Cyrillic, Arabic, Devanagari — spanning 49 of the 58 supported languages), a classic Cavnar & Trenkle “N-Gram-Based Text Categorization” (1994) model disambiguates. Each language has a generated profile: its top-300 character n-grams (orders 1-5, word-boundary-padded, e.g. "the" → "_the_" → {"_","_t","_th",...,"_the_"}), ranked by frequency in a reference corpus (UDHR translations). At detection time, the input’s own ranked n-gram list is compared against each candidate language’s profile via an “out-of-place” distance (sum of rank differences for shared n-grams; a fixed maximum penalty for n-grams absent from the profile). Confidence is a linear rescaling of that distance across the candidate set being compared — a heuristic, not a calibrated probability, so the same input can score differently depending on which languages are being distinguished between (see restrictTo on LanguageDetector).

The tool/generate_scripts.dart and tool/generate_ngram_profiles.dart scripts produce the two .g.dart data files this algorithm depends on; both require network access at codegen time only, never at runtime.

3 Language coverage

58 ISO 639-1 codes, matching betto_lexical’s Stopwords enum exactly (so LanguageDetector.detect() output composes directly with getStopWords(Locale) with no code-translation layer):

4 Known limitations

• Kurdish (ku) is Latin-script only (Kurmanji orthography). Sorani Kurdish, written in a Perso-Arabic script, is misdetected as ar/fa/ur — there is no Sorani-script profile. Accepted trade-off, not a bug. The Kurmanji training text itself is also a minor corpus deviation: it comes from NLTK’s older udhr package (udhr/Kurdish-UTF8), not udhr2 (which only has the Sorani translation) — see tool/generate_ngram_profiles.dart.
• Confidence is relative, not calibrated. See “Algorithm” above — restrictTo narrows the comparison set and can change the confidence value reported for the same input. In particular, narrowing restrictTo to exactly one language always scores that language at confidence 1.0, regardless of how well the text actually fits — there is nothing else to compare against. This is intentional, not a bug.
• Short input has little signal. A manual accuracy spot-check (20 hand-written, held-out short sentences, 5-8 words each, across 16 scripts/language families) scored 15/20 correct. The misses (pt→la, tr→ha, sv→eu) all resolved correctly once given 2-sentence-length input instead of one short clause — the Cavnar-Trenkle technique is validated on document-length text, and a handful of words gives the out-of-place distance little to work with. Coarse accuracy on document-length input, not short queries, is this detector’s actual design target.
• Russian/Ukrainian/Bulgarian confusion. Even with longer (2-sentence) input, Russian and Ukrainian samples both still resolved to Bulgarian in the spot-check above. This is a persistent, known-hard case for character n-gram methods among closely-related Cyrillic Slavic languages, not specific to this implementation — expect some cross-confusion within that group.
• Coarse by design. This detector trades accuracy for zero dependencies and simplicity. It is not intended for high-stakes classification.