quantise function

Uint8List quantise(
  1. Float32List vector
)

Quantises a float32 embedding vector to unsigned 8-bit integers (SQ8).

Uses a fixed symmetric range suitable for L2-normalised vectors whose components lie in [-1.0, 1.0]:

u = clamp(round((f + 1.0) / 2.0 * 255), 0, 255)

This maps -1.0 → 0, 0.0 → 127 (or 128), 1.0 → 255.

Assumptions

  • Input must be an L2-normalised vector (all components in [-1.0, 1.0]). Values marginally outside the range (e.g. due to float rounding) are clamped rather than panicked.
  • No per-vector calibration is required; the fixed range provides adequate accuracy for cosine similarity ranking.
  • vector may have any positive length (384 for BGE Small En v1.5, 1024 for BGE-M3, etc.). A debug-mode assertion fires for empty vectors.

The quantisation error is bounded by 2.0 / 255 ≈ 0.00784 per component (one quantisation step). In practice, round-trip error is ≤ 0.004 per element.

Implementation

Uint8List quantise(Float32List vector) {
  assert(
    vector.isNotEmpty,
    'Embedding vector must be non-empty, got length ${vector.length}',
  );
  final out = Uint8List(vector.length);
  for (var i = 0; i < vector.length; i++) {
    final f = vector[i];
    // Map [-1, 1] → [0, 255] and clamp to guard against float rounding.
    final u = ((f + 1.0) / 2.0 * 255.0).roundToDouble();
    out[i] = min(255, max(0, u.toInt()));
  }
  return out;
}