• Package: betto_zstd
• Version: 0.1.0
• Dart SDK: ^3.12.0
• Upstream C library: Zstandard v1.5.7

1 Purpose and Scope

betto_zstd is a pure-Dart package that wraps the Zstandard (Zstd) C compression library. It exposes a single, unified Dart API that works across all Dart and Flutter target platforms:

• Native (macOS, Linux, Windows, iOS, Android) — via Dart FFI and the Native Assets build system.
• Web — via an Emscripten-compiled WebAssembly (WASM) module loaded at runtime with dart:js_interop.

The package is a direct dependency of KMDB (the Bettongia knowledge-management database) and must remain a pure-Dart package — it must not introduce a dependency on the Flutter SDK. The flutter: section of pubspec.yaml declares the WASM asset but does not pull in Flutter as a code dependency; the Dart library itself uses only Dart SDK libraries.

2 Design Constraints

3 Repository Layout

betto_zstd/
├── bin/
│   └── dartz.dart                  # CLI tool: compress / decompress files
├── docs/
│   ├── plans/                      # Implementation plans (active and completed)
│   └── spec/                       # This specification document
├── example/
│   └── main.dart                   # Minimal usage example
├── hook/
│   └── build.dart                  # Native Assets build hook
├── lib/
│   ├── zstd.dart                   # Public entry point (conditional export)
│   ├── assets/
│   │   └── zstd.wasm               # Pre-built WASM module (~317 KB, committed)
│   └── src/
│       ├── zstd_native.dart        # Native FFI implementation
│       ├── zstd_web.dart           # Web WASM implementation
│       ├── zstd_unsupported.dart   # Stub for unsupported platforms
│       └── third_party/
│           └── zstd.dart           # Auto-generated ffigen bindings (not linted)
├── src/
│   └── zstd_wasm_helpers.c         # WASM-only C helper (i64 wrapper)
├── test/
│   ├── compression_test.dart       # Unit tests (native and web)
│   ├── frame_compat_test.dart      # Cross-platform frame compatibility tests
│   └── fixtures/
│       └── native_compressed.zst  # Golden fixture for cross-platform check
├── third_party/
│   └── zstd/
│       ├── src/zstd.c              # Single-file amalgamation of Zstd C library
│       ├── zstd.h                  # Zstd public header
│       ├── zdict.h                 # Dictionary API header
│       └── zstd_errors.h           # Error codes header
├── integration_test_app/           # Flutter app used for iOS/Android CI tests
├── .github/workflows/ci.yml        # GitHub Actions CI/CD pipeline
├── Makefile                        # Developer and CI task runner
├── VERSION_ZSTD                    # Single source of truth for vendored version
└── pubspec.yaml                    # Package manifest and ffigen config

4 Public API

The public surface is exported from lib/betto_zstd.dart and consists of six symbols:

4.1 ZstdSimple

A synchronous compress/decompress interface over Uint8List.

class ZstdSimple {
  ZstdSimple({int level = 3});  // level defaults to ZSTD_CLEVEL_DEFAULT

  static Future<void> init();   // web: loads WASM; native: no-op
  // On web only, init() also accepts an optional wasmUrl parameter:
  // static Future<void> init({String wasmUrl = '<flutter-asset-path>'});

  String get version;           // Zstd library version string
  Uint8List compress(List<int> data);
  Uint8List decompress(
    List<int> data, {
    int maxOutputBytes = defaultMaxOutputBytes,
  });
}

init() must be awaited once before any use on the web platform (it loads the WASM module). On native platforms it is a synchronous no-op, but callers should always await it so the same call site works on all platforms. The optional wasmUrl parameter is available on native and web; it is ignored on native (accepted so callers do not need platform guards) and overrides the default Flutter asset path on web.

compress() allocates the output buffer using ZSTD_compressBound, performs a single-shot compression, and returns the compressed bytes trimmed to the actual compressed size.

decompress() reads the declared original content size from the Zstd frame header (ZSTD_getFrameContentSize) and rejects it before allocating anything for it if the value is negative or exceeds maxOutputBytes, throwing ZstdLimitExceededException. Only once the declared size passes that check does it allocate the exact output buffer, decompress, and return the result. It also throws if the frame header is invalid or the content size is unknown (streaming frames are not supported).

maxOutputBytes defaults to defaultMaxOutputBytes (64 MiB — see defaultMaxOutputBytes). There is no unbounded/opt-out mode: a caller that needs to decompress something larger passes a larger explicit integer (e.g. 1 << 40). This exists because decompress must size its output buffer from the frame’s declared content size before verifying anything about the frame — and that declared size is a property of untrusted input, not something the caller controls. See Known Limitations for why a bound, rather than streaming, is the mitigation.

4.2 ZstdException

Thrown by compress and decompress when the Zstd library reports an error or the frame header is invalid. Implements Exception, so it is caught by both on ZstdException and on Exception clauses.

class ZstdException implements Exception {
  final String message;
  const ZstdException(this.message);
}

4.2.1 ZstdLimitExceededException

A subtype of ZstdException, thrown by decompress when a frame’s declared decompressed size is negative or exceeds maxOutputBytes. Carries the raw numbers — not just a formatted message — so a caller such as KMDB’s quarantine path can decide what to do without parsing prose:

class ZstdLimitExceededException extends ZstdException {
  final int declaredSize;  // may be negative — see Known Limitations
  final int limit;         // the maxOutputBytes value that was exceeded
  ZstdLimitExceededException(this.declaredSize, this.limit);
}

Because it extends ZstdException, existing on ZstdException handlers catch it without modification.

4.3 defaultMaxOutputBytes

const int defaultMaxOutputBytes = 64 * 1024 * 1024; // 64 MiB

The default value of decompress’s maxOutputBytes parameter. Declared once in lib/src/zstd_limits.dart and imported by all three implementations (native, web, unsupported stub) so the default cannot drift between them. 64 MiB is 64x KMDB’s kMaxDecodedValueBytes (1 MiB) decoded-value ceiling, so it does not fire in normal KMDB use.

Peak transient memory for a decompress call within the cap is a multiple of maxOutputBytes, not the cap itself — approximately 2x on native (the native buffer plus the copy into a Dart-managed Uint8List) and approximately 3x on web (the WASM heap allocation, a sublist copy, and a Uint8List.fromList copy of that), and the WASM heap grows to fit but never shrinks back down.

4.4 minCLevel() / maxCLevel()

Top-level functions returning the minimum and maximum compression levels supported by the underlying library.

5 Platform Dispatch

lib/betto_zstd.dart uses Dart’s conditional export mechanism to select the correct implementation at compile time, and unconditionally exports the platform-independent symbols alongside it:

export 'src/zstd_exception.dart' show ZstdException, ZstdLimitExceededException;
export 'src/zstd_limits.dart' show defaultMaxOutputBytes;
export 'src/zstd_unsupported.dart'
    if (dart.library.ffi) 'src/zstd_native.dart'
    if (dart.library.js_interop) 'src/zstd_web.dart'
    show ZstdSimple, minCLevel, maxCLevel;

The compiler resolves the conditional export at build time based on which Dart SDK libraries are available for the target. The three implementations present identical APIs so callers never need to handle platform differences. ZstdException, ZstdLimitExceededException, and defaultMaxOutputBytes are declared once, outside the conditional export, so their definitions cannot drift between platforms.

6 Native Platform Implementation

File: lib/src/zstd_native.dart

6.1 FFI Binding Strategy

The native implementation uses dart:ffi @Native annotations to declare direct bindings to the compiled shared library. No DynamicLibrary.open call is required; the Native Assets system (via the build hook) registers the library with the Dart runtime at build time using the assetName declared in hook/build.dart.

The set of native functions called:

The generated bindings in lib/src/third_party/zstd.dart (produced by dart run ffigen) are present for reference and historical tooling use, but the active native path uses @Native annotations exclusively, which avoid the DynamicLibrary lookup overhead.

6.2 Memory Management

All native memory is allocated via package:ffi’s malloc allocator. Each allocation is guarded by its own try/finally — the destination allocation’s try is nested inside the source allocation’s try — rather than two malloc calls followed by one finally, so a failure allocating the second buffer can never leak the first (this closes finding F-1, previously present in compress). The pattern for both compress and decompress is:

1. Allocate source buffer, copy List<int> data into it, inside a try.
2. In decompress only: read the declared content size from ZSTD_getFrameContentSize and reject it (throwing ZstdLimitExceededException, with nothing yet allocated for it) if it is negative or exceeds maxOutputBytes.
3. Allocate destination buffer (sized by ZSTD_compressBound or, for decompress, the now-validated declared content size), inside a nested try.
4. Call the Zstd C function.
5. Copy the result into a Dart-managed Uint8List.
6. Free the destination buffer in the inner finally, then the source buffer in the outer finally — both unconditional.

6.3 Build Hook

File: hook/build.dart

The Native Assets build hook is invoked automatically during dart build, flutter build, and dart test. It:

1. Calls _assertVersionPinned() — reads VERSION_ZSTD and parses ZSTD_VERSION_MAJOR, ZSTD_VERSION_MINOR, ZSTD_VERSION_RELEASE from third_party/zstd/zstd.h; throws if they do not match.
2. Constructs a CBuilder.library targeting third_party/zstd/src/zstd.c with LinkModePreference.dynamic.
3. On Windows, defines ZSTD_DLL_EXPORT=1 so MSVC emits __declspec(dllexport) for the public symbols (without this, symbols are absent from the import table and the FFI resolver throws error 127 at runtime).
4. Routes the output library to the app bundle via ToAppBundle().

No iOS companion plugin. native_toolchain_c cross-compiles zstd.c for iOS and Android as part of this hook, so no companion Flutter plugin is needed. This contrasts with packages such as betto_onnxrt, which depend on a pre-built XCFramework that cannot be compiled from source via the native assets toolchain — those packages require an SPM companion plugin (e.g. betto_onnxrt_ios) to deliver the binary and use DynamicLibrary.process() to resolve symbols already linked into the host process. Because zstd is a vendored C source file, the toolchain builds and registers it on every native target without any additional plugin machinery.

7 Web Platform Implementation

File: lib/src/zstd_web.dart

7.1 WASM Module

The WASM module at lib/assets/zstd.wasm (~317 KB) is compiled from third_party/zstd/src/zstd.c and src/zstd_wasm_helpers.c using Emscripten. It is checked in so that the web path works without the Emscripten toolchain at development time. The module is declared as a Flutter asset in pubspec.yaml.

Emscripten flags used for the build (make wasm):

7.2 Initialisation

ZstdSimple.init() fetches the WASM binary via the browser fetch API, instantiates it with WebAssembly.instantiate, and stores the exported function table in a module-level _ZstdExports? _exports variable. Subsequent calls to init() are no-ops. Calling compress or decompress before init() completes throws StateError.

The WASM module imports exactly one host function: env.emscripten_notify_memory_growth(memoryIndex: i32), which fires when ALLOW_MEMORY_GROWTH expands the heap. The shim is a no-op because lib/src/zstd_web.dart re-fetches a fresh Uint8List view over the WASM linear memory on every heap access.

7.3 Memory Management

The WASM module exposes malloc and free from its C runtime. The Dart web implementation uses these to allocate and free buffers within the WASM linear memory, following the same allocate/copy/call/copy-out/free pattern as the native implementation, with the same nested-try structure as native (see Native Memory Management above) and the same declared-size rejection in decompress before the destination buffer is allocated. After any WASM call that may trigger heap growth, the memory view is re-fetched via _heap() to avoid operating on a stale ArrayBuffer reference.

_malloc checks for a null return. Unlike package:ffi’s allocator on native, which throws on allocation failure, the WASM module’s C malloc returns 0 on failure. _malloc (the shared internal helper used at every allocation site in both compress and decompress) throws ZstdException when the underlying malloc returns 0, rather than letting the caller proceed to write at heap offset 0.

7.4 i64 Interop Workaround

File: src/zstd_wasm_helpers.c

ZSTD_getFrameContentSize returns unsigned long long (i64 in 32-bit WASM). JavaScript Number cannot represent arbitrary i64 values without WASM_BIGINT, which would require a more complex interop setup. Instead, zstd_wasm_helpers.c provides ZSTD_getFrameContentSize32, which converts the result to uint32_t:

• ZSTD_CONTENTSIZE_UNKNOWN → 0xFFFFFFFF (arrives in Dart as −1 via sign extension)
• ZSTD_CONTENTSIZE_ERROR → 0xFFFFFFFE (arrives in Dart as −2)
• Values above 4 GiB → treated as ZSTD_CONTENTSIZE_ERROR

This truncation is not a safe simplification on its own — WASM i32 results are sign-extended into a JS Number, so any declared size in [2 GiB, 4 GiB) survives the conversion above but arrives in Dart as a negative number (neither the -1 nor -2 sentinel). Before the maxOutputBytes bound existed, this meant the previously-documented “4 GiB cap” was, in practice, a 2 GiB cap: declaring any size at or above 2 GiB caused decompress to either be rejected outright (if a naive check had existed) or, prior to this fix, reach _malloc with a negative size. ZstdSimple.decompress now rejects any declared size that is negative or exceeds maxOutputBytes before allocating anything for it, which closes this gap regardless of where in the 32-bit range the declared size falls — see defaultMaxOutputBytes. This is the operative bound today, not the 4 GiB (or, pre-fix, 2 GiB) interop ceiling.

8 Unsupported Platform Stub

File: lib/src/zstd_unsupported.dart

When neither dart:ffi nor dart:js_interop is available (a scenario that does not arise in practice with current Dart targets), every public member throws UnsupportedError. This stub ensures the conditional export always resolves to a valid Dart file. Its decompress accepts the same maxOutputBytes named parameter (defaulting to defaultMaxOutputBytes) as the native and web implementations, so the three implementations’ signatures stay identical even though the stub throws before the parameter is ever used.

9 FFI Bindings Generation

The file lib/src/third_party/zstd.dart is auto-generated by dart run ffigen using the configuration in pubspec.yaml. The ffigen config exposes only the four functions needed by the original binding approach: ZSTD_compress, ZSTD_decompress, ZSTD_minCLevel, ZSTD_maxCLevel. This file is excluded from analysis, linting, and license checks.

To regenerate after updating the C header:

dart run ffigen

10 Vendored C Source

The Zstd C library is vendored as a single-file amalgamation, produced by Zstd’s create_single_file_library.sh script:

third_party/zstd/
├── src/zstd.c          # Single-file amalgamation (~1.7 MB)
├── zstd.h              # Public API header
├── zdict.h             # Dictionary API header
└── zstd_errors.h       # Error code definitions

10.1 Version Tracking

VERSION_ZSTD at the repository root is the single source of truth for the vendored version. The build hook enforces that it matches the version macros compiled into zstd.h at every build.

10.2 Updating the Vendored Version

1. Download the desired Zstd release tarball from the Zstd GitHub releases page.
2. Run create_single_file_library.sh from the Zstd source tree.
3. Copy zstd.h, zstd.c, zdict.h, zstd_errors.h to third_party/zstd/.
4. Update VERSION_ZSTD to the new version string (e.g. 1.5.8).
5. Run dart run ffigen to regenerate lib/src/third_party/zstd.dart.
6. Run make wasm to rebuild lib/assets/zstd.wasm (requires Emscripten).
7. Commit all changed files including the updated zstd.wasm.

The deferred make update_zstd automation (which would script steps 1–4) is tracked in docs/plans/plan_betto_zstd_vendor_automation.md.

11 Testing

11.1 Unit Tests (test/compression_test.dart)

Run on both the native VM and Chrome. Tests cover:

• Round-trip correctness at default, minimum, and maximum compression levels.
• Empty input and single-byte round-trips.
• Invalid compression level → ArgumentError.
• Decompressing invalid/truncated data → Exception.
• Decompressing a frame with unknown content size (no FCS field) → Exception.
• Highly compressible data produces smaller output than input.
• Return type is Uint8List.
• maxOutputBytes bound (S-2): decompressing a tiny frame (~8 MiB of zeros, which compresses to a few hundred bytes) with a small maxOutputBytes throws ZstdLimitExceededException with the expected declaredSize/limit values; a payload exactly at the cap succeeds and one byte over throws; a round-trip within the default cap is unchanged.
• Negative-size window: a hand-built frame whose declared content size arrives negative is rejected before allocation rather than reaching the allocator — covered on native (declared size ≥ 2^63) and on web separately (declared size in [2 GiB, 4 GiB), testOn: 'browser'), since the two platforms’ interop representations make different values negative.
• F-1 regression: a compress round-trip stays correct after the nested try/finally restructure (the leak itself is not directly unit-testable from Dart — verified by code review plus this staying green).

11.2 Frame Compatibility Test (test/frame_compat_test.dart)

Verifies that Zstd frames produced by one platform are decodable by the other:

1. Round-trip on current platform — compress and decompress a fixed 1 KB payload (0x00..0xFF × 4) and assert equality.
2. Golden fixture check (native only) — on first run, the native path writes test/fixtures/native_compressed.zst. On subsequent runs (including on Chrome), the fixture is loaded and decompressed; the result must match the known payload. This catches any frame-format drift between the native FFI and WASM paths.
3. Web WASM round-trip — compress and decompress the same payload via the WASM path and assert equality and compression ratio.

The golden fixture file is committed to the repository.

11.3 Integration Tests (integration_test_app/)

A minimal Flutter application used as a test harness for iOS and Android. It does not ship as a user-facing app. Tests exercise the native FFI path on a real device/emulator:

• Round-trip compress/decompress with level 3.
• Empty input round-trip.
• Compression level bounds (minCLevel() < 0, maxCLevel() > 0).

These tests are run locally via make android_test (connected Android emulator/device) and make ios_test (connected iOS simulator/device). Automated CI coverage for iOS and Android is deferred to a post-0.1.0 release.

11.4 Running Tests

The Makefile is the primary interface for running tests. Do not invoke dart test directly — use the appropriate make target so that prerequisites (dependency fetch, coverage tooling) are handled consistently.

12 CI/CD Pipeline

File: .github/workflows/ci.yml

The pipeline runs on every push and pull request to main.

Coverage is uploaded as an artifact (coverage/lcov.info) from the build job.

13 CLI Tool

File: bin/dartz.dart

A command-line utility for file compression and decompression:

dart run bin/dartz.dart <filename>             # compress → <filename>.zst
dart run bin/dartz.dart -d <input.zst> <out>  # decompress

Reports original and compressed/decompressed sizes. Writes errors to stderr and exits with code 1 on failure.

Decompression passes an explicit maxOutputBytes: 1 << 40 rather than relying on defaultMaxOutputBytes (64 MiB): the file being decompressed was chosen locally by whoever is running the tool, so it is trusted input, unlike the untrusted-network-data case the library default is sized for.

14 Developer Workflow

14.1 Makefile Targets

14.2 License Headers

All .dart files under lib/ and test/ (except auto-generated files in lib/src/third_party/, generated docs, and coverage output) must carry an Apache 2.0 header. The addlicense tool is configured via addlicense_config.txt.

15 Dependencies

15.1 Runtime

15.2 Dev

16 Error Handling

17 Known Limitations

• Single-shot only. Both the native and web implementations use the simple one-shot Zstd API. Streaming compression/decompression (for payloads whose uncompressed size is not stored in the frame header, or for very large data) is not supported. This is precisely why decompress bounds its allocation with maxOutputBytes rather than offering an unbounded mode: because the whole output buffer must be sized and allocated up front from the frame’s declared (and untrusted) content size, an unbounded decompress would be a memory-exhaustion vector for any caller handling untrusted input. A future streaming API — decompressing in bounded chunks regardless of the declared size — remains a possible enhancement, but is out of scope here.
• Web content size interop ceiling, corrected. ZSTD_getFrameContentSize32 truncates the 64-bit content size to 32 bits, and previously-documented text described this as a “4 GiB cap”. That is not accurate: because the i32 result is sign-extended into a JS Number, any declared size in [2 GiB, 4 GiB) arrives as a negative number rather than a large positive one — so the real pre-fix ceiling was 2 GiB, not 4 GiB. Only declared sizes at or above 4 GiB (0xFFFFFFFE+) are mapped to ZSTD_CONTENTSIZE_ERROR by the C helper. decompress’s maxOutputBytes bound (default defaultMaxOutputBytes, 64 MiB — see defaultMaxOutputBytes) is the operative limit in practice today, well below either the 2 GiB or 4 GiB interop boundary.
• No dictionary support. The Zstd dictionary API (zdict.h) is vendored but not exposed.