WASM Packages via HTTP Registry

How WASM packages become installable from an HTTP registry using the same flow as JS packages. No special WASM handling in the registry, installer, or package system — just packages with binary assets.

Goal

Today, WASM packages (vim, ratatui-counter, ratatui-files, ratatui-sysmon) are only available via .catalog() — pre-bundled host-side factories that use import.meta.url to resolve .wasm and .mjs assets. This limits them to build-time composition.

After this change, WASM packages follow the same pkg update && pkg install <name> flow as JS packages:

# Same flow for both JS and WASM packages:
$ pkg update                        # fetch index.json from registry
$ pkg install cowsay                # JS package — bundle only
$ pkg install counter               # WASM package — bundle + binary assets

The .catalog() path continues to work unchanged for offline/embedded use.

npm Convention Alignment

This design follows established npm conventions for shipping WASM:

Convention	npm precedent	fishbowl equivalent
Binary assets in the package tarball	esbuild-wasm, sql.js, ffmpeg.wasm ship `.wasm` alongside JS	Registry serves `.wasm` and `.mjs` as assets alongside `bundle.js`
Package entry point loads assets at runtime	`sql.js` uses `locateFile`, esbuild-wasm uses `wasmURL`	Bundle factory receives `AssetResolver`, calls `resolver.resolve('foo.wasm')`
Package.json declares what ships	`"files": ["index.js", "foo.wasm", "foo.mjs"]`	`fishbowl.assets: ["foo.wasm", "foo.mjs"]` in package.json
No special registry treatment	npm registry serves `.wasm` files as regular tarball contents	HTTP registry serves `.wasm` files as regular static files
Host-side resolution via import.meta.url	Standard ESM pattern for co-located assets	Preserved as fallback when no resolver is injected
Install-time asset fetching	`postinstall` scripts download platform binaries	`pkg install` preloads declared assets before bundle evaluation

Key principle: a WASM package is just a package that happens to have binary assets. The registry doesn’t know about WASM. The installer doesn’t know about WASM. The bundle format doesn’t know about WASM. Only the bin function inside the package knows it’s loading a .wasm binary.

This applies equally to both Emscripten and WASI packages. WASI packages are even simpler — no glue file transformation needed, just the .wasm binary as a single asset.

Current State

What works (JS packages via HTTP registry)

pkg update
  → reads /etc/pkg/sources.list → "http://host/registry"
  → urlToVfsPath() → "/net/http/host/registry"
  → reads /net/http/host/registry/index.json via httpFS
  → caches at /pkg/.cache/index.json

pkg install cowsay
  → resolve("cowsay", index, installed) → InstallPlan
  → fetch bundle: /net/http/host/registry/packages/cowsay/1.0.0.bundle.js
  → evaluateBundle(text, nullResolver()) → Extension { bins: { cowsay } }
  → registerExec + write install record

What works (Emscripten WASM packages via catalog)

Unix().catalog('counter', ratauiCounter).build().boot()
  → factory stored in memory as () => Extension
  → pkg install counter → proc.catalogInstall('counter')
  → calls factory() → createWasmBin(import.meta.url, ...) loads .wasm/.mjs
  → Extension { bins: { counter } }

What works (WASI packages via HTTP registry)

pkg install test-wasi-echo
  → resolve("test-wasi-echo", index, installed) → InstallPlan
  → fetch bundle + preload asset: test-wasi-echo.wasm
  → evaluateBundle(text, resolver) → factory receives resolver
  → createWasiBin(resolver, 'test-wasi-echo.wasm') — no glue file
  → Extension { bins: { 'test-wasi-echo': bin } }

WASI packages are simpler than Emscripten packages in the registry: no .mjs glue file means no glue transformation step. The generate-registry tool places WASI .wasm assets flat alongside the bundle (not in a version subdirectory).

Demo registry WASI packages: test-wasi-echo, test-wasi-cat, test-wasi-env.

The gap (Emscripten only — resolved)

createWasmBin() now accepts both string (import.meta.url) and AssetResolver as the first argument. The dual-overload pattern is shared by createWasiBin() which was designed with it from the start.

Existing Infrastructure (no changes needed)

The package system already has full support for binary assets. It’s just never been exercised with WASM packages:

Component	File	What it does
`VersionEntry.assets`	`src/runtime/pkg/types.ts`	Index entries declare asset paths
`InstallStep.assets`	`src/runtime/pkg/types.ts`	Resolved asset VFS paths per install step
Asset preloading	`src/bins/system/pkg.ts:329-376`	Reads all declared assets into memory before evaluation
`cachedResolver()`	`src/runtime/pkg/evaluator.ts`	In-memory cache resolver injected into bundle factory
`AssetResolver`	`src/runtime/pkg/types.ts`	Interface: `resolve(name) → Uint8Array`, `loadModule(name) → unknown`
`evaluateBundle()`	`src/runtime/pkg/evaluator.ts`	Passes resolver to `exports.default(resolver)`
`bundlePackage()`	`tools/generate-registry/src/bundler.ts`	Copies declared assets to registry output, wraps bundle with resolver injection
`httpFS`	`src/kernel/fileservers/http.ts`	Serves binary files (`.wasm`) over HTTP via `fetch()`

Design

Bundle format

WASM bundles use the same CJS wrapper as JS bundles. The factory receives assetResolver and uses it to load binary assets:

// packages/counter/1.0.0.bundle.js (generated by generate-registry)
exports.default = function(assetResolver) {
  // ... bundled code from index.ts + createWasmBin + wasmExec ...
  // createWasmBin internally calls:
  //   assetResolver.resolve('counter.wasm')   → Uint8Array
  //   assetResolver.loadModule('counter.mjs') → glue factory
  return __pkg.default(assetResolver);
};

This is identical to how JS bundles work — exports.default = function(assetResolver) { ... }. The resolver is unused by pure-JS packages (they call nullResolver()). WASM packages use it for their binary assets.

Emscripten glue handling (Emscripten packages only)

WASI packages skip this section entirely — they have no glue file. The generate-registry tool detects WASI packages (no .mjs/.js assets) and places the .wasm asset flat alongside the bundle with no transformation.

The Emscripten .mjs glue file is an ESM module with syntax that cannot be evaluated by new Function():

export default Module (ESM export syntax)
import.meta.url (3 occurrences — ESM-only construct)
await import("module") (top-level await + Node.js dynamic import)

These are parse-time syntax errors in new Function() context, not runtime errors. The evaluateCjsModule() path in loadModule() will fail.

Solution: bundle the glue into the main bundle at registry generation time. The generate-registry tool transforms the glue into a CJS-compatible form during bundlePackage():

esbuild bundles the WASM package entry point (index.ts → bundle.js)
The entry point imports createWasmBin, which imports wasmExec — these are bundled inline
When the resolver path is taken, createWasmBin calls resolver.loadModule('counter.mjs') to get the Emscripten factory
loadModule() in cachedResolver calls evaluateCjsModule() on the glue text

For step 4 to work, the glue must be CJS-compatible. The generate-registry tool must pre-process the glue file during asset preparation:

// Transform at registry build time (in bundler.ts):
// 1. Strip `export default Module;` → append `module.exports = Module;`
// 2. Replace `import.meta.url` → `""` (browser path doesn't use it)
// 3. Remove `await import("module")` Node.js branch (dead code in browser)
// 4. Wrap in async IIFE if needed for the `await` at top level

The transformed glue ships as the .mjs asset in the registry. The original ESM glue remains in the package source for host-side import.meta.url resolution.

Decision: glue transformation happens at registry build time, not at install time. The registry serves pre-processed, CJS-compatible glue. This follows npm’s model where packages are pre-built before publishing. The alternative — transforming at install time in loadModule() — would add complexity to the runtime and make debugging harder.

Registry index

WASM packages declare their assets in index.json, just like the architecture doc already specifies:

{
  "name": "counter",
  "versions": [{
    "version": "1.0.0",
    "description": "Ratatui counter TUI demo",
    "bins": ["counter"],
    "bundle": "packages/counter/1.0.0.bundle.js",
    "assets": [
      "packages/counter/1.0.0/counter.wasm",
      "packages/counter/1.0.0/counter.mjs"
    ]
  }]
}

Registry file layout

registry/
  index.json
  packages/
    cowsay/
      1.0.0.bundle.js          # JS-only — no assets
    counter/
      1.0.0.bundle.js          # Emscripten WASM — bundle (JS wrapper + wasmExec runtime)
      1.0.0/
        counter.wasm            # binary asset (~297KB)
        counter.mjs             # Emscripten glue (pre-processed to CJS)
    test-wasi-echo/
      1.0.0.bundle.js          # WASI — bundle (JS wrapper + wasiExec runtime)
      test-wasi-echo.wasm       # WASI binary (placed flat — no glue, no version subdir)

WASI assets are placed flat alongside the bundle because there is no glue file to version-separate. The generate-registry tool handles this layout automatically.

Install flow (same as JS, assets preloaded)

pkg install counter
  1. Check /pkg/installed/counter.json → not installed
  2. Check proc.catalogInstall → not in catalog
  3. Read cached index → find [email protected] with assets
  4. resolve("counter", index, installed) → InstallPlan with 1 step
  5. Fetch bundle: /net/http/.../packages/counter/1.0.0.bundle.js
  6. Preload assets concurrently:     ← ALREADY IMPLEMENTED in pkg.ts
     - /net/http/.../packages/counter/1.0.0/counter.wasm → Uint8Array
     - /net/http/.../packages/counter/1.0.0/counter.mjs  → Uint8Array
  7. Build cachedResolver(cache, pkgBase)  ← ALREADY IMPLEMENTED
  8. evaluateBundle(text, resolver)         ← ALREADY IMPLEMENTED
     → factory receives resolver
     → createWasmBin uses resolver.resolve() / resolver.loadModule()
     → returns Extension { bins: { counter: counterBin } }
  9. registerExec("counter", fn)
  10. Write /pkg/installed/counter.json

Steps 6-8 are already implemented and wired up. The missing pieces are in createWasmBin (doesn’t accept a resolver) and glue file preparation (ESM → CJS at build time).

Changes Required

1. `createWasmBin` / `createWasiBin` — accept AssetResolver as alternative to import.meta.url

Emscripten: src/runtime/wasm/create-wasm-bin.ts — canonical location within packages/core/. A re-export shim exists at ports/test-helpers/create-wasm-bin.ts for port packages.

WASI: src/runtime/wasm/create-wasi-bin.ts — created with dual-overload support from the start.

Both follow the same pattern. createWasmBin takes (source, wasmFile, glueFile, opts) (4 args — glue required). createWasiBin takes (source, wasmFile, opts) (3 args — no glue). Both accept string | AssetResolver as the first argument:

// Overload 1: host-side (existing) — resolve via import.meta.url
export function createWasmBin(
  metaUrl: string,
  wasmFile: string,
  glueFile: string,
  opts?: CreateWasmBinOptions,
): BinFunction

// Overload 2: registry-side — resolve via AssetResolver
export function createWasmBin(
  resolver: AssetResolver,
  wasmFile: string,
  glueFile: string,
  opts?: CreateWasmBinOptions,
): BinFunction

The bin function checks which was provided:

If string → existing loadAssets(metaUrl, ...) path (host-side, import.meta.url)
If AssetResolver → resolver.resolve(wasmFile) and resolver.loadModule(glueFile)

This follows the dual-resolution pattern from the architecture doc: import.meta.url for host-side, injected resolver for guest-side.

2. WASM package entry points — accept and forward resolver

Files: packages/ratatui-counter/index.ts, packages/ratatui-files/index.ts, packages/ratatui-sysmon/index.ts, packages/vim/index.ts, all test WASM packages

Currently:

const counterBin = createWasmBin(import.meta.url, 'counter.wasm', 'counter.mjs', { ... })

export default function ratauiCounter(): Extension {
  return { bins: { counter: counterBin } }
}

After:

export default function ratauiCounter(resolver?: AssetResolver): Extension {
  const counterBin = createWasmBin(
    resolver ?? import.meta.url,
    'counter.wasm', 'counter.mjs',
    { ttyMode: 'raw', preloadPaths: ['/tmp'] },
  )
  return { bins: { counter: counterBin } }
}

Key change: the createWasmBin call moves inside the factory so it can receive the resolver at call time. Previously it was module-level (bound to import.meta.url at import time).

Vim special case: The vim package wraps createWasmBin’s output with a vimBin function that injects --cmd args for terminal query suppression. This wrapping pattern must be preserved — both baseBin creation and vimBin wrapping move inside the factory:

export default function vim(resolver?: AssetResolver): Extension {
  const baseBin = createWasmBin(resolver ?? import.meta.url, 'vim.wasm', 'vim.js', {
    ttyMode: 'raw',
    env: { TERM: 'xterm-256color', VIMRUNTIME: '/usr/share/vim', VIM: '/usr/share/vim' },
    preloadPaths: computePreloadPaths,
  })
  const vimBin: BinFunction = async (proc): Promise<ExitCode> => {
    const injectedArgv = [proc.argv[0]!, '--cmd', SUPPRESS_QUERIES, ...proc.argv.slice(1)]
    return baseBin({ ...proc, argv: injectedArgv })
  }
  return {
    bins: { vim: vimBin, vi: vimBin },
    env: { TERM: 'xterm-256color', VIMRUNTIME: '/usr/share/vim', VIM: '/usr/share/vim', HOME: '/home' },
    files,
  }
}

3. Package.json `fishbowl.assets` field

Files: Each WASM package’s package.json (or equivalent metadata)

Declare asset files so generate-registry copies them and the index includes them:

{
  "name": "@fishnet/counter",
  "fishbowl": {
    "bins": ["counter"],
    "assets": ["counter.wasm", "counter.mjs"]
  }
}

This parallels npm’s "files" field — declaring what ships with the package.

4. `generate-registry` — handle WASM packages

File: tools/generate-registry/src/bundler.ts

Two changes needed:

4a. esbuild config for WASM entry points:

Verified by testing: esbuild with platform: 'browser' fails on node:url, node:path, node:fs/promises imports in create-wasm-bin.ts (the as string cast does not prevent esbuild from resolving them). Fix: add external: ['node:*'] to the esbuild config.

With external: ['node:*']:

Bundle succeeds at ~23KB (includes wasmExec, installTTYBridge, createWasmBin)
import.meta.url is replaced with empty string (dead code in registry path — resolver is always provided)
Node.js-only loadAssetsNode() code path remains but is never reached (runtime isNode check)

await esbuild.build({
  // ... existing config ...
  external: ['node:*'],
})

4b. Emscripten glue pre-processing:

Add a glue transformation step in bundlePackage() for assets ending in .mjs or .js (Emscripten glue):

// In bundlePackage(), after copying each asset:
if (asset.endsWith('.mjs') || isEmscriptenGlue(assetContent)) {
  const transformed = transformGlueToCjs(assetContent)
  await fs.writeFile(assetOutPath, transformed)
}

The transformation:

Replace export default <Name> with module.exports = <Name>
Replace import.meta.url with ""
Remove the await import("module") Node.js branch
Wrap the async function body if top-level await remains after branch removal

5. Demo web registry — add a WASM package

Files: demo/web/public/registry/index.json, new asset files in demo/web/public/registry/packages/counter/

Add counter to the demo HTTP registry alongside cowsay/fortune/greeting:

Copy counter.wasm to public/registry/packages/counter/1.0.0/
Pre-process counter.mjs (ESM → CJS) and copy to same location
Generate 1.0.0.bundle.js via generate-registry or hand-write
Add entry to index.json with assets field

This proves the flow works end-to-end in the browser.

What Does NOT Change

Catalog path — .catalog('counter', ratauiCounter) continues to work. The factory is called without a resolver, so resolver ?? import.meta.url falls back to import.meta.url.
Registry protocol — No new endpoints, no WASM-specific or WASI-specific handling. Assets are regular static files served by httpFS.
pkg bin — No changes. Asset preloading and resolver injection already work for both Emscripten and WASI packages.
evaluateBundle — No changes. Already passes resolver to factory.
AssetResolver interface — No changes. resolve() and loadModule() are sufficient. WASI packages only use resolve() (no loadModule() since there’s no glue file).
cachedResolver — No changes. Already caches binary data by VFS path.
httpFS — No changes. Already serves binary content via fetch().

Testing

New tests required

Glue transformation unit test — verify transformGlueToCjs() produces valid CJS from actual Emscripten output. Test against both Emscripten format variants (Pattern A: older, single async function Module; Pattern B: newer, IIFE wrapper around createModule).
WASM registry integration test — exercise the full pkg install flow for a WASM package from a file:// registry: index with assets → resolve → fetch bundle → preload assets → evaluate → register bin → run bin.
End-to-end browser test — pkg update && pkg install counter && counter in the web demo.

Existing tests (no changes expected)

All 19 WASM test packages in test/wasm/packages.test.ts — they use the catalog path, which is unchanged.
All package system tests (test/pkg/) — JS-only packages, unaffected.
The 1418 existing tests should continue passing.

Migration Path

~~Promote createWasmBin from packages/test-helpers/ to src/runtime/wasm/~~ (done)
Add AssetResolver | string overload to createWasmBin
Update WASM package entry points to accept resolver, move createWasmBin inside factory
Write transformGlueToCjs() in generate-registry with unit tests against real Emscripten output
Add external: ['node:*'] to generate-registry esbuild config
Add fishbowl.assets to WASM package.json files
Generate a WASM registry bundle and verify it evaluates correctly
Add a WASM package to the demo web registry
Verify: pkg update && pkg install counter && counter works in browser
All existing tests continue to pass (catalog path unchanged)

Risks

Risk	Mitigation
Emscripten glue ESM → CJS transformation is fragile	Two known format variants; test against real glue files from both C and Rust compilation. Pin Emscripten version in build scripts.
esbuild inlines entire WASM runtime (~23KB) into each package bundle	Acceptable for v1. Each package is self-contained. If deduplication is needed later, extract `wasmExec` to a shared runtime package.
esbuild replaces `import.meta.url` with empty string	Dead code in registry path — resolver is always provided. The `typeof first === 'string'` check in `createWasmBin` treats empty string as string path (which is fine — that code path is never reached when a resolver is injected).
Large `.wasm` files slow down `pkg install`	Assets are fetched concurrently. Future optimization: `WebAssembly.compileStreaming` for `.wasm` assets.
Moving `createWasmBin` inside factory changes caching semantics	Each factory call gets its own `cachedWasm`/`cachedFactory` closure. This is correct — each `pkg install` or `.use()` produces independent state. Multiple invocations of the same bin within a session share the cache.
Node.js imports remain in bundle as dead code	`external: ['node:*']` keeps them as external references. The `isNode` runtime check prevents them from being reached in browser. If this causes issues, the bundler can strip the `loadAssetsNode` function entirely with a custom esbuild plugin.