Engine

An Engine holds one block of WebAssembly memory (at most 4 GB) and, where the runtime allows it, one worker thread per core. Tables are made through it and keep their columns in its memory. Each entry point exports its own Engine: the Node one reads files and spawns worker threads, the browser one uses Web Workers on cross-origin-isolated pages, and the portable one runs on one thread with no Node or browser APIs. The core methods below are the same in all three.

import { Engine } from 'vectrojs';            // Node: worker threads on every core
import { Engine } from 'vectrojs/browser';    // browsers: Web Workers when isolated, else one thread
import { Engine } from 'vectrojs/portable';   // Deno, Bun, Cloudflare Workers: one thread

Making an engine

new Engine(options?)

Starts the threads and compiles the static kernels. In Node the worker threads are unref’d, so a script can exit without close(), but closing is still the clean way to stop them.

Name	Type	Default	Meaning
threads	number	all cores	Threads to use, the calling thread included. Needs a platform with workers and SharedArrayBuffer; otherwise 1.
jit	true, false or 'auto'	'auto'	true compiles one SIMD kernel per expression; false runs expressions on the vectorized interpreter; 'auto' compiles wherever the runtime allows it.
wasmModule	WebAssembly.Module	none	The precompiled dist/vectro.wasm, for runtimes that can’t compile WebAssembly (Cloudflare Workers). Single-threaded only.
initialMemoryMB	number	16 on one thread, 0 with threads	Memory to reserve at the start. Plain (non-shared) memory moves when it grows, so starting bigger avoids copies.
spin	number	20000 (Node), 0 (browser)	How long idle workers spin before they sleep.
minParallel	object	tuned per operation	Row counts below which an operation runs on the calling thread alone.
wasm	boolean	true	false uses JavaScript kernels for the low-level array operations.
platform	object	set by the entry point	Thread support. Node passes nodePlatform; leave it alone.

const engine = new Engine();                  // Node: every core
const small = new Engine({ threads: 2 });

On Cloudflare Workers, bundle the precompiled module and pass it in:

import { Engine } from 'vectrojs/portable';
import wasmModule from 'vectrojs/dist/vectro.wasm';
const engine = new Engine({ wasmModule });

See On the edge.

canCompileWasm()

Returns true when the runtime can compile WebAssembly from bytes. Cloudflare Workers and Vercel Edge return false; there the engine needs wasmModule. Exported from every entry point.

nodePlatform

The object the Node entry passes as platform: cpus(), spawn(workerData) and receive(port). Exported for completeness; you don’t need it to make an engine.

Making tables

Each of these returns a Table whose columns live in this engine’s memory.

engine.table(columns, options?)

Builds a table from columns, copying them into engine memory. A column is a Float64Array, Float32Array or number[] (numbers), a string[] (a category column: each distinct string gets a code), an array of Date objects (a date column), or { codes, labels } (codes into a labels array, a category column with no string lookups). All columns must have the same length.

Name	Type	Default	Meaning
type	Float64Array or Float32Array	Float64Array	Storage for every column. Float32Array halves memory but loses precision; tables with date columns are refused (times don’t fit in float32).
dates	string[]	[]	Columns that hold dates as milliseconds or ISO text. Arrays of Date objects are dates anyway.
compress	boolean	false	Store each column in 1, 2 or 4 bytes per row where every value comes back exactly. See Compressed columns.

Column names must be letters, digits and _, not starting with a digit.

import { Engine } from 'vectrojs';

const engine = new Engine();
const orders = engine.table({
  region: ['EU', 'US', 'EU', 'APAC'],
  price: [120, 80, 45, 300],
  qty: [2, 1, 4, 1],
  ts: [new Date('2024-03-01'), new Date('2024-03-02'), new Date('2024-03-02'), new Date('2024-03-05')],
});
console.log(orders.length, orders.columnNames);
console.log(orders.filter("region == 'EU'").agg({ n: 'count()', revenue: 'sum(price * qty)' }));
await engine.close();

engine.tableFromRows(rows, options?)

Builds a table from an array of plain objects. Column names and kinds come from the first row: a string there makes a category column, anything else a number column. Takes type as in table(). Each row is visited once, so prefer table() when you already have columns.

engine.parseCSV(text, options?)

Parses CSV given as a string or bytes (Uint8Array, Buffer) into a table, on every thread: the bytes are cut at row ends and each thread parses its share straight into the columns. Column types are guessed from a sample of rows: numbers, dates (ISO text), categories (text that repeats) and free text (text where most values differ, which can be returned but not filtered or grouped on).

Name	Type	Default	Meaning
delimiter	string	','	One character, not a quote or newline.
header	boolean	true	false: no header line; columns are named col0, col1…
types	object	{}	Per column: 'number', 'category', 'string' (free text), 'text' (category or string, from the sample) or 'date'.
columns	string[]	all	Load only these columns.
type	Float64Array or Float32Array	Float64Array	Number storage.
sampleRows	number	1000	Rows read to guess the types.
compress	boolean	false	Compress the columns after parsing.

Header names are cleaned to work in expressions: Order Date becomes Order_Date, a leading digit gets _ in front, and repeated names get _2, _3. A UTF-8 byte order mark is skipped. Blank fields load as missing (NaN for numbers, '' for text).

Also parseCSVAsync(text, options?), which parses on the worker threads and returns a promise.

import { Engine } from 'vectrojs';

const engine = new Engine();
const csv = 'day,region,amount\n2024-03-01,EU,10.5\n2024-03-01,US,7\n2024-03-02,EU,3\n';
const t = engine.parseCSV(csv);
console.log(t.groupBy('region').agg({ total: 'sum(amount)' }).toRows());
const typed = await engine.parseCSVAsync(csv, { types: { region: 'string' }, columns: ['region', 'amount'] });
console.log(typed.toRows(2));
await engine.close();

engine.readCSV(path, options?)

Node only. Reads a CSV file straight into engine memory and parses it as parseCSV does (same options, compress included). Also readCSVAsync(path, options?), which reads with async I/O and parses on the worker threads.

import { mkdtempSync, writeFileSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { Engine } from 'vectrojs';

const dir = mkdtempSync(join(tmpdir(), 'vectro-'));
const file = join(dir, 'sales.tsv');
writeFileSync(file, 'region\tamount\nEU\t10\nUS\t20\nEU\t5\n');

const engine = new Engine();
const sales = engine.readCSV(file, { delimiter: '\t' });
console.log(sales.sum('amount'), await (await engine.readCSVAsync(file, { delimiter: '\t' })).countAsync());
await engine.close();
rmSync(dir, { recursive: true, force: true });

engine.fromArrow(bytes, options?)

Loads Arrow IPC bytes (the stream format or the file format, .arrow / .feather, LZ4-compressed bodies included) as a table. Options: columns (names to read; a column of a type Vectro can’t read is an error unless left out) and type. See Arrow.

Datasets

Node methods for datasets, tables kept on disk and read a chunk at a time. Each returns or takes the objects documented there.

engine.openDataset(path)

Opens a .vectro file for queries and returns a Dataset. Only the footer is read. Also openDatasetAsync(source), which also takes a URL (read with HTTP range requests), a Blob or File, or a file object from blobFile or urlFile.

engine.createDataset(path, options?)

Returns a DatasetWriter for path: append() tables or columns as often as needed, then close(). Options: rowsPerChunk (default 4,194,304), append (add to an existing dataset with the same columns), bitPack (default true). Also createDatasetAsync(target, options?), which also takes a writable file object such as memoryFile() and returns a promise of the writer.

engine.writeDataset(path, source, options?)

Writes a table (the rows matching its filter), a lazy query, a dataset, or { column: values } as a dataset, and returns { path, rows, chunks, bytes }. A query over a dataset is written chunk by chunk, so its result needn’t fit in memory. Options as for createDataset. Also writeDatasetAsync(target, source, options?).

engine.csvToDataset(csvPath, path, options?)

Converts a CSV file of any size into a dataset, reading it in pieces of pieceBytes (default 64 MB). The column types come from the first piece. Takes the CSV options of parseCSV, the writer options of createDataset, and onProgress, called after each piece with { bytes, size, rows }. Returns the writer’s summary. Also csvToDatasetAsync(csv, out, options?), where csv may also be a URL or a Blob and out a writable file object.

In the browser

The vectrojs/browser engine has the core methods above and these instead of the Node file methods. All of them are async.

Method	Does
loadCSV(url, options?)	Fetches a CSV and parses it (on the workers when there are any).
openDataset(source)	Opens a dataset from a URL, a File or Blob, or a file object.
writeDataset(target, source, options?)	Writes into browser storage (OPFS) by name, to a FileSystemFileHandle, or to a file object such as memoryFile().
csvToDataset(csv, target, options?)	Converts a CSV (File, Blob or URL) of any size; chunks go to disk as they are made.
createDataset(target, options?)	A writer with appendAsync and closeAsync.
storedDatasets()	The .vectro files in browser storage, as [{ name, size }].
openStoredDataset(name)	Opens one of them.
storedDatasetFile(name)	One of them as a File, to download or open.
removeStoredDataset(name)	Deletes one of them.

On a page served with Cross-Origin-Opener-Policy: same-origin and Cross-Origin-Embedder-Policy: require-corp the engine runs Web Workers on every core; the async methods use them while the page stays responsive, and sync calls run on the page’s thread alone. Other pages get the one-thread engine. threadsAvailable(), also exported from vectrojs/browser, tells which you’ll get. For multi-core work on pages without those headers, use SplitEngine. See In the browser.

import { Engine } from 'vectrojs/browser';
const engine = new Engine();
const orders = await engine.loadCSV('/orders.csv');
await orders.filter("region == 'EU'").sumAsync('price * qty');

State and shutdown

engine.busy

A getter: true while an async query is running on the worker threads. A sync call made then throws, because it would overwrite the running task; await the async call first, or use async calls throughout. Async calls queue up and run one at a time.

engine.close()

Stops the worker threads and returns a promise.

Arrays in engine memory

These are the building blocks the tables use. Only arrays that live in engine memory get the WebAssembly SIMD kernels.

engine.alloc(Type, n)

Allocates a typed array of n elements of Type (for example Float64Array) in engine memory and returns it. Throws a RangeError past 4 GB.

engine.shared(Type, n)

The same as alloc.

engine.toShared(array)

Copies a normal typed array into engine memory and returns the copy.

engine.live(array)

Returns a working view of an array from alloc. Plain (non-shared) memory gets a new buffer when it grows, which empties older views; live() rebuilds the view. Shared memory never goes stale, and every engine method calls live() on its inputs, so you need it only for your own reads and writes after other allocations.

engine.free(array)

Returns an array from alloc to the engine. Throws for arrays it didn’t allocate.

engine.interpreterScratch(bytes)

Reserves (once) the scratch area the vectorized interpreter keeps its registers in, and returns its byte offset. The engine calls it itself; you don’t need to.

Kernels on arrays

Whole-array operations on all threads. Inputs are arrays from alloc or toShared (or views of a SharedArrayBuffer); small arrays run on the calling thread.

Method	Returns
engine.sum(a)	The sum of a.
engine.min(a), engine.max(a)	The smallest, largest value.
engine.dot(a, b)	The sum of a[i] * b[i].
engine.countGreater(a, x)	How many values are greater than x.
engine.add(a, b, out)	out, set to a[i] + b[i].
engine.scale(a, k, c, out)	out, set to a[i] * k + c.
engine.sort(a)	a, sorted in place (ascending; threads sort chunks, then merge).
engine.countBytes(bytes, pattern)	Occurrences of pattern (a string or bytes) in bytes.
engine.heavy(a)	The sum of sqrt(x) * sin(x) + cos(x / 2): a compute-bound benchmark kernel.

engine.compile(spec)

Compiles an expression into one SIMD kernel and returns it; compiled kernels are cached by their text. spec is a reduction ('sum(a * 2 + 1)'), an array of up to 8 reductions (['sum(a)', 'max(a)']), an object of them ({ total: 'sum(a)' }), or an elementwise expression ('a * k + b'). Reductions are sum, mean, min, max and count. The kernel has run(vars, out?) and runAsync(vars, out?): vars maps each name to a Float64Array or Float32Array from engine memory or to a number. All arrays must have the same type and length. A reduction returns a number, an array or an object, in the shape of spec; an elementwise expression writes into out and returns it. The expression language is in Expressions.

engine.eval(spec, vars, out?)

compile(spec).run(vars, out). Also evalAsync(spec, vars, out?).

engine.evalG(spec, vars, out?)

The generator form of eval, used inside the engine’s own operations (it yields “run this on the threads” requests to a driver). Use eval or evalAsync instead.

import { Engine } from 'vectrojs';

const engine = new Engine();
const a = engine.alloc(Float64Array, 1_000_000);
for (let i = 0; i < a.length; i++) a[i] = i % 100;
console.log(engine.sum(a), engine.max(a), engine.countGreater(a, 90));
console.log(engine.eval(['sum(a)', 'mean(a)', 'count(a > 50)'], { a }));

const out = engine.alloc(Float64Array, a.length);
engine.eval('a * k + 1', { a, k: 2 }, out);
console.log(out[3], await engine.evalAsync({ total: 'sum(a * a)' }, { a }));
engine.free(out);
engine.free(a);
await engine.close();