Table

A Table is a set of equal-length columns in one engine’s memory. Number and date columns hold one float per row (dates as milliseconds since 1970, UTC); text columns are dictionary-encoded, so region == 'EU' compiles to a number comparison. Tables come from engine.table(), parseCSV(), readCSV(), fromArrow() and the results of other operations; you rarely call the constructor or the statics yourself.

Filters are lazy: t.filter(p) returns a view with a condition attached, and the next aggregation runs filter and aggregation as one fused pass. Every method that does work has an …Async twin that runs on the worker threads and returns a promise; in browsers and on one thread the twins are what you use. Expressions are Vectro’s small language, documented in Expressions.

import { Table } from 'vectrojs';            // Node
import { Table } from 'vectrojs/browser';    // browsers
import { Table } from 'vectrojs/portable';   // edge runtimes

Making tables

Table.fromColumns(engine, columns, options?)

What engine.table(columns, options) calls: builds a table from { name: values }. Values are a Float64Array, Float32Array or number[], a string[] (category), an array of Date objects (date), or { codes, labels }. Options: type (Float64Array or Float32Array), dates (names of columns given as milliseconds or ISO text), compress. Throws if the columns differ in length or a name isn’t letters, digits and _.

Table.fromRows(engine, rows, options?)

What engine.tableFromRows(rows, options) calls: one column per key of the first row; a string there makes a category column. Option: type.

Shape

t.length

A getter: the rows the table holds, ignoring any filter. For the rows a filter matches, use t.count().

t.columnNames

A getter: the column names, in order.

t.type

A getter: the storage type of the number columns, Float64Array or Float32Array.

t.column(name)

The column’s array in engine memory, without a copy and ignoring the filter: numbers as they are, dates as milliseconds, category columns as codes into their labels. After compress() it holds the packed codes; use toColumns() for values. Throws for an unknown column.

t.memoryBytes()

Returns { bytes, uncompressed }: the bytes the columns take now and without packing.

t.compress()

Stores each number and category column in 1, 2 or 4 bytes per row where every value decodes back exactly; other columns stay as they are. Works in place (views of the table keep working) and returns the table. Answers don’t change. Also compressAsync(). See Compressed columns.

Filtering and computing

t.filter(expr, params?)

Returns a view of the same columns with the condition expr attached; nothing is computed or copied. Filtering a filtered view combines both conditions with &&. params gives values for names that aren’t columns ({ min: 100 } for price > min); a Date param becomes milliseconds. A param can’t have a column’s name, and can’t change a value an earlier filter set.

orders.filter("region == 'EU' && price > min", { min: 100 })
orders.filter("ts >= '2024-03-01'")                     // date columns compare with ISO text
orders.filter("like(sku, 'EU-%') || in(region, 'US', 'CA')")

Where a filter has column < number parts, the zone maps (min and max per block of 16,384 rows, built on first use) skip blocks that can’t match. See Zone maps.

t.derive(name, expr, params?)

Returns a new table with one more column, expr computed for every row in one fused pass. The new table shares the other columns and keeps the filter; it owns only the new column. The name must not exist yet. A derived dayStart(ts), monthStart(ts) and the like stays a date column; a plain copy of a text column stays text. Also deriveAsync.

t.select(...names)

Returns a view with only these columns, in this order, without a copy. Columns the filter reads stay available to it. Names may also be passed as one array.

Aggregating

t.agg(spec, params?)

Computes several aggregations over the rows matching the filter in one pass and returns { name: value }. Each value of spec is one of:

Aggregation	Gives
count()	Rows.
count(expr)	Rows where expr holds (a condition), or where it has a value (any other expression).
sum(expr)	The sum, skipping missing values.
mean(expr)	The mean of the values present.
min(expr), max(expr)	The smallest, largest value present; NaN if there is none.
countDistinct(expr)	Distinct values, exact; missing values don’t count.
approxDistinct(expr)	The same on a table; split tables and streams use HyperLogLog sketches.

Also aggAsync. See Filter, derive, aggregate and Missing values.

t.sum(expr, params?)

agg({ v: 'sum(expr)' }).v. Likewise t.mean, t.min, t.max, and t.count(expr?) (count() without an expression). Also sumAsync, meanAsync, minAsync, maxAsync, countAsync.

t.groupBy(keys)

keys is a column name or an array of them: category columns or any numbers (IDs, amounts; NaN is its own group, -0 groups with 0). Returns an object with agg(spec, params?) and aggAsync(spec, params?), which take the aggregations of t.agg and return a new table: one row per group, the key columns first, sorted by key (text in dictionary order, numbers ascending, NaN last, earlier keys first). An aggregation can’t have a key’s name, and free-text columns can’t be keys. See Group by.

import { Engine } from 'vectrojs';

const engine = new Engine();
const orders = engine.table({
  region: ['EU', 'US', 'EU', 'APAC', 'US', 'EU'],
  customer: [1, 2, 1, 3, 4, 5],
  price: [120, 80, 45, 300, 99, NaN],
  qty: [2, 1, 4, 1, 3, 10],
});

console.log(orders.agg({ n: 'count()', priced: 'count(price)', avg: 'mean(price)', big: 'count(price > 100)' }));
console.log(orders.filter('qty >= q', { q: 2 }).sum('price * qty'));

const byRegion = orders.groupBy('region').agg({
  orders: 'count()',
  revenue: 'sum(price * qty)',
  customers: 'countDistinct(customer)',
});
console.log(byRegion.toRows());
await engine.close();

Sorting

t.sort(by)

Returns a new table with the matching rows in order of by: a column name, an expression ('price * qty'), or an array of them, earlier keys first. A leading - sorts large to small ('-price', '-(price * qty)'). Text sorts by label, ties keep table order, and missing values come last. 50,000 rows or more sort on every core. Also sortAsync. See Sort and top-k.

t.top(k, by)

The first k rows of sort(by), as a new table, without sorting everything: rows past a cutoff estimated from a sample are skipped on every thread, and only the rest are sorted. Also topAsync.

Joining

t.join(other, options)

A hash join with another table of the same engine. other is the side that is hashed, so put the smaller table there. Returns a new table: rows in this table’s order, each followed by its matches in other’s order.

Name	Type	Default	Meaning
on	string, string[] or { left, right }	required	A key column both tables have, several of them, or the key’s name on each side.
how	'inner' or 'left'	'inner'	'left' keeps rows with no match once, with NaN (numbers) or '' (text) in other’s columns.
suffix	string	'_right'	Added to other’s column names that this table already has.

Text keys match by label; a missing (NaN) key never matches. Also joinAsync. See Joins.

import { Engine } from 'vectrojs';

const engine = new Engine();
const orders = engine.table({ cust: [1, 2, 1, 3], price: [120, 80, 45, 300] });
const customers = engine.table({ id: [1, 2], tier: ['gold', 'basic'], price: [0, 0] });

const joined = orders.join(customers, { on: { left: 'cust', right: 'id' }, how: 'left' });
console.log(joined.columnNames);
console.log(joined.toRows());
console.log(joined.groupBy('tier').agg({ spend: 'sum(price)' }).toRows());
await engine.close();

Getting rows out

t.collect()

Returns a new table holding only the rows that match the filter, copied on every thread straight to their final place. Without a filter it returns this table itself, not a copy. Also collectAsync.

t.head(n)

The first n matching rows as a new table. When the table has n rows or fewer it returns the collected table (this table itself if there is no filter). Also headAsync.

t.slice(start, end?)

Matching rows start up to end (default: the end) as a new table. Also sliceAsync.

t.toRows(limit?)

The matching rows as plain objects, at most limit: numbers as numbers (missing values NaN), text as strings, dates as Date objects (null when missing). Also toRowsAsync.

t.toColumns()

The matching rows as plain columns, copied out of engine memory: a Float64Array (or Float32Array) per number or date column, { codes, labels } per category column, string[] per free-text column. Also toColumnsAsync.

t.toArrow(options?)

The matching rows as Arrow IPC bytes: the stream format, or with { file: true } the file format (.arrow / .feather). Categories become dictionary-encoded strings, dates Timestamp(ms, UTC). Also toArrowAsync. See Arrow.

t.lazy()

Starts a lazy query over this table: the steps are recorded and optimized as a whole before anything runs.

Memory

t.dispose()

Frees the columns this table allocated. Views (filter, select) allocate nothing, and a derive result owns only its new column, so disposing a view leaves the table it came from intact; disposing a table breaks views of it. Engine memory is capped at 4 GB and isn’t garbage collected, so dispose results you no longer need.

import { Engine } from 'vectrojs';

const engine = new Engine();
const t = engine.table({ day: [3, 1, 2, 1, 3], amount: [10, 20, 30, 40, 50] });
const big = t.filter('amount > 15');
const kept = big.collect();
console.log(t.length, big.length, kept.length);
console.log(t.derive('double', 'amount * 2').top(2, '-double').toRows());
console.log(t.sort(['day', '-amount']).toColumns());
console.log(t.slice(1, 3).toRows(), t.memoryBytes(), t.compress().memoryBytes());
kept.dispose();
await engine.close();