Tables

A table is a set of named columns of the same length, held in the engine’s WebAssembly memory. Every column is an array of numbers: number columns as they are, text as category codes, dates as milliseconds. That is what lets one compiled loop read any column the same way.

From columns

engine.table(columns) takes an object of column name to values. Values can be plain arrays or any typed array.

import { Engine } from 'vectrojs';

const engine = new Engine();
const t = engine.table({
  price: [12.5, 8, 30, 4.25],
  qty: new Int32Array([1, 3, 2, 10]),
  weight: new Float32Array([0.5, 1.25, 2, 0.1]),
});

console.log(t.length, t.columnNames, t.type.name);   // 4 [ 'price', 'qty', 'weight' ] Float64Array
console.log(t.sum('price * qty'));

await engine.close();

The values are copied into engine memory once, as float64 by default, whatever array they came in. Pass { type: Float32Array } to store every number column as float32: half the memory, about 7 significant digits.

Column names are letters, digits and _, not starting with a digit, so that they can be used in expressions. Infinity and NaN are taken.

Text becomes categories

An array of strings becomes a category column: each distinct label gets a code (0, 1, 2… in the order the labels first appear), and the column stores the codes. Comparisons, in(), like() and group-bys work on the codes, so text filters run as fast as number filters.

import { Engine } from 'vectrojs';

const engine = new Engine();
const t = engine.table({
  region: ['EU', 'US', 'EU', 'APAC'],
  price: [10, 20, 30, 40],
});

console.log(t.column('region'));                // Float64Array [0, 1, 0, 2]: the codes
console.log(t.filter("region == 'EU'").sum('price'));

// codes and labels you already have: no string is looked at per row
const fast = engine.table({
  region: { codes: new Uint8Array([0, 1, 1, 0, 2]), labels: ['EU', 'US', 'APAC'] },
  price: [1, 2, 3, 4, 5],
});
console.log(fast.groupBy('region').agg({ total: 'sum(price)' }).toRows());

await engine.close();

Giving { codes, labels } skips the step of looking every string up, which matters for millions of rows. Two labels that are the same string share one code.

CSV files can also have free-text columns (types: { note: 'string' }): values kept as text, returned in rows, but not usable in expressions. Load a column as a category to filter or group on it. See CSV, JSON lines, Arrow.

Dates

A date column holds milliseconds since 1970-01-01 UTC. An array of Date objects becomes a date column by itself; for milliseconds or ISO text, name the column in dates.

import { Engine } from 'vectrojs';

const engine = new Engine();
const t = engine.table({
  ts: ['2024-03-01T09:30:00Z', '2024-03-02', '2024-04-15T18:00:00+02:00'],
  shipped: [new Date('2024-03-03'), new Date('2024-03-05'), null],
  amount: [100, 250, 75],
}, { dates: ['ts'] });

console.log(t.filter("ts >= '2024-03-02'").toRows());
console.log(t.derive('month', 'month(ts)').select('ts', 'month').toRows());

await engine.close();

Rows come back with Date objects, and a missing date (null, empty text) as null. Text with an offset such as +02:00 is converted to UTC. Time zones other than UTC are not supported yet. Date functions and comparisons are on the Expressions page.

Missing values

A missing number is NaN; missing text is ''. Aggregations skip missing values as SQL does: see Missing values.

In a plain array, null and undefined load as missing: [1, null, 3] is stored as 1, NaN, 3. (A typed array has no null: Float64Array.from([1, null]) already holds 0.)

From rows

engine.tableFromRows(rows), or Table.fromRows(engine, rows), converts an array of objects. The columns and their kinds come from the first row: a string there makes a category column, anything else a number column.

import { Engine, Table } from 'vectrojs';

const engine = new Engine();
const rows = [
  { city: 'Paris', temp: 18.5, rain: 2 },
  { city: 'Oslo', temp: 9.0, rain: 5 },
  { city: 'Paris', temp: 21.0, rain: 0 },
];

const t = engine.tableFromRows(rows);       // same as Table.fromRows(engine, rows)
console.log(t.groupBy('city').agg({ avg: 'mean(temp)', rain: 'sum(rain)' }).toRows());

await engine.close();

It is a one-time conversion, row by row, so it is slower than columns for big inputs. A Date in the first row makes that column a date column, and null loads as missing.

Shape and columns

Member	What it gives
length	Rows in the table. On a filtered view this is still the table’s rows; use count() for the matching ones
columnNames	The column names, in order
type	Float64Array or Float32Array: how numbers are stored
column(name)	The column’s array in engine memory, without a copy. Text columns give their codes
memoryBytes()	{ bytes, uncompressed }: what the columns take now, and without compression

column() returns a view into engine memory, so it is cheap but tied to the engine. On a single-thread engine (pages without isolation, the portable build) memory moves when it grows and older views become empty; engine.live(array) gives a working view again. Node engines use shared memory, which never moves.

Getting data out

import { Engine } from 'vectrojs';

const engine = new Engine();
const t = engine.table({
  region: ['EU', 'US', 'EU'],
  price: [10, 20, 30],
});

console.log(t.toRows());          // [{ region: 'EU', price: 10 }, ...]
console.log(t.toRows(1));         // at most 1 row
console.log(t.filter('price > 15').toColumns());
// { region: { codes: Uint32Array [1, 0], labels: ['EU', 'US'] }, price: Float64Array [20, 30] }

await engine.close();

• toRows(limit): plain objects, with text as strings and dates as Date. Fine for results; slow for millions of rows.
• toColumns(): one array per column, copied out of engine memory and safe to keep. Numbers as Float64Array (or Float32Array), categories as { codes, labels }, free text as string[].
• toArrow(): Arrow IPC bytes for pandas, Polars or DuckDB. See CSV, JSON lines, Arrow.
• head(n), slice(start, end) and select('a', 'b') make smaller tables.

All of them apply the table’s filter first. Each has an …Async version (toRowsAsync(), toColumnsAsync()…).

Memory and dispose

An engine’s memory grows as tables are made, up to 4 GB (the most a 32-bit WebAssembly address can reach). For more data than that, use datasets, which read files a chunk at a time.

table.dispose() frees the columns a table made. Which columns a table owns depends on how it was made:

Made by	Owns
engine.table, readCSV, parseCSV, fromArrow, tableFromRows	All its columns
filter, select	Nothing: a view of the same columns
derive	Only the new column; the others belong to the source table
collect, groupBy().agg, sort, top, join, head, slice	All its columns (new arrays)

import { Engine } from 'vectrojs';

const engine = new Engine();
const t = engine.table({ x: Float64Array.from({ length: 1_000_000 }, (_, i) => i) });
console.log(t.memoryBytes());               // { bytes: 8000000, uncompressed: 8000000 }

const big = t.filter('x >= 500000').collect();   // a new table with its own columns
console.log(big.length);                    // 500000
big.dispose();                              // its memory can be reused now

const withY = t.derive('y', 'x * 2');
withY.dispose();                            // frees y only; t still works
console.log(t.sum('x'));

await engine.close();

In a script you can let engine.close() clean up. In a long-running server, dispose result tables you are done with, or memory keeps growing. Don’t dispose a table while views of it (filters, selects, derives) are still in use: they read the same arrays. collect() on a table with no filter, and head(n) when n covers every row, return the table itself rather than a copy, so dispose only one of the two.