Dataset and DatasetWriter

A dataset is a table kept in one .vectro file. The file holds chunks of rows (4,194,304 by default); each column of each chunk is stored compressed and packed to the bit, and a footer holds the schema, one dictionary per text column and the min and max of every column for each chunk and each 64K-row block. Opening one reads only the footer. A query is a lazy plan run chunk by chunk with the normal kernels: only the columns it uses are read, chunks and blocks its filters rule out are skipped, and partial results (totals, groups, top-k) are merged as chunks go by. Its size isn’t limited by memory. See Datasets bigger than memory.

In Node, datasets are read and written by path. In browsers and edge runtimes they are opened read-only from a URL (HTTP range requests), a File or Blob, and written to browser storage or a file object.

import { Engine, Dataset, DatasetWriter } from 'vectrojs';      // Node: engine.openDataset(path) and friends
import { Engine } from 'vectrojs/browser';                      // engine.openDataset(url | file), engine.writeDataset(name, ...)
import { Engine, openDatasetAsync, writeDatasetAsync, createDatasetAsync, csvToDatasetAsync } from 'vectrojs/portable';

You normally make these objects through the engine: engine.openDataset(), openDatasetAsync(), createDataset(), writeDataset(), csvToDataset() and their async forms, documented on Engine.

Dataset

new Dataset(engine, path)

Node: opens the file at path and reads its footer. engine.openDataset(path) does the same. Sources read asynchronously (a URL, a Blob) need openDatasetAsync.

openDatasetAsync(engine, source)

Exported from vectrojs/portable, vectrojs/browser and, as engine.openDatasetAsync(source), from Node. source is a path (Node), an http(s): URL, a Blob or File, or a file object from blobFile or urlFile. Reads the footer and returns a promise of the Dataset. A URL must answer Range: bytes=a-b requests with 206.

Properties

Name	Meaning
ds.length	A getter: rows in the dataset.
ds.columnNames	A getter: the column names.
ds.chunkCount	A getter: the number of chunks.
ds.path	The file’s path, URL or name.
ds.bytes	The file size.
ds.lastRun	What the last query read: { chunks, read, skipped, rows, bytesRead, ms }.

Query steps

ds.lazy() starts a Query. ds.filter(expr, params?), ds.derive(name, expr, params?), ds.select(...names), ds.sort(by), ds.top(k, by), ds.limit(n, offset?), ds.join(other, options) and ds.groupBy(keys) are the same steps on ds.lazy(), so each returns a Query, not a table. Run it with collect(), toRows(), agg() or count(). With the …Async calls the next chunk is read while the current one is computed.

A join’s other side (a table, or a dataset’s matching rows) is hashed once for all chunks. When both sides are datasets and the other side has more than 16M matching rows, both are split to disk by key range (a grace hash join), and rows come out range by range rather than in this dataset’s order. Group-bys with more than 8M groups of keys that aren’t labels or whole-number IDs spill partial groups to disk the same way.

ds.agg(spec, params?)

Totals over every row: { name: value }, aggregations as in table.agg (countDistinct and approxDistinct both count exactly here). Also aggAsync.

ds.count(expr?)

Rows, or rows where expr holds. Also countAsync(expr?).

ds.sum(expr, params?)

agg({ v: 'sum(expr)' }).v; likewise ds.mean, ds.min, ds.max. These four have no async form: use aggAsync.

ds.head(n)

The first n rows as a Table; only the chunks needed are read. Also headAsync.

ds.toRows(limit?)

Rows as plain objects; with a finite limit only those rows are read. Also toRowsAsync.

ds.collect()

The whole dataset as a table in memory. Also collectAsync.

ds.explain()

The optimized plan of ds.lazy(), as text.

ds.runPlan(plan, sync)

Called by the planner to run a plan that reads this dataset. Not for direct use.

ds.close()

Closes the file. Returns a promise.

import { mkdtempSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { Engine, sql } from 'vectrojs';

const dir = mkdtempSync(join(tmpdir(), 'vectro-'));
const engine = new Engine();
const n = 100_000;
const orders = engine.table({
  day: Float64Array.from({ length: n }, (_, i) => Math.floor(i / 1000)),
  region: Array.from({ length: n }, (_, i) => ['EU', 'US', 'APAC'][i % 3]),
  price: Float64Array.from({ length: n }, (_, i) => (i % 500) + 0.99),
});

const path = join(dir, 'orders.vectro');
const { rows, chunks } = engine.writeDataset(path, orders, { rowsPerChunk: 25_000 });
console.log(rows, chunks);

const ds = engine.openDataset(path);
console.log(ds.length, ds.chunkCount, ds.columnNames);
console.log(ds.filter('day >= 90').groupBy('region').agg({ n: 'count()', revenue: 'sum(price)' }).toRows());
console.log(ds.lastRun.read, 'of', ds.lastRun.chunks, 'chunks read');
console.log(await sql('SELECT COUNT(DISTINCT day) AS days FROM ds WHERE region = \'EU\'', { ds }).toRowsAsync());
await ds.close();
await engine.close();
rmSync(dir, { recursive: true, force: true });

DatasetWriter

new DatasetWriter(engine, path, options?)

Opens path (Node) or a writable file object for writing. engine.createDataset(path, options) does the same; createDatasetAsync(engine, target, options) (or engine.createDatasetAsync(target, options)) also reads the existing footer of a browser file before appending.

Name	Type	Default	Meaning
rowsPerChunk	number	4,194,304	Rows per chunk; the last one may be shorter.
append	boolean	false	Add to an existing dataset with the same columns instead of replacing it.
bitPack	boolean	true	false stores packed codes in whole bytes: bigger files, no unpacking on read.

A dataset is written by one writer at a time.

w.append(source, lo?, hi?)

Adds rows: a Table (the rows matching its filter, or its rows lo to hi) or { column: values }. The first append fixes the columns and their kinds; later ones must match. Rows wait in memory until a chunk is full. Returns the writer. Also appendAsync, which also waits until the bytes are written (browser files write in the background).

w.close()

Writes the remaining rows and the footer and returns { path, rows, chunks, bytes }. At least one append is needed (it may have no rows). Also closeAsync.

w.abort()

Stops without a footer. The file is left unreadable as a dataset; a browser file is discarded.

writeDatasetAsync(engine, target, source, options?)

Exported from vectrojs/portable and vectrojs/browser; in Node it is engine.writeDatasetAsync(target, source, options). Writes a table, a query, a dataset or { column: values } to a path (Node) or a writable file object, with the async calls, and returns the summary. A query over a dataset is written chunk by chunk; a full sort written this way is an external sort.

csvToDatasetAsync(engine, csv, out, options?)

Exported from vectrojs/portable and vectrojs/browser; in Node it is engine.csvToDatasetAsync(csv, out, options). Converts a CSV of any size (a path in Node, a URL, a Blob or File) to a dataset at out (a path in Node, or a writable file object), reading pieceBytes (default 64 MB) at a time. Takes the CSV options, the writer options, and onProgress, called after each piece with { bytes, size, rows }.

import { mkdtempSync, writeFileSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { Engine } from 'vectrojs';

const dir = mkdtempSync(join(tmpdir(), 'vectro-'));
const engine = new Engine();

const w = engine.createDataset(join(dir, 'events.vectro'), { rowsPerChunk: 2 });
w.append({ kind: ['click', 'view', 'view'], ms: [12, 40, 33] });
w.append(engine.table({ kind: ['click'], ms: [8] }));
console.log(w.close().chunks);

const more = engine.createDataset(join(dir, 'events.vectro'), { append: true });
more.append({ kind: ['buy'], ms: [100] });
console.log(more.close().rows);

writeFileSync(join(dir, 'clicks.csv'), 'kind,ms\nclick,5\nview,9\nclick,7\n');
console.log(engine.csvToDataset(join(dir, 'clicks.csv'), join(dir, 'clicks.vectro'), { onProgress: (p) => console.log('progress', p) }).rows);

const events = engine.openDataset(join(dir, 'events.vectro'));
const slow = engine.writeDataset(join(dir, 'slow.vectro'), events.filter('ms > 30'));
console.log(slow.rows, events.groupBy('kind').agg({ n: 'count()', avg: 'mean(ms)' }).toRows());
await events.close();
await engine.close();
rmSync(dir, { recursive: true, force: true });