Compressed columns

A plain column takes 8 bytes per row (4 on a Float32Array table), whatever it holds. Most real columns need far less: a quantity from 1 to 9, a region code, a price in cents, a timestamp on whole seconds. table.compress() stores each such column in 1, 2 or 4 bytes per row, and only when every value comes back exactly.

import { Engine } from 'vectrojs';

const engine = new Engine();
const n = 1_000_000;
const regions = ['EU', 'US', 'APAC', 'LATAM'];
const orders = engine.table({
  qty:    Float64Array.from({ length: n }, (_, i) => 1 + (i % 9)),
  price:  Float64Array.from({ length: n }, (_, i) => (i % 40_000) / 100),         // cents
  ts:     Float64Array.from({ length: n }, (_, i) => Date.UTC(2024, 0, 1) + i * 1000), // whole seconds
  region: Array.from({ length: n }, (_, i) => regions[i % 4]),
  score:  Float64Array.from({ length: n }, () => Math.random()),                  // can't be packed
});

const before = orders.sum('price * qty');
console.log(orders.memoryBytes());
orders.compress();
console.log(orders.memoryBytes());
console.log(orders.sum('price * qty') === before);
await engine.close();

memoryBytes() returns { bytes, uncompressed }: what the columns take now, and what they would take unpacked. Here 40 MB becomes 16 MB; the random score column stays at 8 bytes per row and the other four shrink.

Three ways to turn it on

table.compress();                               // in place; returns the table
await table.compressAsync();                    // the same on the worker threads
engine.table(columns, { compress: true });      // pack while building
engine.parseCSV(text, { compress: true });      // pack after parsing (also parseCSVAsync)

compress() works in place: the table and any views of it (filters, selects) keep working, and the old arrays are freed.

What gets packed

Each number and category column is checked on its own, on every thread. It is packed if its values fit one of two shapes with small whole-number codes:

Shape	Fits	Example
base + code * step	whole numbers, with step 1; or times on whole seconds, minutes, hours or days	counts, IDs, codes, timestamps
code / scale	decimals with up to 4 places (scale 10, 100, 1,000 or 10,000)	prices in cents, rates in basis points

The codes go in the smallest of three sizes they fit:

Size	Codes
1 byte	0 to 254
2 bytes	0 to 65,534
4 bytes	about plus or minus 2.1 billion

The largest code of each size is reserved for a missing value, so NaN survives packing. Category columns already hold small codes (one per label), so a column with up to 255 labels takes 1 byte per row.

A column stays as it is when:

• any value wouldn’t come back exactly (random decimals, more than 4 decimal places);
• it holds Infinity, -Infinity or -0;
• it is free text (long, mostly unique strings loaded as string).

Every candidate is checked value by value with the same arithmetic the kernels use to decode it (on a Float32Array table, float32 arithmetic). If one value would differ, that encoding is rejected. That is why packing never changes an answer.

What it costs at query time

The fused kernels that run filters, aggregates and derived columns read packed columns directly: a SIMD load of the codes, widened, converted and scaled inside the loop. Group-by kernels do the same. Scans read 2 to 8 times fewer bytes, which often makes up for the extra arithmetic.

Sort, top and collect gather the codes themselves, so their results stay packed. Joins and the other operations decode the columns they need first; a join’s result is a plain table.

From the README (npm run bench:pack, 10M orders):

	Plain	Compressed
memory	458 MB	143 MB (3.2x smaller)
time to compress		0.34 s
a new column	4.0 ms	3.0 ms
filtered revenue	2.4 ms	2.9 ms
group by region	4.7 ms	4.4 ms
top 10	8.4 ms	12.9 ms

Queries run about as fast as on plain columns: some a little faster, top-k a little slower. The whole test suite also runs with every table compressed (npm run test:compressed), checking that answers are the same.

Results stay small

import { Engine } from 'vectrojs';

const engine = new Engine();
const n = 200_000;
const t = engine.table({
  store: Float64Array.from({ length: n }, (_, i) => i % 120),
  qty:   Float64Array.from({ length: n }, (_, i) => i % 9),
  price: Float64Array.from({ length: n }, (_, i) => (i % 5000) / 100),
}, { compress: true });

const kb = (x) => Math.round(x.memoryBytes().bytes / 1024) + ' KB';
console.log('table', kb(t), 'unpacked', Math.round(t.memoryBytes().uncompressed / 1024) + ' KB');
console.log('sorted', kb(t.sort('-price')));
console.log('filtered', kb(t.filter('qty > 4').collect()));
console.log('top', t.top(3, '-price').toRows());
console.log('group', t.groupBy('store').agg({ units: 'sum(qty)' }).toRows(2));
await engine.close();

When to use it

• Tables that are large next to the 4 GB an engine can hold: packing lets roughly three times as many rows fit, by the README’s 10M-order example.
• In the browser, where memory is tighter and pages share a device with everything else.
• Tables you load once and query many times. Compressing 10M rows took 0.34 s.

Packing is worth skipping for small, short-lived tables, and for tables of mostly random decimals, where few columns would pack.