Introducing Vectro

Most JavaScript apps that need to add up, group or search a lot of data send it somewhere else first: a database, a warehouse, a Python service. Vectro does that work where the data already is, inside your Node server or your user’s browser tab.

import { Engine, sql } from 'vectrojs';

const engine = new Engine();               // one worker per core
const orders = engine.readCSV('orders.csv');

sql(`SELECT region, COUNT(*) AS orders, SUM(price * qty) AS revenue
     FROM orders GROUP BY region ORDER BY revenue DESC`, { orders }).toRows();

What it is

Vectro keeps your data in columns in WebAssembly memory. When a query first runs, it is compiled into a SIMD WebAssembly loop, and every core runs that loop over its own share of the rows. There is no native addon to build and no dependency to install: it is JavaScript that writes its own WebAssembly.

• SQL and a dataframe API. Joins (including outer joins), subqueries, CTEs, window functions, percentiles, text and date functions; or filter, groupBy and join from JavaScript.
• Files bigger than memory. A .vectro dataset is read one chunk at a time, and every block of rows keeps its smallest and largest values, so a query skips what it doesn’t need.
• Everywhere JavaScript runs. Node 20 and later, browsers (Chrome, Safari and Firefox are checked on every change) and edge runtimes such as Cloudflare Workers.
• Your BI tools. vectro serve speaks the PostgreSQL protocol, HTTP, OData and Arrow Flight SQL, so Power BI, Tableau, Metabase and Grafana can read your tables.

How fast

We compare against DuckDB because it is the best embedded analytics engine we know, and beating it on its own turf is the bar we set. On the same machine, with the same SQL and the same answers:

• 20 of 20 SQL benchmark queries over 10 million rows were faster in Vectro, from 1.3 to 4.7 times.
• Over a billion rows on disk, Vectro was level or faster on 7 of 8 queries and used 0.37 GB of memory where DuckDB used 6.6 GB. The top 10 rows by price took 11 ms against 827 ms, because Vectro skipped every block that couldn’t hold a winning row.
• DuckDB is still faster at one shape we measured: a join over a file with a GROUP BY on the joined table’s columns (1.3 times at a billion rows).

Every number, the scripts that produced it and the machine it ran on are on the benchmarks page. Or skip our numbers and race them in your own browser: Vectro, DuckDB-WASM and Arquero get the same rows and the same 13 questions, and every answer is checked.

Try it

npm install vectrojs

Then the quick start takes about a minute. The source is on GitHub, under the MIT license.

This is version 0.1.0. The dataset file format may still change before 1.0, and the streaming API is a prototype. If something is slow, wrong or missing, please open an issue, or write to [email protected].