Files: blobFile, urlFile, memoryFile

Datasets read and write through small file objects. In Node you pass paths and never see them. Elsewhere, and in Node when the data isn’t in a file, these functions make them: blobFile and urlFile to read, memoryFile and writableFile to write. engine.openDataset() in the browser and openDatasetAsync() everywhere also accept a Blob, File or URL directly and wrap it for you.

import { blobFile, urlFile, memoryFile } from 'vectrojs';                     // Node, also 'vectrojs/portable'
import { blobFile, urlFile, memoryFile, writableFile, storageAvailable } from 'vectrojs/browser';

Reading

blobFile(blob, name?)

A read-only file object over a Blob or File (anything with slice()), read asynchronously. name defaults to blob.name, else 'dataset'. Open it with openDatasetAsync.

urlFile(url, options?)

A read-only file object over a URL, read with HTTP range requests: only the parts a query needs are fetched, with neighbouring reads merged into one request. Returns a promise; it first sends Range: bytes=0-0 to learn the size, and throws if the server doesn’t answer 206 with a Content-Range.

Name	Type	Default	Meaning
fetch	function	globalThis.fetch	The fetch to use.
headers	object	{}	Headers sent with every request, for example Authorization.

import { createServer } from 'node:http';
import { Engine, urlFile, memoryFile } from 'vectrojs';

const engine = new Engine();
const made = memoryFile('orders.vectro');
await engine.writeDatasetAsync(made, engine.table({ region: ['EU', 'US', 'EU'], price: [120, 80, 45] }));
const bytes = new Uint8Array(await made.blob().arrayBuffer());

// a server that answers range requests
const server = createServer((req, res) => {
  const [, a, b] = /bytes=(\d+)-(\d+)/.exec(req.headers.range ?? '') ?? [];
  if (a === undefined) { res.writeHead(200); return res.end(bytes); }
  res.writeHead(206, { 'content-range': `bytes ${a}-${b}/${bytes.length}` });
  res.end(bytes.subarray(Number(a), Number(b) + 1));
});
await new Promise((r) => server.listen(0, '127.0.0.1', r));

const file = await urlFile(`http://127.0.0.1:${server.address().port}/orders.vectro`, { headers: { 'x-demo': '1' } });
const orders = await engine.openDatasetAsync(file);
console.log(file.size, await orders.filter("region == 'EU'").aggAsync({ total: 'sum(price)' }));

await orders.close();
server.close();
await engine.close();

Writing

memoryFile(name?)

A writable file object kept in memory, for runtimes without file storage or for a dataset you want to upload or download. name defaults to 'dataset.vectro'. Write to it with writeDatasetAsync, createDatasetAsync or csvToDatasetAsync; afterwards file.blob() returns the dataset as a Blob, which openDatasetAsync opens.

import { Engine, memoryFile, blobFile, openDatasetAsync, writeDatasetAsync } from 'vectrojs/portable';

const engine = new Engine();
const events = engine.table({ kind: ['click', 'view', 'click'], ms: [12, 40, 8] });

const file = memoryFile('events.vectro');
console.log(await writeDatasetAsync(engine, file, events));

const blob = file.blob();
const ds = await openDatasetAsync(engine, blobFile(blob, 'events.vectro'));
console.log(ds.path, blob.size, await ds.filter("kind == 'click'").countAsync());
await ds.close();
await engine.close();

writableFile(target, options?)

Browser only (vectrojs/browser). A writable file object over browser storage (OPFS, navigator.storage.getDirectory()): target is a file name there, or a FileSystemFileHandle such as one from showSaveFilePicker(). Option: append (default false), to add to an existing dataset. Returns a promise. Writes go to the file’s writable stream in order, in the background; the writer’s appendAsync and closeAsync wait for them. Nothing is committed until the file closes, and an abort discards it. The browser engine’s writeDataset, csvToDataset and createDataset call it for you when given a name or a handle.

import { Engine, writableFile, storageAvailable, memoryFile } from 'vectrojs/browser';
const engine = new Engine();
const target = storageAvailable() ? await writableFile('eu.vectro') : memoryFile('eu.vectro');
await engine.writeDataset(target, orders.filter("region == 'EU'"));

storageAvailable()

Browser only. true when the browser has file storage for datasets (navigator.storage.getDirectory). Without it, write to memoryFile() and download the result.

Your own file objects

Anything with the same shape works. A file to read has name, size, async read(u8, pos) (fill u8 from byte pos) and close(). A file to write has name, existed (false for a new file), write(u8, pos), truncate(size) and close(), and optionally abort(), and flush() and done() returning promises that settle when pending writes are on disk.