Datasets, storage & files

Every run leaves something behind: rows in a dataset, a queryable table per workflow, and files. This page is the map of where that output lands, how to query and search it, how long it is kept, and the caps the file store enforces.

Everything here is scoped to your organization: an id that is not yours answers 404, never a 403 that would confirm it exists.

datasets ▸ one per source

Datasets

A dataset is the accumulated output of one source — a crawl or a workflow. The list tells you where each one came from and how fresh it is: every entry carries a source_type (crawl or workflow), its run_count and its last_updated timestamp.

Endpoint	Does
`GET /api/v1/datasets`	List your datasets with `source_type`, `run_count`, `last_updated`.
`GET /api/v1/datasets/{id}/records`	Page through one dataset’s records.
`GET /api/v1/datasets/search`	One query across every dataset you hold (semantics below).
`DELETE /api/v1/datasets/{id}` · `…/records`	Delete a dataset, or just its records. Requires the `datasets:delete` scope.

tables ▸ rows from runs

Workflow tables

Each workflow also exposes its output as one table. A run that extracted a list contributes one row per record — a run that scraped 40 products adds 40 rows, not one blob. Every row carries its provenance: run_id, run_at and status, and the inputs the run was called with surface as input.<name> columns, with secret values redacted.

Endpoint	Does
`GET /api/v1/workflows/{id}/data`	The table itself: filter, sort, paginate.
`GET /api/v1/workflows/{id}/data/facets`	Distinct values per column, for building filter UIs.
`GET /api/v1/workflows/{id}/data/export`	The same table as a download (formats below).

Param	Meaning
`q`	Substring match across all data fields and inputs.
`filter`	Repeatable `column:substring` pairs.
`filters`	JSON clauses, for conditions `filter` can’t express.
`sort_by` / `sort_dir`	A data column, an `input.<name>` column, or `run_at` \| `status` \| `duration_ms`.
`limit` / `offset`	`limit` 1–500, default 50.
`include_inputs`	Adds the `input.<name>` columns to the payload.
`collection`	Pivots a nested array into one row per item.

search ▸ across everything

Search

GET /api/v1/datasets/search runs one query across every dataset you hold. The same call is datasets.search in every SDK:

const hits = await client.datasets.search("invoice 2291", { limit: 20 });

hits = client.datasets.search("invoice 2291", limit=20)

hits, err := client.Datasets.Search(ctx, "invoice 2291", nil)

let hits = agent.datasets().search("invoice 2291").await?;

curl "http://127.0.0.1:8131/v1/datasets/search?q=invoice+2291" \
  -H "Authorization: Bearer $WRIT_TOKEN"

The semantics are deliberately small, and worth knowing exactly:

Space-separated terms are ANDed; each term matches case-insensitively as a prefix.
Phrase and boolean operators are intentionally unsupported; a query takes at most 8 terms.
Candidates are capped at the 500 most recent matches — the response sets a truncated flag when the cap is hit.
Snippets show 80 characters of context around the match; limit is 1–200, default 50.

export ▸ four formats

Export formats

Tables and datasets render in four formats:

Format	Default for
`json`	API responses.
`csv`	Downloads.
`markdown`	—
`html`	—

retention ▸ how long

Retention

Record	Kept for
Runs	90 days
Logs	90 days
Detected changes	90 days
Audit events	400 days (~13 months)

These are the default windows.

files ▸ the byte side

Files

Files live in per-tenant object storage, addressed by a stable file_… handle, and are served through the Files API at /api/v1/files (upload, list, retrieve, download, delete). A download is a 302 to a signed, single-object link that expires in 600 s, so the storage host and its credentials are never exposed.

Per-file cap: 100 MB — it applies to every file, however it was created.
Files produced by workflow_output, ai_session or streaming are ephemeral: a 24 h TTL, unless you promote them to the library.
Executable content types are denied by default.
Ownership checks answer 404 for anything that is not yours — never a 403 that leaks existence.

quota ▸ per plan

Storage quota

Each plan carries a file-storage quota for the organization:

Plan	Storage
Free	1 GB
Starter	2 GB
Pro	5 GB
Growth	50 GB
Scale	200 GB
Enterprise	1000 GB

A full quota answers a storage 402 — a different error from a credits 402. Freeing space (or a bigger plan) fixes the first; funds fix the second.

byo ▸ your own bucket

Bring your own storage

By default the bytes sit in Writ’s managed store. You can instead point Writ at an S3-compatible bucket you control — s3, minio, r2 or spaces — in Settings → Storage. The secret key is write-only (encrypted at rest, never returned), a server-side connection test probes the bucket before it goes live, and switching your default provider never breaks downloads of files already stored.

What next

Workflows: where the rows come from.
API reference: request and response shapes, status codes.
Automations & webhooks: push output onward instead of polling for it.
Billing & usage: what a credits 402 means and the three ways out.