Turn a whole site
into clean data.

A whole-site crawl is one POST with a key. Writ fans the seed across the fleet and lands the result as a callable dataset.

# No account, no key — a few same-domain pages, one level deep.
curl -X POST https://api.usewrit.app/v1/keyless/crawl \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "limit": 3}'

# With a key: the whole site, as a job you poll.
curl -X POST https://api.usewrit.app/api/crawl \
  -H "Authorization: Bearer $WRIT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "max_depth": 3, "page_budget": 500}'

curl https://api.usewrit.app/api/crawl/8125 -H "Authorization: Bearer $WRIT_API_KEY"

⚠No account needed to try it: the keyless tier crawls a few same-domain pages, one level deep, against a daily allowance. Whole-site scope, logins and a callable dataset need a key.

Point your assistant at Writ and it can crawl a site directly, then read the rows back without you writing a parser.

claude mcp add writ-cloud \
  -e WRIT_API_KEY=wt_your_api_key \
  -- npx -y writ-mcp

# then, one line in the session:
writ_crawl_site({ url: "https://example.com",
                 intent: "product pages with specs",
                 page_budget: 500, save_as: "example-catalog" })

writ_crawl_status({ crawl_id: 8125 })

Config-file clients — Claude Desktop · Cursor · Windsurf

Pass save_as and the crawl becomes re-runnable and callable, so later runs can reuse recent data instead of re-crawling.

npm install github:usewrit/writ-sdks --workspace typescript

import { CloudApi } from "@usewrit/agent-sdk";

const cloud = new CloudApi({ apiKey: process.env.WRIT_API_KEY });
const job = await cloud.crawl({ url: "https://example.com", max_depth: 3, page_budget: 500 });
const status = await cloud.crawlStatus(job.id);
console.log(status.pages_done, "/", status.pages_discovered);

pip install "git+https://github.com/usewrit/writ-sdks#subdirectory=python"

from writ_agent import Cloud

cloud = Cloud(api_key=os.environ["WRIT_API_KEY"])
job = cloud.crawl("https://example.com", max_depth=3, page_budget=500)
status = cloud.crawl_status(job["id"])
print(status["pages_done"], "/", status["pages_discovered"])

go get github.com/usewrit/writ-sdks/go

client := writ.New(writ.WithAPIKey(os.Getenv("WRIT_API_KEY")))
job, err := client.Cloud.Crawl(ctx, writ.CrawlStartParams{
    URL:        "https://example.com",
    MaxDepth:   writ.Ptr(int64(3)),
    PageBudget: writ.Ptr(int64(500)),
})
status, _ := client.Cloud.CrawlStatus(ctx, job.ID)

cargo add writ-client --git https://github.com/usewrit/writ-sdks

use writ_client::{CloudClient, CrawlStartParams};

let cloud = CloudClient::from_env()?;
let job = cloud.crawl(&CrawlStartParams {
    url: "https://example.com".into(),
    max_depth: Some(3),
    page_budget: Some(500),
    ..Default::default()
}).await?;
let status = cloud.crawl_status(job.id).await?;