Evals & Golden Datasets
The datasets behind every score. Public splits are downloadable and CC-BY-4.0; holdouts never leave. Each row links to its design doc and JSON schema.
Datasets
| Dataset | items | public / holdout | truth source | refresh | download |
|---|---|---|---|---|---|
| web_search · queryset | 62 | public (30% holdout excluded) | verified golden URLs | quarterly +25% rotation | jsonl |
| web_extraction · WCXB | 150 | public | WCXB dev (Foley) · 20% audited | quarterly resample | json ↓ |
| planted · sentinel pages | — | holdout until snapshot | we publish them (unique token) | probing · curve pending first crawl | protocol only |
| kyb · core cohort | 1000 | 700 / 300 | SoS · EDGAR · Companies House | registry deltas, daily | first run July |
Row Schema — web_search example
{"id": "ws-0002",
"query": "california contractor license lookup official",
"stratum": "navigational_docs",
"golden_urls": ["https://www.cslb.ca.gov/onlineservices/checklicenseII/checklicense.aspx"],
"split": "public",
"verified_at": "2026-05-02T00:00:00Z"}
Anchor-phrase rules for extraction rows: 3–6 phrases per page, each under 15 words, drawn from distinct sections, never from boilerplate. Full schemas live in the repo with the scoring docstrings published verbatim on the methodology page.
Overfit Gap — current snapshot
The web_search-2026-q3 figures are reported over the public split only (n=299, descriptive form); the 30% private holdout is excluded from this run. The public-minus-holdout overfit gap is therefore — (pending) for every vendor until a holdout pass is scored. The column exists so that the day a vendor tunes to the public set, it shows — but no gap is fabricated before that measurement lands.
No overfit-gap numbers are published until the holdout is scored. Per-vendor gaps render as — (pending), never 0.