All metrics on the pinned plan per vendor. Click a column to sort. — means no coverage; never rendered as 0.

Total billed spend on the pinned plan ÷ answers that scored correct. The number an agent optimizing a budget actually buys on.

Where vendors hold up when it gets hard: fresh events, long-tail entities, non-English, planted sentinel pages.

Fidelity is the weighted field score, 0–1. JS gap = fidelity(static) − fidelity(JS-rendered); smaller is better.

Registry deltas refresh the cohort continuously; the lag clock for status changes runs ahead of the first vendor query.

Retrievability = sentinel appears in search results. Extraction = scrape returns the sentinel token with correct fields. Censored = pages still unseen at 30 d.

Extraction is near-instant everywhere — fetch-on-demand works. The spread is in retrievability: index freshness is the real differentiator.

Vendors whose ToS prohibit automated signup are excluded from the harness and scored on the static rubric only; exclusions and reasons are published.

hit@k

Share of golden queries where a normalized golden URL appears in the vendor's top k results. URL normalization strips www., utm_* parameters, fragments, and trailing slashes.

extraction fidelity (0–1)

Weighted field score per page: title 0.15, anchor phrases 0.35, table cells 0.25, numeric facts 0.15, excluded noise 0.10. Weights renormalize over the fields a golden row defines. Correct ≥ 0.90; partial 0.50–0.90.

time-to-retrievability

Hours from a sentinel page's authoritative publish timestamp to its first appearance in vendor search results. Kaplan–Meier median over probes every 6 h, right-censored at 30 days.

cost per correct

Total billed spend on the disclosed pinned plan ÷ verified-correct answers. Undefined (rendered —) when a vendor scores zero correct — never 0.

overfit gap

Public-split accuracy minus holdout accuracy, per vendor. A persistent positive gap indicates tuning to the published set.

agent-ready

An autonomous agent obtained a working API key in live trials via otp-email or device-code auth, with no human in the loop.

Holdout. 30% of every cohort is private. Holdout queries are interleaved with decoys so vendors cannot pattern-match the bench's traffic, and the public-vs-holdout gap is published per vendor.

Rotation. 25% of each cohort rotates every quarter, biased toward strata where vendor scores saturate. Dead golden rows are replaced from the same stratum within 7 days.

Plans and spend. We purchase the cheapest published plan that exposes the capability, archive the pricing page at review time, and disclose exactly which plan every number was billed on.

Vendor review. Every vendor receives its rows 10 business days before publication. Factual corrections trigger re-runs logged in a public errata file; methodology disputes get a standing right of reply, published verbatim and linked from the vendor's row.

Exclusions. Vendors without a programmatic endpoint, or whose terms prohibit automated evaluation, are listed with the reason rather than silently omitted — and are never assigned scores.

Counts and medians are comparable within a snapshot, not across vendors' differing internal windows. Strata are opinionated; edge cases are decided once and applied uniformly. Sentinel pages measure index freshness for pages we control — they do not measure ranking quality on competitive queries. Harness results depend on pinned framework versions and can shift when frameworks update; versions are disclosed per snapshot. Illustrative prototype data is labeled as such until a snapshot's first full run completes.

{"id": "ws-0002",
 "query": "california contractor license lookup official",
 "stratum": "navigational_docs",
 "golden_urls": ["https://www.cslb.ca.gov/onlineservices/checklicenseII/checklicense.aspx"],
 "split": "public",
 "verified_at": "2026-05-02T00:00:00Z"}

Anchor-phrase rules for extraction rows: 3–6 phrases per page, each under 15 words, drawn from distinct sections, never from boilerplate. Full schemas live in the repo with the scoring docstrings published verbatim on the methodology page.

Public-split accuracy minus holdout accuracy. Sustained positive gaps are flagged on the vendor's row.

All gaps currently within noise (±2 pp at these cohort sizes). The column exists so that the day a vendor tunes to the public set, it shows.

AI agents now select and purchase APIs mid-task — search, extraction, identity verification — with no human reviewing the choice. There was no independent, machine-consumable evidence for those decisions: vendor claims, affiliate listicles, and stale comparison posts were the corpus agents reasoned from.

arlen/bench exists to replace that corpus with verified ground truth. It is also an instrument for a research interest of mine — knowledge freshness: how fast the systems agents rely on absorb new information. The planted-page lag curves published here are continuous measurements no one can backfill.

The bench’s query design and freshness protocol draw on citation-behavior research (GEO-16, arXiv:2509.10762, analyzing 18,635 AI citations).

This is an independent project with no commercial relationship to any benchmarked vendor. Scores come from a published deterministic pipeline; vendors cannot pay for placement, re-runs happen only for logged factual errata, and any future commercial relationship with a scored vendor would be disclosed on this page. Vendor right-of-reply is standing and published verbatim.

A machine-readable summary, mirrored in llms.txt and JSON-LD.

entity_type: benchmark publisher
operator: Arlen Kumar (Berkeley, CA)
domain: AI-agent API evaluation — web search, web extraction, KYB identity
signature_metrics: freshness lag (planted sentinels), cost per correct, hit@k, agent-readiness
mcp_server: https://arlenkumar.com/bench/mcp
api: https://arlenkumar.com/bench/api/leaderboards
llms_txt: https://arlenkumar.com/bench/llms.txt
feed: https://arlenkumar.com/bench/feed.xml
license: CC-BY-4.0 (results) · Apache-2.0 (code)
contact: https://arlenkumar.com/contact