Question 1

What is the most accurate web search API for AI agents?

Accepted Answer

On the verified web_search-2026-q3 snapshot (n=299 public queries, descriptive form), Exa leads hit@5 at 80.9%, ahead of SerpAPI at 69.2%: their 95% Wilson confidence intervals are disjoint and McNemar p≈0.001, so Exa is statistically separable as the clear leader, and both are ahead of Brave (58.9%) and Tavily (49.4%). Only hit@1 and hit@5 are measured this run, so no cost, freshness or latency leader is claimed.

Question 2

How does arlen/bench measure freshness?

Accepted Answer

Sentinel pages carrying a unique token are deployed to sentinel.arlenkumar.com and probed for first appearance in each vendor's results, recording a not_indexed vs ranked_below_k discriminator. Freshness lag will be reported as the Kaplan-Meier median time from publication to first retrievability, right-censored at 30 days. For web_search-2026-q3 this curve is pending the first crawl, so the freshness numbers are not yet published.

Question 3

Were cost, freshness or latency measured for web search in this snapshot?

Accepted Answer

No. The web_search-2026-q3 run measured only hit@1 and hit@5 over 299 public queries. Cost-per-correct and latency metering and the sentinel freshness instrument have not yet been run, so those columns render as an em-dash (pending), never zero, and no cost, freshness or latency leader is claimed. Serper and Firecrawl were not scored on web search this snapshot and are excluded; agent onboarding was not tested in this run.

Date	Snapshot	Status	What changed
2026-06-14	build · reproduction gate	integrity	Hardened the cross-surface consistency gate to validate the published reproduction files (-rows.json / -results.json), not only the snapshot contract: n_rows must equal the snapshot cohort, the row-array length must match its own n_rows, the snapshot_id must match, and all of a snapshot's reproduction files must share one generated_at. A rows/results header can no longer drift from the leaderboard and claims — the build fails first. Prompted by a stale cached n_rows:62 view of the re-scored n=299 files (the live origin already read n=299).
2026-06-14	web_search-2026-q3	verified (reproducible, pre-review)	Re-scored on the expanded cohort: n=299 public queries (up from 62), 3 reps/query, 3587 result records. Exa 80.9% now leads SerpAPI 69.2% on hit@5 with disjoint 95% Wilson CIs (McNemar p≈0.001) — the top pair, previously a tie at n=62, is now statistically separable. hit@1/hit@5 measured; cost/freshness/latency still pending. Rows/results JSON republished.
2026-06-12	web_search-2026-q3	verified (reproducible, pre-review)	First verified scored run: n=62 public queries, hit@1/hit@5 measured (cost/freshness/latency pending). Exa 79.0% and SerpAPI 71.0% tied on hit@5 (McNemar p=0.40). Replaces the prior illustrative placeholder. Golden cohort expanded to 435 public for the next run.
(prior)	web_search	illustrative	Placeholder figures only; every value was marked status=illustrative and "must not be cited." No vendor run had been scored.
2026-06-10	web_extraction-2026-q2	verified (audited, pre-review)	Real scored run over 150 WCXB pages; independent 20% human audit, 30/30 agreed. Vendor right-of-reply pre-review (rows not yet sent).

Changelog & Snapshot History

Snapshot Revision History