Question 1

What is the most accurate web search API for AI agents?

Accepted Answer

On the verified web_search-2026-q3 snapshot (n=62 public queries, descriptive form), Exa and SerpAPI are tied at the top of hit@5 at 79.0% and 71.0% respectively: their 95% Wilson confidence intervals overlap and McNemar p=0.40, so they are not statistically separable at this sample size, and both are clearly ahead of Brave (54.8%) and Tavily (43.5%). Only hit@1 and hit@5 are measured this run, so no cost, freshness or latency leader is claimed.

Question 2

How does arlen/bench measure freshness?

Accepted Answer

Sentinel pages carrying a unique token are deployed to sentinel.arlenkumar.com and probed for first appearance in each vendor's results, recording a not_indexed vs ranked_below_k discriminator. Freshness lag will be reported as the Kaplan-Meier median time from publication to first retrievability, right-censored at 30 days. For web_search-2026-q3 this curve is pending the first crawl, so the freshness numbers are not yet published.

Question 3

Were cost, freshness or latency measured for web search in this snapshot?

Accepted Answer

No. The web_search-2026-q3 run measured only hit@1 and hit@5 over 62 public queries. Cost-per-correct and latency metering and the sentinel freshness instrument have not yet been run, so those columns render as an em-dash (pending), never zero, and no cost, freshness or latency leader is claimed. Serper and Firecrawl were not scored on web search this snapshot and are excluded; agent onboarding was not tested in this run.

Date	Snapshot	Status	What changed
2026-06-12	web_search-2026-q3	verified (reproducible, pre-review)	First verified scored run: n=62 public queries, hit@1/hit@5 measured (cost/freshness/latency pending). Exa 79.0% and SerpAPI 71.0% tied on hit@5 (McNemar p=0.40). Replaces the prior illustrative placeholder. Golden cohort expanded to 435 public for the next run.
(prior)	web_search	illustrative	Placeholder figures only; every value was marked status=illustrative and "must not be cited." No vendor run had been scored.
2026-06-10	web_extraction-2026-q2	verified (audited, pre-review)	Real scored run over 150 WCXB pages; independent 20% human audit, 30/30 agreed. Vendor right-of-reply pre-review (rows not yet sent).

Changelog & Snapshot History

Snapshot Revision History