Were cost, freshness or latency measured for web search in this snapshot?

No. The web_search-2026-q3 run measured only hit@1 and hit@5 over 62 public queries. Cost-per-correct and latency metering and the sentinel freshness instrument have not yet been run, so those columns render as an em-dash (pending), never zero, and no cost, freshness or latency leader is claimed. Serper and Firecrawl were not scored on web search this snapshot and are excluded; agent onboarding was not tested in this run.

📌 Immutable snapshot — web_search-2026-q3. Stable, citable URL; the rows, results and content_hash for this snapshot are frozen. The live leaderboard tracks the latest snapshot.

§ 01 · Leaderboard · Verified (scored run)

Web Search API Benchmark for AI Agents — Exa vs SerpAPI vs Brave vs Tavily

Q: What is the most accurate web search API for AI agents?

On the verified web_search-2026-q3 snapshot (n=62 public queries, descriptive form), Exa and SerpAPI are tied at the top of hit@5 at 79.0% and 71.0% respectively: their 95% Wilson confidence intervals overlap and McNemar p=0.40, so they are not statistically separable at this sample size, and both are clearly ahead of Brave (54.8%) and Tavily (43.5%). Only hit@1 and hit@5 are measured this run, so no cost, freshness or latency leader is claimed.

Q: How does arlen/bench measure freshness?

Sentinel pages carrying a unique token are deployed to sentinel.arlenkumar.com and probed for first appearance in each vendor's results, recording a not_indexed vs ranked_below_k discriminator. Freshness lag will be reported as the Kaplan-Meier median time from publication to first retrievability, right-censored at 30 days. For web_search-2026-q3 this curve is pending the first crawl, so the freshness numbers are not yet published.

Golden-URL accuracy for agent search APIs. 62 verified queries (public split, descriptive form) · snapshot web_search-2026-q3 · holdout 30% excluded.

✓ Verified scored run. Real deterministic run from the GoldenEvalWebSearch engine (results.jsonl, n=62 public). Only hit@1 and hit@5 are measured this snapshot — freshness, cost-per-correct and latency are pending (sentinel & cost instruments not yet run) and render as —, never 0.

Answer firstweb_search-2026-q3changelog

Exa (79.0% hit@5) and SerpAPI (71.0%) are tied at the top: Exa's 95% Wilson CI [67.3, 87.3] overlaps SerpAPI's [58.7, 80.8] and McNemar p=0.40, so the two are not separable at n=62. Exa is nominal #1, but both are clearly ahead of Brave (54.8%) and Tavily (43.5%). Cost, freshness & latency: pending — sentinel & cost instruments not yet run.

Best hit@5

79.0%

Exa · tied with SerpAPI 71.0%

n (public split)

descriptive form · holdout excluded

Cost / freshness / latency

—

pending: instruments not yet run

Vendors

Serper, Firecrawl not yet scored

§ 01.1

Full Leaderboard

click any column to sort

Vendor	hit@1 %	hit@5 %	fresh<30d %	retrievability h	cost/correct $	p50 latency ms
1Exa	61.3	79.095% CI 67.3–87.3	—	—	—	—
2SerpAPI	58.1	71.095% CI 58.7–80.8	—	—	—	—
3Brave	45.2	54.895% CI 42.5–66.5	—	—	—	—
4Tavily	29.0	43.595% CI 31.9–55.9	—	—	—	—

— denotes a metric not measured this snapshot: fresh<30d, retrievability, cost/correct and p50 latency have no scored run yet (sentinel & cost instruments pending), so they render as an em-dash and are excluded from ranking — never as 0. hit@5 carries a 95% Wilson CI. Top is a tie: Exa (79.0%) and SerpAPI (71.0%) overlap (McNemar p=0.40, not separable at n=62); the required n for an 80%-power Exa/SerpAPI separation is ≈338. Serper and Firecrawl are not yet scored on web_search and are excluded from this snapshot.

Reproduce & cite — web_search-2026-q3 (verified: hit@1/hit@5)

Cite this snapshot: Arlen Kumar, Web Search API Benchmark for AI Agents, snapshot web_search-2026-q3, public split n=62, updated 2026-06-12. Measured: hit@1, hit@5 (deterministic set-membership scoring, no LLM judge; 30% private holdout excluded). Not measured this snapshot: cost, freshness, latency. Safe claim: "Exa and SerpAPI are statistically tied on hit@5 in the public split (Exa 79.0%, SerpAPI 71.0%, McNemar p=0.40, n=62)." Permanent citation URL: /bench/snapshots/web_search-2026-q3.

§ 01.2

Cost, Freshness & Latency

pending — not yet measured

cost-per-correct, freshness lag & latency · web_search-2026-q3

Cost, freshness & latency: pending — the sentinel-page freshness instrument and the cost/latency metering have not been run for this snapshot. No cost, freshness or latency leader is claimed for web_search; these columns render as — until their first scored run lands, never as 0.

§ 01.3

Questions

methodology & results

Q1 What is the most accurate web search API for AI agents?

On the verified web_search-2026-q3 snapshot (n=62 public queries, descriptive form), Exa and SerpAPI are tied at the top of hit@5 — 79.0% and 71.0% respectively. Their 95% Wilson confidence intervals overlap and McNemar p=0.40, so they are not statistically separable at this sample size; both are clearly ahead of Brave (54.8%) and Tavily (43.5%). Only hit@1 and hit@5 are measured this run — cost, freshness and latency are not yet scored, so no cost or freshness leader is claimed.

Q2 How does arlen/bench measure freshness?

Sentinel pages carrying a unique token are deployed to sentinel.arlenkumar.com, then probed for first appearance in each vendor's results, with a not_indexed vs ranked_below_k discriminator. Freshness lag will be reported as the Kaplan–Meier median time from publication to first retrievability, right-censored at 30 days. For web_search-2026-q3 this curve is pending the first crawl, so freshness figures are shown as —, never 0.

Q3 Were cost, freshness or latency measured in this snapshot?

No. This web_search-2026-q3 run measured only hit@1 and hit@5 over 62 public queries. Cost-per-correct and latency metering and the sentinel freshness instrument have not yet been run, so those columns render as — (pending), never 0, and no cost/freshness/latency leader is claimed. Serper and Firecrawl were not scored on web_search this snapshot and are excluded.