Sourcery Search Benchmark: Search API Retrieval Eval
Disclaimer
Independent<br>Nobody paid me to run this, no provider saw it before it went up, and there are no affiliate links on this page. Some of it ran on credits I had from hackathons, friends, etc.<br>Self-funded<br>The prices on this page are what I got charged. There are good pay-as-you-go plans and free tiers to try each of these. I spent around $40.08 across the eight providers.<br>Checkable<br>Every question-rating pair can be checked in the explorer, with the sentence the judge wrote to justify its rating. Read them there.<br>Pooling<br>Every page is graded once, so if several providers returned the same link I only ran it past the judges once, to control cost. There is a real methodological argument for pooling too. A page is detached from whoever returned it before anyone grades it, so there is no provider name for a model to be swayed by. 12,954 question-page pairs collapse to 7,496 unique ones, and grading the unique set meant 42% fewer judge calls. Where several providers returned the same URL, the longest extraction is the one that got graded, so extraction quality is averaged away on those pages. Would not pool next time with a larger budget.<br>Context management<br>To preserve cost and context size, each result is truncated to 1600 characters before a judge sees it. That is a realistic slice of an agent's context budget and it was also the cheapest way to make this many ratings affordable, but it cuts hardest against the providers that return whole documents, so a bigger slice could reorder the bottom of the table. The reason I do not think it is only measuring truncation is Exa and Brave, which run past the budget on 93% and 77% of their results and still place fourth and second. More on that in the limits.<br>Experimental design<br>This run was designed as a follow-up to the first one on Sourcery. Making it agnostic to the model's answer was my takeaway from run 1, which found very small differences between search APIs largely because the models already knew a lot of the answers. Run 2 overhauls the scoring, widens the field of both providers and judge models, and raises the latency threshold so the slower providers get to finish. Looking at you, Bright Data.
I thought it important to be upfront about the limits of this run. If there is something you would want done differently, DM me on X. There is more in the methodology, and what this cannot tell you.
How to pick your provider
What you're building decides the best provider for you. Three of these eight are tied on the score below and they still aren't interchangeable: one returns snippets, one returns prose with the links stripped out, one returns excerpts with the links left in. So pick on the job at hand, and weigh what your code can check: it can see an empty result and handle it, but it is much harder to account for a confident near-miss.
Perplexitybest overall
Clear of the field on the score and on the cheapest tier. Fully answers 49.9% of the time, about 1.4× anyone else. But to be clear, with Perplexity you are getting excerpts not whole pages. Only 40% of its results even reach the 1600-character budget, against 98% of Firecrawl's. So if your agents need to read a document end to end rather than a summary, providers like Tavily, Firecrawl and Exa are your best bets.
Exabest for agents
Keeps structure and strips navigation: 89% of its pages have headings, with only 7% having links. Top-cluster relevance, and it goes toe to toe with Parallel while having a much better developer experience and dashboard. The best option when you are feeding text to a model and link noise costs you context.
Parallelbest balance
Third, close enough to Brave and Exa that the order between the three of them is noise, on the cheapest tier at $0.005 a query, and quick. The thing that sets it apart is that alongside Perplexity it is the only arm that does not drop between the base and the hard half, while everyone else gets worse exactly where you needed the help. 89% of its pages keep headings and 36% keep links, so it sits between Exa's stripped prose and Tavily's whole documents.
Bravesummaries only
Ranks second on relevance and returns real page text on 1% of results, and its content is search snippets joined together. Fine for grounding a summary, but wrong for anything that has to read a page.
Serperbest raw search
Links and snippets, with no page content, at $0.001 a query. The cheapest, and the best of them if you already have a scraper and only need the index.
Firecrawlhardest to fool
On the 12 questions with no answer, it produced the fewest convincing near-misses, beating Perplexity on all 12. That is 12 questions, so read it as a signal rather than a rank. Most expensive of the eight.
Tavilybest for full documents
Full page markdown on 98% of results, with headings, links and images intact. Outscores Firecrawl and costs less. If your agent needs the document rather than a summary of it, start...