Meituan LoHoSearch Benchmark: 544 Questions, 61 Tool Calls, and Real Agent Cost
Editorial standards and source policy: Editorial standards, Team. Content links to primary sources; see Methodology.
Last checked: 2026-07-28.
Key numbers first
The construction graph contains 7.62 million Wikipedia entities and 265 million directed edges. The final benchmark has 544 human-verified questions across 11 domains; 75.5% passed human review directly, 22.3% were accepted after edits, and 2.2% were discarded. Meituan reports GPT-5.5 at 34.74%. DeepSeek-V4-Flash scores 10.02%, while average tool calls on successful traces rise from 35 on BrowseComp to 61 on LoHoSearch.
The useful part of this benchmark is not another universal leaderboard. Longer tasks expose breaks in search, memory, planning, navigation, and tool use. Scores come from the publisher's environment and settings. They can justify an internal rerun, but they cannot replace one.
What is actually being measured
Read the result in four layers: how tasks or environments were constructed, which tools a model could call, how many steps each run received, and how the final score was computed. A single percentage hides tool calls, retries, latency, and external-search cost. Two systems with the same accuracy can have very different budgets and failure surfaces when one needs 60 calls and the other needs 20.
How the 544 questions were built
LoHoSearch does not ask a model to improvise obscure trivia. It treats English Wikipedia pages as entities, article links as directed edges, and Wikidata P31 values as types. It samples both trees and denser graphs: trees expand the candidate set, while graph structures add crossing constraints and cycles. Generated questions then pass automated checks, a search-agent validation step, filters for ambiguous or easy items, and human review.
This addresses a structural weakness of human-authored benchmarks. Authors tend to start from entities they already know and cannot precisely control candidate-space size or relationship complexity. The graph makes both measurable. Yet automated construction is not automatically clean: 2.2% of items were discarded for serious problems and 22.3% needed annotator edits, so human review remains part of the benchmark's quality claim.
Reading 34.74% and 10.02%
Meituan reports GPT-5.5 at 34.74%. DeepSeek-V4-Flash falls from 58.84% on BrowseComp to 10.02% on LoHoSearch; graph questions score 8.01% versus 11.89% for trees. The model is not simply unable to search. It loses intermediate evidence when the candidate set grows and constraints must be joined.
Repeated sampling raises the chance that one run finds the answer: pass@N rises from 9.3% at N=1 to 38.3% at N=16. But best-of-N selection reaches only 24.6%, well below the pass@16 ceiling. The system can occasionally produce the right answer without reliably recognizing it, making verification and aggregation part of production cost.
Long-horizon search is a systems problem
Successful traces average 61 tool calls instead of 35, and the median rises from 26 to 59. Every extra call adds link decay, rate limits, duplicated text, context compression, and billing variance. Discard-all plus Verify raises 10.02% to 16.82%, a 6.8-point gain, versus roughly 14 points for comparable strategies on BrowseComp. Simple summarization or context resets do not preserve which evidence satisfies which constraint.
An internal run should therefore store calls per question, unique domains, repeat-visit rate, live-citation rate, conflicting evidence, tokens, retries, and wall time. A three-point accuracy gain is not automatically useful if it requires three times the calls and twice the latency.
Turn a public benchmark into an internal test
Select 20 to 60 tasks with known answers and divide them into single-step, multi-step, and conflicting-information groups. Lock model version, tools, maximum steps, and budget, then save full traces. Record invalid actions, wasted searches, working citations, recoveries, latency, and total cost alongside success rate. Public scores shortlist candidates; this task pack decides deployment.
Practical case
A 12-person research team runs 40 vendor checks each week. It selects 60 questions with known answers and splits them into single-constraint, three-constraint, and conflicting-source groups. Every run records page requests, search calls, citation validity, elapsed time, and model cost. Acceptance requires at least 80% answer accuracy, one working source for every material claim, and p95 latency below eight minutes. Automation pauses after two rounds of broken citations, search loops, or cost above 1.5 times the human baseline.
Common reading mistakes
- Presenting a vendor result as independent reproduction.
- Assuming the same model name means identical inference settings.
- Reporting success without calls, failed traces, latency, or cost.
- Using static question-answering results to predict sustained action in a changing environment.
- Making a procurement decision when sources, versions, or run settings are missing.