LiveSearch: Questions as Executable Verifier Code for a Live Benchmark of Professional-Source Financial Search Agents
Finance answers must be current; benchmarks agents on live, source-grounded retrieval with verifiable queries instead of static QA.
The explainer · 3-minute read · 251 words
The claim
A financial benchmark recomputes its gold answers at evaluation time, so memorized answers go stale.
The problem
In finance, the current answer is the one that matters.
Most benchmarks freeze gold answers at annotation time.
Stale gold rewards recall of old data, not live search.
How it works
Each question is stored as executable verifier code.
The code makes typed calls to a professional financial data service.
A declarative extractor turns the responses into the gold answer.
The gold is computed when the evaluation runs, with no human in the loop.
A daily job re-executes every question to keep answers current.
Figures
What changes
Construction is automatic too: a language model verbalizes programs but never chooses answers.
Under general web search, six tool-calling models reach 70.9% to 76.7% accuracy.
Over three days, 70% of served questions change their gold value.
Limits
Accuracy depends on access to professional data services.
Daily re-execution costs compute and needs stable tool endpoints.
The benchmark covers finance search, not other tool-use domains.
Source · arXiv wire