Benchmarking LLM agents on meta-analysis articles from Nature Portfolio.
A meta-analysis statistically combines results from independent studies that answer the same research question, usually inside a systematic review that retrieves candidate studies and screens them against predefined eligibility criteria — organized as Population, Intervention/Exposure, Comparison, and Outcome (PI/ECO).
MetaSyn turns 422 Nature Portfolio source reviews into linked benchmark tasks. Each task pairs a research question and PI/ECO fields with the published included-study list and reference synthesis, letting retrieval, protocol-based evidence selection, and report synthesis be evaluated end to end against one shared 140,585-article PubMed corpus.
Retrieval, evidence selection, and synthesis are usually evaluated in isolation. MetaSyn keeps them linked to the same review, so a system's losses at each stage are attributable rather than confounded.
Each stage is scored against the same review, so retrieval, selection, and synthesis outcomes stay attributable to one another rather than evaluated in isolation.
THUIR/MetaSyn ships as two configs that share one ID space: reviews (the 422 tasks, split disjointly into train/test by source review) and corpus (the 140,585-record shared PubMed pool every task searches). A review's matched_corpus_ids point into the corpus, so retrieval, selection, and synthesis can all be scored against the same stable IDs.
reviews · train / test, 422 rowsgeneration_results/review_<ID> and evaluation_results/review_<ID>.corpus · train split only, 140,585 rowsmatched_corpus_ids and source_review_corpus_ids.{heading, text} blocks with PMC full text, when available — otherwise a record is abstract-only.Field names above are taken directly from src/metasyn/data.py, query.py, and agent_tools.py. Code and project-authored annotations are MIT licensed; PubMed metadata, abstracts, and PMC-derived text retain their upstream terms — see the dataset card for field-level detail.
Validated on Python 3.10 / 3.11. Every LLM call goes through a standard OpenAI-compatible client configured in .env; the dataset itself loads through the regular Hugging Face datasets library.
# create an environment and install
python -m venv .venv
source .venv/bin/activate
pip install -e .
cp .env.example .env
from metasyn.data import load_reviews, load_corpus, review_by_id
train_reviews = load_reviews("train") # 336 reviews
test_reviews = load_reviews("test") # 86 held-out reviews
corpus = load_corpus() # 140,585 shared PubMed records
review = review_by_id(63)
review["Research_Question"], review["Population"], review["Outcome"]
review["matched_corpus_ids"] # gold included-study corpus IDs
from metasyn.query import build_protocol_query
# joins Research_Question + PI/ECO fields; the source-review title is never included
query = build_protocol_query(review)
python scripts/build_index.py --output artifacts/ma_retriever_index python scripts/run_retrieval.py \ --retriever ma_retriever --review-id 63 \ --index artifacts/ma_retriever_index --top-k 200 \ --output outputs/retrieval_63.json python scripts/run_rag.py \ --review-id 63 --index artifacts/ma_retriever_index \ --top-k 200 --mode actual --output outputs/rag_63
python scripts/evaluate.py \
--result outputs/rag_63/results.json \
--report outputs/rag_63/report.md \
--output outputs/rag_63/evaluation.json \
--id-only # drop --id-only to add evaluator-dependent metrics
from metasyn.agent_tools import AgentCorpusTools, build_agent_prompt
# leakage-safe search/fetch tools, scoped to one review's task
tools = AgentCorpusTools(retriever, review, k=20, max_distinct_articles=200)
hits = tools.search("focused protocol query")
article = tools.fetch(hits[0]["corpus_id"])
prompt = build_agent_prompt(review)
# expose tools.search / tools.fetch through any agent framework's tool
# interface (used as-is for GPT-Researcher and Open Deep Research)
See the full README and BASELINES.md for exact reproduction settings, the strict-JSON evaluator schema, and MA-Retriever training.
Twelve baseline configurations are released end to end: three retrievers, three one‑pass RAG generators, and three agentic / deep‑research systems.
Query the shared corpus with the research question and PI/ECO fields — never the source-review title — and retrieve a candidate pool.
retrieval_recallretrieval_precisionScreen the retrieved pool against the review's eligibility criteria and output a final included-article list.
inclusion_recall / precision / f1screening_accuracyconditional_retention, post_retrieval_lossWrite the final synthesis: inclusion/exclusion reasoning, conclusion direction, and key insights, judged against the published review.
inclusion_consistency, exclusion_consistencyconclusion_direction_accuracyinsights_consistency, structure_qualityTwelve baseline configurations — three retrievers, three one‑pass RAG generators, and three agentic / deep‑research systems — are released end to end, but their retrieval and evidence-selection numbers are intentionally left off this page. Baseline scores age quickly as retrievers and models improve, and a snapshot here would read as a leaderboard rather than the reference point it's meant to be.
evaluation_results/ — all 1,032 review–configuration records.
generation_results/ — the original report and result pair behind each score.
BASELINES.md — the configuration behind every released number.
arXiv paper — discussion and analysis of the baseline runs.
Code and project-authored annotations are MIT licensed. PubMed metadata, abstracts, identifiers, and PMC-derived text retain their upstream terms.
@misc{metasyn2026,
title = {Benchmarking {LLM} Agents on Meta-Analysis Articles from {Nature} Portfolio},
author = {Anzhe Xie and Weihang Su and Yujia Zhou and Yiqun Liu and Min Zhang and Qingyao Ai},
year = {2026},
eprint = {2606.17041},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
url = {https://arxiv.org/abs/2606.17041}
}