PaperBench: Evaluating AI’s Ability to Replicate AI Research

2026

Open publication workspace · Sign in to read the full PDF.

AI-generated summary

PaperBench: Evaluating AI’s Ability to Replicate AI Research

1) PaperBench is a novel benchmark designed to rigorously evaluate AI agents' capabilities in replicating complex machine learning research papers from scratch, offering a crucial measure of AI autonomy in scientific discovery.
2) This benchmark includes 20 ICML 2024 Spotlight and Oral papers, author-developed rubrics with over 8,000 gradable tasks, and an LLM-based grading system.
3) The study reveals that current AI agents show limited ability to fully replicate research papers, with the best-performing agent achieving only 21.0% of the replication score, highlighting the complexity of long-horizon AI R&D tasks.
Tags: AI research replication, benchmark, machine learning, AI agents, evaluation

Check the original publication for accuracy and context.