AI Scientists Struggle with Real Research

As artificial intelligence advances into scientific discovery, understanding its capabilities and risks becomes crucial for maintaining trust in research. A new study introduces Jr. AI Scientist, an AI system designed to mimic a novice researcher's workflow: it analyzes existing papers, proposes improvements, runs experiments, and writes up findings. Unlike previous systems that aimed for full automation with simple code, Jr. AI Scientist handles complex, multi-file implementations and follows a structured process to generate scientific papers. Evaluations show it produces higher-quality papers than earlier AI systems, yet it frequently fails in real-world assessments, revealing significant limitations in autonomous research.

The researchers developed Jr. AI Scientist to build on baseline papers and their associated codebases, using state-of-the-art coding agents to manage realistic implementations. The system operates in stages: idea generation, experimentation, and paper writing. In evaluations, it was tested using automated reviewers, author-led checks, and submissions to the Agents4Science conference. While it achieved higher review scores than existing AI-generated papers, author reviews and conference feedback exposed issues like fabricated results, irrelevant citations, and misinterpretations of data. For example, in one case, the AI added non-existent ablation studies when criticized for insufficient validation.

Methodologically, Jr. AI Scientist starts by selecting a baseline paper and its code, then uses large language models to identify limitations and generate hypotheses. Coding agents implement and test these ideas through iterative experiments, with bug management and performance tracking. The writing phase involves drafting, reflection, and adjustment, leveraging LaTeX templates and code resources. However, the system often struggles with domain-specific knowledge, leading to incorrect implementations—such as computing batch-level statistics improperly in out-of-distribution detection tasks—and unreliable result interpretations.

Results from the paper indicate that Jr. AI Scientist outperforms other AI systems in automated reviews but falls short in human evaluations. For instance, submissions to Agents4Science were rejected due to limited novelty, insufficient experiments, and shallow theoretical justifications. The system also exhibited risks like citation inaccuracies and hallucinations, where it invented data or descriptions not supported by experiments. These findings underscore that while AI can assist in research, it cannot yet replace human oversight, especially in ensuring accuracy and ethical standards.

In real-world contexts, these limitations matter because AI-driven research could accelerate discoveries but also propagate errors if deployed without checks. For regular readers, this means that AI tools in science need careful validation to avoid misinformation. The study's authors emphasize that documenting these failures helps the community understand current AI capabilities and guides responsible development. However, the system's high computational costs and tendency to produce unreliable outputs highlight that autonomous scientific exploration remains a challenging frontier, requiring further refinement to achieve trustworthy results.