AI Struggles to Reconstruct Code from Real Repositories

Large language models have become essential tools for software development, but a new benchmark reveals they still struggle with one of the most fundamental tasks: understanding how code fits together across entire repositories. Researchers have introduced R E CUBE, a benchmark that directly tests how well AI models can leverage repository-level context to generate functionally correct code. show that even state-of-the-art models like GPT-5 achieve only a 37.57% strict pass rate when reconstructing masked files from real-world codebases, indicating that current AI systems lack the architectural understanding needed for complex software engineering.

The core from the R E CUBE benchmark is that AI models find it particularly challenging to implement code that integrates properly across multiple files. The benchmark evaluates models on their ability to reconstruct a masked Python file using only other source files, dependency specifications, and documentation from the same codebase as context. This task requires understanding not just individual functions but how those functions interact with the broader system. The researchers found that models perform significantly worse on external test cases, which verify cross-file interactions, compared to internal tests that check self-contained logic, with an average 9.55% gap in strict pass rates.

To create this benchmark, the researchers collected 20 high-quality GitHub repositories created after January 2025, each with at least 10,000 stars and 1,000 forks to ensure codebase quality. They manually curated these repositories into 40 functional subsets representing distinct, self-contained functionalities, then masked 366 target Python files by removing import statements and replacing function bodies with NotImplementedError while preserving signatures and docstrings. For each instance, models receive a repository-level context that includes the masked files, environment dependencies with version descriptions from PyPI, and relevant documentation files, formatted with XML-style delimiter tags to help models distinguish components.

Across eight models in four experimental settings reveal consistent patterns of difficulty. In the full-context setting where models receive all information in a single prompt, GPT-5 achieved the highest strict pass rate at 37.57% and average pass rate at 60.43%, while smaller open-source models like Qwen3-Coder 30B reached 25.34% strict pass rate. Performance varied significantly across functional domains, with LLM services and agent frameworks showing higher tractability (up to 44.19% strict pass rate) while voice and speech processing tasks proved substantially more difficult. The researchers also introduced a Caller-Centric Exploration toolkit that uses dependency graphs to guide agents toward relevant caller files, which consistently improved performance across all models, with gains of up to 7.56% in strict pass rate over basic agentic frameworks.

These have important for both AI research and practical software development. The benchmark demonstrates that current AI coding assistants may be limited to surface-level pattern matching rather than deep architectural understanding, which could affect their reliability in real-world development scenarios. The researchers note that their CCE toolkit shows promise for improving AI's repository navigation, but the overall low performance suggests fundamental s remain. For developers, this means AI tools may struggle with tasks requiring cross-file integration or understanding of complex dependencies, potentially limiting their usefulness for maintaining and extending large codebases.

The study acknowledges several limitations, including R E CUBE's narrow domain scope that predominantly focuses on LLM-related projects and its restriction to Python files. The high computational cost of evaluation also limited the study to a small subset of models, particularly in agent-based settings. Additionally, because the benchmark uses recent, popular repositories, it may favor prevalent codebase patterns rather than representing the full diversity of software engineering practices. The researchers plan to address these limitations in future work by evaluating a broader range of models and incorporating additional frameworks to further assess the effectiveness of their proposed toolkit.