Study Maps How Users Build Mental Models of AI Systems

People develop internal explanations of how AI systems work, but these mental models often diverge from reality. A new review paper in AI & Society finds that this gap between user understanding and actual system behavior represents a critical safety challenge for modern artificial intelligence deployment.

The study, titled "Users' mental models within human-AI interaction: a systematic scoping review," analyzed 52 empirical studies on how people understand and predict AI behavior. Researchers discovered that while interest in this area is exploding, the field lacks shared concepts and rigorous long-term research on how understanding evolves over time.

Mental models determine whether users rely on AI recommendations, challenge them, or ignore them entirely. When people overestimate capabilities, they may trust flawed outputs and surrender situational awareness. When they underestimate, they reject potentially valuable assistance. The core problem is calibration—users need accurate internal models that neither inflate nor dismiss AI capabilities.

Modern systems complicate this challenge because they are probabilistic, adaptive, and generative rather than rule-based tools. These systems write text, make recommendations, classify images, and increasingly appear as collaborators or advisors. Unlike traditional software with predictable logic, contemporary artificial intelligence operates through statistical patterns that even developers struggle to fully explain.

The review connects current challenges to decades of human-computer interaction research showing that people naturally respond socially to computers, even when they know the systems lack consciousness. This tendency intensifies as AI systems become more conversational and seemingly autonomous.

The fragmentation problem matters because without shared frameworks, each research group develops its own theories and methods. This makes it difficult to build cumulative knowledge or establish best practices for designing AI systems that support appropriate user understanding.

For practitioners, this research suggests that technical performance alone won't determine real-world success. Systems must also support accurate mental models through transparency mechanisms, appropriate uncertainty communication, and interface design that reflects actual capabilities rather than anthropomorphic assumptions.

The historical parallel to early computer interaction research offers some guidance. Decades ago, researchers learned that users needed conceptual models matching system behavior—a lesson that becomes more complex as systems grow less predictable.

Looking ahead, the field needs longitudinal studies tracking how mental models shift over extended use, standardized measurement approaches, and practical design guidelines. Until then, developers should assume users will form their own explanations regardless of intended design, making it crucial to align system behavior with reasonable expectations.

FAQ

What are AI mental models and why do they matter?
Mental models are the internal explanations people build to understand how systems work. They determine trust levels and usage patterns, directly affecting safety and effectiveness.

How many studies were reviewed in this research?
The systematic review analyzed 52 empirical studies on human-AI interaction and mental model formation.

Why is this research important for AI development?
Accurate mental models help users calibrate trust appropriately, preventing both over-reliance on flawed outputs and rejection of useful assistance.

What makes modern AI harder to understand than traditional software?
Contemporary systems are probabilistic, adaptive, and generative rather than following explicit rules, making their behavior less predictable even to developers.