AI Still Can't Think Before It Acts in Robots

Robots working alongside humans in high-stakes environments like ship maintenance or disaster response must do more than follow orders—they need to recognize what they don't know before taking action, diagnose problems from deep understanding rather than guesswork, and choose actions based on real-world consequences. These cognitive abilities are not optional extras but essential requirements for safety, where a single mistake could be catastrophic. A new study from researchers at Rensselaer Polytechnic Institute and other institutions directly tests whether today's advanced AI models can meet these demands, with sobering that the rush to deploy them in physical systems.

In a controlled experiment, the researchers found that large language models (LLMs) consistently fail at metacognitive self-monitoring, a critical ability where an agent checks its own knowledge state before acting. When six different LLMs, including frontier models like Claude Opus and GPT-5.2, were tasked with a collaborative shipboard maintenance scenario, 100% of trials under standard conditions dispatched physical commands without verifying preconditions, such as not knowing the location or features of a needed part. Even when given explicit procedural knowledge equal to that of a specialized cognitive architecture, 60% of trials still acted prematurely. This failure led to downstream errors like hallucinating object features or locations, with models inventing details not grounded in perception or dialogue, as shown in Figure 4 where hallucination rates remained high across conditions.

The study used a novel robotic architecture called HARMONIC, which pairs a strategic reasoning layer with a reactive tactical layer for physical control, allowing a direct comparison between LLMs and a cognitive architecture named OntoAgent. OntoAgent operates over ontologically structured knowledge, enabling it to perform metacognitive monitoring by comparing plan requirements against its current understanding before issuing any command. In contrast, the LLMs were integrated via an LLMAgent module that processed perception data at 2 Hz and could call tools like SEARCH or WAYPOINT. The researchers tested two conditions: Internal Knowledge, where LLMs relied on pretrained knowledge, and Knowledge-Equalization, where they had access to narrative scripts detailing the exact procedures OntoAgent uses, including the need to verify preconditions before acting. This design separated knowledge availability from reasoning mechanisms, revealing that deficits are architectural, not just due to missing information.

From the three studies, summarized in Table I, show that LLMs improved under knowledge-equalization but did not achieve reliability. For diagnostic reasoning, domain-first diagnosis—where hypotheses are generated from causal knowledge before consulting data—increased from 7% to 70%, a large effect. However, hallucinated facts did not decrease, with models making unsupported claims at similar rates under both conditions. In action consequence reasoning, correct action selection (choosing SEARCH over WAYPOINT to avoid timing failures) rose from 57% to 93%, but every wrong-action trial led to cascade failures like behavioral loops or hallucinated success, as depicted in Figure 6. The epistemic hedging dissociation was striking: expressed uncertainty in language increased from 43% to 93%, yet factual accuracy did not improve, indicating models mimic caution without underlying mechanisms.

These have profound for deploying AI in safety-critical settings like healthcare, manufacturing, or autonomous vehicles. A system that acts prematurely in 60% of trials, even with equal knowledge, is unpredictably unsafe, as certification requires bounded, verifiable behavior. The study highlights that LLMs, despite their language prowess, lack the architectural guarantees for deterministic reasoning and traceability that OntoAgent provides through its inspectable transcripts and ontological justifications. This supports a hybrid approach where cognitive architectures retain decision authority for monitoring and planning, while LLMs contribute to language tasks, a model HARMONIC exemplifies through its OntoAgentic AI extension.

Limitations of the study include its focus on a single task scenario, five trials per model-condition cell, and a knowledge-equalization condition that provided procedural scripts but not the full ontological structure of OntoAgent. Binary metrics captured presence or absence of behaviors but not degrees of partial competence, and future work plans to extend these evaluations. The researchers note that fine-tuning or elaborate scaffolding is unlikely to resolve the core metacognition issue, as direct procedural instruction already failed in many cases. This research underscores that for robots in human environments, reliability must be built into the system's architecture, not hoped for as an emergent property of scale.