AI Learns Safer Decisions from Past Data

Artificial intelligence systems that learn from data without real-world interaction could transform fields like robotics and healthcare, where trial-and-error is costly or dangerous. In a study published as arXiv:2511.02567v1, researchers from Tsinghua University introduce an algorithm called Adaptive Neighborhood-constrained Q-learning (ANQ) that addresses a key challenge in offline reinforcement learning: avoiding errors when AI evaluates actions not present in its training dataset. This advance allows AI to learn effective decision-making policies from static datasets, reducing the need for risky exploration.

The researchers found that ANQ restricts the AI's action selection to neighborhoods around actions in the dataset, preventing overestimation of untested actions while allowing flexibility to find better alternatives. This approach mitigates the over-conservatism seen in prior methods, which either limit choices too strictly or struggle with inaccurate behavior modeling. Theoretically, the neighborhood constraint bounds errors under certain conditions and approximates the least restrictive constraint without requiring complex modeling of the behavior policy.

Methodologically, the team employed a bilevel optimization framework. In the inner level, the algorithm maximizes the Q-function—a measure of action quality—separately within each action's neighborhood. In the outer level, it implicitly maximizes over all available actions using techniques like expectile regression. An adaptive criterion adjusts neighborhood sizes based on action advantages, with larger radii for low-advantage actions to encourage broader search and smaller radii for high-advantage actions to limit errors. The final policy is extracted through weighted regression toward optimized actions.

Results from standard benchmarks, including Gym-MuJoCo and AntMaze tasks, show ANQ achieving state-of-the-art performance. For example, in locomotion tasks, it attained normalized scores like 111.7 for walker2d-medium-expert and 107.0 for hopper-medium-expert, outperforming baselines such as CQL, IQL, and SPOT. In challenging AntMaze environments, ANQ scored up to 96.0 in umaze tasks, demonstrating strong generalization. The algorithm also exhibited robustness in noisy and limited data scenarios, maintaining performance even when expert data ratios decreased or portions of the dataset were discarded, as shown in Figures 1 and 2 of the paper.

This work matters because it enables more reliable AI decision-making in real-world applications where data collection is expensive or hazardous, such as autonomous driving or medical treatment planning. By reducing reliance on perfect datasets and avoiding modeling complexities, ANQ could accelerate the deployment of AI systems in safety-critical domains. However, limitations include the potential non-optimality of the adaptive neighborhood design; incorporating additional factors like uncertainty quantification might lead to further improvements. The method's efficacy in highly stochastic environments remains an area for future investigation.