The Hidden Assumptions Behind AI Safety: A New Study Reveals Critical Gaps in Cyber-Physical Systems

In the high-stakes world of artificial intelligence and autonomous systems, formal guarantees—those mathematical promises of safety, reliability, and performance—are the bedrock of trust. But what if those guarantees rest on shaky, unspoken foundations? A groundbreaking survey from researchers at the University of Florida delves into the often-overlooked realm of assumptions in cyber-physical systems (CPS), uncovering that many of the assurances we rely on are built atop a patchwork of implicit and underspecified conditions. By analyzing 104 papers from 2014 to 2024, the study extracted 423 assumptions and 321 guarantees, revealing that modeling abstractions dominate while critical elements like perception, sensing, and neural components are frequently left in the shadows. This systematic examination not only highlights prevalent trends but also sounds an alarm for the AI industry, where GPUs power everything from self-driving cars to industrial robots, urging a more rigorous approach to assumption reporting to prevent catastrophic failures in real-world deployments.

To uncover these insights, the researchers employed a meticulous grounded-theory ology, scouring literature from control, verification, and runtime assurance domains. They defined assumptions as falsifiable predicates that bridge abstract guarantees—proven mathematically—to operational realities in deployed systems, using a universe of discourse that includes real-world systems, environments, models, and artifacts. Through open, axial, and selective coding, they categorized assumptions into four high-level groups: Modeling (e.g., system abstractions and model validity), External Constraints (like environment and human behavior), Physical (covering dynamics and actuation), and Interface and Estimation (encompassing sensing and perception). This rigorous process involved tagging each assumption with specific labels and language features, from algebraic expressions to probabilistic statements, ensuring a comprehensive analysis of how assumptions support guarantees across diverse CPS applications.

Paint a stark picture of current practices: modeling assumptions account for over half (53.8%) of all instances, with system abstraction alone making up 35.5%, indicating a heavy reliance on theoretical models rather than real-world interfaces. In contrast, sensing and perception assumptions are notably sparse, representing just 8.5% combined, despite their critical role in safety-critical systems like autonomous vehicles. Guarantees are similarly skewed, with safety dominating at 27%, followed by feasibility and performance, while robustness and resilience lag behind. Language-wise, assumptions lean heavily on algebraic and logical features (e.g., variables and arithmetic), with uncertainty-related terms like probability and distribution comprising only 14.1%, suggesting a preference for deterministic bounds over stochastic clarity. Co-occurrence analyses further reveal that initialization conditions often underpin safety and robustness claims, hinting at localized rather than global assurances, and neural assumptions are rare, often buried under broader modeling tags.

These have profound for the tech industry, particularly as AI and GPUs drive advancements in robotics, autonomous systems, and smart infrastructure. The underspecification of perception and sensing assumptions, for instance, could lead to unsafe behaviors in real-world scenarios where sensor noise or environmental changes occur, undermining the very guarantees that justify deployment. Similarly, the lack of explicit neural assumptions complicates accountability in learning-enabled systems, making it hard to trace failures to specific components. By calling for clearer reporting on initialization dependence, sensing constraints, neural properties, and uncertainty, the study advocates for a cultural shift in CPS research—one that prioritizes transparency and testability to enhance reproducibility and trust. This is especially urgent in fields like autonomous driving and industrial automation, where assumption failures could result in financial losses, legal liabilities, or even loss of life.

However, the study acknowledges its limitations, including a corpus focused on papers with explicit guarantees, which may underrepresent practical deployments, and the subjective nature of manual coding that could miss nuances in assumption taxonomy. External validity is also constrained by the sampling from specific venues and years, potentially skewing insights toward theoretical over applied work. Despite this, the research offers a foundational framework for future efforts, encouraging expanded surveys, domain-specific replications, and community engagement through an open database. As AI continues to evolve, embracing these guidelines could bridge the gap between abstract proofs and real-world resilience, ensuring that the systems we build are not just theoretically sound but practically assured.

Reference: Li, C., Faghfoorian, S., & Ruchkin, I. (2025). What Does It Take to Get Guarantees? Systematizing Assumptions in Cyber-Physical Systems. arXiv.