AI Now Checks Legal Rules More Accurately

Businesses and organizations face a growing challenge: ensuring they comply with complex regulations like the GDPR, which governs data privacy across the European Union. These rules are dense, interlinked, and full of exceptions, making manual checks error-prone and costly. A new AI framework, GraphCompliance, addresses this by combining structured graphs with large language models (LLMs) to automate compliance judgments, reducing errors and improving reliability for real-world scenarios.

The key finding from the research is that GraphCompliance significantly boosts accuracy in determining regulatory compliance. In tests using 300 GDPR-inspired scenarios, it improved performance by 4.1 to 7.2 percentage points over LLM-only methods and retrieval-augmented generation (RAG) approaches. It also lowered false positive rates and reduced tendencies toward under- or over-prediction, meaning it's better at correctly identifying when rules are violated or followed.

Methodologically, the framework works by aligning two types of graphs. First, a Policy Graph is built from regulatory texts, breaking them into structured units that capture obligations, prohibitions, and cross-references. Second, a Context Graph formalizes real-world scenarios into entity-relation triples, such as 'IT operations manager exports EHR data.' These graphs are then processed through a Compliance Gate, which uses an LLM to make judgments based on pre-analyzed, simplified information, focusing the AI on nuanced reasoning rather than raw text interpretation.

Results analysis shows clear gains: GraphCompliance achieved micro-F1 scores up to 59.0% and macro-F1 up to 60.2% in evaluations, outperforming baselines like raw LLMs and GraphRAG. For instance, in handling decision-tree-like rules (as in Figure 1(b)), it avoided logic breaks that plague keyword-based methods. Ablation studies confirmed that each component—Policy Graph, Context Graph, anchoring, and reference traversal—contributes critically, with removals causing performance drops of up to 10.2 percentage points. The framework also demonstrated robustness across different LLMs and prompt variations, with low sensitivity to paraphrasing.

In context, this matters because automating compliance can save time and reduce risks in sectors like healthcare and finance, where errors can lead to penalties or data breaches. By structuring regulations into graphs, GraphCompliance helps AI navigate cross-references and exceptions more reliably, making it suitable for applications where verifiable, auditable decisions are essential. This approach could streamline audits and enhance trust in AI-driven regulatory systems.

Limitations noted in the paper include dependencies on the quality of graph construction, which could affect judgment accuracy if not automated robustly. Future work aims to extend the framework to broader domains and improve graph-building automation to reduce manual efforts.