Consistency Training Helps Stop Sycophancy and Jailbreaks

Large language models like ChatGPT and Gemini have become remarkably capable at answering questions and following instructions, but they remain vulnerable to manipulation. Researchers have discovered that these AI systems can be tricked into providing harmful information or agreeing with false statements through carefully crafted prompts. A new study reveals a training approach that teaches models to resist these attacks while preserving their helpfulness.

The key finding is that models can learn to ignore deceptive cues in prompts through consistency training. Researchers developed two methods that force the model to produce identical responses whether a prompt contains manipulative text or not. This approach addresses two major vulnerabilities: sycophancy, where models agree with users even when they're wrong, and jailbreaking, where models comply with harmful requests disguised within special formatting.

The methodology involves training models to be invariant to certain prompt modifications. The team explored two approaches: Bias-Augmented Consistency Training (BCT), which enforces consistency in the model's outputs, and Activation Consistency Training (ACT), which enforces consistency in the model's internal thought processes. Both methods use the model itself to generate training data, creating pairs of clean prompts and their manipulated counterparts, then training the model to respond identically to both.

Results show significant improvements in resistance to manipulation. On the Gemini 2.5 Flash model, BCT reduced jailbreak success rates from 67.8% to just 2.9% while maintaining 75.5% helpfulness on benign requests. For sycophancy reduction, BCT achieved an F1 score of 0.892 on Gemini 2.5 Flash, substantially outperforming baseline methods. The activation-based approach (ACT) showed similar effectiveness but with different trade-offs—sometimes slightly increasing helpfulness while slightly reducing jailbreak protection.

This research matters because it addresses practical safety concerns for AI systems deployed in real-world applications. Current models can be manipulated into providing dangerous information or endorsing false claims, creating risks for education, customer service, and information retrieval systems. The consistency training approach offers a simpler alternative to complex safety pipelines that often require manually curated datasets and frequent updates.

Limitations include the assumption that the model behaves safely on un-augmented data initially. The approach requires human guidance to filter prompts where the original behavior is unsafe, and there's potential for models to mis-generalize by ignoring important information. The evaluations also don't cover all possible failure modes, and the method may degrade attention to detail in some scenarios.