AI Struggles to Recognize Religious Language

Artificial intelligence systems are surprisingly bad at identifying religious language in environmental discussions, according to new research that reveals fundamental limitations in how machines understand human communication. The study shows that even the most advanced language models frequently miss spiritual references that are obvious to human readers, raising questions about AI's ability to navigate complex cultural contexts.

Researchers from Vrije Universiteit Amsterdam discovered that large language models consistently underperform compared to simple rule-based methods when detecting religious language in climate change discourse. The team analyzed over 880,000 sentences from environmental organizations' websites, comparing a traditional keyword approach against AI systems including GPT-4o mini and Llama 3.3 70B. Surprisingly, the rule-based method using a predefined vocabulary of religious terms consistently identified more religious language than the sophisticated AI models.

The researchers employed two distinct approaches. The rule-based method used a hierarchical tree of religious concepts drawn from ecotheology literature, while the AI approach relied on large language models operating in a zero-shot setting—meaning they received no specific training for the task. Both methods were applied to content from nine environmental organizations, including both secular groups like Greenpeace and religiously-affiliated organizations.

The results revealed striking patterns. When analyzing sentences mentioning "Mother Earth," the rule-based method automatically flagged all 1,229 instances as religious language. However, the AI models only recognized 33.8% to 37.2% of these cases as religious. Even more telling was the handling of "sacred earth" references—while the rule-based approach identified all 52 instances, the AI models missed between 25% and 88.5% of these clearly spiritual references.

The study highlights a fundamental tension in how religious language is defined. The AI models tended to focus on explicit religious vocabulary and references to established religious texts, often missing more subtle spiritual expressions. For example, the models frequently dismissed terms like "sacred earth" and "Mother Earth" as merely metaphorical or descriptive, even when human readers would recognize their spiritual significance.

This research matters because it reveals critical limitations in how AI systems interpret human language and culture. As environmental organizations increasingly use spiritual language to convey the urgency of climate action, AI's inability to properly recognize these communications could lead to misunderstandings in automated content analysis, policy development, and public discourse.

The study also uncovered significant inconsistencies in AI responses. Identical sentences were sometimes classified differently by the same model, and the reasoning provided often seemed arbitrary. This suggests that current AI systems lack a stable framework for understanding religious and spiritual concepts, instead relying on statistical patterns from their training data that may not capture nuanced cultural meanings.

While the research focused specifically on climate discourse, the implications extend to any domain where understanding cultural and spiritual context matters—from healthcare communications to legal documents to educational materials. The findings suggest that for now, AI cannot be trusted as an objective classifier of religious language and should be used cautiously in contexts requiring cultural sensitivity.