How LLMs Are Revolutionizing Argument Mining—And Where They Still Fail

In the rapidly evolving landscape of artificial intelligence, large language models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks, from creative writing to complex reasoning. Yet one particularly challenging domain—argument mining—has remained a critical testbed for evaluating true language understanding. A comprehensive new study from researchers at AGH University of Krakow and the University of Silesia provides the most extensive evaluation to date of how modern LLMs perform at automatically identifying and classifying argumentative components in natural language. Their reveal both impressive progress and persistent limitations that could shape the future of AI-powered discourse analysis.

The research team conducted a systematic evaluation of nine state-of-the-art LLMs, including GPT-5.2, Llama 4, DeepSeek R1, and several open-weight models, across two established argument mining benchmarks: the UKP corpus and Args.me dataset. These corpora contain thousands of manually annotated arguments on controversial topics ranging from abortion and gun control to nuclear energy and school uniforms. The study employed advanced prompting strategies including Chain-of-Thought reasoning, prompt rephrasing, and certainty-based classification to push the models to their limits. What emerged was a nuanced picture of where LLMs excel and where they fundamentally struggle with the complexities of human argumentation.

Quantitatively, show substantial progress over previous approaches. GPT-5.2 achieved the highest accuracy at 78.0% on the UKP corpus and 91.9% on Args.me, significantly outperforming traditional machine learning s like BERT (57.7% on UKP) and LSTM architectures (45% on UKP). Perhaps more surprisingly, the open-weight model gpt-oss-120b performed nearly as well as the proprietary GPT-5.2 when enhanced with voting strategies, achieving 82.1% accuracy on UKP and 89.6% on Args.me. The researchers found that prompt engineering and ensemble techniques could boost performance by 2-8 percentage points, with multi-prompt voting strategies proving particularly effective for mitigating individual prompt failures.

However, the qualitative error analysis revealed systematic limitations that aggregate metrics mask. The models consistently struggled with contrastive discourse structures marked by words like "but" or "however," often misinterpreting which clause contained the main argumentative conclusion. They frequently failed to resolve referential dependencies involving pronouns like "it" or "this," losing track of what exactly was being argued about. Perhaps most concerning was the tendency to infer argumentative intent based on topic associations rather than textual evidence—for instance, systematically misclassifying statements about the death penalty as opposing arguments regardless of their actual content. These failures suggest that while LLMs can recognize surface patterns, they lack deeper understanding of discourse structure and pragmatic inference.

Of this research extend beyond academic benchmarks to real-world applications in legal analysis, policy debate monitoring, and social media moderation. The finding that smaller open-weight models can approach state-of-the-art performance with proper prompting suggests more accessible and cost-effective argument mining systems could be developed. However, the persistent failure modes around implicit criticism, complex reasoning, and topic bias indicate that fully automated systems should be deployed cautiously in high-stakes contexts. As the authors conclude, future progress will require not just larger models but architectural innovations specifically targeting discourse-level reasoning and training regimes that incorporate argumentation theory alongside next-token prediction.