AI's Hidden Reasoning Flaws Threaten Cancer Care

A new study reveals that large language models like GPT-4, despite performing well on standard medical tests, commit reasoning errors that mirror human cognitive biases when interpreting real clinical oncology notes. These errors occur in nearly one-quarter of interpretations and are strongly linked to recommendations that deviate from established cancer care guidelines, potentially putting patients at risk. the current reliance on accuracy-focused benchmarks for evaluating medical AI and highlight the urgent need to assess how these models think, not just what they conclude.

The researchers found that GPT-4 made errors in 23.1% of its interpretations of oncology notes, with reasoning failures accounting for 85.4% of these mistakes. The most common errors were confirmation bias, where the model selectively attended to data aligning with preliminary impressions while dismissing contradictory evidence, and anchoring bias, where it over-weighted initial clinical . These cognitive biases, familiar in human clinical reasoning, manifested in the AI's chain-of-thought responses, showing it could reach correct answers through faulty logic. The error rates increased notably in recommendation tasks, especially for managing metastatic disease, precisely when clinical stakes are highest and decisions most complex.

To uncover these flaws, the team developed a novel three-tier hierarchical taxonomy to classify reasoning errors in GPT-4's responses. They analyzed 600 chain-of-thought reasoning outputs generated from 40 real oncology progress notes for breast and pancreatic cancer, using prompts designed for extraction, analysis, and recommendation tasks aligned with clinical phases like presentation, evaluation, and management. The taxonomy mapped computational failures onto established cognitive bias frameworks, with high inter-rater agreement (κ≥0.85) ensuring reproducibility. This framework was then validated on an independent cohort of 24 prostate cancer consult notes spanning localized to metastatic disease, generating 822 additional responses to assess error patterns across different clinical contexts.

The data shows a clear correlation between reasoning errors and clinically meaningful risks. In recommendation tasks, responses containing reasoning errors received significantly lower clinical impact scores on a 5-point scale, with an average drop that indicated potential harm. These outputs were also more likely to be discordant with National Comprehensive Cancer Network guidelines. For example, among recommendations rated as potentially harmful, specific error subtypes like confirmation bias, anchoring bias, and stark omission were disproportionately represented. Error rates varied by task and context, with analysis prompts showing 19.8% errors in management contexts, while recommendation prompts saw increased errors in advanced disease stages like metastatic castration-resistant prostate cancer.

This research has critical for the safe deployment of AI in healthcare. It exposes a fundamental gap in current evaluation paradigms that prioritize endpoint accuracy over logical validity, as models can pass medical exams while making reasoning errors that lead to unsafe recommendations in real-world scenarios. The study suggests that prospective monitoring of reasoning quality could serve as an early warning system, with error types like confirmation bias being targetable through debiasing prompts or mandatory human review for high-risk queries. However, automated attempts to detect these errors using other LLMs like Claude and Gemini showed limited success, achieving reasonable sensitivity for error presence but poor subtype classification, underscoring the need for human oversight in clinical workflows.

Despite its insights, the study has several limitations. It focused on a single model, GPT-4-32k, though this allowed in-depth characterization of a widely used system. The assessment of clinical harm relied on expert judgment using NCCN guidelines in a retrospective, note-based simulation, not actual patient outcomes. The use of zero-shot prompting reflects realistic usage but may not capture how engineered prompts could reduce errors. Additionally, the research did not compare LLM errors directly against human clinician errors on identical cases, limiting the ability to contextualize these failures within existing clinical practice. These constraints highlight the need for further studies to validate across models and real-world settings.