AI Fails at Basic Medical Math

When artificial intelligence systems can write poetry and generate computer code, you might assume they can handle basic medical arithmetic. A new study reveals this assumption is dangerously wrong, exposing critical gaps in AI's ability to perform the fundamental calculations that doctors use daily for patient care.

The researchers found that large language models consistently fail at medical calculations, with most models scoring below 30% accuracy on a comprehensive test of 709 different medical calculation tasks. Even the best-performing model without specialized training achieved only 31.1% accuracy, meaning it got nearly 7 out of 10 calculations wrong. This performance gap exists despite these models being trained on enormous amounts of medical text and data.

The research team developed a specialized training environment called MedCalc-Env that uses reinforcement learning to improve calculation skills. In this system, the AI model receives immediate feedback on its calculations, allowing it to learn from mistakes in a simulated medical setting. When they applied this method to the Qwen2.5-32B model, accuracy improved dramatically from 25.4% to 40.8% - a significant 15.4 percentage point gain that demonstrates the potential for targeted training to address specific weaknesses.

The data shows clear patterns in where AI models fail. The most common errors involve incorrect recall of medical formulas and rules, accounting for the majority of mistakes. Models also struggle with extracting the right numbers from patient information and making simple arithmetic errors, even when they identify the correct formula. The reinforcement learning approach specifically reduced these knowledge and calculation errors, though extraction problems remained challenging.

For healthcare professionals and patients, these findings matter because medical calculations aren't academic exercises - they directly impact treatment decisions. Doctors use calculations like the Cockcroft-Gault formula to determine kidney function for medication dosing, the CHA2DS2-VASc score to assess stroke risk in heart patients, and the Glasgow Coma Scale to evaluate consciousness in emergency situations. An AI that miscalculates these could recommend wrong drug doses or misclassify patient risk levels.

The study acknowledges important limitations. Even after specialized training, the best model still got about 60% of calculations wrong. The researchers identified persistent challenges with unit conversions, multi-condition logic, and understanding complex medical contexts. The training focused on structured data, while real medical records often contain unstructured text with redundant information and synonyms that could further challenge AI systems.

This research provides a crucial reality check about AI's current capabilities in healthcare. While language models show promise for many medical applications, their inability to reliably perform basic calculations suggests they're not ready to replace human judgment in critical clinical decisions. The findings highlight the need for continued development and rigorous testing before deploying these systems in real healthcare settings where mathematical accuracy can be a matter of life and death.