Who Does Your Algorithm Fail? Investigating Age and Ethnic Bias in the MAMA-MIA Dataset

Automated medical imaging systems promise to revolutionize cancer diagnosis, but new research reveals they may be failing an important demographic: younger patients. A comprehensive audit of artificial intelligence tools for breast cancer detection shows these systems consistently underperform for women under 40, potentially creating dangerous disparities in healthcare outcomes.

Researchers discovered that automated segmentation models for breast cancer detection exhibit significant age-related bias, with younger patients receiving systematically worse performance. The study found that models trained to identify tumors in medical scans performed substantially better for older patients than for younger ones, even when accounting for other factors. This bias persisted across different evaluation metrics and remained significant even when researchers controlled for dataset imbalances.

The investigation used the MAMA-MIA dataset, a large collection of breast MRI scans from multiple medical centers. Researchers employed an "unawareness" auditing approach, meaning they evaluated model performance without explicitly programming the AI to consider demographic factors. They trained standard neural network models using 5-fold cross-validation and compared performance across three age groups: young (under 40), middle-aged (40-55), and older (over 55) patients.

The data revealed clear patterns of bias. As shown in Figure 1, younger patients consistently received lower-quality tumor segmentations. The Demographic Parity Difference metric showed an 8.87% performance gap between age groups, meaning younger patients were significantly less likely to receive high-quality segmentations. The Disparate Impact Ratio of 0.699 indicates that older patients were nearly 50% more likely to receive optimal performance compared to younger patients. These quantitative findings were supported by expert ratings of segmentation quality.

Perhaps most concerning, this bias wasn't merely a result of imbalanced training data. When researchers created a balanced cohort with equal representation across age groups, the performance disparities remained statistically significant (ANOVA p=0.0260). This suggests the bias is intrinsic to how these models learn patterns from medical data, rather than simply reflecting dataset composition.

The implications for real-world healthcare are substantial. If deployed without addressing these biases, automated diagnostic systems could lead to delayed cancer detection and suboptimal treatment planning for younger patients. Given that breast cancer in younger women often presents differently and can be more aggressive, these performance gaps could have serious consequences for patient outcomes.

The study also revealed that simply looking at overall performance metrics can mask significant disparities. When researchers aggregated results across all demographic groups, some fairness metrics appeared acceptable. However, when they disaggregated the analysis by age and ethnicity subgroups, clear patterns of bias emerged. This "masking effect" means that superficial evaluations of AI systems might miss critical fairness issues.

While the research provides clear evidence of age-related bias, several limitations remain. The study doesn't identify the specific mechanisms causing these disparities—whether they stem from physiological differences in breast tissue, variations in image quality, or other factors. Additionally, the analysis focused primarily on segmentation performance rather than downstream diagnostic accuracy. Future research will need to investigate whether these segmentation biases translate to actual differences in cancer detection rates and patient outcomes.

The findings underscore the importance of rigorous fairness auditing before deploying AI systems in clinical settings. As healthcare increasingly relies on automated tools, ensuring these technologies serve all patients equitably becomes not just a technical challenge but an ethical imperative.