AI Generates Medical Reports from Cancer Scans

A new artificial intelligence system can automatically generate detailed medical reports from cancer scans, potentially easing the burden on radiologists while maintaining clinical accuracy. The PETAR-4B model developed by researchers at University of Wisconsin-Madison and Microsoft creates localized findings for positron emission tomography/computed tomography (PET/CT) scans, which are crucial for cancer diagnosis and treatment monitoring but produce some of the longest and most complex radiology reports.

The key breakthrough lies in the model's ability to connect specific regions of interest in medical images with descriptive text. Unlike previous systems that treated scans as whole images, PETAR-4B uses segmentation masks to focus on individual lesions and abnormalities, then generates precise descriptions of what it observes. This mask-aware approach allows the AI to provide localized findings rather than general summaries, matching how radiologists actually analyze and report on scans.

Researchers built the system using a novel four-stage training process. They first created PETAR-11K, the largest dataset of its kind containing 11,356 PET/CT exams paired with lesion-level descriptions extracted from radiology reports. The training began with pretraining on anatomical region classification, followed by embedding alignment to connect visual features with text, projector alignment to map between modalities, and finally full fine-tuning for generating lesion-specific findings. The model processes both PET metabolic data and CT structural information, using focal prompts to maintain high resolution for small lesions that typically occupy less than 1% of the scan volume.

Evaluation results demonstrate substantial improvements over existing methods. PETAR-4B achieved a 56% improvement in METEOR score, 46% improvement in ROUGE-L, and 29% improvement in BLEU-4 compared to the best-performing baseline model. More importantly, it showed strong performance on clinical relevance metrics, with RaTEScore improving from 0.6271 to 0.7084 and GREEN score increasing from 0.0705 to 0.2260. Human evaluation by five nuclear medicine physicians confirmed the model's ability to produce clinically coherent reports with accurate localization and interpretation.

This technology matters because PET/CT scans play a critical role in oncology for diagnosis, staging, and treatment response assessment, yet they generate reports that are three times longer than typical radiology reports. The growing volume of these complex scans contributes to radiologist burnout, while the dispersed nature of lesions across whole-body scans makes manual interpretation particularly challenging. An automated system that can accurately describe localized findings could help streamline workflow while maintaining the detailed analysis needed for cancer care.

The research acknowledges several limitations. The model currently focuses on PET/CT scans and may not generalize to other imaging modalities without additional training. It also relies on existing segmentation tools to identify regions of interest, though the authors suggest it could work with FDA-cleared single-click segmentation tools in clinical practice. Future work will explore converting the generated findings into templated reports and integrating the system into clinical workflow tools.