AI Adapts to Diagnose New Diseases Quickly

As the COVID-19 pandemic surged, people increasingly turned to online symptom checkers for quick health assessments, but these tools often failed to consider multiple diseases at once. This limitation meant that while a user might be checked for COVID-19, other potential illnesses could be overlooked, highlighting a critical gap in digital healthcare. A new AI approach addresses this by enabling diagnostic models to rapidly incorporate new diseases, such as COVID-19, while maintaining their ability to accurately identify a wide range of existing conditions.

The researchers developed a machine learning model that integrates COVID-19 into differential diagnosis—the process of considering multiple possible diseases based on symptoms. By training the model with a combination of synthetic data from an expert system and real-world data from a COVID-19 symptom checker, it learned to include COVID-19 in its assessments without sacrificing performance on other diagnoses. For instance, in cases where symptoms overlapped with respiratory infections, the model correctly placed COVID-19 among the top possible diseases, as shown in qualitative examples from the paper.

To build this model, the team used two main datasets: one simulated from an expert system with 830 diseases and over 2,000 findings, and another from a publicly deployed COVID-19 assessment tool that collected user symptoms and risk factors. They combined these to train a deep learning model that processes demographic and symptom inputs separately, using embeddings and pooling techniques to handle uncertainties in diagnosis. The model was optimized with a loss function that prioritizes considering all plausible diseases, even with partial information, ensuring it errs on the side of caution in medical scenarios.

The results demonstrate the model's effectiveness. On the Semigran dataset, which includes 45 standardized patient cases, the base model achieved a top-3 accuracy of 85.8%, close to human doctors at 84.3%. When enhanced with COVID-19 data, it maintained high accuracy, with the COVID-inclusive version scoring 84.4% for top-3 recall. In COVID-19-specific tests, the model identified COVID-19 in 73% of cases within the top five predictions, and when using all available symptoms, it reached 100% for top-3 and top-5 accuracy. This shows that the approach can accurately model new diseases without degrading performance on others, as detailed in the paper's tables.

This advancement matters because it makes online symptom checkers more reliable and comprehensive, especially during health crises. For everyday users, it means getting a holistic view of potential health issues—like distinguishing between COVID-19 and strep throat—based on symptoms alone. By leveraging prior medical knowledge through synthetic data, the method works even when real data is scarce, offering a scalable solution for future outbreaks. It bridges the gap between rigid rule-based systems and data-driven models, providing a flexible tool that can adapt as new medical evidence emerges.

However, the study has limitations. The model's performance dropped in cases where symptoms did not sufficiently overlap with those in the training data, indicating that it relies on shared findings to include new diseases effectively. Additionally, the evaluation used a limited set of COVID-19 cases, and the approach may need further testing with diverse populations to ensure generalizability. The paper notes that without all specific symptoms, such as unique exposure risks, the model might not always prioritize COVID-19, suggesting areas for improvement in handling novel disease characteristics.