PersonaDrift: A New Benchmark for Tracking Subtle Language Changes in Dementia Care

In the nuanced world of dementia care, caregivers often notice gradual shifts in communication long before clinical diagnoses are confirmed. People living with dementia (PLwD) may become less expressive, more repetitive, or drift off-topic in subtle ways that signal cognitive decline. While these changes are informally observed, computational tools have struggled to provide structured, long-term tracking of such behavioral drift. A new research initiative from the University of Toronto introduces PersonaDrift, a synthetic benchmark designed specifically to evaluate machine learning and statistical s for detecting progressive changes in daily communication patterns. This framework represents a significant step toward developing privacy-aware monitoring tools that could help caregivers detect early signs of change and provide timely support, bridging the gap between informal observation and computational analysis.

The PersonaDrift benchmark simulates 60-day interaction logs between digital reminder systems and synthetic users modeled after real PLwD, based on extensive interviews with caregivers. These caregiver-informed personas vary in tone, modality, and communication habits, enabling realistic diversity in behavior that mirrors actual user patterns. The researchers focused on two forms of longitudinal change that caregivers identified as particularly salient: flattened sentiment (reduced emotional tone and verbosity) and off-topic replies (semantic drift). These anomalies are injected progressively at different rates—slow (15–20 days), medium (10–14 days), and fast (6–8 days)—to emulate naturalistic cognitive trajectories. The simulation pipeline includes routine schedule generation, random event injection, response generation using large language models conditioned on persona attributes, and controlled anomaly injection with severity scaling over time, creating a structured environment for ical evaluation.

From evaluating multiple detection approaches reveal distinct patterns in what types of changes are detectable with different s. For flattened sentiment detection, simple statistical s like Cumulative Sum Control Chart (CUSUM) consistently delivered strong performance across all progression speeds, with personas achieving F1 scores above 0.98 and negligible detection delay, particularly for users with stable communication patterns. In contrast, off-topic reply detection proved more challenging, requiring more sophisticated approaches. While GRU-based sequence models using BERT embeddings achieved high ROC AUC values (typically >0.95), indicating strong ranking ability, their F1 scores remained moderate (generally 0.4–0.7), reflecting difficulty in defining clear anomaly thresholds. The most striking finding emerged from comparing personalized versus generalized classifiers: personalized models trained on individual users' data achieved near-perfect performance (F1 and ROC AUC >0.95), while generalized classifiers showed clear weaknesses, with F1 scores dipping below 0.9 and more substantial drops for expressive users.

Of these are substantial for both technology development and clinical practice. The strong performance of personalized models underscores that semantic drift is highly individualized and that generic thresholds cannot reliably detect meaningful change in cognitively sensitive interactions. This suggests that future NLP systems for dementia monitoring must incorporate user-specific baselines as a core requirement rather than an optional enhancement. Additionally, the benchmark reveals how performance varies with interaction modality and expressiveness—terse, typed responses led to clearer drift signals, while voice-based or emotionally dynamic personas exhibited higher baseline variance, resulting in more false positives and delayed detection. These insights provide crucial guidance for developers creating adaptive systems that must account for diverse communication styles and natural variability in real-world deployment scenarios.

Despite its contributions, PersonaDrift has several limitations that point to important directions for future research. The benchmark currently includes only two anomaly types (affective flattening and semantic drift), while other common features of cognitive decline such as repetition, lexical retrieval issues, and syntactic simplification are not yet represented. The simulation is text-based, excluding important multimodal cues like vocal tone and speech timing that could improve ecological validity. Additionally, the personas represent relatively stable traits rather than the dynamic, evolving communication patterns often observed in PLwD. Future work could expand the benchmark to include additional anomaly types, incorporate multimodal signals, simulate evolving user traits over longer timelines, and test more adaptive modeling approaches like memory-augmented networks and few-shot personalization techniques. As the researchers note, while PersonaDrift provides a controlled testbed, the issues uncovered here are likely to be even more severe in real deployments, making this benchmark-first approach essential for identifying failure modes early in development.