AI Assistants Struggle to Remember Who You Are

As artificial intelligence assistants become more integrated into daily life, their ability to remember and adapt to individual users is crucial. However, current large language models (LLMs) often fall short in personalization, a gap that has been difficult to measure due to a lack of realistic benchmarks. Researchers from the University of Science and Technology of China and the National University of Singapore have introduced AlpsBench, a new evaluation tool built from real human-AI dialogues, to systematically test how well AI can manage personalized information. This benchmark uncovers significant weaknesses in existing models, suggesting that the dream of a truly personalized AI assistant remains out of reach for now.

The key finding from AlpsBench is that AI models struggle across multiple aspects of personalization. In tests on frontier LLMs like GPT-5.2, Gemini-3 Flash, and DeepSeek Reasoner, the researchers discovered that models have limited reliability in extracting latent user traits from conversations. For example, in Task 1 (personalized information extraction), the best-performing model, Gemini-3 Flash, achieved only a 51.67 score, while others like Llama-4 Maverick scored as low as 22.07. This indicates that even top models cannot consistently identify and structure personal details from dialogue, a fundamental requirement for personalized assistance.

Ology behind AlpsBench involves a four-step pipeline based on real-world data. First, the team collected 2,500 long-term interaction sequences from the WildChat dataset, which contains authentic human-LLM dialogues with 6 to 249 turns each. Next, they used an LLM to extract structured memories—such as user preferences and traits—from these conversations, followed by human annotation to verify accuracy. Finally, they constructed four evaluation tasks: extraction, updating, retrieval, and utilization of personalized information. This approach ensures the benchmark reflects real conversational diversity and implicit signals, unlike previous benchmarks that relied on synthetic data and lacked natural complexity.

From the experiments reveal specific performance ceilings and s. In Task 2 (personalized information updating), GPT-5.2 scored 81.49, the highest among models, but this still falls short of ideal reliability. The study also found that memory updating performance on new-memory addition and conflict-memory modification shows a weak association, meaning models are inconsistent in how they handle different types of updates. For Task 3 (personalized information retrieval), accuracy declines sharply as distractor memories increase; with 1000 distractors, some models like GPT-4.1-mini dropped to 0.7295 recall, highlighting sensitivity to noise. Additionally, memory-oriented systems, such as A-Mem and EverMemOS, sometimes underperform their backbone models in extraction recall, possibly due to storage-policy biases that prioritize broad dialogue content over relevant personalization.

Of these are significant for the development of personalized AI assistants. AlpsBench shows that no single model excels across all dimensions of utilization, such as persona awareness, preference following, and emotional intelligence. For instance, in Task 4, Gemini-3 Flash scored 0.6895 on persona awareness, indicating room for improvement. Memory systems can enhance certain capabilities but introduce "personalization bias," often at the expense of emotional intelligence or virtual-reality awareness. This suggests that current designs focus too much on memory capacity rather than effective usage, which could hinder real-world deployment where emotional resonance and accurate filtering are key.

Limitations of the study, as noted in the paper, include the computational cost of running AI assistants and the expense of high-quality manual annotation, which constrained data filtering. The benchmark also prioritizes users with implicit memories to enhance , potentially skewing the dataset. Future work will involve continuously updating AlpsBench with evolving real-world conversations to maintain its relevance. These constraints underscore the need for more efficient retrieval architectures and logic-aware layers to improve personalization without prohibitive costs, pointing toward ongoing research opportunities in this critical area of AI development.