This paper highlights the importance of personalization in the context of personalized AI assistants, particularly private AI models that leverage private user data. We focus on evaluating the ability of AI models to access and interpret users' private data (e.g., conversation history, user-AI interactions, app usage history) to understand users' personal information (e.g., biographical information, preferences, social relationships, etc.). Recognizing the limited availability of publicly available datasets due to the sensitive nature of these data, we present a synthetic data generation pipeline that generates private documents that simulate diverse and realistic user profiles and personal activities. Building on this, we propose a benchmark, PersonaBench, to evaluate the performance of AI models that understand private information extracted from simulated private user data. Using a Retrieval-Augmented Generation (RAG) pipeline, we evaluate the performance of AI models that understand private information extracted from simulated private user data. Our results reveal that current RAG-based AI models struggle to extract personal information from user documents and answer private questions, highlighting the need for improved methodologies to enhance AI's personalization capabilities.
