Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

Sleepless Nights, Sugary Days: Creating Synthetic Users with Health Conditions for Realistic Coaching Agent Interactions

Created by
  • Haebom

Author

Taedong Yun, Eric Yang, Mustafa Safdari, Jong Ha Lee, Vaishnavi Vinod Kumar, S. Sara Mahdavi, Jonathan Amar, Derek Peyton, Reut Aharony, Andreas Michaelides, Logan Schneider, Isaac Galatzer-Levy, Yugang Jia, John Canny, Arthur Gretton, Maja Matari c

Outline

This paper presents an end-to-end framework for generating synthetic users to evaluate conversational agents designed to encourage positive behavior change, such as health and lifestyle coaching. Focusing specifically on sleep and diabetes management, synthetic users are generated based on health and lifestyle factors to ensure realistic interactions. First, structured data is generated based on real-world health and lifestyle factors, in addition to basic demographic and behavioral characteristics. Second, a full profile of the synthetic user is developed based on the generated structured data. Interactions between the synthetic user and the coaching agent are simulated using generative agent-based models, such as Concordia, or by prompting a language model. Using two independently developed agents for sleep and diabetes coaching as case studies, we demonstrate the effectiveness of this framework by analyzing the coaching agent's understanding of the synthetic user's needs and challenges. Finally, through multiple blind evaluations of user-coach interactions by human experts, we demonstrate that synthetic users with health and behavioral characteristics more accurately represent real-world users with those characteristics. The proposed framework establishes a foundation for the efficient development of conversational agents through extensive, realistic, and robust simulations of interactions.

Takeaways, Limitations

Takeaways:
Providing an efficient and realistic synthetic user-generated framework for evaluating health and lifestyle coaching agents.
Improving the accuracy of agent performance evaluation through synthetic users with characteristics similar to real users.
Presenting the possibility of simulating various interactions using generative agent-based models and language models.
Contributes to increasing efficiency and reducing development costs in the conversational agent development process.
Limitations:
As this is a case study limited to sleep and diabetes management, generalizability to other areas needs to be verified.
The accuracy of the results may be affected by the quality and quantity of data used to generate synthetic users.
The generalizability of the results may be limited by the size and composition of the blind evaluation participants.
There is a possibility that the behavioral patterns of synthetic users may not perfectly reflect the complexity of real users.
👍