Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

Evaluating AI Counseling in Japanese: Counselor, Client, and Evaluator Roles Assessed by Motivational Interviewing Criteria

Created by
  • Haebom

Author

Keita Kiuchi, Yoshikazu Fujimoto, Hideyuki Goto, Tomonori Hosokawa, Makoto Nishimura, Yosuke Sato, Izumi Sezai

Outline

This study is the first to comprehensively evaluate the performance of large-scale language models (LLMs) across three counseling roles in a Japanese therapy setting. We simultaneously evaluated counselor AI systems (GPT-4-turbo, Claude-3-Opus-SMDP using zero-shot prompting or structured multi-step conversation prompting (SMDP), client AI simulations, and evaluator AI systems (o3, Claude-3.7-Sonnet, Gemini-2.5-pro). Experienced human experts (n=15) evaluated the AI-generated conversations using the Motivational Interviewing Treatment Integrity (MITI) Coding Manual 4.2.1. Implementation of SMDP significantly improved the performance of the counselor AI on all MITI global assessments compared to zero-shot prompting, with no significant differences between GPT-SMDP and Opus-SMDP. The evaluator AI performed similarly to human raters in facilitating change conversations, but systematically overestimated maintenance conversation de-escalation and overall quality metrics. Gemini showed model-specific biases, such as prioritizing power sharing, o3 prioritizing technical proficiency, and Sonnet prioritizing emotional expression. The client AI simulations showed a limited emotional range and unusually high compliance, suggesting the need for improved realism. These results set a benchmark for non-English AI-assisted counseling and suggest important areas for improvement through advanced prompt engineering, augmented search generation, and goal-oriented fine-tuning, with important implications for the development of culturally sensitive AI mental health tools.

Takeaways, Limitations

•
Takeaways:
◦
To provide the first comprehensive evaluation of the LLM's counseling role performance in a Japanese language therapeutic setting.
◦
SMDP prompting technique is proven to be effective in improving the performance of counseling AI.
◦
Presenting the possibility of utilizing AI evaluation systems and their limitations (tendency to overestimate).
◦
Suggest areas for improvement, including model-specific bias and lack of realism in client AI simulations.
◦
Presenting important Takeaways for the development of culturally sensitive AI mental health tools.
•
Limitations:
◦
Limited emotional range and unrealistically high compliance of client AI simulations.
◦
Inconsistent evaluation results from the evaluation AI (especially, relaxation of maintenance dialogue and overestimation of overall quality).
◦
Limited sample size (15 human experts).
◦
The need to more comprehensively consider different types of counseling and cultural backgrounds.
👍
Made with Slashpage