This paper presents OnGoal, an interface that effectively assesses and manages users' goal achievement in long-term conversations with a large-scale language model (LLM). OnGoal facilitates the effective navigation of complex conversations by providing real-time goal congruence feedback via LLM-based assessments, explanations with examples of assessment results, and an overview of goal progress over time. In a writing task study involving 20 participants, we compared OnGoal with a basic chat interface without goal tracking. We found that participants using OnGoal reduced the time and effort required to achieve their goals and explored novel prompting strategies for error resolution. This suggests that goal tracking and visualization can enhance engagement and resilience in LLM conversations. Our findings provide guidance for future LLM chat interface designs that enable feedback to improve LLM performance and enhance goal communication, cognitive load reduction, and interactivity.
