As multi-turn dialogues with large language models (LLMs) grow longer and
more complex, how can users better evaluate and review progress on their
conversational goals? We present OnGoal, an LLM chat interface that helps users
better manage goal progress. OnGoal provides real-time feedback on goal
alignment through LLM-assisted evaluation, explanations for evaluation results
with examples, and overviews of goal progression over time, enabling users to
navigate complex dialogues more effectively. Through a study with 20
participants on a writing task, we evaluate OnGoal against a baseline chat
interface without goal tracking. Using OnGoal, participants spent less time and
effort to achieve their goals while exploring new prompting strategies to
overcome miscommunication, suggesting tracking and visualizing goals can
enhance engagement and resilience in LLM dialogues. Our findings inspired
design implications for future LLM chat interfaces that improve goal
communication, reduce cognitive load, enhance interactivity, and enable
feedback to improve LLM performance.