An efficient multi-turn dialogue evaluator aggregates multiple LLM judgments into a single model to assess dialogue quality with reduced computational cost.
Evaluating the conversational abilities of large language models (LLMs)
remains a challenging task. Current mainstream approaches primarily rely on the
“LLM-as-a-judge” paradigm, where an LLM is prompted to serve as an evaluator
to assess dialogue quality. However, such methods often suffer from various
biases, which undermine the reliability and consistency of the evaluation
results. To mitigate these biases, recent methods employ multiple LLMs as
judges and aggregate their judgments to select the optimal assessment. Although
effective, this multi-judge approach incurs significant computational overhead
during inference. In this paper, we propose an efficient multi-turn dialogue
evaluator that captures the collective wisdom of multiple LLM judges by
aggregating their preference knowledge into a single model. Our approach
preserves the advantages of diverse multi-judge feedback while drastically
reducing the evaluation cost, enabling fast and flexible dialogue quality
assessment. Extensive experiments on seven single rating and pairwise
comparison dialogue evaluation benchmarks demonstrate that our method
outperforms existing baselines across diverse scenarios, showcasing its
efficiency and robustness.