Existing studies on bias mitigation methods for large language models (LLMs)
use diverse baselines and metrics to evaluate debiasing performance, leading to
inconsistent comparisons among them. Moreover, their evaluations are mostly
based on the comparison between LLMs’ probabilities of biased and unbiased
contexts, which ignores the gap between such evaluations and real-world use
cases where users interact with LLMs by reading model responses and expect fair
and safe outputs rather than LLMs’ probabilities. To enable consistent
evaluation across debiasing methods and bridge this gap, we introduce
BiasFreeBench, an empirical benchmark that comprehensively compares eight
mainstream bias mitigation techniques (covering four prompting-based and four
training-based methods) on two test scenarios (multi-choice QA and open-ended
multi-turn QA) by reorganizing existing datasets into a unified query-response
setting. We further introduce a response-level metric, Bias-Free Score, to
measure the extent to which LLM responses are fair, safe, and
anti-stereotypical. Debiasing performances are systematically compared and
analyzed across key dimensions: the prompting vs. training paradigm, model
size, and generalization of different training strategies to unseen bias types.
We will publicly release our benchmark, aiming to establish a unified testbed
for bias mitigation research.