Search-augmented large language models (LLMs) have advanced
information-seeking tasks by integrating retrieval into generation, reducing
users’ cognitive burden compared to traditional search systems. Yet they remain
insufficient for fully addressing diverse user needs, which requires
recognizing how the same query can reflect different intents across users and
delivering information in preferred forms. While recent systems such as ChatGPT
and Gemini attempt personalization by leveraging user histories, systematic
evaluation of such personalization is under-explored. To address this gap, we
propose BESPOKE, the realistic benchmark for evaluating personalization in
search-augmented LLMs. BESPOKE is designed to be both realistic, by collecting
authentic chat and search histories directly from humans, and diagnostic, by
pairing responses with fine-grained preference scores and feedback. The
benchmark is constructed through long-term, deeply engaged human annotation,
where human annotators contributed their own histories, authored queries with
detailed information needs, and evaluated responses with scores and diagnostic
feedback. Leveraging BESPOKE, we conduct systematic analyses that reveal key
requirements for effective personalization in information-seeking tasks,
providing a foundation for fine-grained evaluation of personalized
search-augmented LLMs. Our code and data are available at
https://augustinlib.github.io/BESPOKE/.