Patrick Wu and Jiayi Pan and Xinyan Hu and Wei-Lin Chiang and Anastasios Angelopoulos and Trevor Darrell and Narges Norouzi and Joseph Gonzalez

EECS Department, University of California, Berkeley

Technical Report No. UCB/EECS-2025-177

October 22, 2025

http://www2.eecs.berkeley.edu/Pubs/TechRpts/2025/EECS-2025-177.pdf

Search-augmented language models combine web search with Large Language Models (LLMs) to improve response groundedness and freshness. However, analyzing these systems remains challenging: existing datasets are limited in scale and narrow in scope, often constrained to static, single-turn, fact-checking questions. In this work, we introduce Search Arena, a crowd-sourced, large-scale, human-preference dataset of over 24,000 paired multi-turn user interactions with search-augmented LLMs. The dataset spans diverse intents and languages, and contains full system traces with around 12,000 human preference votes. Our analysis reveals that user preferences are influenced by the number of citations, even when the cited content does not directly support the attributed claims, uncovering a gap between perceived and actual credibility. Furthermore, user preferences vary across cited sources, revealing that community-driven platforms are generally preferred and static encyclopedic sources are not always appropriate and reliable. To assess performance across different settings, we conduct cross-arena analyses by testing search-augmented LLMs in a general-purpose chat environment and conventional LLMs in search-intensive settings. We find that web search does not degrade and may even improve performance in non-search settings; however, the quality in search settings is significantly affected if solely relying on the model's parametric knowledge. We open-sourced the dataset to support future research in this direction.

Advisors: Trevor Darrell and Joseph Gonzalez

---

BibTeX citation:

@mastersthesis{Wu:EECS-2025-177,
    Author= {Wu, Patrick and Pan, Jiayi and Hu, Xinyan and Chiang, Wei-Lin and Angelopoulos, Anastasios and Darrell, Trevor and Norouzi, Narges and Gonzalez, Joseph},
    Title= {Search Arena: Analyzing Search-Augmented Large Language Models},
    School= {EECS Department, University of California, Berkeley},
    Year= {2025},
    Month= {Oct},
    Url= {http://www2.eecs.berkeley.edu/Pubs/TechRpts/2025/EECS-2025-177.html},
    Number= {UCB/EECS-2025-177},
    Note= {The work has been co-lead with Tsung-Han Wu and advised by Trevor Darrell, Narges Norouzi, and Joseph Gonzalez.
Project contributors: Logan King, Tianle Li, Jiayi Pan, Xinyan Hu, Wei-Lin Chiang, Anastasios N. Angelopoulos.
Preprint: https://arxiv.org/abs/2506.05334},
    Abstract= {Search-augmented language models combine web search with Large Language Models (LLMs) to improve response groundedness and freshness. However, analyzing these systems remains challenging: existing datasets are limited in scale and narrow in scope, often constrained to static, single-turn, fact-checking questions. In this work, we introduce **Search Arena**, a crowd-sourced, large-scale, human-preference dataset of over 24,000 paired multi-turn user interactions with search-augmented LLMs. The dataset spans diverse intents and languages, and contains full system traces with around 12,000 human preference votes. Our analysis reveals that user preferences are influenced by the number of citations, even when the cited content does not directly support the attributed claims, uncovering a gap between perceived and actual credibility. Furthermore, user preferences vary across cited sources, revealing that community-driven platforms are generally preferred and static encyclopedic sources are not always appropriate and reliable. To assess performance across different settings, we conduct cross-arena analyses by testing search-augmented LLMs in a general-purpose chat environment and conventional LLMs in search-intensive settings. We find that web search does not degrade and may even improve performance in non-search settings; however, the quality in search settings is significantly affected if solely relying on the model's parametric knowledge. We open-sourced the dataset to support future research in this direction.},
}

EndNote citation:

%0 Thesis
%A Wu, Patrick 
%A Pan, Jiayi 
%A Hu, Xinyan 
%A Chiang, Wei-Lin 
%A Angelopoulos, Anastasios 
%A Darrell, Trevor 
%A Norouzi, Narges 
%A Gonzalez, Joseph 
%T Search Arena: Analyzing Search-Augmented Large Language Models
%I EECS Department, University of California, Berkeley
%D 2025
%8 October 22
%@ UCB/EECS-2025-177
%U http://www2.eecs.berkeley.edu/Pubs/TechRpts/2025/EECS-2025-177.html
%F Wu:EECS-2025-177