Justin Shao

EECS Department, University of California, Berkeley

Technical Report No. UCB/EECS-2024-114

May 16, 2024

This publication is archived. It is kept only for reference purposes, so it is no longer being updated and may not meet accessibility standards. If you need this content in a different format, please email webteam@eecs.berkeley.edu.

http://www2.eecs.berkeley.edu/Pubs/TechRpts/2024/Archive/EECS-2024-114.pdf

Multiple Choice Questions (MCQs) are a prevalent evaluation method used across many popular LLM benchmarks. Typically, these evaluations rely on first-token probabilities to deduce the model’s proposed answer. However, previous studies have demonstrated that first-token probabilities are vulnerable to prompt perturbations. In this study, we broaden our examination to explore the performance disparity between direct free-generation and the assessment of MCQs using first-token probabilities. Our experiments confirm the unreliability of first-token probabilities, as they often do not align with generation results. Additionally, we uncover a surprising finding: LLMs tend to struggle with arithmetic MCQs, even though they can reliably generate the correct answers.

Advisors: Jiantao Jiao


BibTeX citation:

@mastersthesis{Shao:EECS-2024-114,
    Author= {Shao, Justin},
    Title= {First Token Probabilities are Unreliable Indicators for LLM Knowledge},
    School= {EECS Department, University of California, Berkeley},
    Year= {2024},
    Month= {May},
    Url= {http://www2.eecs.berkeley.edu/Pubs/TechRpts/2024/EECS-2024-114.html},
    Number= {UCB/EECS-2024-114},
    Abstract= {Multiple Choice Questions (MCQs) are a prevalent evaluation method used across many popular LLM benchmarks. Typically, these evaluations rely on first-token probabilities to deduce the model’s proposed answer. However, previous studies have demonstrated that first-token probabilities are vulnerable to prompt perturbations. In this study, we broaden our examination to explore the performance disparity between direct free-generation and the assessment of MCQs using first-token probabilities. Our experiments confirm the unreliability of first-token probabilities, as they often do not align with generation results. Additionally, we uncover a surprising finding: LLMs tend to struggle with arithmetic MCQs, even though they can reliably generate the correct answers.},
}

EndNote citation:

%0 Thesis
%A Shao, Justin 
%T First Token Probabilities are Unreliable Indicators for LLM Knowledge
%I EECS Department, University of California, Berkeley
%D 2024
%8 May 16
%@ UCB/EECS-2024-114
%U http://www2.eecs.berkeley.edu/Pubs/TechRpts/2024/EECS-2024-114.html
%F Shao:EECS-2024-114