Mind the Gap: A Closer Look at Tokenization for Multiple-Choice Question
  Answering with LLMs
When evaluating large language models (LLMs) with multiple-choice question
answering (MCQA), it is common to end the prompt with the string "Answer:" to
facilitate automated answer extraction via next-token probabilities. However,
there is no consensus on how to tokenize the space following the colon, often
overlooked as a trivial choice. In this paper, we uncover accuracy differences
of up to 11% due to this (seemingly irrelevant) tokenization variation as well
as reshuffled model rankings, raising concerns about the reliability of LLM
comparisons in prior work. Surprisingly, we are able to recommend one specific
strategy -- tokenizing the space together with the answer letter -- as we
observe consistent and statistically significant performance improvements.
Additionally, it improves model calibration, enhancing the reliability of the
model's confidence estimates. Our findings underscore the importance of careful
evaluation design and highlight the need for standardized, transparent
evaluation protocols to ensure reliable and comparable results.
68cd2c4b3df9ac65e93dc787
Mario Sanz-Guerrero
claimed_verified
2025-09-19T10:24:53.862Z
64d4b251071f4a335c97264e
/avatars/c14ae3eea034c1b39186a194425e9989.svg
Mario Sanz
user
mario-sanz
68cd2c4b3df9ac65e93dc788
Minh Duc Bui
admin_assigned
2025-09-19T10:28:19.439Z
65e0e6fa4394fc3d1b59627a
https://cdn-avatars.huggingface.co/v1/production/uploads/65e0e6fa4394fc3d1b59627a/rlKw_UdH3MpmpgUcchMgB.jpeg
Minh Duc Bui
user
MinhDucBui
68cd2c4b3df9ac65e93dc789
Katharina von der Wense
Mind the Gap: A Closer Look at Tokenization for Multiple-Choice Question
  Answering with LLMs
When evaluating large language models (LLMs) with multiple-choice question
answering (MCQA), it is common to end the prompt with the string "Answer:" to
facilitate automated answer extraction via next-token probabilities. However,
there is no consensus on how to tokenize the space following the colon, often
overlooked as a trivial choice. In this paper, we uncover accuracy differences
of up to 11% due to this (seemingly irrelevant) tokenization variation as well
as reshuffled model rankings, raising concerns about the reliability of LLM
comparisons in prior work. Surprisingly, we are able to recommend one specific
strategy -- tokenizing the space together with the answer letter -- as we
observe consistent and statistically significant performance improvements.
Additionally, it improves model calibration, enhancing the reliability of the
model's confidence estimates. Our findings underscore the importance of careful
evaluation design and highlight the need for standardized, transparent
evaluation protocols to ensure reliable and comparable results.
Tokenizing the space with the answer letter in multiple-choice question answering improves LLM accuracy and calibration.
large language models multiple-choice question answering tokenization next-token probabilities model calibration evaluation protocols