The Curious Case of Factual (Mis)Alignment between LLMs' Short- and
  Long-Form Answers
Large language models (LLMs) can correctly answer "When was Einstein born?"
yet fail to provide the same date when writing about Einstein's life revealing
a fundamental inconsistency in how models access factual knowledge across task
complexities. While models display impressive accuracy on factual
question-answering benchmarks, the reliability gap between simple and complex
queries remains poorly understood, eroding their trustworthiness. In this work,
we introduce Short-Long Form Alignment for Factual Question Answering (SLAQ), a
controlled evaluation framework that compares LLMs' answers to the same factual
questions asked (a) in isolation (short) vs. (b) integrated into complex
queries (long). Looking at 16 LLMs across 600 queries, we find a systematic
misalignment of answers to the corresponding short and long queries. We further
uncover position-dependent accuracy loss and momentum effects where consecutive
correct or incorrect answers create self-reinforcing patterns. Through
mechanistic analysis, we find that aligned facts activate overlapping model
internals, and that metrics based on mechanistic similarity can predict
short-long answer alignment with up to 78% accuracy. Our work establishes
factual consistency over query complexity as an important aspect of LLMs'
trustworthiness and challenges current evaluation practices, which implicitly
assume that good performance for simple factual queries implies reliability in
more complex knowledge-seeking tasks too.
68ee46459b77b5223f6661dd
Saad Obaid ul Islam
claimed_verified
2025-10-14T14:24:06.666Z
6182910a68444be3259d8b67
/avatars/b47b0609f61b1192a1337fd7c9f8a75b.svg
Saad Obaid ul Islam
user
saadob12
68ee46459b77b5223f6661de
Anne Lauscher
68ee46459b77b5223f6661df
Goran Glavaš
The Curious Case of Factual (Mis)Alignment between LLMs' Short- and
  Long-Form Answers
Large language models (LLMs) can correctly answer "When was Einstein born?"
yet fail to provide the same date when writing about Einstein's life revealing
a fundamental inconsistency in how models access factual knowledge across task
complexities. While models display impressive accuracy on factual
question-answering benchmarks, the reliability gap between simple and complex
queries remains poorly understood, eroding their trustworthiness. In this work,
we introduce Short-Long Form Alignment for Factual Question Answering (SLAQ), a
controlled evaluation framework that compares LLMs' answers to the same factual
questions asked (a) in isolation (short) vs. (b) integrated into complex
queries (long). Looking at 16 LLMs across 600 queries, we find a systematic
misalignment of answers to the corresponding short and long queries. We further
uncover position-dependent accuracy loss and momentum effects where consecutive
correct or incorrect answers create self-reinforcing patterns. Through
mechanistic analysis, we find that aligned facts activate overlapping model
internals, and that metrics based on mechanistic similarity can predict
short-long answer alignment with up to 78% accuracy. Our work establishes
factual consistency over query complexity as an important aspect of LLMs'
trustworthiness and challenges current evaluation practices, which implicitly
assume that good performance for simple factual queries implies reliability in
more complex knowledge-seeking tasks too.
LLMs exhibit inconsistent factual knowledge retrieval between simple and complex queries, highlighting a reliability gap that undermines trustworthiness.
Large language models factual question-answering controlled evaluation framework short-long form alignment position-dependent accuracy momentum effects mechanistic analysis factual consistency query complexity trustworthiness