KITAB: Evaluating LLMs on Constraint Satisfaction for Information
  Retrieval
We study the ability of state-of-the art models to answer constraint
satisfaction queries for information retrieval (e.g., 'a list of ice cream
shops in San Diego'). In the past, such queries were considered to be tasks
that could only be solved via web-search or knowledge bases. More recently,
large language models (LLMs) have demonstrated initial emergent abilities in
this task. However, many current retrieval benchmarks are either saturated or
do not measure constraint satisfaction. Motivated by rising concerns around
factual incorrectness and hallucinations of LLMs, we present KITAB, a new
dataset for measuring constraint satisfaction abilities of language models.
KITAB consists of book-related data across more than 600 authors and 13,000
queries, and also offers an associated dynamic data collection and constraint
verification approach for acquiring similar test data for other authors. Our
extended experiments on GPT4 and GPT3.5 characterize and decouple common
failure modes across dimensions such as information popularity, constraint
types, and context availability. Results show that in the absence of context,
models exhibit severe limitations as measured by irrelevant information,
factual errors, and incompleteness, many of which exacerbate as information
popularity decreases. While context availability mitigates irrelevant
information, it is not helpful for satisfying constraints, identifying
fundamental barriers to constraint satisfaction. We open source our
contributions to foster further research on improving constraint satisfaction
abilities of future models.
65386075a5ab055cf2d101fc
Marah I Abdin
claimed_verified
2023-10-26T07:33:42.440Z
649af1e3ca20306aeef590b8
https://cdn-avatars.huggingface.co/v1/production/uploads/649af1e3ca20306aeef590b8/Tvy8yf7zDO2FoNs03tWj6.jpeg
Marah Abdin
user
marah-abdin
65386075a5ab055cf2d101fd
Suriya Gunasekar
admin_assigned
2023-10-26T11:48:43.000Z
63e1b4f77fbb6ae4d4f36aa4
/avatars/7febc332d393e0bb94255104527ffe2a.svg
Gunasekar
user
suriyagunasekar
65386075a5ab055cf2d101fe
Varun Chandrasekaran
65386075a5ab055cf2d101ff
Jerry Li
65386075a5ab055cf2d10200
Mert Yuksekgonul
65386075a5ab055cf2d10201
Rahee Ghosh Peshawaria
admin_assigned
2023-10-26T11:49:42.221Z
6529c0f3ff2202020e7851d0
/avatars/05fa3c902862b22c7bf506a664736dd7.svg
Rahee Peshawaria
user
raghosh
65386075a5ab055cf2d10202
Ranjita Naik
admin_assigned
2023-10-26T11:49:49.303Z
63d8512207cd1aa3c4a1a835
/avatars/cd5056c920a6b18eb93e2e9d0c5dc7c4.svg
Ranjita Naik
user
ranjita
65386075a5ab055cf2d10203
Besmira Nushi
admin_assigned
2023-10-26T11:49:56.490Z
638ee2ed848625f9fad11034
https://cdn-avatars.huggingface.co/v1/production/uploads/1670309301345-638ee2ed848625f9fad11034.jpeg
Besmira Nushi
user
nushib
KITAB: Evaluating LLMs on Constraint Satisfaction for Information
  Retrieval
We study the ability of state-of-the art models to answer constraint
satisfaction queries for information retrieval (e.g., 'a list of ice cream
shops in San Diego'). In the past, such queries were considered to be tasks
that could only be solved via web-search or knowledge bases. More recently,
large language models (LLMs) have demonstrated initial emergent abilities in
this task. However, many current retrieval benchmarks are either saturated or
do not measure constraint satisfaction. Motivated by rising concerns around
factual incorrectness and hallucinations of LLMs, we present KITAB, a new
dataset for measuring constraint satisfaction abilities of language models.
KITAB consists of book-related data across more than 600 authors and 13,000
queries, and also offers an associated dynamic data collection and constraint
verification approach for acquiring similar test data for other authors. Our
extended experiments on GPT4 and GPT3.5 characterize and decouple common
failure modes across dimensions such as information popularity, constraint
types, and context availability. Results show that in the absence of context,
models exhibit severe limitations as measured by irrelevant information,
factual errors, and incompleteness, many of which exacerbate as information
popularity decreases. While context availability mitigates irrelevant
information, it is not helpful for satisfying constraints, identifying
fundamental barriers to constraint satisfaction. We open source our
contributions to foster further research on improving constraint satisfaction
abilities of future models.
KITAB, a new dataset, evaluates constraint satisfaction in large language models, highlighting their limitations in handling irrelevant information, factual errors, and incomplete responses, especially without context.
large language models LLMs constraint satisfaction KITAB information popularity constraint types context availability irrelevant information factual errors incompleteness