Narrowing the Knowledge Evaluation Gap: Open-Domain Question Answering
  with Multi-Granularity Answers
Factual questions typically can be answered correctly at different levels of
granularity. For example, both ``August 4, 1961'' and ``1961'' are correct
answers to the question ``When was Barack Obama born?''. Standard question
answering (QA) evaluation protocols, however, do not explicitly take this into
account and compare a predicted answer against answers of a single granularity
level. In this work, we propose GRANOLA QA, a novel evaluation setting where a
predicted answer is evaluated in terms of accuracy and informativeness against
a set of multi-granularity answers. We present a simple methodology for
enriching existing datasets with multi-granularity answers, and create
GRANOLA-EQ, a multi-granularity version of the EntityQuestions dataset. We
evaluate a range of decoding methods on GRANOLA-EQ, including a new algorithm,
called Decoding with Response Aggregation (DRAG), that is geared towards
aligning the response granularity with the model's uncertainty. Our experiments
show that large language models with standard decoding tend to generate
specific answers, which are often incorrect. In contrast, when evaluated on
multi-granularity answers, DRAG yields a nearly 20 point increase in accuracy
on average, which further increases for rare entities. Overall, this reveals
that standard evaluation and decoding schemes may significantly underestimate
the knowledge encapsulated in LMs.
659e0b010ce6bc9fbd08909b
Gal Yona
admin_assigned
2024-01-10T09:36:27.520Z
64637577c615cbc124477498
/avatars/6bd01b4a6ea1e3a51d739d4104eb4ca9.svg
Gal Yona
user
galyona
659e0b010ce6bc9fbd08909c
Roee Aharoni
admin_assigned
2024-01-10T09:36:34.010Z
6363bf2b123a5d5cd4a8fe7c
https://cdn-avatars.huggingface.co/v1/production/uploads/1667756718733-6363bf2b123a5d5cd4a8fe7c.jpeg
Roee Aharoni
user
roeeaharoni
659e0b010ce6bc9fbd08909d
Mor Geva
admin_assigned
2024-01-10T09:36:41.188Z
610b729f9da682cd54ad9adf
https://cdn-avatars.huggingface.co/v1/production/uploads/1628140189042-noauth.jpeg
Mor Geva
user
mega
Narrowing the Knowledge Evaluation Gap: Open-Domain Question Answering
  with Multi-Granularity Answers
Factual questions typically can be answered correctly at different levels of
granularity. For example, both ``August 4, 1961'' and ``1961'' are correct
answers to the question ``When was Barack Obama born?''. Standard question
answering (QA) evaluation protocols, however, do not explicitly take this into
account and compare a predicted answer against answers of a single granularity
level. In this work, we propose GRANOLA QA, a novel evaluation setting where a
predicted answer is evaluated in terms of accuracy and informativeness against
a set of multi-granularity answers. We present a simple methodology for
enriching existing datasets with multi-granularity answers, and create
GRANOLA-EQ, a multi-granularity version of the EntityQuestions dataset. We
evaluate a range of decoding methods on GRANOLA-EQ, including a new algorithm,
called Decoding with Response Aggregation (DRAG), that is geared towards
aligning the response granularity with the model's uncertainty. Our experiments
show that large language models with standard decoding tend to generate
specific answers, which are often incorrect. In contrast, when evaluated on
multi-granularity answers, DRAG yields a nearly 20 point increase in accuracy
on average, which further increases for rare entities. Overall, this reveals
that standard evaluation and decoding schemes may significantly underestimate
the knowledge encapsulated in LMs.
GRANOLA QA evaluates model answers against multi-granularity responses, revealing that a new decoding method improves accuracy and highlights limitations in standard evaluation methods.
GRANOLA QA multidisciplinary evaluation multi-granularity answers decoding methods Decoding with Response Aggregation (DRAG) large language models standard decoding accuracy rare entities