Investigating Answerability of LLMs for Long-Form Question Answering
As we embark on a new era of LLMs, it becomes increasingly crucial to
understand their capabilities, limitations, and differences. Toward making
further progress in this direction, we strive to build a deeper understanding
of the gaps between massive LLMs (e.g., ChatGPT) and smaller yet effective
open-source LLMs and their distilled counterparts. To this end, we specifically
focus on long-form question answering (LFQA) because it has several practical
and impactful applications (e.g., troubleshooting, customer service, etc.) yet
is still understudied and challenging for LLMs. We propose a
question-generation method from abstractive summaries and show that generating
follow-up questions from summaries of long documents can create a challenging
setting for LLMs to reason and infer from long contexts. Our experimental
results confirm that: (1) our proposed method of generating questions from
abstractive summaries pose a challenging setup for LLMs and shows performance
gaps between LLMs like ChatGPT and open-source LLMs (Alpaca, Llama) (2)
open-source LLMs exhibit decreased reliance on context for generated questions
from the original document, but their generation capabilities drop
significantly on generated questions from summaries -- especially for longer
contexts (>1024 tokens)
65079e401704c7eb0a9b6273
Meghana Moorthy Bhat
claimed_verified
2023-09-18T19:43:01.520Z
650889c4ce81b73e633d3a31
/avatars/67572890f939149ccc6f078786e0ab8f.svg
Meghana Bhat
user
meghanabhat
65079e401704c7eb0a9b6274
Rui Meng
claimed_verified
2023-09-18T19:42:56.989Z
61b022b39f13f47207d335cf
/avatars/9cf00a775b80230aa32ef68ea7377833.svg
Rui
user
memray
65079e401704c7eb0a9b6275
Ye Liu
65079e401704c7eb0a9b6276
Yingbo Zhou
admin_assigned
2023-09-18T10:11:08.107Z
649bc93758d8b19de0c7785f
/avatars/3ed9473aee23d99f4ee949d3705089ea.svg
Yingbo Zhou
user
yingbozhou
65079e401704c7eb0a9b6277
Semih Yavuz
admin_assigned
2023-09-18T10:11:16.798Z
64f8a68e33fd3f0fcb7810b5
/avatars/cb5ea85e9ed513efaf24a9f36869f98f.svg
Semih Yavuz
user
syavuz
Investigating Answerability of LLMs for Long-Form Question Answering
As we embark on a new era of LLMs, it becomes increasingly crucial to
understand their capabilities, limitations, and differences. Toward making
further progress in this direction, we strive to build a deeper understanding
of the gaps between massive LLMs (e.g., ChatGPT) and smaller yet effective
open-source LLMs and their distilled counterparts. To this end, we specifically
focus on long-form question answering (LFQA) because it has several practical
and impactful applications (e.g., troubleshooting, customer service, etc.) yet
is still understudied and challenging for LLMs. We propose a
question-generation method from abstractive summaries and show that generating
follow-up questions from summaries of long documents can create a challenging
setting for LLMs to reason and infer from long contexts. Our experimental
results confirm that: (1) our proposed method of generating questions from
abstractive summaries pose a challenging setup for LLMs and shows performance
gaps between LLMs like ChatGPT and open-source LLMs (Alpaca, Llama) (2)
open-source LLMs exhibit decreased reliance on context for generated questions
from the original document, but their generation capabilities drop
significantly on generated questions from summaries -- especially for longer
contexts (>1024 tokens)
A question-generation method using abstractive summaries reveals performance gaps between large and small LLMs in long-form question answering tasks.
abstractive summaries long-form question answering (LFQA) long contexts