PDFTriage: Question Answering over Long, Structured Documents
Large Language Models (LLMs) have issues with document question answering
(QA) in situations where the document is unable to fit in the small context
length of an LLM. To overcome this issue, most existing works focus on
retrieving the relevant context from the document, representing them as plain
text. However, documents such as PDFs, web pages, and presentations are
naturally structured with different pages, tables, sections, and so on.
Representing such structured documents as plain text is incongruous with the
user's mental model of these documents with rich structure. When a system has
to query the document for context, this incongruity is brought to the fore, and
seemingly trivial questions can trip up the QA system. To bridge this
fundamental gap in handling structured documents, we propose an approach called
PDFTriage that enables models to retrieve the context based on either structure
or content. Our experiments demonstrate the effectiveness of the proposed
PDFTriage-augmented models across several classes of questions where existing
retrieval-augmented LLMs fail. To facilitate further research on this
fundamental problem, we release our benchmark dataset consisting of 900+
human-generated questions over 80 structured documents from 10 different
categories of question types for document QA.
65090c23c2d4c8fc37854a7e
Jon Saad-Falcon
admin_assigned
2023-09-20T13:12:31.495Z
6035790c56d1aa1de2953e47
/avatars/c5fba6f1aed48d3b61298ee085018ee2.svg
Jon Saad-Falcon
user
jonsaadfalcon
65090c23c2d4c8fc37854a7f
Joe Barrow
admin_assigned
2023-09-20T13:13:13.217Z
64b71b089ebb7e6c7dce4d38
/avatars/b5601682b6500f30369ad7660e88bb55.svg
Joe Barrow
user
jbarrow
65090c23c2d4c8fc37854a80
Alexa Siu
65090c23c2d4c8fc37854a81
Ani Nenkova
65090c23c2d4c8fc37854a82
Ryan A. Rossi
admin_assigned
2023-09-20T13:13:35.102Z
62a3ab83e4dd6252344d27cd
/avatars/7ca8510f70a58dc207b104240e30c35c.svg
Ryan A. Rossi
user
ryanrossi
65090c23c2d4c8fc37854a83
Franck Dernoncourt
extracted_confirmed
2023-09-19T18:56:10.355Z
62c5947524171688a9feb992
/avatars/5a151713b9eae8dc566f5957acee3475.svg
Franck Dernoncourt
user
Franck-Dernoncourt
PDFTriage: Question Answering over Long, Structured Documents
Large Language Models (LLMs) have issues with document question answering
(QA) in situations where the document is unable to fit in the small context
length of an LLM. To overcome this issue, most existing works focus on
retrieving the relevant context from the document, representing them as plain
text. However, documents such as PDFs, web pages, and presentations are
naturally structured with different pages, tables, sections, and so on.
Representing such structured documents as plain text is incongruous with the
user's mental model of these documents with rich structure. When a system has
to query the document for context, this incongruity is brought to the fore, and
seemingly trivial questions can trip up the QA system. To bridge this
fundamental gap in handling structured documents, we propose an approach called
PDFTriage that enables models to retrieve the context based on either structure
or content. Our experiments demonstrate the effectiveness of the proposed
PDFTriage-augmented models across several classes of questions where existing
retrieval-augmented LLMs fail. To facilitate further research on this
fundamental problem, we release our benchmark dataset consisting of 900+
human-generated questions over 80 structured documents from 10 different
categories of question types for document QA.
PDFTriage enables LLMs to retrieve context effectively from structured documents like PDFs and web pages for better document QA performance.
LLMs document question answering QA context length PDFTriage