DocGraphLM: Documental Graph Language Model for Information Extraction
Advances in Visually Rich Document Understanding (VrDU) have enabled
information extraction and question answering over documents with complex
layouts. Two tropes of architectures have emerged -- transformer-based models
inspired by LLMs, and Graph Neural Networks. In this paper, we introduce
DocGraphLM, a novel framework that combines pre-trained language models with
graph semantics. To achieve this, we propose 1) a joint encoder architecture to
represent documents, and 2) a novel link prediction approach to reconstruct
document graphs. DocGraphLM predicts both directions and distances between
nodes using a convergent joint loss function that prioritizes neighborhood
restoration and downweighs distant node detection. Our experiments on three
SotA datasets show consistent improvement on IE and QA tasks with the adoption
of graph features. Moreover, we report that adopting the graph features
accelerates convergence in the learning process during training, despite being
solely constructed through link prediction.
659b6ac6eff07dcf1ff16a0c
Dongsheng Wang
admin_assigned
2024-01-08T11:37:20.833Z
64863e4e60b30c3615b92812
/avatars/e4a271b53e567f9423bcd9e62cf77889.svg
Dongsheng Wang
user
dswang2011
659b6ac6eff07dcf1ff16a0d
Zhiqiang Ma
extracted_pending
2024-01-08T03:23:50.788Z
64b702e8fa7eabaae5f0cbfb
/avatars/512690f60997ae81b79b999a634a1861.svg
Zhiqiang Ma
user
zqmaster
659b6ac6eff07dcf1ff16a0e
Armineh Nourbakhsh
claimed_verified
2024-01-08T08:47:23.876Z
5e79436f30dc073f817a2b91
/avatars/5ffac317818689106be1a00ee69cded0.svg
Armi Nouri
user
arminouri
659b6ac6eff07dcf1ff16a0f
Kang Gu
659b6ac6eff07dcf1ff16a10
Sameena Shah
DocGraphLM: Documental Graph Language Model for Information Extraction
Advances in Visually Rich Document Understanding (VrDU) have enabled
information extraction and question answering over documents with complex
layouts. Two tropes of architectures have emerged -- transformer-based models
inspired by LLMs, and Graph Neural Networks. In this paper, we introduce
DocGraphLM, a novel framework that combines pre-trained language models with
graph semantics. To achieve this, we propose 1) a joint encoder architecture to
represent documents, and 2) a novel link prediction approach to reconstruct
document graphs. DocGraphLM predicts both directions and distances between
nodes using a convergent joint loss function that prioritizes neighborhood
restoration and downweighs distant node detection. Our experiments on three
SotA datasets show consistent improvement on IE and QA tasks with the adoption
of graph features. Moreover, we report that adopting the graph features
accelerates convergence in the learning process during training, despite being
solely constructed through link prediction.
A new framework, DocGraphLM, combines pre-trained language models with graph semantics to improve information extraction and question answering on documents with complex layouts.
transformer-based models Graph Neural Networks DocGraphLM joint encoder architecture link prediction joint loss function neighborhood restoration distant node detection