FormNetV2: Multimodal Graph Contrastive Learning for Form Document
  Information Extraction
The recent advent of self-supervised pre-training techniques has led to a
surge in the use of multimodal learning in form document understanding.
However, existing approaches that extend the mask language modeling to other
modalities require careful multi-task tuning, complex reconstruction target
designs, or additional pre-training data. In FormNetV2, we introduce a
centralized multimodal graph contrastive learning strategy to unify
self-supervised pre-training for all modalities in one loss. The graph
contrastive objective maximizes the agreement of multimodal representations,
providing a natural interplay for all modalities without special customization.
In addition, we extract image features within the bounding box that joins a
pair of tokens connected by a graph edge, capturing more targeted visual cues
without loading a sophisticated and separately pre-trained image embedder.
FormNetV2 establishes new state-of-the-art performance on FUNSD, CORD, SROIE
and Payment benchmarks with a more compact model size.
6454624706728ff79a382477
Chen-Yu Lee
extracted_confirmed
2023-08-08T08:38:20.262Z
64d1fe784dfd5df7076bfd1a
/avatars/e2bb108d1f6b1383f0e3f263c4747b3d.svg
Chen-Yu Lee
user
chenyulee
6454624706728ff79a382478
Chun-Liang Li
6454624706728ff79a382479
Hao Zhang
6454624706728ff79a38247a
Timothy Dozat
6454624706728ff79a38247b
Vincent Perot
claimed_verified
2023-11-14T09:38:20.186Z
64e3f501bbee0c24b95e20c2
/avatars/bae7b5297e779d91da21ff50389dd90a.svg
Vincent
user
vperot
6454624706728ff79a38247c
Guolong Su
6454624706728ff79a38247d
Xiang Zhang
6454624706728ff79a38247e
Kihyuk Sohn
6454624706728ff79a38247f
Nikolai Glushnev
6454624706728ff79a382480
Renshen Wang
6454624706728ff79a382481
Joshua Ainslie
6454624706728ff79a382482
Shangbang Long
6454624706728ff79a382483
Siyang Qin
6454624706728ff79a382484
Yasuhisa Fujii
6454624706728ff79a382485
Nan Hua
6454624706728ff79a382486
Tomas Pfister
FormNetV2: Multimodal Graph Contrastive Learning for Form Document
  Information Extraction
The recent advent of self-supervised pre-training techniques has led to a
surge in the use of multimodal learning in form document understanding.
However, existing approaches that extend the mask language modeling to other
modalities require careful multi-task tuning, complex reconstruction target
designs, or additional pre-training data. In FormNetV2, we introduce a
centralized multimodal graph contrastive learning strategy to unify
self-supervised pre-training for all modalities in one loss. The graph
contrastive objective maximizes the agreement of multimodal representations,
providing a natural interplay for all modalities without special customization.
In addition, we extract image features within the bounding box that joins a
pair of tokens connected by a graph edge, capturing more targeted visual cues
without loading a sophisticated and separately pre-trained image embedder.
FormNetV2 establishes new state-of-the-art performance on FUNSD, CORD, SROIE
and Payment benchmarks with a more compact model size.
FormNetV2 uses a centralized multimodal graph contrastive learning approach to improve self-supervised pre-training and achieves state-of-the-art performance on form understanding benchmarks with a compact model.
self-supervised pre-training multimodal learning mask language modeling graph contrastive learning multimodal representations bounding box image features visual cues FUNSD CORD SROIE Payment benchmarks