Efficient Medical VIE via Reinforcement Learning
Visual Information Extraction (VIE) converts unstructured document images
into structured formats like JSON, critical for medical applications such as
report analysis and online consultations. Traditional methods rely on OCR and
language models, while end-to-end multimodal models offer direct JSON
generation. However, domain-specific schemas and high annotation costs limit
their effectiveness in medical VIE. We base our approach on the Reinforcement
Learning with Verifiable Rewards (RLVR) framework to address these challenges
using only 100 annotated samples. Our approach ensures dataset diversity, a
balanced precision-recall reward mechanism to reduce hallucinations and improve
field coverage, and innovative sampling strategies to enhance reasoning
capabilities. Fine-tuning Qwen2.5-VL-7B with our RLVR method, we achieve
state-of-the-art performance on medical VIE tasks, significantly improving F1,
precision, and recall. While our models excel on tasks similar to medical
datasets, performance drops on dissimilar tasks, highlighting the need for
domain-specific optimization. Case studies further demonstrate the value of
reasoning during training and inference for VIE.
685234100164cd1316710508
Lijun Liu
685234100164cd1316710509
Ruiyang Li
claimed_verified
2025-06-18T10:59:03.729Z
66864a38369b09d564ba2ce4
/avatars/866699ffee8a1d7cf2c9bebe0d3d58fe.svg
lry
user
lryyyy
685234100164cd131671050a
Zhaocheng Liu
claimed_verified
2025-06-18T10:59:00.988Z
633e570be7d5ce7bfe037a53
https://cdn-avatars.huggingface.co/v1/production/uploads/633e570be7d5ce7bfe037a53/zV8ULv4Mu7YIGZ8D3JtmK.jpeg
Zhaocheng Liu
user
zhaocheng
685234100164cd131671050b
Chenglin Zhu
685234100164cd131671050c
Chong Li
685234100164cd131671050d
Jiehan Cheng
685234100164cd131671050e
Qiang Ju
claimed_verified
2025-06-23T08:20:37.713Z
62dcdb86d36b2070f928a51e
/avatars/a341e4305217f8abd14cff97201a24aa.svg
sdujq
user
sdujq
685234100164cd131671050f
Jian Xie
Efficient Medical VIE via Reinforcement Learning
Visual Information Extraction (VIE) converts unstructured document images
into structured formats like JSON, critical for medical applications such as
report analysis and online consultations. Traditional methods rely on OCR and
language models, while end-to-end multimodal models offer direct JSON
generation. However, domain-specific schemas and high annotation costs limit
their effectiveness in medical VIE. We base our approach on the Reinforcement
Learning with Verifiable Rewards (RLVR) framework to address these challenges
using only 100 annotated samples. Our approach ensures dataset diversity, a
balanced precision-recall reward mechanism to reduce hallucinations and improve
field coverage, and innovative sampling strategies to enhance reasoning
capabilities. Fine-tuning Qwen2.5-VL-7B with our RLVR method, we achieve
state-of-the-art performance on medical VIE tasks, significantly improving F1,
precision, and recall. While our models excel on tasks similar to medical
datasets, performance drops on dissimilar tasks, highlighting the need for
domain-specific optimization. Case studies further demonstrate the value of
reasoning during training and inference for VIE.
An RLVR framework using fine-tuned Qwen2.5-VL-7B achieves state-of-the-art performance in medical VIE with limited annotated samples, enhancing reasoning and balance between precision and recall.
Reinforcement Learning with Verifiable Rewards (RLVR) JSON generation multimodal models dataset diversity precision-recall reward mechanism hallucinations field coverage sampling strategies fine-tuning Qwen2.5-VL-7B F1 score case studies