OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in
  Financial Domain
As a typical and practical application of Large Language Models (LLMs),
Retrieval-Augmented Generation (RAG) techniques have gained extensive
attention, particularly in vertical domains where LLMs may lack domain-specific
knowledge. In this paper, we introduce an omnidirectional and automatic RAG
benchmark, OmniEval, in the financial domain. Our benchmark is characterized by
its multi-dimensional evaluation framework, including (1) a matrix-based RAG
scenario evaluation system that categorizes queries into five task classes and
16 financial topics, leading to a structured assessment of diverse query
scenarios; (2) a multi-dimensional evaluation data generation approach, which
combines GPT-4-based automatic generation and human annotation, achieving an
87.47\% acceptance ratio in human evaluations on generated instances; (3) a
multi-stage evaluation system that evaluates both retrieval and generation
performance, result in a comprehensive evaluation on the RAG pipeline; and (4)
robust evaluation metrics derived from rule-based and LLM-based ones, enhancing
the reliability of assessments through manual annotations and supervised
fine-tuning of an LLM evaluator. Our experiments demonstrate the
comprehensiveness of OmniEval, which includes extensive test datasets and
highlights the performance variations of RAG systems across diverse topics and
tasks, revealing significant opportunities for RAG models to improve their
capabilities in vertical domains. We open source the code of our benchmark in
https://github.com/RUC-NLPIR/OmniEval{https://github.com/RUC-NLPIR/OmniEval}.
67625f697fa2dfcfa4a26eb1
Shuting Wang
claimed_verified
2024-12-18T09:51:53.983Z
66d81fe818e470c7312c0f2a
https://cdn-avatars.huggingface.co/v1/production/uploads/66d81fe818e470c7312c0f2a/HwPcM_yDB0nAgjz9bT_Cs.jpeg
ShutingWang
user
ShootingWong
67625f697fa2dfcfa4a26eb2
Jiejun Tan
claimed_verified
2024-12-18T12:38:31.592Z
62e52483a944e2a56cd2c6ca
https://cdn-avatars.huggingface.co/v1/production/uploads/62e52483a944e2a56cd2c6ca/pG44O-1qD00q5CEJMMyFQ.jpeg
Jiejun Tan
user
zstanjj
67625f697fa2dfcfa4a26eb3
Zhicheng Dou
admin_assigned
2024-12-18T12:52:00.291Z
66f0bf59e9d50ec57febf751
/avatars/be97941e60064e5dd806c6fe9db3c537.svg
Zhicheng Dou
user
douzc
67625f697fa2dfcfa4a26eb4
Ji-Rong Wen
admin_assigned
2024-12-18T12:52:06.433Z
64b8c89052b7353d8c6a1013
/avatars/cd59fffe81f6b07b4519540b8ff3d95f.svg
Ji-Rong Wen
user
jrwen
OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in
  Financial Domain
As a typical and practical application of Large Language Models (LLMs),
Retrieval-Augmented Generation (RAG) techniques have gained extensive
attention, particularly in vertical domains where LLMs may lack domain-specific
knowledge. In this paper, we introduce an omnidirectional and automatic RAG
benchmark, OmniEval, in the financial domain. Our benchmark is characterized by
its multi-dimensional evaluation framework, including (1) a matrix-based RAG
scenario evaluation system that categorizes queries into five task classes and
16 financial topics, leading to a structured assessment of diverse query
scenarios; (2) a multi-dimensional evaluation data generation approach, which
combines GPT-4-based automatic generation and human annotation, achieving an
87.47\% acceptance ratio in human evaluations on generated instances; (3) a
multi-stage evaluation system that evaluates both retrieval and generation
performance, result in a comprehensive evaluation on the RAG pipeline; and (4)
robust evaluation metrics derived from rule-based and LLM-based ones, enhancing
the reliability of assessments through manual annotations and supervised
fine-tuning of an LLM evaluator. Our experiments demonstrate the
comprehensiveness of OmniEval, which includes extensive test datasets and
highlights the performance variations of RAG systems across diverse topics and
tasks, revealing significant opportunities for RAG models to improve their
capabilities in vertical domains. We open source the code of our benchmark in
https://github.com/RUC-NLPIR/OmniEval{https://github.com/RUC-NLPIR/OmniEval}.
OmniEval is a comprehensive benchmark for Retrieval-Augmented Generation (RAG) in the financial domain, offering a structured evaluation framework with automatic generation, human annotation, and robust metrics.
Retrieval-Augmented Generation (RAG) OmniEval GPT-4 multi-dimensional evaluation scenario evaluation financial topics multi-stage evaluation evaluation metrics LLM evaluator supervised fine-tuning