KOFFVQA: An Objectively Evaluated Free-form VQA Benchmark for Large
  Vision-Language Models in the Korean Language
The recent emergence of Large Vision-Language Models(VLMs) has resulted in a
variety of different benchmarks for evaluating such models. Despite this, we
observe that most existing evaluation methods suffer from the fact that they
either require the model to choose from pre-determined responses, sacrificing
open-endedness, or evaluate responses using a judge model, resulting in
subjective and unreliable evaluation. In addition, we observe a lack of
benchmarks for VLMs in the Korean language, which are necessary as a separate
metric from more common English language benchmarks, as the performance of
generative language models can differ significantly based on the language being
used. Therefore, we present KOFFVQA, a general-purpose free-form visual
question answering benchmark in the Korean language for the evaluation of VLMs.
Our benchmark consists of 275 carefully crafted questions each paired with an
image and grading criteria covering 10 different aspects of VLM performance.
The grading criteria eliminate the problem of unreliability by allowing the
judge model to grade each response based on a pre-determined set of rules. By
defining the evaluation criteria in an objective manner, even a small
open-source model can be used to evaluate models on our benchmark reliably. In
addition to evaluating a large number of existing VLMs on our benchmark, we
also experimentally verify that our method of using pre-existing grading
criteria for evaluation is much more reliable than existing methods. Our
evaluation code is available at https://github.com/maum-ai/KOFFVQA
67eb567141abf40cd86e0e15
Yoonshik Kim
claimed_verified
2025-04-01T07:46:53.729Z
67038a66eb760972bcb62c70
/avatars/8cc82af8f11cae994bc83f4bd99b51bc.svg
김윤식
user
yoonshik1205
67eb567141abf40cd86e0e16
Jaeyoon Jung
claimed_verified
2025-04-01T07:46:56.151Z
646484cfb90150b2706df03b
https://cdn-avatars.huggingface.co/v1/production/uploads/646484cfb90150b2706df03b/8ocSbXBSbrruhlhcxwzEt.png
Jaeyoon Jung
user
lastdefiance20
KOFFVQA: An Objectively Evaluated Free-form VQA Benchmark for Large
  Vision-Language Models in the Korean Language
The recent emergence of Large Vision-Language Models(VLMs) has resulted in a
variety of different benchmarks for evaluating such models. Despite this, we
observe that most existing evaluation methods suffer from the fact that they
either require the model to choose from pre-determined responses, sacrificing
open-endedness, or evaluate responses using a judge model, resulting in
subjective and unreliable evaluation. In addition, we observe a lack of
benchmarks for VLMs in the Korean language, which are necessary as a separate
metric from more common English language benchmarks, as the performance of
generative language models can differ significantly based on the language being
used. Therefore, we present KOFFVQA, a general-purpose free-form visual
question answering benchmark in the Korean language for the evaluation of VLMs.
Our benchmark consists of 275 carefully crafted questions each paired with an
image and grading criteria covering 10 different aspects of VLM performance.
The grading criteria eliminate the problem of unreliability by allowing the
judge model to grade each response based on a pre-determined set of rules. By
defining the evaluation criteria in an objective manner, even a small
open-source model can be used to evaluate models on our benchmark reliably. In
addition to evaluating a large number of existing VLMs on our benchmark, we
also experimentally verify that our method of using pre-existing grading
criteria for evaluation is much more reliable than existing methods. Our
evaluation code is available at https://github.com/maum-ai/KOFFVQA
KOFFVQA is a Korean language-based visual question answering benchmark for evaluating vision-language models, featuring 275 questions and objective grading criteria to enhance reliability.
Large Vision-Language Models VLMs visual question answering grading criteria Korean language reliability evaluation benchmark