CDM: A Reliable Metric for Fair and Accurate Formula Recognition
  Evaluation
Formula recognition presents significant challenges due to the complicated
structure and varied notation of mathematical expressions. Despite continuous
advancements in formula recognition models, the evaluation metrics employed by
these models, such as BLEU and Edit Distance, still exhibit notable
limitations. They overlook the fact that the same formula has diverse
representations and is highly sensitive to the distribution of training data,
thereby causing the unfairness in formula recognition evaluation. To this end,
we propose a Character Detection Matching (CDM) metric, ensuring the evaluation
objectivity by designing a image-level rather than LaTex-level metric score.
Specifically, CDM renders both the model-predicted LaTeX and the ground-truth
LaTeX formulas into image-formatted formulas, then employs visual feature
extraction and localization techniques for precise character-level matching,
incorporating spatial position information. Such a spatially-aware and
character-matching method offers a more accurate and equitable evaluation
compared with previous BLEU and Edit Distance metrics that rely solely on
text-based character matching. Experimentally, we evaluated various formula
recognition models using CDM, BLEU, and ExpRate metrics. Their results
demonstrate that the CDM aligns more closely with human evaluation standards
and provides a fairer comparison across different models by eliminating
discrepancies caused by diverse formula representations.
66dab2f7006a9245b9284a70
Bin Wang
claimed_verified
2024-09-06T09:06:04.441Z
63ae9ff5557befe297a76f90
https://cdn-avatars.huggingface.co/v1/production/uploads/1672388558183-noauth.jpeg
Bin Wang
user
wanderkid
66dab2f7006a9245b9284a71
Fan Wu
66dab2f7006a9245b9284a72
Linke Ouyang
admin_assigned
2024-09-06T09:24:45.266Z
66753c556f2ac48ee625d7d1
/avatars/8f7c252675fd8a096794d12971903722.svg
Linke Ouyang
user
ouyanglinke
66dab2f7006a9245b9284a73
Zhuangcheng Gu
66dab2f7006a9245b9284a74
Rui Zhang
66dab2f7006a9245b9284a75
Renqiu Xia
admin_assigned
2024-09-06T09:23:41.774Z
65b7ae76768464877cdb2e39
https://cdn-avatars.huggingface.co/v1/production/uploads/65b7ae76768464877cdb2e39/uAWPo4tpkqbZoDeEkc7y0.jpeg
Renqiu Xia
user
renqiux0302
66dab2f7006a9245b9284a76
Bo Zhang
admin_assigned
2024-09-06T09:23:35.959Z
643dfd235aafbdca3a5792c0
/avatars/ce8553cf5936012c692e08054ee27937.svg
Bo Zhang
user
BoZhang
66dab2f7006a9245b9284a77
Conghui He
admin_assigned
2024-09-06T09:22:58.520Z
63f9fca8d4349b157a109eec
/avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
Conghui He
user
conghui
CDM: A Reliable Metric for Fair and Accurate Formula Recognition
  Evaluation
Formula recognition presents significant challenges due to the complicated
structure and varied notation of mathematical expressions. Despite continuous
advancements in formula recognition models, the evaluation metrics employed by
these models, such as BLEU and Edit Distance, still exhibit notable
limitations. They overlook the fact that the same formula has diverse
representations and is highly sensitive to the distribution of training data,
thereby causing the unfairness in formula recognition evaluation. To this end,
we propose a Character Detection Matching (CDM) metric, ensuring the evaluation
objectivity by designing a image-level rather than LaTex-level metric score.
Specifically, CDM renders both the model-predicted LaTeX and the ground-truth
LaTeX formulas into image-formatted formulas, then employs visual feature
extraction and localization techniques for precise character-level matching,
incorporating spatial position information. Such a spatially-aware and
character-matching method offers a more accurate and equitable evaluation
compared with previous BLEU and Edit Distance metrics that rely solely on
text-based character matching. Experimentally, we evaluated various formula
recognition models using CDM, BLEU, and ExpRate metrics. Their results
demonstrate that the CDM aligns more closely with human evaluation standards
and provides a fairer comparison across different models by eliminating
discrepancies caused by diverse formula representations.
A new Character Detection Matching (CDM) metric is proposed for formula recognition evaluation, offering greater accuracy and fairness by considering spatial and character-level matching of image-formatted formulas.
Character Detection Matching CDM visual feature extraction localization techniques LaTeX image-formatted formulas spatial position information formula recognition BLEU Edit Distance ExpRate human evaluation standards