oMeBench: Towards Robust Benchmarking of LLMs in Organic Mechanism
  Elucidation and Reasoning
Organic reaction mechanisms are the stepwise elementary reactions by which
reactants form intermediates and products, and are fundamental to understanding
chemical reactivity and designing new molecules and reactions. Although large
language models (LLMs) have shown promise in understanding chemical tasks such
as synthesis design, it is unclear to what extent this reflects genuine
chemical reasoning capabilities, i.e., the ability to generate valid
intermediates, maintain chemical consistency, and follow logically coherent
multi-step pathways. We address this by introducing oMeBench, the first
large-scale, expert-curated benchmark for organic mechanism reasoning in
organic chemistry. It comprises over 10,000 annotated mechanistic steps with
intermediates, type labels, and difficulty ratings. Furthermore, to evaluate
LLM capability more precisely and enable fine-grained scoring, we propose oMeS,
a dynamic evaluation framework that combines step-level logic and chemical
similarity. We analyze the performance of state-of-the-art LLMs, and our
results show that although current models display promising chemical intuition,
they struggle with correct and consistent multi-step reasoning. Notably, we
find that using prompting strategy and fine-tuning a specialist model on our
proposed dataset increases performance by 50% over the leading closed-source
model. We hope that oMeBench will serve as a rigorous foundation for advancing
AI systems toward genuine chemical reasoning.
68ee9f1b510125ca000ccf52
Ruiling Xu
claimed_verified
2025-10-15T03:13:36.903Z
67c604e3476768c5580ba8df
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/NQjSegrFH1ZdiqAR08_nc.png
Ruiling Xu
user
CarolXRL
68ee9f1b510125ca000ccf53
Yifan Zhang
claimed_verified
2025-10-15T03:13:33.464Z
67f4358ffb93d7694823f55e
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/oYoMzHH5LtgE-cA3KDSmW.png
Yifan Zhang
user
skylarzyf
68ee9f1b510125ca000ccf54
Qingyun Wang
claimed_verified
2025-10-15T03:13:39.029Z
628ba530ac304a69264afb75
https://cdn-avatars.huggingface.co/v1/production/uploads/1653319185560-628ba530ac304a69264afb75.jpeg
Qingyun Wang
user
eaglew
68ee9f1b510125ca000ccf55
Carl Edwards
68ee9f1b510125ca000ccf56
Heng Ji
oMeBench: Towards Robust Benchmarking of LLMs in Organic Mechanism
  Elucidation and Reasoning
Organic reaction mechanisms are the stepwise elementary reactions by which
reactants form intermediates and products, and are fundamental to understanding
chemical reactivity and designing new molecules and reactions. Although large
language models (LLMs) have shown promise in understanding chemical tasks such
as synthesis design, it is unclear to what extent this reflects genuine
chemical reasoning capabilities, i.e., the ability to generate valid
intermediates, maintain chemical consistency, and follow logically coherent
multi-step pathways. We address this by introducing oMeBench, the first
large-scale, expert-curated benchmark for organic mechanism reasoning in
organic chemistry. It comprises over 10,000 annotated mechanistic steps with
intermediates, type labels, and difficulty ratings. Furthermore, to evaluate
LLM capability more precisely and enable fine-grained scoring, we propose oMeS,
a dynamic evaluation framework that combines step-level logic and chemical
similarity. We analyze the performance of state-of-the-art LLMs, and our
results show that although current models display promising chemical intuition,
they struggle with correct and consistent multi-step reasoning. Notably, we
find that using prompting strategy and fine-tuning a specialist model on our
proposed dataset increases performance by 50% over the leading closed-source
model. We hope that oMeBench will serve as a rigorous foundation for advancing
AI systems toward genuine chemical reasoning.
A benchmark and evaluation framework for assessing the chemical reasoning capabilities of large language models in organic reaction mechanisms.
large language models organic mechanism reasoning benchmark annotated mechanistic steps intermediates type labels difficulty ratings dynamic evaluation framework step-level logic chemical similarity prompting strategy fine-tuning