RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World
  Scenarios
This paper introduces RuleArena, a novel and challenging benchmark designed
to evaluate the ability of large language models (LLMs) to follow complex,
real-world rules in reasoning. Covering three practical domains -- airline
baggage fees, NBA transactions, and tax regulations -- RuleArena assesses LLMs'
proficiency in handling intricate natural language instructions that demand
long-context understanding, logical reasoning, and accurate mathematical
computation. Two key attributes distinguish RuleArena from traditional
rule-based reasoning benchmarks: (1) it extends beyond standard first-order
logic representations, and (2) it is grounded in authentic, practical
scenarios, providing insights into the suitability and reliability of LLMs for
real-world applications. Our findings reveal several notable limitations in
LLMs: (1) they struggle to identify and apply the appropriate rules, frequently
becoming confused by similar but distinct regulations, (2) they cannot
consistently perform accurate mathematical computations, even when they
correctly identify the relevant rules, and (3) in general, they perform poorly
in the benchmark. These results highlight significant challenges in advancing
LLMs' rule-guided reasoning capabilities in real-life applications.
675bca99935c1403e2c788a2
Ruiwen Zhou
admin_assigned
2024-12-16T11:18:29.856Z
66ad5084bcede3c1e3445395
https://cdn-avatars.huggingface.co/v1/production/uploads/66ad5084bcede3c1e3445395/egBqNuArOy0rz7aGVB-T6.jpeg
Ruiwen Zhou
user
rwzhou
675bca99935c1403e2c788a3
Wenyue Hua
admin_assigned
2024-12-16T11:18:35.880Z
639a25aba2b0b1c9d85a51e8
https://cdn-avatars.huggingface.co/v1/production/uploads/639a25aba2b0b1c9d85a51e8/pphz-MK62hPNbBkMHAkeR.jpeg
Wenyue Hua
user
wenyueH
675bca99935c1403e2c788a4
Liangming Pan
admin_assigned
2024-12-16T11:18:41.535Z
628cd837ef14f971b699e4d4
/avatars/cf64c88e71c730fc0a54df59e17bffbb.svg
Liangming Pan
user
Liangming
675bca99935c1403e2c788a5
Sitao Cheng
admin_assigned
2024-12-16T11:18:48.386Z
66d74853f3af37296e52e921
/avatars/0683450a86ef3aa96f0813beaa078fa2.svg
Sitao Cheng
user
Stauskas
675bca99935c1403e2c788a6
Xiaobao Wu
admin_assigned
2024-12-16T11:18:54.348Z
64cb02869e30a46f7b80b355
/avatars/81ce4ba78826b54f0e1b53eeaff87ee6.svg
Xiaobao Wu
user
bobxwu
675bca99935c1403e2c788a7
En Yu
675bca99935c1403e2c788a8
William Yang Wang
admin_assigned
2024-12-16T11:19:13.368Z
6476607efb22e3b77f3f1452
/avatars/22f40c81e6b6e09bdb66a5afd978f98b.svg
William Yang Wang
user
wangwilliamyang
RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World
  Scenarios
This paper introduces RuleArena, a novel and challenging benchmark designed
to evaluate the ability of large language models (LLMs) to follow complex,
real-world rules in reasoning. Covering three practical domains -- airline
baggage fees, NBA transactions, and tax regulations -- RuleArena assesses LLMs'
proficiency in handling intricate natural language instructions that demand
long-context understanding, logical reasoning, and accurate mathematical
computation. Two key attributes distinguish RuleArena from traditional
rule-based reasoning benchmarks: (1) it extends beyond standard first-order
logic representations, and (2) it is grounded in authentic, practical
scenarios, providing insights into the suitability and reliability of LLMs for
real-world applications. Our findings reveal several notable limitations in
LLMs: (1) they struggle to identify and apply the appropriate rules, frequently
becoming confused by similar but distinct regulations, (2) they cannot
consistently perform accurate mathematical computations, even when they
correctly identify the relevant rules, and (3) in general, they perform poorly
in the benchmark. These results highlight significant challenges in advancing
LLMs' rule-guided reasoning capabilities in real-life applications.
RuleArena evaluates large language models' proficiency in handling complex, real-world rules across practical domains, revealing notable limitations in rule application, mathematical computation, and overall performance.