EvalTree: Profiling Language Model Weaknesses via Hierarchical
  Capability Trees
An ideal model evaluation should achieve two goals: identifying where the
model fails and providing actionable improvement guidance. Toward these goals
for Language Model (LM) evaluations, we formulate the problem of generating a
weakness profile, a set of weaknesses expressed in natural language, given an
LM's performance on every individual instance in a benchmark. We introduce a
suite of quantitative assessments to compare different weakness profiling
methods. We also propose a weakness profiling method EvalTree. It constructs a
capability tree where each node represents a capability described in natural
language and is linked to a subset of benchmark instances that specifically
evaluate this capability; it then extracts nodes where the LM performs poorly
to generate a weakness profile. On the MATH and WildChat benchmarks, we show
that EvalTree outperforms baseline weakness profiling methods by identifying
weaknesses more precisely and comprehensively. Weakness profiling further
enables weakness-guided data collection, and training data collection guided by
EvalTree-identified weaknesses improves LM performance more than other data
collection strategies. We also show how EvalTree exposes flaws in Chatbot
Arena's human-voter-based evaluation practice. To facilitate future work, we
release our code and an interface that allows practitioners to interactively
explore the capability trees built by EvalTree.
67dade8168f86d8f53993ed8
Zhiyuan Zeng
claimed_verified
2025-03-20T10:46:00.793Z
64a85e23b6512b8328f9d9e2
/avatars/4a6b35752d3f76cb03278f52b3b43426.svg
Zhiyuan Zeng
user
ZhiyuanZeng
67dade8168f86d8f53993ed9
Yizhong Wang
67dade8168f86d8f53993eda
Hannaneh Hajishirzi
67dade8168f86d8f53993edb
Pang Wei Koh
EvalTree: Profiling Language Model Weaknesses via Hierarchical
  Capability Trees
An ideal model evaluation should achieve two goals: identifying where the
model fails and providing actionable improvement guidance. Toward these goals
for Language Model (LM) evaluations, we formulate the problem of generating a
weakness profile, a set of weaknesses expressed in natural language, given an
LM's performance on every individual instance in a benchmark. We introduce a
suite of quantitative assessments to compare different weakness profiling
methods. We also propose a weakness profiling method EvalTree. It constructs a
capability tree where each node represents a capability described in natural
language and is linked to a subset of benchmark instances that specifically
evaluate this capability; it then extracts nodes where the LM performs poorly
to generate a weakness profile. On the MATH and WildChat benchmarks, we show
that EvalTree outperforms baseline weakness profiling methods by identifying
weaknesses more precisely and comprehensively. Weakness profiling further
enables weakness-guided data collection, and training data collection guided by
EvalTree-identified weaknesses improves LM performance more than other data
collection strategies. We also show how EvalTree exposes flaws in Chatbot
Arena's human-voter-based evaluation practice. To facilitate future work, we
release our code and an interface that allows practitioners to interactively
explore the capability trees built by EvalTree.
EvalTree generates precise and comprehensive weakness profiles for language models by constructing a capability tree and identifying areas of poor performance, leading to improved data collection and model performance.
Language Model weakness profile capability tree MATH benchmark WildChat benchmark Chatbot Arena human-voter-based evaluation