HSCodeComp: A Realistic and Expert-level Benchmark for Deep Search
  Agents in Hierarchical Rule Application
Effective deep search agents must not only access open-domain and
domain-specific knowledge but also apply complex rules-such as legal clauses,
medical manuals and tariff rules. These rules often feature vague boundaries
and implicit logic relationships, making precise application challenging for
agents. However, this critical capability is largely overlooked by current
agent benchmarks.
  To fill this gap, we introduce HSCodeComp, the first realistic, expert-level
e-commerce benchmark designed to evaluate deep search agents in hierarchical
rule application. In this task, the deep reasoning process of agents is guided
by these rules to predict 10-digit Harmonized System Code (HSCode) of products
with noisy but realistic descriptions. These codes, established by the World
Customs Organization, are vital for global supply chain efficiency. Built from
real-world data collected from large-scale e-commerce platforms, our proposed
HSCodeComp comprises 632 product entries spanning diverse product categories,
with these HSCodes annotated by several human experts.
  Extensive experimental results on several state-of-the-art LLMs, open-source,
and closed-source agents reveal a huge performance gap: best agent achieves
only 46.8% 10-digit accuracy, far below human experts at 95.0%. Besides,
detailed analysis demonstrates the challenges of hierarchical rule application,
and test-time scaling fails to improve performance further.
68fad2aef158a71c5a2f57c5
Yiqian Yang
68fad2aef158a71c5a2f57c6
Tian Lan
claimed_verified
2025-10-24T16:12:52.576Z
627ca439a09edfe62239c671
https://cdn-avatars.huggingface.co/v1/production/uploads/1652335660508-noauth.jpeg
Tian Lan
user
GMFTBY
68fad2aef158a71c5a2f57c7
Qianghuai Jia
68fad2aef158a71c5a2f57c8
Li Zhu
68fad2aef158a71c5a2f57c9
Hui Jiang
68fad2aef158a71c5a2f57ca
Hang Zhu
68fad2aef158a71c5a2f57cb
Longyue Wang
68fad2aef158a71c5a2f57cc
Weihua Luo
68fad2aef158a71c5a2f57cd
Kaifu Zhang
HSCodeComp: A Realistic and Expert-level Benchmark for Deep Search
  Agents in Hierarchical Rule Application
Effective deep search agents must not only access open-domain and
domain-specific knowledge but also apply complex rules-such as legal clauses,
medical manuals and tariff rules. These rules often feature vague boundaries
and implicit logic relationships, making precise application challenging for
agents. However, this critical capability is largely overlooked by current
agent benchmarks.
  To fill this gap, we introduce HSCodeComp, the first realistic, expert-level
e-commerce benchmark designed to evaluate deep search agents in hierarchical
rule application. In this task, the deep reasoning process of agents is guided
by these rules to predict 10-digit Harmonized System Code (HSCode) of products
with noisy but realistic descriptions. These codes, established by the World
Customs Organization, are vital for global supply chain efficiency. Built from
real-world data collected from large-scale e-commerce platforms, our proposed
HSCodeComp comprises 632 product entries spanning diverse product categories,
with these HSCodes annotated by several human experts.
  Extensive experimental results on several state-of-the-art LLMs, open-source,
and closed-source agents reveal a huge performance gap: best agent achieves
only 46.8% 10-digit accuracy, far below human experts at 95.0%. Besides,
detailed analysis demonstrates the challenges of hierarchical rule application,
and test-time scaling fails to improve performance further.
HSCodeComp evaluates deep search agents' hierarchical rule application in predicting product HS Codes, revealing significant performance gaps compared to human experts.
deep search agents hierarchical rule application Harmonized System Code (HSCode) global supply chain efficiency LLMs open-source agents closed-source agents test-time scaling