DITING: A Multi-Agent Evaluation Framework for Benchmarking Web Novel
  Translation
Large language models (LLMs) have substantially advanced machine translation
(MT), yet their effectiveness in translating web novels remains unclear.
Existing benchmarks rely on surface-level metrics that fail to capture the
distinctive traits of this genre. To address these gaps, we introduce DITING,
the first comprehensive evaluation framework for web novel translation,
assessing narrative and cultural fidelity across six dimensions: idiom
translation, lexical ambiguity, terminology localization, tense consistency,
zero-pronoun resolution, and cultural safety, supported by over 18K
expert-annotated Chinese-English sentence pairs. We further propose AgentEval,
a reasoning-driven multi-agent evaluation framework that simulates expert
deliberation to assess translation quality beyond lexical overlap, achieving
the highest correlation with human judgments among seven tested automatic
metrics. To enable metric comparison, we develop MetricAlign, a meta-evaluation
dataset of 300 sentence pairs annotated with error labels and scalar quality
scores. Comprehensive evaluation of fourteen open, closed, and commercial
models reveals that Chinese-trained LLMs surpass larger foreign counterparts,
and that DeepSeek-V3 delivers the most faithful and stylistically coherent
translations. Our work establishes a new paradigm for exploring LLM-based web
novel translation and provides public resources to advance future research.
68edd273de1fee572713a963
Enze Zhang
claimed_verified
2025-10-14T07:29:47.167Z
67e659905c1ed903debed72a
/avatars/b112b1f5903806e55aabd16aafdeee10.svg
zez
user
Everything-is-Ok
68edd273de1fee572713a964
Jiaying Wang
claimed_verified
2025-10-14T07:29:42.425Z
6852d8a124a68790bddff660
/avatars/0c07abf23c1a099335a7fa0ca15db7ca.svg
Jiaying Wang
user
winniwang
68edd273de1fee572713a965
Mengxi Xiao
claimed_verified
2025-10-14T07:29:44.631Z
663adb42e14047f710dc1d29
/avatars/7ca49d67a4a8b4cf0ee896e07646715f.svg
Mengxi Xiao
user
ElsaShaw
68edd273de1fee572713a966
Jifei Liu
claimed_verified
2025-10-15T15:25:11.506Z
6852d79ec084bc0658b7ec4c
/avatars/1b68a590fd81b92f4860d6855409e31b.svg
Liu jifei
user
whuviolet
68edd273de1fee572713a967
Ziyan Kuang
claimed_verified
2025-10-15T15:25:13.629Z
67fd030140e7539fcc5f6521
/avatars/d79c0eed928d964c3b014a0cf1f01d72.svg
Ziyan Kuang
user
plumjane
68edd273de1fee572713a968
Rui Dong
68edd273de1fee572713a969
Eric Dong
68edd273de1fee572713a96a
Sophia Ananiadou
68edd273de1fee572713a96b
Min Peng
68edd273de1fee572713a96c
Qianqian Xie
DITING: A Multi-Agent Evaluation Framework for Benchmarking Web Novel
  Translation
Large language models (LLMs) have substantially advanced machine translation
(MT), yet their effectiveness in translating web novels remains unclear.
Existing benchmarks rely on surface-level metrics that fail to capture the
distinctive traits of this genre. To address these gaps, we introduce DITING,
the first comprehensive evaluation framework for web novel translation,
assessing narrative and cultural fidelity across six dimensions: idiom
translation, lexical ambiguity, terminology localization, tense consistency,
zero-pronoun resolution, and cultural safety, supported by over 18K
expert-annotated Chinese-English sentence pairs. We further propose AgentEval,
a reasoning-driven multi-agent evaluation framework that simulates expert
deliberation to assess translation quality beyond lexical overlap, achieving
the highest correlation with human judgments among seven tested automatic
metrics. To enable metric comparison, we develop MetricAlign, a meta-evaluation
dataset of 300 sentence pairs annotated with error labels and scalar quality
scores. Comprehensive evaluation of fourteen open, closed, and commercial
models reveals that Chinese-trained LLMs surpass larger foreign counterparts,
and that DeepSeek-V3 delivers the most faithful and stylistically coherent
translations. Our work establishes a new paradigm for exploring LLM-based web
novel translation and provides public resources to advance future research.
A new evaluation framework, DITING, and a reasoning-driven multi-agent evaluation framework, AgentEval, are introduced to assess the quality of web novel translations, revealing that Chinese-trained LLMs outperform larger foreign models.
large language models machine translation web novels DITING narrative fidelity cultural fidelity idiom translation lexical ambiguity terminology localization tense consistency zero-pronoun resolution cultural safety AgentEval reasoning-driven multi-agent evaluation MetricAlign meta-evaluation DeepSeek-V3 stylistic coherence