NeKo: Toward Post Recognition Generative Correction Large Language
  Models with Task-Oriented Experts
Construction of a general-purpose post-recognition error corrector poses a
crucial question: how can we most effectively train a model on a large mixture
of domain datasets? The answer would lie in learning dataset-specific features
and digesting their knowledge in a single model. Previous methods achieve this
by having separate correction language models, resulting in a significant
increase in parameters. In this work, we present Mixture-of-Experts as a
solution, highlighting that MoEs are much more than a scalability tool. We
propose a Multi-Task Correction MoE, where we train the experts to become an
``expert'' of speech-to-text, language-to-text and vision-to-text datasets by
learning to route each dataset's tokens to its mapped expert. Experiments on
the Open ASR Leaderboard show that we explore a new state-of-the-art
performance by achieving an average relative 5.0% WER reduction and
substantial improvements in BLEU scores for speech and translation tasks. On
zero-shot evaluation, NeKo outperforms GPT-3.5 and Claude-Opus with 15.5% to
27.6% relative WER reduction in the Hyporadise benchmark. NeKo performs
competitively on grammar and post-OCR correction as a multi-task model.
67335d4cd40c698f6d001c1f
Yen-Ting Lin
67335d4cd40c698f6d001c20
Chao-Han Huck Yang
extracted_pending
2024-11-12T13:51:09.350Z
629e1b71bb6419817ed7566c
https://cdn-avatars.huggingface.co/v1/production/uploads/629e1b71bb6419817ed7566c/0ZCt-11eQtRDCOk9AozOp.jpeg
Huck Yang
user
huckiyang
67335d4cd40c698f6d001c21
Zhehuai Chen
67335d4cd40c698f6d001c22
Piotr Zelasko
67335d4cd40c698f6d001c23
Xuesong Yang
claimed_verified
2025-06-13T07:43:42.121Z
64b708fa53d91a364aa32bbf
https://cdn-avatars.huggingface.co/v1/production/uploads/64b708fa53d91a364aa32bbf/f8gnDFShkI3VBhadko8qr.jpeg
Xuesong Yang
user
magicyoung8
67335d4cd40c698f6d001c24
Zih-Ching Chen
67335d4cd40c698f6d001c25
Krishna C Puvvada
67335d4cd40c698f6d001c26
Szu-Wei Fu
67335d4cd40c698f6d001c27
Ke Hu
67335d4cd40c698f6d001c28
Jun Wei Chiu
67335d4cd40c698f6d001c29
Jagadeesh Balam
67335d4cd40c698f6d001c2a
Boris Ginsburg
67335d4cd40c698f6d001c2b
Yu-Chiang Frank Wang
NeKo: Toward Post Recognition Generative Correction Large Language
  Models with Task-Oriented Experts
Construction of a general-purpose post-recognition error corrector poses a
crucial question: how can we most effectively train a model on a large mixture
of domain datasets? The answer would lie in learning dataset-specific features
and digesting their knowledge in a single model. Previous methods achieve this
by having separate correction language models, resulting in a significant
increase in parameters. In this work, we present Mixture-of-Experts as a
solution, highlighting that MoEs are much more than a scalability tool. We
propose a Multi-Task Correction MoE, where we train the experts to become an
``expert'' of speech-to-text, language-to-text and vision-to-text datasets by
learning to route each dataset's tokens to its mapped expert. Experiments on
the Open ASR Leaderboard show that we explore a new state-of-the-art
performance by achieving an average relative 5.0% WER reduction and
substantial improvements in BLEU scores for speech and translation tasks. On
zero-shot evaluation, NeKo outperforms GPT-3.5 and Claude-Opus with 15.5% to
27.6% relative WER reduction in the Hyporadise benchmark. NeKo performs
competitively on grammar and post-OCR correction as a multi-task model.
A Multi-Task Correction MoE enhances error correction across different domains, achieving state-of-the-art performance with improved WER and BLEU scores.
Mixture-of-Experts Multi-Task Correction MoE speech-to-text language-to-text vision-to-text Open ASR Leaderboard WER BLEU zero-shot evaluation NeKo GPT-3.5 Claude-Opus Hyporadise post-OCR correction