Mistral-C2F: Coarse to Fine Actor for Analytical and Reasoning
  Enhancement in RLHF and Effective-Merged LLMs
Despite the advances in Large Language Models (LLMs), exemplified by models
like GPT-4 and Claude, smaller-scale LLMs such as Llama and Mistral often
struggle with generating in-depth and coherent dialogues. This paper presents a
novel two-step Coarse-to-Fine Actor model to address the inherent limitations
in conversational and analytical capabilities of small-sized LLMs. Our approach
begins with the Policy-based Coarse Actor, employing a technique we term
"Continuous Maximization". The Coarse Actor establishes an enhanced,
knowledge-rich pool adept at aligning with human preference styles in analysis
and reasoning. Through the RLHF process, it employs Continuous Maximization, a
strategy that dynamically and adaptively extends the output length limit,
enabling the generation of more detailed and analytical content. Subsequently,
the Fine Actor refines this analytical content, addressing the generation of
excessively redundant information from the Coarse Actor. We introduce a
"Knowledge Residue Merger" approach, refining the content from the Coarse Actor
and merging it with an existing Instruction model to improve quality,
correctness, and reduce redundancies. We applied our methodology to the popular
Mistral model, creating Mistral-C2F, which has demonstrated exceptional
performance across 11 general language tasks and the MT-Bench Dialogue task,
outperforming similar-scale models and even larger models with 13B and 30B
parameters. Our model has significantly improved conversational and analytical
reasoning abilities.
666b9e39c636d47bd57e7ff4
Chen Zheng
claimed_verified
2024-06-18T07:20:15.429Z
63be66194a2beec6555d49d6
https://cdn-avatars.huggingface.co/v1/production/uploads/63be66194a2beec6555d49d6/Ck-VAuADj9BCDxU0NsdTF.jpeg
CHEN ZHENG
user
zhengchenphd
666b9e39c636d47bd57e7ff5
Ke Sun
666b9e39c636d47bd57e7ff6
Xun Zhou
admin_assigned
2024-06-14T11:40:08.786Z
62533db4a06ec75172eeabe7
/avatars/b1a4dad90afae5c00df97233a97777db.svg
xunzhou
user
xunzhou
Mistral-C2F: Coarse to Fine Actor for Analytical and Reasoning
  Enhancement in RLHF and Effective-Merged LLMs
Despite the advances in Large Language Models (LLMs), exemplified by models
like GPT-4 and Claude, smaller-scale LLMs such as Llama and Mistral often
struggle with generating in-depth and coherent dialogues. This paper presents a
novel two-step Coarse-to-Fine Actor model to address the inherent limitations
in conversational and analytical capabilities of small-sized LLMs. Our approach
begins with the Policy-based Coarse Actor, employing a technique we term
"Continuous Maximization". The Coarse Actor establishes an enhanced,
knowledge-rich pool adept at aligning with human preference styles in analysis
and reasoning. Through the RLHF process, it employs Continuous Maximization, a
strategy that dynamically and adaptively extends the output length limit,
enabling the generation of more detailed and analytical content. Subsequently,
the Fine Actor refines this analytical content, addressing the generation of
excessively redundant information from the Coarse Actor. We introduce a
"Knowledge Residue Merger" approach, refining the content from the Coarse Actor
and merging it with an existing Instruction model to improve quality,
correctness, and reduce redundancies. We applied our methodology to the popular
Mistral model, creating Mistral-C2F, which has demonstrated exceptional
performance across 11 general language tasks and the MT-Bench Dialogue task,
outperforming similar-scale models and even larger models with 13B and 30B
parameters. Our model has significantly improved conversational and analytical
reasoning abilities.
A two-step Coarse-to-Fine Actor model enhances small-sized LLMs' conversational and analytical capabilities through Continuous Maximization and Knowledge Residue Merger, achieving superior performance in various tasks.
Large Language Models LLMs GPT-4 Claude Llama Mistral Coarse-to-Fine Actor model Policy-based Coarse Actor Continuous Maximization RLHF Fine Actor Knowledge Residue Merger Mistral-C2F MT-Bench Dialogue task