MoC: Mixtures of Text Chunking Learners for Retrieval-Augmented
  Generation System
Retrieval-Augmented Generation (RAG), while serving as a viable complement to
large language models (LLMs), often overlooks the crucial aspect of text
chunking within its pipeline. This paper initially introduces a dual-metric
evaluation method, comprising Boundary Clarity and Chunk Stickiness, to enable
the direct quantification of chunking quality. Leveraging this assessment
method, we highlight the inherent limitations of traditional and semantic
chunking in handling complex contextual nuances, thereby substantiating the
necessity of integrating LLMs into chunking process. To address the inherent
trade-off between computational efficiency and chunking precision in LLM-based
approaches, we devise the granularity-aware Mixture-of-Chunkers (MoC)
framework, which consists of a three-stage processing mechanism. Notably, our
objective is to guide the chunker towards generating a structured list of
chunking regular expressions, which are subsequently employed to extract chunks
from the original text. Extensive experiments demonstrate that both our
proposed metrics and the MoC framework effectively settle challenges of the
chunking task, revealing the chunking kernel while enhancing the performance of
the RAG system.
67d270d88f3def6bcbb87b6b
Jihao Zhao
claimed_verified
2025-03-13T08:23:58.864Z
658e85bb5b7553ca5c29ba89
https://cdn-avatars.huggingface.co/v1/production/uploads/658e85bb5b7553ca5c29ba89/KK6UpS9agtrxevvBoup5N.jpeg
Jihao Zhao
user
Robot2050
67d270d88f3def6bcbb87b6c
Zhiyuan Ji
67d270d88f3def6bcbb87b6d
Zhaoxin Fan
67d270d88f3def6bcbb87b6e
Hanyu Wang
67d270d88f3def6bcbb87b6f
Simin Niu
67d270d88f3def6bcbb87b70
Bo Tang
67d270d88f3def6bcbb87b71
Feiyu Xiong
67d270d88f3def6bcbb87b72
Zhiyu Li
MoC: Mixtures of Text Chunking Learners for Retrieval-Augmented
  Generation System
Retrieval-Augmented Generation (RAG), while serving as a viable complement to
large language models (LLMs), often overlooks the crucial aspect of text
chunking within its pipeline. This paper initially introduces a dual-metric
evaluation method, comprising Boundary Clarity and Chunk Stickiness, to enable
the direct quantification of chunking quality. Leveraging this assessment
method, we highlight the inherent limitations of traditional and semantic
chunking in handling complex contextual nuances, thereby substantiating the
necessity of integrating LLMs into chunking process. To address the inherent
trade-off between computational efficiency and chunking precision in LLM-based
approaches, we devise the granularity-aware Mixture-of-Chunkers (MoC)
framework, which consists of a three-stage processing mechanism. Notably, our
objective is to guide the chunker towards generating a structured list of
chunking regular expressions, which are subsequently employed to extract chunks
from the original text. Extensive experiments demonstrate that both our
proposed metrics and the MoC framework effectively settle challenges of the
chunking task, revealing the chunking kernel while enhancing the performance of
the RAG system.
A dual-metric evaluation method and granularity-aware Mixture-of-Chunkers framework improve text chunking quality in Retrieval-Augmented Generation systems.
Retrieval-Augmented Generation dual-metric evaluation Boundary Clarity Chunk Stickiness semantic chunking large language models granularity-aware Mixture-of-Chunkers chunking regular expressions