Meta-Chunking: Learning Efficient Text Segmentation via Logical
  Perception
Retrieval-Augmented Generation (RAG), while serving as a viable complement to
large language models (LLMs), often overlooks the crucial aspect of text
chunking within its pipeline, which impacts the quality of knowledge-intensive
tasks. This paper introduces the concept of Meta-Chunking, which refers to a
granularity between sentences and paragraphs, consisting of a collection of
sentences within a paragraph that have deep linguistic logical connections. To
implement Meta-Chunking, we designed two strategies based on LLMs: Margin
Sampling Chunking and Perplexity Chunking. The former employs LLMs to perform
binary classification on whether consecutive sentences need to be segmented,
making decisions based on the probability difference obtained from margin
sampling. The latter precisely identifies text chunk boundaries by analyzing
the characteristics of perplexity distribution. Additionally, considering the
inherent complexity of different texts, we propose a strategy that combines
Meta-Chunking with dynamic merging to achieve a balance between fine-grained
and coarse-grained text chunking. Experiments conducted on eleven datasets
demonstrate that Meta-Chunking can more efficiently improve the performance of
single-hop and multi-hop question answering based on RAG. For instance, on the
2WikiMultihopQA dataset, it outperforms similarity chunking by 1.32 while only
consuming 45.8% of the time. Our code is available at
https://github.com/IAAR-Shanghai/Meta-Chunking.
6717157f69532dd147c7579e
Jihao Zhao
claimed_verified
2024-10-22T08:00:47.204Z
658e85bb5b7553ca5c29ba89
https://cdn-avatars.huggingface.co/v1/production/uploads/658e85bb5b7553ca5c29ba89/KK6UpS9agtrxevvBoup5N.jpeg
Jihao Zhao
user
Robot2050
6717157f69532dd147c7579f
Zhiyuan Ji
6717157f69532dd147c757a0
Pengnian Qi
admin_assigned
2024-10-22T09:15:31.720Z
64be260cf8f28a19b0e02f35
/avatars/cc55af21b31589f87dbdf9fed65be6bc.svg
pengnian qi
user
MoCun
6717157f69532dd147c757a1
Simin Niu
admin_assigned
2024-10-22T09:15:37.051Z
66daea8776dbaaa372eabec5
/avatars/1e5fbe4ff06bb6121c7029253b76b79f.svg
siminniu
user
siminniu
6717157f69532dd147c757a2
Bo Tang
6717157f69532dd147c757a3
Feiyu Xiong
6717157f69532dd147c757a4
Zhiyu Li
admin_assigned
2024-10-22T09:16:33.336Z
62a155e615eeab266b2f2243
/avatars/e89ef156e73af028e3ce3664e6cb4e62.svg
Zhiyu Li
user
jimi888
Meta-Chunking: Learning Efficient Text Segmentation via Logical
  Perception
Retrieval-Augmented Generation (RAG), while serving as a viable complement to
large language models (LLMs), often overlooks the crucial aspect of text
chunking within its pipeline, which impacts the quality of knowledge-intensive
tasks. This paper introduces the concept of Meta-Chunking, which refers to a
granularity between sentences and paragraphs, consisting of a collection of
sentences within a paragraph that have deep linguistic logical connections. To
implement Meta-Chunking, we designed two strategies based on LLMs: Margin
Sampling Chunking and Perplexity Chunking. The former employs LLMs to perform
binary classification on whether consecutive sentences need to be segmented,
making decisions based on the probability difference obtained from margin
sampling. The latter precisely identifies text chunk boundaries by analyzing
the characteristics of perplexity distribution. Additionally, considering the
inherent complexity of different texts, we propose a strategy that combines
Meta-Chunking with dynamic merging to achieve a balance between fine-grained
and coarse-grained text chunking. Experiments conducted on eleven datasets
demonstrate that Meta-Chunking can more efficiently improve the performance of
single-hop and multi-hop question answering based on RAG. For instance, on the
2WikiMultihopQA dataset, it outperforms similarity chunking by 1.32 while only
consuming 45.8% of the time. Our code is available at
https://github.com/IAAR-Shanghai/Meta-Chunking.
Meta-Chunking improves RAG's performance in text chunking for knowledge-intensive tasks by using LLMs to dynamically segment text, balancing fine-grained and coarse-grained chunking.
Retrieval-Augmented Generation Meta-Chunking large language models Margin Sampling Chunking Perplexity Chunking single-hop question answering multi-hop question answering dynamic merging 2WikiMultihopQA