CS-Sum: A Benchmark for Code-Switching Dialogue Summarization and the
  Limits of Large Language Models
Code-switching (CS) poses a significant challenge for Large Language Models
(LLMs), yet its comprehensibility remains underexplored in LLMs. We introduce
CS-Sum, to evaluate the comprehensibility of CS by the LLMs through CS dialogue
to English summarization. CS-Sum is the first benchmark for CS dialogue
summarization across Mandarin-English (EN-ZH), Tamil-English (EN-TA), and
Malay-English (EN-MS), with 900-1300 human-annotated dialogues per language
pair. Evaluating ten LLMs, including open and closed-source models, we analyze
performance across few-shot, translate-summarize, and fine-tuning (LoRA, QLoRA
on synthetic data) approaches. Our findings show that though the scores on
automated metrics are high, LLMs make subtle mistakes that alter the complete
meaning of the dialogue. To this end, we introduce 3 most common type of errors
that LLMs make when handling CS input. Error rates vary across CS pairs and
LLMs, with some LLMs showing more frequent errors on certain language pairs,
underscoring the need for specialized training on code-switched data.
682d4ccc3b5f51f4218e12b4
Sathya Krishnan Suresh
admin_assigned
2025-05-21T10:18:55.833Z
62f0e457bc8201db9ef47f89
https://cdn-avatars.huggingface.co/v1/production/uploads/62f0e457bc8201db9ef47f89/zOhDptwZpDGaugKCBOWB2.jpeg
Sathya Krishnan
user
SkAndMl
682d4ccc3b5f51f4218e12b5
Tanmay Surana
682d4ccc3b5f51f4218e12b6
Lim Zhi Hao
682d4ccc3b5f51f4218e12b7
Eng Siong Chng
CS-Sum: A Benchmark for Code-Switching Dialogue Summarization and the
  Limits of Large Language Models
Code-switching (CS) poses a significant challenge for Large Language Models
(LLMs), yet its comprehensibility remains underexplored in LLMs. We introduce
CS-Sum, to evaluate the comprehensibility of CS by the LLMs through CS dialogue
to English summarization. CS-Sum is the first benchmark for CS dialogue
summarization across Mandarin-English (EN-ZH), Tamil-English (EN-TA), and
Malay-English (EN-MS), with 900-1300 human-annotated dialogues per language
pair. Evaluating ten LLMs, including open and closed-source models, we analyze
performance across few-shot, translate-summarize, and fine-tuning (LoRA, QLoRA
on synthetic data) approaches. Our findings show that though the scores on
automated metrics are high, LLMs make subtle mistakes that alter the complete
meaning of the dialogue. To this end, we introduce 3 most common type of errors
that LLMs make when handling CS input. Error rates vary across CS pairs and
LLMs, with some LLMs showing more frequent errors on certain language pairs,
underscoring the need for specialized training on code-switched data.
LLMs exhibit high automated metric scores but make subtle errors in code-switching dialogue summarization, highlighting the need for specialized training on CS data.
LLMs code-switching (CS) CS-Sum Mandarin-English Tamil-English Malay-English dialogue summarization few-shot translate-summarize fine-tuning LoRA QLoRA synthetic data