DIWALI - Diversity and Inclusivity aWare cuLture specific Items for
  India: Dataset and Assessment of LLMs for Cultural Text Adaptation in Indian
  Context
Large language models (LLMs) are widely used in various tasks and
applications. However, despite their wide capabilities, they are shown to lack
cultural alignment ryan-etal-2024-unintended,
alkhamissi-etal-2024-investigating and produce biased generations
naous-etal-2024-beer due to a lack of cultural knowledge and competence.
Evaluation of LLMs for cultural awareness and alignment is particularly
challenging due to the lack of proper evaluation metrics and unavailability of
culturally grounded datasets representing the vast complexity of cultures at
the regional and sub-regional levels. Existing datasets for culture specific
items (CSIs) focus primarily on concepts at the regional level and may contain
false positives. To address this issue, we introduce a novel CSI dataset for
Indian culture, belonging to 17 cultural facets. The dataset comprises sim8k
cultural concepts from 36 sub-regions. To measure the cultural competence of
LLMs on a cultural text adaptation task, we evaluate the adaptations using the
CSIs created, LLM as Judge, and human evaluations from diverse
socio-demographic region. Furthermore, we perform quantitative analysis
demonstrating selective sub-regional coverage and surface-level adaptations
across all considered LLMs. Our dataset is available here:
https://huggingface.co/datasets/nlip/DIWALI{https://huggingface.co/datasets/nlip/DIWALI},
project
webpage\href{https://nlip-lab.github.io/nlip/publications/diwali/{https://nlip-lab.github.io/nlip/publications/diwali/}},
and our codebase with model outputs can be found here:
https://github.com/pramitsahoo/culture-evaluation{https://github.com/pramitsahoo/culture-evaluation}.
68d24c281ca7156988a8eea9
Pramit Sahoo
claimed_verified
2025-09-23T10:04:44.425Z
65dc385490bd042d5b87194e
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/MbUdAN__mLxjEQSkzMQGh.jpeg
Pramit Sahoo
user
pramitsahoo
68d24c281ca7156988a8eeaa
Maharaj Brahma
claimed_verified
2025-09-23T10:04:46.695Z
604a4a8edca2c7ac7508b847
https://cdn-avatars.huggingface.co/v1/production/uploads/604a4a8edca2c7ac7508b847/N_HFCXuEaJ0nZw7BKY2Jj.jpeg
Maharaj Brahma
user
mrajbrahma
68d24c281ca7156988a8eeab
Maunendra Sankar Desarkar
DIWALI - Diversity and Inclusivity aWare cuLture specific Items for
  India: Dataset and Assessment of LLMs for Cultural Text Adaptation in Indian
  Context
Large language models (LLMs) are widely used in various tasks and
applications. However, despite their wide capabilities, they are shown to lack
cultural alignment ryan-etal-2024-unintended,
alkhamissi-etal-2024-investigating and produce biased generations
naous-etal-2024-beer due to a lack of cultural knowledge and competence.
Evaluation of LLMs for cultural awareness and alignment is particularly
challenging due to the lack of proper evaluation metrics and unavailability of
culturally grounded datasets representing the vast complexity of cultures at
the regional and sub-regional levels. Existing datasets for culture specific
items (CSIs) focus primarily on concepts at the regional level and may contain
false positives. To address this issue, we introduce a novel CSI dataset for
Indian culture, belonging to 17 cultural facets. The dataset comprises sim8k
cultural concepts from 36 sub-regions. To measure the cultural competence of
LLMs on a cultural text adaptation task, we evaluate the adaptations using the
CSIs created, LLM as Judge, and human evaluations from diverse
socio-demographic region. Furthermore, we perform quantitative analysis
demonstrating selective sub-regional coverage and surface-level adaptations
across all considered LLMs. Our dataset is available here:
https://huggingface.co/datasets/nlip/DIWALI{https://huggingface.co/datasets/nlip/DIWALI},
project
webpage\href{https://nlip-lab.github.io/nlip/publications/diwali/{https://nlip-lab.github.io/nlip/publications/diwali/}},
and our codebase with model outputs can be found here:
https://github.com/pramitsahoo/culture-evaluation{https://github.com/pramitsahoo/culture-evaluation}.
A new dataset for Indian culture is introduced to evaluate the cultural competence of large language models, focusing on sub-regional cultural facets and providing a framework for human and model-based evaluations.
large language models cultural alignment biased generations cultural knowledge evaluation metrics culturally grounded datasets cultural facets cultural text adaptation LLM as Judge human evaluations socio-demographic region quantitative analysis sub-regional coverage surface-level adaptations