DialectGen: Benchmarking and Improving Dialect Robustness in Multimodal
  Generation
Contact languages like English exhibit rich regional variations in the form
of dialects, which are often used by dialect speakers interacting with
generative models. However, can multimodal generative models effectively
produce content given dialectal textual input? In this work, we study this
question by constructing a new large-scale benchmark spanning six common
English dialects. We work with dialect speakers to collect and verify over 4200
unique prompts and evaluate on 17 image and video generative models. Our
automatic and human evaluation results show that current state-of-the-art
multimodal generative models exhibit 32.26% to 48.17% performance degradation
when a single dialect word is used in the prompt. Common mitigation methods
such as fine-tuning and prompt rewriting can only improve dialect performance
by small margins (< 7%), while potentially incurring significant performance
degradation in Standard American English (SAE). To this end, we design a
general encoder-based mitigation strategy for multimodal generative models. Our
method teaches the model to recognize new dialect features while preserving SAE
performance. Experiments on models such as Stable Diffusion 1.5 show that our
method is able to simultaneously raise performance on five dialects to be on
par with SAE (+34.4%), while incurring near zero cost to SAE performance.
68f1d2d56e0bef323a68ff5e
Yu Zhou
claimed_verified
2025-10-18T04:49:07.041Z
62d9a2ac51e8289052c22e42
https://cdn-avatars.huggingface.co/v1/production/uploads/62d9a2ac51e8289052c22e42/d7QLmvyTnA4OvF4FTsNXh.jpeg
Yu Zhou
user
bryanzhou008
68f1d2d56e0bef323a68ff5f
Sohyun An
68f1d2d56e0bef323a68ff60
Haikang Deng
claimed_verified
2025-10-17T05:56:24.216Z
624cf6ba5ee9cb3d45588971
/avatars/f0fc5e353951d3e501542dab559bbdb5.svg
Haikang Deng
user
hk
68f1d2d56e0bef323a68ff61
Da Yin
68f1d2d56e0bef323a68ff62
Clark Peng
68f1d2d56e0bef323a68ff63
Cho-Jui Hsieh
68f1d2d56e0bef323a68ff64
Kai-Wei Chang
68f1d2d56e0bef323a68ff65
Nanyun Peng
DialectGen: Benchmarking and Improving Dialect Robustness in Multimodal
  Generation
Contact languages like English exhibit rich regional variations in the form
of dialects, which are often used by dialect speakers interacting with
generative models. However, can multimodal generative models effectively
produce content given dialectal textual input? In this work, we study this
question by constructing a new large-scale benchmark spanning six common
English dialects. We work with dialect speakers to collect and verify over 4200
unique prompts and evaluate on 17 image and video generative models. Our
automatic and human evaluation results show that current state-of-the-art
multimodal generative models exhibit 32.26% to 48.17% performance degradation
when a single dialect word is used in the prompt. Common mitigation methods
such as fine-tuning and prompt rewriting can only improve dialect performance
by small margins (< 7%), while potentially incurring significant performance
degradation in Standard American English (SAE). To this end, we design a
general encoder-based mitigation strategy for multimodal generative models. Our
method teaches the model to recognize new dialect features while preserving SAE
performance. Experiments on models such as Stable Diffusion 1.5 show that our
method is able to simultaneously raise performance on five dialects to be on
par with SAE (+34.4%), while incurring near zero cost to SAE performance.
A new benchmark and encoder-based mitigation strategy improve multimodal generative models' performance on dialectal textual input without degrading performance on Standard American English.
multimodal generative models dialectal textual input large-scale benchmark dialect speakers image and video generative models performance degradation fine-tuning prompt rewriting encoder-based mitigation strategy Stable Diffusion 1.5