Counterfactual Reasoning for Steerable Pluralistic Value Alignment in Large Language Models
Hanze Guo, Jing Yao, Xiao Zhou, Xiaoyuan Yi, Xing Xie
As large language models (LLMs) become increasingly integrated into applications serving users across diverse cultures, communities and demographics, it is critical to align LLMs with pluralistic human values beyond average norms (e.g. HHH). In psychological and social value theories such as Schwartz’s Value Theory, each value profile is represented by multi-dimensional values paired with priorities. However, existing methods encounter two challenges when aligning with such fine-grained value objectives: they often treat multiple values as independent and equally important, ignoring their interdependence and relative priorities (value complexity); they struggle to precisely control nuanced value priorities, especially those underrepresented ones (value steerability). To handle these challenges, we propose COUPLE, a COUnterfactual reasoning framework for PLuralistic valuE alignment. It introduces a Structural Causal Model (SCM) to feature the causal relationship between high-level value dimensions and behaviors. Moreover, it applies counterfactual reasoning to generate a new output aligned with any desired value objective. Benefitting from explicit causal modeling, COUPLE also provides better interpretability.  We evaluate COUPLE on two datasets with different value systems and demonstrate that COUPLE advances other baselines across diverse types of value objectives. Our code is available at the anonymous repository: https://anonymous.4open.science/r/COUPLE-7D83.