Can Large Language Models Transform Computational Social Science?
Large Language Models (LLMs) like ChatGPT are capable of successfully
performing many language processing tasks zero-shot (without the need for
training data). If this capacity also applies to the coding of social phenomena
like persuasiveness and political ideology, then LLMs could effectively
transform Computational Social Science (CSS). This work provides a road map for
using LLMs as CSS tools. Towards this end, we contribute a set of prompting
best practices and an extensive evaluation pipeline to measure the zero-shot
performance of 13 language models on 24 representative CSS benchmarks. On
taxonomic labeling tasks (classification), LLMs fail to outperform the best
fine-tuned models but still achieve fair levels of agreement with humans. On
free-form coding tasks (generation), LLMs produce explanations that often
exceed the quality of crowdworkers' gold references. We conclude that today's
LLMs can radically augment the CSS research pipeline in two ways: (1) serving
as zero-shot data annotators on human annotation teams, and (2) bootstrapping
challenging creative generation tasks (e.g., explaining the hidden meaning
behind text). In summary, LLMs can significantly reduce costs and increase
efficiency of social science analysis in partnership with humans.
6458461ef8bdc3512d1f5673
Caleb Ziems
admin_assigned
2023-06-02T13:58:09.891Z
60b95db8337c67f46490d6bb
https://cdn-avatars.huggingface.co/v1/production/uploads/1663610682205-60b95db8337c67f46490d6bb.jpeg
Caleb Ziems
user
cjziems
6458461ef8bdc3512d1f5674
William Held
claimed_verified
2024-03-13T13:27:10.598Z
632116accafe12f481a473cb
https://cdn-avatars.huggingface.co/v1/production/uploads/1666207676653-632116accafe12f481a473cb.jpeg
Will Held
user
WillHeld
6458461ef8bdc3512d1f5675
Omar Shaikh
admin_assigned
2023-06-02T13:58:40.538Z
632601d42a0af3bd4522aed7
/avatars/4c02961e3a944c026c6444094d4642aa.svg
Omar Shaikh
user
oshaikh13
6458461ef8bdc3512d1f5676
Jiaao Chen
admin_assigned
2023-06-02T13:59:13.867Z
6327c45946ffcfc27cfa1ee3
/avatars/f64b36e978fc369bfdfebc16057cd67e.svg
Chen
user
Jiaao
6458461ef8bdc3512d1f5677
Zhehao Zhang
6458461ef8bdc3512d1f5678
Diyi Yang
Can Large Language Models Transform Computational Social Science?
Large Language Models (LLMs) like ChatGPT are capable of successfully
performing many language processing tasks zero-shot (without the need for
training data). If this capacity also applies to the coding of social phenomena
like persuasiveness and political ideology, then LLMs could effectively
transform Computational Social Science (CSS). This work provides a road map for
using LLMs as CSS tools. Towards this end, we contribute a set of prompting
best practices and an extensive evaluation pipeline to measure the zero-shot
performance of 13 language models on 24 representative CSS benchmarks. On
taxonomic labeling tasks (classification), LLMs fail to outperform the best
fine-tuned models but still achieve fair levels of agreement with humans. On
free-form coding tasks (generation), LLMs produce explanations that often
exceed the quality of crowdworkers' gold references. We conclude that today's
LLMs can radically augment the CSS research pipeline in two ways: (1) serving
as zero-shot data annotators on human annotation teams, and (2) bootstrapping
challenging creative generation tasks (e.g., explaining the hidden meaning
behind text). In summary, LLMs can significantly reduce costs and increase
efficiency of social science analysis in partnership with humans.
LLMs can enhance Computational Social Science by serving as zero-shot annotators and bootstrapping creative generation tasks, improving efficiency and reducing costs in social science analysis.
Large Language Models LLMs zero-shot performance prompting best practices evaluation pipeline CSS Computational Social Science taxonomic labeling tasks free-form coding tasks fine-tuned models data annotators human annotation teams creative generation tasks