Increasing Diversity While Maintaining Accuracy: Text Data Generation
  with Large Language Models and Human Interventions
Large language models (LLMs) can be used to generate text data for training
and evaluating other models. However, creating high-quality datasets with LLMs
can be challenging. In this work, we explore human-AI partnerships to
facilitate high diversity and accuracy in LLM-based text data generation. We
first examine two approaches to diversify text generation: 1) logit
suppression, which minimizes the generation of languages that have already been
frequently generated, and 2) temperature sampling, which flattens the token
sampling probability. We found that diversification approaches can increase
data diversity but often at the cost of data accuracy (i.e., text and labels
being appropriate for the target domain). To address this issue, we examined
two human interventions, 1) label replacement (LR), correcting misaligned
labels, and 2) out-of-scope filtering (OOSF), removing instances that are out
of the user's domain of interest or to which no considered label applies. With
oracle studies, we found that LR increases the absolute accuracy of models
trained with diversified datasets by 14.4%. Moreover, we found that some models
trained with data generated with LR interventions outperformed LLM-based
few-shot classification. In contrast, OOSF was not effective in increasing
model accuracy, implying the need for future work in human-in-the-loop text
data generation.
64812ec63fb124fc984fdcda
John Joon Young Chung
extracted_confirmed
2023-06-08T01:29:28.157Z
630c92f6c169245d78fcfa19
/avatars/a4dd46b1b61f43530a1c934de29420af.svg
John Joon Young Chung
user
johnr0
64812ec63fb124fc984fdcdb
Ece Kamar
64812ec63fb124fc984fdcdc
Saleema Amershi
extracted_pending
2024-06-02T22:47:10.572Z
665cf655d2b102a5bbb21150
/avatars/9cd4926132844f1900aae9b2903e0f37.svg
Saleema Amershi
user
samershi
Increasing Diversity While Maintaining Accuracy: Text Data Generation
  with Large Language Models and Human Interventions
Large language models (LLMs) can be used to generate text data for training
and evaluating other models. However, creating high-quality datasets with LLMs
can be challenging. In this work, we explore human-AI partnerships to
facilitate high diversity and accuracy in LLM-based text data generation. We
first examine two approaches to diversify text generation: 1) logit
suppression, which minimizes the generation of languages that have already been
frequently generated, and 2) temperature sampling, which flattens the token
sampling probability. We found that diversification approaches can increase
data diversity but often at the cost of data accuracy (i.e., text and labels
being appropriate for the target domain). To address this issue, we examined
two human interventions, 1) label replacement (LR), correcting misaligned
labels, and 2) out-of-scope filtering (OOSF), removing instances that are out
of the user's domain of interest or to which no considered label applies. With
oracle studies, we found that LR increases the absolute accuracy of models
trained with diversified datasets by 14.4%. Moreover, we found that some models
trained with data generated with LR interventions outperformed LLM-based
few-shot classification. In contrast, OOSF was not effective in increasing
model accuracy, implying the need for future work in human-in-the-loop text
data generation.
Human-AI partnerships improve text data generation quality and accuracy through label replacement, while diversification introduces some accuracy trade-offs.
logit suppression temperature sampling label replacement out-of-scope filtering