TrueTeacher: Learning Factual Consistency Evaluation with Large Language
  Models
Factual consistency evaluation is often conducted using Natural Language
Inference (NLI) models, yet these models exhibit limited success in evaluating
summaries. Previous work improved such models with synthetic training data.
However, the data is typically based on perturbed human-written summaries,
which often differ in their characteristics from real model-generated summaries
and have limited coverage of possible factual errors. Alternatively, large
language models (LLMs) have recently shown promising results in directly
evaluating generative tasks, but are too computationally expensive for
practical use. Motivated by these limitations, we introduce TrueTeacher, a
method for generating synthetic data by annotating diverse model-generated
summaries using a LLM. Unlike prior work, TrueTeacher does not rely on
human-written summaries, and is multilingual by nature. Experiments on the TRUE
benchmark show that a student model trained using our data, substantially
outperforms both the state-of-the-art model with similar capacity, and the LLM
teacher. In a systematic study, we compare TrueTeacher to existing synthetic
data generation methods and demonstrate its superiority and robustness to
domain-shift. Using the the mFACE dataset, we also show that our method
generalizes to multilingual scenarios. Finally, we release a large-scale
synthetic dataset with 1.4M examples generated using TrueTeacher.
6466fc375993aa7666cc0a5b
Zorik Gekhman
claimed_verified
2023-05-19T14:33:40.975Z
61499a8f8517f77f09a3a990
/avatars/fa15d117e8ce8c50669455d3e483408b.svg
Zorik
user
zorik
6466fc375993aa7666cc0a5c
Jonathan Herzig
6466fc375993aa7666cc0a5d
Roee Aharoni
6466fc375993aa7666cc0a5e
Chen Elkind
6466fc375993aa7666cc0a5f
Idan Szpektor
TrueTeacher: Learning Factual Consistency Evaluation with Large Language
  Models
Factual consistency evaluation is often conducted using Natural Language
Inference (NLI) models, yet these models exhibit limited success in evaluating
summaries. Previous work improved such models with synthetic training data.
However, the data is typically based on perturbed human-written summaries,
which often differ in their characteristics from real model-generated summaries
and have limited coverage of possible factual errors. Alternatively, large
language models (LLMs) have recently shown promising results in directly
evaluating generative tasks, but are too computationally expensive for
practical use. Motivated by these limitations, we introduce TrueTeacher, a
method for generating synthetic data by annotating diverse model-generated
summaries using a LLM. Unlike prior work, TrueTeacher does not rely on
human-written summaries, and is multilingual by nature. Experiments on the TRUE
benchmark show that a student model trained using our data, substantially
outperforms both the state-of-the-art model with similar capacity, and the LLM
teacher. In a systematic study, we compare TrueTeacher to existing synthetic
data generation methods and demonstrate its superiority and robustness to
domain-shift. Using the the mFACE dataset, we also show that our method
generalizes to multilingual scenarios. Finally, we release a large-scale
synthetic dataset with 1.4M examples generated using TrueTeacher.
TrueTeacher, a method for generating synthetic data using a large language model, improves factual consistency evaluation of summaries compared to existing models and synthetic data techniques.
Natural Language Inference (NLI) large language models (LLMs) synthetic data model-generated summaries factual consistency evaluation TrueTeacher student model TRUE benchmark mFACE dataset multilingual scenarios