Fantastic Questions and Where to Find Them: FairytaleQA -- An Authentic Dataset for Narrative Comprehension
Question answering (QA) is a fundamental means to facilitate assessment and
training of narrative comprehension skills for both machines and young
children, yet there is scarcity of high-quality QA datasets carefully designed
to serve this purpose. In particular, existing datasets rarely distinguish
fine-grained reading skills, such as the understanding of varying narrative
elements. Drawing on the reading education research, we introduce FairytaleQA,
a dataset focusing on narrative comprehension of kindergarten to eighth-grade
students. Generated by educational experts based on an evidence-based
theoretical framework, FairytaleQA consists of 10,580 explicit and implicit
questions derived from 278 children-friendly stories, covering seven types of
narrative elements or relations. Our dataset is valuable in two folds: First,
we ran existing QA models on our dataset and confirmed that this annotation
helps assess models' fine-grained learning skills. Second, the dataset supports
question generation (QG) task in the education domain. Through benchmarking
with QG models, we show that the QG model trained on FairytaleQA is capable of
asking high-quality and more diverse questions.