The Bystander Affect Detection (BAD) Dataset for Failure Detection in HRI
For a robot to repair its own error, it must first know it has made a
mistake. One way that people detect errors is from the implicit reactions from
bystanders -- their confusion, smirks, or giggles clue us in that something
unexpected occurred. To enable robots to detect and act on bystander responses
to task failures, we developed a novel method to elicit bystander responses to
human and robot errors. Using 46 different stimulus videos featuring a variety
of human and machine task failures, we collected a total of 2452 webcam videos
of human reactions from 54 participants. To test the viability of the collected
data, we used the bystander reaction dataset as input to a deep-learning model,
BADNet, to predict failure occurrence. We tested different data labeling
methods and learned how they affect model performance, achieving precisions
above 90%. We discuss strategies to model bystander reactions and predict
failure and how this approach can be used in real-world robotic deployments to
detect errors and improve robot performance. As part of this work, we also
contribute with the "Bystander Affect Detection" (BAD) dataset of bystander
reactions, supporting the development of better prediction models.