LogicInference: A New Dataset for Teaching Logical Inference to seq2seq Models
Machine learning models such as Transformers or LSTMs struggle with tasks
that are compositional in nature such as those involving reasoning/inference.
Although many datasets exist to evaluate compositional generalization, when it
comes to evaluating inference abilities, options are more limited. This paper
presents LogicInference, a new dataset to evaluate the ability of models to
perform logical inference. The dataset focuses on inference using propositional
logic and a small subset of first-order logic, represented both in semi-formal
logical notation, as well as in natural language. We also report initial
results using a collection of machine learning models to establish an initial
baseline in this dataset.