Can Large Reasoning Models do Analogical Reasoning under Perceptual
  Uncertainty?
This work presents a first evaluation of two state-of-the-art Large Reasoning
Models (LRMs), OpenAI's o3-mini and DeepSeek R1, on analogical reasoning,
focusing on well-established nonverbal human IQ tests based on Raven's
progressive matrices. We benchmark with the I-RAVEN dataset and its more
difficult extension, I-RAVEN-X, which tests the ability to generalize to longer
reasoning rules and ranges of the attribute values. To assess the influence of
visual uncertainties on these nonverbal analogical reasoning tests, we extend
the I-RAVEN-X dataset, which otherwise assumes an oracle perception. We adopt a
two-fold strategy to simulate this imperfect visual perception: 1) we introduce
confounding attributes which, being sampled at random, do not contribute to the
prediction of the correct answer of the puzzles and 2) smoothen the
distributions of the input attributes' values. We observe a sharp decline in
OpenAI's o3-mini task accuracy, dropping from 86.6% on the original I-RAVEN to
just 17.0% -- approaching random chance -- on the more challenging I-RAVEN-X,
which increases input length and range and emulates perceptual uncertainty.
This drop occurred despite spending 3.4x more reasoning tokens. A similar trend
is also observed for DeepSeek R1: from 80.6% to 23.2%. On the other hand, a
neuro-symbolic probabilistic abductive model, ARLC, that achieves
state-of-the-art performances on I-RAVEN, can robustly reason under all these
out-of-distribution tests, maintaining strong accuracy with only a modest
reduction from 98.6% to 88.0%. Our code is available at
https://github.com/IBM/raven-large-language-models.
67d82c6e6d6baf84fba32ea1
Giacomo Camposampiero
admin_assigned
2025-03-17T15:07:03.399Z
6395beb3b5af054482d87456
/avatars/b24ae0d2e3fea8ad716d1b7225d9e722.svg
Giacomo Camposampiero
user
gcamposampie
67d82c6e6d6baf84fba32ea2
Michael Hersche
extracted_pending
2025-03-17T14:06:39.558Z
668f96ca35cddb688ad9f2cf
/avatars/eb4febcf47b424732ee0a73a81c3b6dd.svg
Michael Hersche
user
mihoesche
67d82c6e6d6baf84fba32ea3
Roger Wattenhofer
extracted_confirmed
2025-05-06T12:49:07.228Z
64afd0c109d6573c1d89dbf7
/avatars/ed6bd7133517590f46847a0930d7a049.svg
Roger Wattenhofer
user
Rogrrr
67d82c6e6d6baf84fba32ea4
Abu Sebastian
67d82c6e6d6baf84fba32ea5
Abbas Rahimi
Can Large Reasoning Models do Analogical Reasoning under Perceptual
  Uncertainty?
This work presents a first evaluation of two state-of-the-art Large Reasoning
Models (LRMs), OpenAI's o3-mini and DeepSeek R1, on analogical reasoning,
focusing on well-established nonverbal human IQ tests based on Raven's
progressive matrices. We benchmark with the I-RAVEN dataset and its more
difficult extension, I-RAVEN-X, which tests the ability to generalize to longer
reasoning rules and ranges of the attribute values. To assess the influence of
visual uncertainties on these nonverbal analogical reasoning tests, we extend
the I-RAVEN-X dataset, which otherwise assumes an oracle perception. We adopt a
two-fold strategy to simulate this imperfect visual perception: 1) we introduce
confounding attributes which, being sampled at random, do not contribute to the
prediction of the correct answer of the puzzles and 2) smoothen the
distributions of the input attributes' values. We observe a sharp decline in
OpenAI's o3-mini task accuracy, dropping from 86.6% on the original I-RAVEN to
just 17.0% -- approaching random chance -- on the more challenging I-RAVEN-X,
which increases input length and range and emulates perceptual uncertainty.
This drop occurred despite spending 3.4x more reasoning tokens. A similar trend
is also observed for DeepSeek R1: from 80.6% to 23.2%. On the other hand, a
neuro-symbolic probabilistic abductive model, ARLC, that achieves
state-of-the-art performances on I-RAVEN, can robustly reason under all these
out-of-distribution tests, maintaining strong accuracy with only a modest
reduction from 98.6% to 88.0%. Our code is available at
https://github.com/IBM/raven-large-language-models.
State-of-the-art Large Reasoning Models, o3-mini and DeepSeek R1, significantly decline in analogical reasoning accuracy under perceptual uncertainty compared to a neuro-symbolic probabilistic abductive model, ARLC, which maintains robust performance.
Large Reasoning Models LRMs OpenAI's o3-mini DeepSeek R1 analogical reasoning Raven's progressive matrices I-RAVEN dataset I-RAVEN-X visual uncertainties confounding attributes neuro-symbolic probabilistic abductive model ARLC perceptual uncertainty