Non Verbis, Sed Rebus: Large Language Models are Weak Solvers of Italian
  Rebuses
Rebuses are puzzles requiring constrained multi-step reasoning to identify a
hidden phrase from a set of images and letters. In this work, we introduce a
large collection of verbalized rebuses for the Italian language and use it to
assess the rebus-solving capabilities of state-of-the-art large language
models. While general-purpose systems such as LLaMA-3 and GPT-4o perform poorly
on this task, ad-hoc fine-tuning seems to improve models' performance. However,
we find that performance gains from training are largely motivated by
memorization. Our results suggest that rebus solving remains a challenging test
bed to evaluate large language models' linguistic proficiency and sequential
instruction-following skills.
66ac7b62c019f71636813668
Gabriele Sarti
extracted_confirmed
2024-08-02T06:29:32.764Z
5e7749883d77a72421292d07
https://cdn-avatars.huggingface.co/v1/production/uploads/5e7749883d77a72421292d07/M4AmBReZk_otxCIG3o0bL.jpeg
Gabriele Sarti
user
gsarti
66ac7b62c019f71636813669
Tommaso Caselli
extracted_pending
2024-08-02T06:23:31.564Z
60cde5cd4061635e43b28f44
https://cdn-avatars.huggingface.co/v1/production/uploads/60cde5cd4061635e43b28f44/f9JlUdh39gfoCnESqMtic.jpeg
Tommaso Caselli
user
tommasoc
66ac7b62c019f7163681366a
Malvina Nissim
extracted_pending
2024-08-02T06:23:31.564Z
6246f49c0569474a9623e05a
https://cdn-avatars.huggingface.co/v1/production/uploads/1648817396108-6246f49c0569474a9623e05a.jpeg
Malvina Nissim
user
mnissim
66ac7b62c019f7163681366b
Arianna Bisazza
admin_assigned
2024-08-02T13:33:08.481Z
61e050c73a1781f66b4e9ab6
/avatars/9c91a18cdc53587422311fd13a14833e.svg
Arianna Bisazza
user
arianna-bis
Non Verbis, Sed Rebus: Large Language Models are Weak Solvers of Italian
  Rebuses
Rebuses are puzzles requiring constrained multi-step reasoning to identify a
hidden phrase from a set of images and letters. In this work, we introduce a
large collection of verbalized rebuses for the Italian language and use it to
assess the rebus-solving capabilities of state-of-the-art large language
models. While general-purpose systems such as LLaMA-3 and GPT-4o perform poorly
on this task, ad-hoc fine-tuning seems to improve models' performance. However,
we find that performance gains from training are largely motivated by
memorization. Our results suggest that rebus solving remains a challenging test
bed to evaluate large language models' linguistic proficiency and sequential
instruction-following skills.
State-of-the-art language models perform poorly on rebus-solving tasks, with performance improvements primarily attributed to memorization rather than enhanced linguistic skills.
rebus-solving large language models LLaMA-3 GPT-4o fine-tuning memorization linguistic proficiency sequential instruction-following