Exploring the Abilities of Large Language Models to Solve Proportional
  Analogies via Knowledge-Enhanced Prompting
Making analogies is fundamental to cognition. Proportional analogies, which
consist of four terms, are often used to assess linguistic and cognitive
abilities. For instance, completing analogies like "Oxygen is to Gas as <blank>
is to <blank>" requires identifying the semantic relationship (e.g., "type of")
between the first pair of terms ("Oxygen" and "Gas") and finding a second pair
that shares the same relationship (e.g., "Aluminum" and "Metal"). In this work,
we introduce a 15K Multiple-Choice Question Answering (MCQA) dataset for
proportional analogy completion and evaluate the performance of contemporary
Large Language Models (LLMs) in various knowledge-enhanced prompt settings.
Specifically, we augment prompts with three types of knowledge: exemplar,
structured, and targeted. Our results show that despite extensive training
data, solving proportional analogies remains challenging for current LLMs, with
the best model achieving an accuracy of 55%. Notably, we find that providing
targeted knowledge can better assist models in completing proportional
analogies compared to providing exemplars or collections of structured
knowledge.
674f5e7fada64263fc369469
Thilini Wijesiriwardene
674f5e7fada64263fc36946a
Ruwan Wickramarachchi
674f5e7fada64263fc36946b
Sreeram Vennam
674f5e7fada64263fc36946c
Vinija Jain
674f5e7fada64263fc36946d
Aman Chadha
claimed_verified
2024-12-04T09:46:13.632Z
63a4754927f1f64ed7238dac
https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg
Aman Chadha
user
amanchadha
674f5e7fada64263fc36946e
Amitava Das
674f5e7fada64263fc36946f
Ponnurangam Kumaraguru
claimed_verified
2025-02-10T09:51:19.013Z
67a6a4b7f379cef464950268
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/yaMsT2jYsj6mvvYU1gvO_.jpeg
ponnurangam kumaraguru
user
pk-profgiri
674f5e7fada64263fc369470
Amit Sheth
Exploring the Abilities of Large Language Models to Solve Proportional
  Analogies via Knowledge-Enhanced Prompting
Making analogies is fundamental to cognition. Proportional analogies, which
consist of four terms, are often used to assess linguistic and cognitive
abilities. For instance, completing analogies like "Oxygen is to Gas as <blank>
is to <blank>" requires identifying the semantic relationship (e.g., "type of")
between the first pair of terms ("Oxygen" and "Gas") and finding a second pair
that shares the same relationship (e.g., "Aluminum" and "Metal"). In this work,
we introduce a 15K Multiple-Choice Question Answering (MCQA) dataset for
proportional analogy completion and evaluate the performance of contemporary
Large Language Models (LLMs) in various knowledge-enhanced prompt settings.
Specifically, we augment prompts with three types of knowledge: exemplar,
structured, and targeted. Our results show that despite extensive training
data, solving proportional analogies remains challenging for current LLMs, with
the best model achieving an accuracy of 55%. Notably, we find that providing
targeted knowledge can better assist models in completing proportional
analogies compared to providing exemplars or collections of structured
knowledge.
LLMs struggle with completing proportional analogies, even with augmented knowledge, with targeted knowledge proving most beneficial.
Large Language Models MCQA dataset proportional analogies exemplar structured targeted knowledge