A Tale of Trust and Accuracy: Base vs. Instruct LLMs in RAG Systems
Retrieval Augmented Generation (RAG) represents a significant advancement in
artificial intelligence combining a retrieval phase with a generative phase,
with the latter typically being powered by large language models (LLMs). The
current common practices in RAG involve using "instructed" LLMs, which are
fine-tuned with supervised training to enhance their ability to follow
instructions and are aligned with human preferences using state-of-the-art
techniques. Contrary to popular belief, our study demonstrates that base models
outperform their instructed counterparts in RAG tasks by 20% on average under
our experimental settings. This finding challenges the prevailing assumptions
about the superiority of instructed LLMs in RAG applications. Further
investigations reveal a more nuanced situation, questioning fundamental aspects
of RAG and suggesting the need for broader discussions on the topic; or, as
Fromm would have it, "Seldom is a glance at the statistics enough to understand
the meaning of the figures".
6679223125771fc36d9ac38b
Florin Cuconasu
claimed_verified
2024-06-24T13:10:24.561Z
62bfff6788fdef8ecde8c45b
/avatars/637da88b8dce01bdc2a4c1319cc882e3.svg
Florin Cuconasu
user
florin-hf
6679223125771fc36d9ac38c
Giovanni Trappolini
admin_assigned
2024-06-24T14:02:18.536Z
631e51736d6a5870f3d7d0fe
/avatars/9eb93a278d01c3abd6dc28c708f0d107.svg
Giovanni Trappolini
user
rarinpear
6679223125771fc36d9ac38d
Nicola Tonellotto
admin_assigned
2024-06-24T14:02:23.972Z
66505657f329f5aa4e1e7466
/avatars/f5af1279f6761c2a12cf7fe4781301ad.svg
Nicola Tonellotto
user
ntonellotto
6679223125771fc36d9ac38e
Fabrizio Silvestri
admin_assigned
2024-06-24T14:02:29.026Z
6075d282134c000d1ae10d54
/avatars/8691622e1f6325bc378b2268c119b9a0.svg
Fabrizio Silvestri
user
fabrizio-silvestri
A Tale of Trust and Accuracy: Base vs. Instruct LLMs in RAG Systems
Retrieval Augmented Generation (RAG) represents a significant advancement in
artificial intelligence combining a retrieval phase with a generative phase,
with the latter typically being powered by large language models (LLMs). The
current common practices in RAG involve using "instructed" LLMs, which are
fine-tuned with supervised training to enhance their ability to follow
instructions and are aligned with human preferences using state-of-the-art
techniques. Contrary to popular belief, our study demonstrates that base models
outperform their instructed counterparts in RAG tasks by 20% on average under
our experimental settings. This finding challenges the prevailing assumptions
about the superiority of instructed LLMs in RAG applications. Further
investigations reveal a more nuanced situation, questioning fundamental aspects
of RAG and suggesting the need for broader discussions on the topic; or, as
Fromm would have it, "Seldom is a glance at the statistics enough to understand
the meaning of the figures".
Retrieval Augmented Generation (RAG) large language models (LLMs) instructed LLMs fine-tuned supervised training human preferences base models instructed counterparts RAG tasks experimental settings