FreshLLMs: Refreshing Large Language Models with Search Engine
  Augmentation
Most large language models (LLMs) are trained once and never updated; thus,
they lack the ability to dynamically adapt to our ever-changing world. In this
work, we perform a detailed study of the factuality of LLM-generated text in
the context of answering questions that test current world knowledge.
Specifically, we introduce FreshQA, a novel dynamic QA benchmark encompassing a
diverse range of question and answer types, including questions that require
fast-changing world knowledge as well as questions with false premises that
need to be debunked. We benchmark a diverse array of both closed and
open-source LLMs under a two-mode evaluation procedure that allows us to
measure both correctness and hallucination. Through human evaluations involving
more than 50K judgments, we shed light on limitations of these models and
demonstrate significant room for improvement: for instance, all models
(regardless of model size) struggle on questions that involve fast-changing
knowledge and false premises. Motivated by these results, we present
FreshPrompt, a simple few-shot prompting method that substantially boosts the
performance of an LLM on FreshQA by incorporating relevant and up-to-date
information retrieved from a search engine into the prompt. Our experiments
show that FreshPrompt outperforms both competing search engine-augmented
prompting methods such as Self-Ask (Press et al., 2022) as well as commercial
systems such as Perplexity.AI. Further analysis of FreshPrompt reveals that
both the number of retrieved evidences and their order play a key role in
influencing the correctness of LLM-generated answers. Additionally, instructing
the LLM to generate concise and direct answers helps reduce hallucination
compared to encouraging more verbose answers. To facilitate future work, we
release FreshQA at github.com/freshllms/freshqa and commit to updating it at
regular intervals.
651f65f34376ae05d622d76a
Tu Vu
admin_assigned
2023-10-09T09:05:19.608Z
636828a644e19ccad213063a
/avatars/e1551d4ee9ca94018103af3bf1249cf2.svg
Tu Vu
user
tuvu
651f65f34376ae05d622d76b
Mohit Iyyer
651f65f34376ae05d622d76c
Xuezhi Wang
651f65f34376ae05d622d76d
Noah Constant
651f65f34376ae05d622d76e
Jerry Wei
651f65f34376ae05d622d76f
Jason Wei
651f65f34376ae05d622d770
Chris Tar
claimed_verified
2024-09-26T08:26:54.499Z
623e5bd5d10f369bb5f2d392
https://cdn-avatars.huggingface.co/v1/production/uploads/623e5bd5d10f369bb5f2d392/u3nib56InZsTxPrdYPAd1.jpeg
Chris Tar
user
ctar
651f65f34376ae05d622d771
Yun-Hsuan Sung
651f65f34376ae05d622d772
Denny Zhou
claimed_verified
2024-02-20T18:27:22.740Z
65d4d62d493aeaf0bb295f1b
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vXMxaoWwFziul8TqIhQw8.jpeg
Denny Zhou
user
dennyzhou
651f65f34376ae05d622d773
Quoc Le
651f65f34376ae05d622d774
Thang Luong
FreshLLMs: Refreshing Large Language Models with Search Engine
  Augmentation
Most large language models (LLMs) are trained once and never updated; thus,
they lack the ability to dynamically adapt to our ever-changing world. In this
work, we perform a detailed study of the factuality of LLM-generated text in
the context of answering questions that test current world knowledge.
Specifically, we introduce FreshQA, a novel dynamic QA benchmark encompassing a
diverse range of question and answer types, including questions that require
fast-changing world knowledge as well as questions with false premises that
need to be debunked. We benchmark a diverse array of both closed and
open-source LLMs under a two-mode evaluation procedure that allows us to
measure both correctness and hallucination. Through human evaluations involving
more than 50K judgments, we shed light on limitations of these models and
demonstrate significant room for improvement: for instance, all models
(regardless of model size) struggle on questions that involve fast-changing
knowledge and false premises. Motivated by these results, we present
FreshPrompt, a simple few-shot prompting method that substantially boosts the
performance of an LLM on FreshQA by incorporating relevant and up-to-date
information retrieved from a search engine into the prompt. Our experiments
show that FreshPrompt outperforms both competing search engine-augmented
prompting methods such as Self-Ask (Press et al., 2022) as well as commercial
systems such as Perplexity.AI. Further analysis of FreshPrompt reveals that
both the number of retrieved evidences and their order play a key role in
influencing the correctness of LLM-generated answers. Additionally, instructing
the LLM to generate concise and direct answers helps reduce hallucination
compared to encouraging more verbose answers. To facilitate future work, we
release FreshQA at github.com/freshllms/freshqa and commit to updating it at
regular intervals.
FreshQA evaluates LLMs' factuality and hallucination in responding to current knowledge questions, revealing limitations and demonstrating improvements through FreshPrompt, an augmented prompting method using search engine data.
large language models LLMs dynamic QA benchmark FreshQA fast-changing world knowledge false premises two-mode evaluation human evaluations FreshPrompt Self-Ask Perplexity.AI search engine-augmented prompting