Retrospective Learning from Interactions
Multi-turn interactions between large language models (LLMs) and users
naturally include implicit feedback signals. If an LLM responds in an
unexpected way to an instruction, the user is likely to signal it by rephrasing
the request, expressing frustration, or pivoting to an alternative task. Such
signals are task-independent and occupy a relatively constrained subspace of
language, allowing the LLM to identify them even if it fails on the actual
task. This creates an avenue for continually learning from interactions without
additional annotations. We introduce ReSpect, a method to learn from such
signals in past interactions via retrospection. We deploy ReSpect in a new
multimodal interaction scenario, where humans instruct an LLM to solve an
abstract reasoning task with a combinatorial solution space. Through thousands
of interactions with humans, we show how ReSpect gradually improves task
completion rate from 31% to 82%, all without any external annotation.
6711c3ba7da5c0a86f61f0c5
Zizhao Chen
admin_assigned
2024-10-18T12:54:54.582Z
634642c17393804ce0d83879
/avatars/36ef4df22b9faaf76a8a03e2c55a66e8.svg
zizhao chen
user
czz
6711c3ba7da5c0a86f61f0c6
Mustafa Omer Gul
admin_assigned
2024-10-18T12:55:00.071Z
640261cc06c715b934025f4e
https://cdn-avatars.huggingface.co/v1/production/uploads/640261cc06c715b934025f4e/aXnh-ZGvEahSovuivi6dr.jpeg
Mustafa Omer Gul
user
momergul
6711c3ba7da5c0a86f61f0c7
Yiwei Chen
6711c3ba7da5c0a86f61f0c8
Gloria Geng
6711c3ba7da5c0a86f61f0c9
Anne Wu
admin_assigned
2024-10-18T12:55:59.916Z
64515bda110343638e38b3df
/avatars/a69ccb8cc40a58d49fa907a00a1397fd.svg
Anne Wu
user
anneyouw
6711c3ba7da5c0a86f61f0ca
Yoav Artzi
admin_assigned
2024-10-18T12:56:05.282Z
637c23489495870ef76d7880
/avatars/4fd2f23d9a487b1ec0c749c1e11b8ffc.svg
Yoav Artzi
user
yoavartzi
Retrospective Learning from Interactions
Multi-turn interactions between large language models (LLMs) and users
naturally include implicit feedback signals. If an LLM responds in an
unexpected way to an instruction, the user is likely to signal it by rephrasing
the request, expressing frustration, or pivoting to an alternative task. Such
signals are task-independent and occupy a relatively constrained subspace of
language, allowing the LLM to identify them even if it fails on the actual
task. This creates an avenue for continually learning from interactions without
additional annotations. We introduce ReSpect, a method to learn from such
signals in past interactions via retrospection. We deploy ReSpect in a new
multimodal interaction scenario, where humans instruct an LLM to solve an
abstract reasoning task with a combinatorial solution space. Through thousands
of interactions with humans, we show how ReSpect gradually improves task
completion rate from 31% to 82%, all without any external annotation.
ReSpect, a method for learning from implicit feedback signals in interactions, significantly improves an LLM's task completion rate in abstract reasoning without external annotations.
large language models LLMs implicit feedback rephrasing frustration alternative tasks retrospection multimodal interaction abstract reasoning combinatorial solution space task completion rate