Policy Improvement using Language Feedback Models
We introduce Language Feedback Models (LFMs) that identify desirable
behaviour - actions that help achieve tasks specified in the instruction - for
imitation learning in instruction following. To train LFMs, we obtain feedback
from Large Language Models (LLMs) on visual trajectories verbalized to language
descriptions. First, by using LFMs to identify desirable behaviour to imitate,
we improve in task-completion rate over strong behavioural cloning baselines on
three distinct language grounding environments (Touchdown, ScienceWorld, and
ALFWorld). Second, LFMs outperform using LLMs as experts to directly predict
actions, when controlling for the number of LLM output tokens. Third, LFMs
generalize to unseen environments, improving task-completion rate by 3.5-12.0%
through one round of adaptation. Finally, LFM can be modified to provide
human-interpretable feedback without performance loss, allowing human
verification of desirable behaviour for imitation learning.
65cae68b5495933ab08201e0
Victor Zhong
admin_assigned
2024-02-13T08:22:34.814Z
64b7f8ad324038715956501e
https://cdn-avatars.huggingface.co/v1/production/uploads/64b7f8ad324038715956501e/FS8mWUV4zk4r2TsQuDFpE.jpeg
Victor Zhong
user
vzhong
65cae68b5495933ab08201e1
Dipendra Misra
65cae68b5495933ab08201e2
Xingdi Yuan
65cae68b5495933ab08201e3
Marc-Alexandre Côté
admin_assigned
2024-02-13T08:23:13.012Z
62960afb91760de5e547e544
/avatars/bebed5926b04d066d4c0e49d246acd78.svg
Marc-Alexandre Côté
user
MarcCote
Policy Improvement using Language Feedback Models
We introduce Language Feedback Models (LFMs) that identify desirable
behaviour - actions that help achieve tasks specified in the instruction - for
imitation learning in instruction following. To train LFMs, we obtain feedback
from Large Language Models (LLMs) on visual trajectories verbalized to language
descriptions. First, by using LFMs to identify desirable behaviour to imitate,
we improve in task-completion rate over strong behavioural cloning baselines on
three distinct language grounding environments (Touchdown, ScienceWorld, and
ALFWorld). Second, LFMs outperform using LLMs as experts to directly predict
actions, when controlling for the number of LLM output tokens. Third, LFMs
generalize to unseen environments, improving task-completion rate by 3.5-12.0%
through one round of adaptation. Finally, LFM can be modified to provide
human-interpretable feedback without performance loss, allowing human
verification of desirable behaviour for imitation learning.
Language Feedback Models (LFMs) enhance task completion rates in instruction following by identifying desirable behaviors from visual trajectories, surpassing direct action prediction by Large Language Models (LLMs) and generalizing to new environments.
Language Feedback Models LFMs Large Language Models LLMs imitation learning instruction following task-completion rate behavioral cloning visual trajectories language descriptions generalization human-interpretable feedback