LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction
  Tuning
Generating instructional images of human daily actions from an egocentric
viewpoint serves a key step towards efficient skill transfer. In this paper, we
introduce a novel problem -- egocentric action frame generation. The goal is to
synthesize the action frame conditioning on the user prompt question and an
input egocentric image that captures user's environment. Notably, existing
egocentric datasets lack the detailed annotations that describe the execution
of actions. Additionally, the diffusion-based image manipulation models fail to
control the state change of an action within the corresponding egocentric image
pixel space. To this end, we finetune a visual large language model (VLLM) via
visual instruction tuning for curating the enriched action descriptions to
address our proposed problem. Moreover, we propose to Learn EGOcentric (LEGO)
action frame generation using image and text embeddings from VLLM as additional
conditioning. We validate our proposed model on two egocentric datasets --
Ego4D and Epic-Kitchens. Our experiments show prominent improvement over prior
image manipulation models in both quantitative and qualitative evaluation. We
also conduct detailed ablation studies and analysis to provide insights on our
method.
65727c32e887face3eb87572
Bolin Lai
extracted_confirmed
2023-12-08T18:56:59.365Z
64dbcdaa70446182be84353e
https://cdn-avatars.huggingface.co/v1/production/uploads/64dbcdaa70446182be84353e/wjzSeLKOdg3krOlkn24ea.jpeg
Bolin Lai
user
bolinlai
65727c32e887face3eb87573
Xiaoliang Dai
admin_assigned
2023-12-08T11:03:52.545Z
6549417b3ce45eb764faf993
/avatars/d310f475d0697f5f13b3d4141ea0ccaf.svg
Xiaoliang Dai
user
daixl1992
65727c32e887face3eb87574
Lawrence Chen
admin_assigned
2023-12-08T11:05:21.721Z
6303ff0c7b50dd9d0a3777e8
/avatars/1c65d5936fe0cafd1fadd1102fad28db.svg
Lawrence Chen
user
lawrencechen
65727c32e887face3eb87575
Guan Pang
claimed_verified
2023-12-11T08:16:41.294Z
6573a458d7f487de5f9bb44a
/avatars/4aa5626af43855ad770186f0d4191a01.svg
Guan Pang
user
tbpangolin
65727c32e887face3eb87576
James M. Rehg
65727c32e887face3eb87577
Miao Liu
LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction
  Tuning
Generating instructional images of human daily actions from an egocentric
viewpoint serves a key step towards efficient skill transfer. In this paper, we
introduce a novel problem -- egocentric action frame generation. The goal is to
synthesize the action frame conditioning on the user prompt question and an
input egocentric image that captures user's environment. Notably, existing
egocentric datasets lack the detailed annotations that describe the execution
of actions. Additionally, the diffusion-based image manipulation models fail to
control the state change of an action within the corresponding egocentric image
pixel space. To this end, we finetune a visual large language model (VLLM) via
visual instruction tuning for curating the enriched action descriptions to
address our proposed problem. Moreover, we propose to Learn EGOcentric (LEGO)
action frame generation using image and text embeddings from VLLM as additional
conditioning. We validate our proposed model on two egocentric datasets --
Ego4D and Epic-Kitchens. Our experiments show prominent improvement over prior
image manipulation models in both quantitative and qualitative evaluation. We
also conduct detailed ablation studies and analysis to provide insights on our
method.
A model for generating egocentric action frames from user prompts and images using a fine-tuned visual large language model and learned embeddings improves upon existing image manipulation models.
visual large language model visual instruction tuning egocentric action frame generation action frame synthesis enriched action descriptions image embeddings text embeddings Ego4D Epic-Kitchens