MesaTask: Towards Task-Driven Tabletop Scene Generation via 3D Spatial
  Reasoning
The ability of robots to interpret human instructions and execute
manipulation tasks necessitates the availability of task-relevant tabletop
scenes for training. However, traditional methods for creating these scenes
rely on time-consuming manual layout design or purely randomized layouts, which
are limited in terms of plausibility or alignment with the tasks. In this
paper, we formulate a novel task, namely task-oriented tabletop scene
generation, which poses significant challenges due to the substantial gap
between high-level task instructions and the tabletop scenes. To support
research on such a challenging task, we introduce MesaTask-10K, a large-scale
dataset comprising approximately 10,700 synthetic tabletop scenes with manually
crafted layouts that ensure realistic layouts and intricate inter-object
relations. To bridge the gap between tasks and scenes, we propose a Spatial
Reasoning Chain that decomposes the generation process into object inference,
spatial interrelation reasoning, and scene graph construction for the final 3D
layout. We present MesaTask, an LLM-based framework that utilizes this
reasoning chain and is further enhanced with DPO algorithms to generate
physically plausible tabletop scenes that align well with given task
descriptions. Exhaustive experiments demonstrate the superior performance of
MesaTask compared to baselines in generating task-conforming tabletop scenes
with realistic layouts. Project page is at https://mesatask.github.io/
68da105b0177a6054b013ba7
Jinkun Hao
claimed_verified
2025-09-29T05:52:19.633Z
64edb581067fbb625f893628
/avatars/d59893d6f1f752bb73255bd78b325fe9.svg
hao
user
wuzhi-hao
68da105b0177a6054b013ba8
Naifu Liang
claimed_verified
2025-09-29T11:26:29.306Z
68da238ef2f999edd0fed048
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/57HzxM8PgI18NOXQT7OIS.jpeg
Naifu Liang
user
nfliang
68da105b0177a6054b013ba9
Zhen Luo
68da105b0177a6054b013baa
Xudong Xu
68da105b0177a6054b013bab
Weipeng Zhong
68da105b0177a6054b013bac
Ran Yi
68da105b0177a6054b013bad
Yichen Jin
68da105b0177a6054b013bae
Zhaoyang Lyu
68da105b0177a6054b013baf
Feng Zheng
68da105b0177a6054b013bb0
Lizhuang Ma
68da105b0177a6054b013bb1
Jiangmiao Pang
MesaTask: Towards Task-Driven Tabletop Scene Generation via 3D Spatial
  Reasoning
The ability of robots to interpret human instructions and execute
manipulation tasks necessitates the availability of task-relevant tabletop
scenes for training. However, traditional methods for creating these scenes
rely on time-consuming manual layout design or purely randomized layouts, which
are limited in terms of plausibility or alignment with the tasks. In this
paper, we formulate a novel task, namely task-oriented tabletop scene
generation, which poses significant challenges due to the substantial gap
between high-level task instructions and the tabletop scenes. To support
research on such a challenging task, we introduce MesaTask-10K, a large-scale
dataset comprising approximately 10,700 synthetic tabletop scenes with manually
crafted layouts that ensure realistic layouts and intricate inter-object
relations. To bridge the gap between tasks and scenes, we propose a Spatial
Reasoning Chain that decomposes the generation process into object inference,
spatial interrelation reasoning, and scene graph construction for the final 3D
layout. We present MesaTask, an LLM-based framework that utilizes this
reasoning chain and is further enhanced with DPO algorithms to generate
physically plausible tabletop scenes that align well with given task
descriptions. Exhaustive experiments demonstrate the superior performance of
MesaTask compared to baselines in generating task-conforming tabletop scenes
with realistic layouts. Project page is at https://mesatask.github.io/
MesaTask, an LLM-based framework with a Spatial Reasoning Chain, generates realistic tabletop scenes aligned with task descriptions using DPO algorithms.
LLM-based framework Spatial Reasoning Chain object inference spatial interrelation reasoning scene graph construction DPO algorithms