F-HOI: Toward Fine-grained Semantic-Aligned 3D Human-Object Interactions
Existing 3D human object interaction (HOI) datasets and models simply align
global descriptions with the long HOI sequence, while lacking a detailed
understanding of intermediate states and the transitions between states. In
this paper, we argue that fine-grained semantic alignment, which utilizes
state-level descriptions, offers a promising paradigm for learning semantically
rich HOI representations. To achieve this, we introduce Semantic-HOI, a new
dataset comprising over 20K paired HOI states with fine-grained descriptions
for each HOI state and the body movements that happen between two consecutive
states. Leveraging the proposed dataset, we design three state-level HOI tasks
to accomplish fine-grained semantic alignment within the HOI sequence.
Additionally, we propose a unified model called F-HOI, designed to leverage
multimodal instructions and empower the Multi-modal Large Language Model to
efficiently handle diverse HOI tasks. F-HOI offers multiple advantages: (1) It
employs a unified task formulation that supports the use of versatile
multimodal inputs. (2) It maintains consistency in HOI across 2D, 3D, and
linguistic spaces. (3) It utilizes fine-grained textual supervision for direct
optimization, avoiding intricate modeling of HOI states. Extensive experiments
reveal that F-HOI effectively aligns HOI states with fine-grained semantic
descriptions, adeptly tackling understanding, reasoning, generation, and
reconstruction tasks.
66a05e60007a749299a6efb9
Jie Yang
66a05e60007a749299a6efba
Xuesong Niu
admin_assigned
2024-07-24T08:02:02.291Z
64620ca8a489ecb6b6b5b4e4
/avatars/9bf8060cb7bc59c0369ddf14c50aa480.svg
Xuesong Niu
user
nxsEdson
66a05e60007a749299a6efbb
Nan Jiang
66a05e60007a749299a6efbc
Ruimao Zhang
admin_assigned
2024-07-24T08:04:18.088Z
65324848220f490dbaa3b8c9
/avatars/cec7dcadf4657bb4f61e2c9a8075521b.svg
Zhang
user
Ruimao
66a05e60007a749299a6efbd
Siyuan Huang
admin_assigned
2024-07-24T08:05:05.841Z
63c7a33121bd95f80ed74652
/avatars/7dd59afea785a2bff0ec2b757abd474e.svg
Siyuan Huang
user
thuhsy
F-HOI: Toward Fine-grained Semantic-Aligned 3D Human-Object Interactions
Existing 3D human object interaction (HOI) datasets and models simply align
global descriptions with the long HOI sequence, while lacking a detailed
understanding of intermediate states and the transitions between states. In
this paper, we argue that fine-grained semantic alignment, which utilizes
state-level descriptions, offers a promising paradigm for learning semantically
rich HOI representations. To achieve this, we introduce Semantic-HOI, a new
dataset comprising over 20K paired HOI states with fine-grained descriptions
for each HOI state and the body movements that happen between two consecutive
states. Leveraging the proposed dataset, we design three state-level HOI tasks
to accomplish fine-grained semantic alignment within the HOI sequence.
Additionally, we propose a unified model called F-HOI, designed to leverage
multimodal instructions and empower the Multi-modal Large Language Model to
efficiently handle diverse HOI tasks. F-HOI offers multiple advantages: (1) It
employs a unified task formulation that supports the use of versatile
multimodal inputs. (2) It maintains consistency in HOI across 2D, 3D, and
linguistic spaces. (3) It utilizes fine-grained textual supervision for direct
optimization, avoiding intricate modeling of HOI states. Extensive experiments
reveal that F-HOI effectively aligns HOI states with fine-grained semantic
descriptions, adeptly tackling understanding, reasoning, generation, and
reconstruction tasks.
Semantic-HOI dataset and F-HOI model improve 3D human-object interaction representation by focusing on detailed state-level descriptions and multimodal instructions.
Semantic-HOI F-HOI HOI states fine-grained descriptions multimodal instructions Multi-modal Large Language Model unified task formulation HOI tasks direct optimization