Learning to Generate Human-Human-Object Interactions from Textual Descriptions
Jeonghyeon Na, Sangwon Baik, Inhee Lee, Junyoung Lee, Hanbyul Joo
The way humans interact with each other, including interpersonal distances, spatial configuration, and motion, varies significantly across different situations. To enable machines to understand such complex, context-dependent behaviors, it is essential to model multiple people in relation to the surrounding scene context.In this paper, we present a novel research problem to model the correlations between two people engaged in a shared interaction scenario defined by an object. We refer to this formulation as Human-Human-Object Interactions (HHOIs).To overcome the lack of dedicated datasets for HHOIs, we present a newly captured HHOIs dataset and a method to synthesize HHOI data by leveraging image generative models. Using these data sources, we train a score-based diffusion model to integrate the two modalities into a unified generative framework capable of synthesizing complete HHOIs in a single sampling process.Experimental results show that our method generates realistic multi-person scenes conditioned on both object and textual descriptions, outperforming previous approaches that focus only on single-person HOIs.Finally, we present an extension toward text-guided generation of multi-person interactions beyond dyads.