GuideFlow3D: Optimization-Guided Rectified Flow For Appearance Transfer
Transferring appearance to 3D assets using different representations of the
appearance object - such as images or text - has garnered interest due to its
wide range of applications in industries like gaming, augmented reality, and
digital content creation. However, state-of-the-art methods still fail when the
geometry between the input and appearance objects is significantly different. A
straightforward approach is to directly apply a 3D generative model, but we
show that this ultimately fails to produce appealing results. Instead, we
propose a principled approach inspired by universal guidance. Given a
pretrained rectified flow model conditioned on image or text, our training-free
method interacts with the sampling process by periodically adding guidance.
This guidance can be modeled as a differentiable loss function, and we
experiment with two different types of guidance including part-aware losses for
appearance and self-similarity. Our experiments show that our approach
successfully transfers texture and geometric details to the input 3D asset,
outperforming baselines both qualitatively and quantitatively. We also show
that traditional metrics are not suitable for evaluating the task due to their
inability of focusing on local details and comparing dissimilar inputs, in
absence of ground truth data. We thus evaluate appearance transfer quality with
a GPT-based system objectively ranking outputs, ensuring robust and human-like
assessment, as further confirmed by our user study. Beyond showcased scenarios,
our method is general and could be extended to different types of diffusion
models and guidance functions.
68f7229e24c44893631119eb
Sayan Deb Sarkar
claimed_verified
2025-10-21T14:02:55.826Z
650ec19e6620b0c57e2a551b
/avatars/c26c03fa920d857120f03c9ccb9f1d7a.svg
Sayan Deb Sarkar
user
sayandsarkar
68f7229e24c44893631119ec
Sinisa Stekovic
68f7229e24c44893631119ed
Vincent Lepetit
68f7229e24c44893631119ee
Iro Armeni
GuideFlow3D: Optimization-Guided Rectified Flow For Appearance Transfer
Transferring appearance to 3D assets using different representations of the
appearance object - such as images or text - has garnered interest due to its
wide range of applications in industries like gaming, augmented reality, and
digital content creation. However, state-of-the-art methods still fail when the
geometry between the input and appearance objects is significantly different. A
straightforward approach is to directly apply a 3D generative model, but we
show that this ultimately fails to produce appealing results. Instead, we
propose a principled approach inspired by universal guidance. Given a
pretrained rectified flow model conditioned on image or text, our training-free
method interacts with the sampling process by periodically adding guidance.
This guidance can be modeled as a differentiable loss function, and we
experiment with two different types of guidance including part-aware losses for
appearance and self-similarity. Our experiments show that our approach
successfully transfers texture and geometric details to the input 3D asset,
outperforming baselines both qualitatively and quantitatively. We also show
that traditional metrics are not suitable for evaluating the task due to their
inability of focusing on local details and comparing dissimilar inputs, in
absence of ground truth data. We thus evaluate appearance transfer quality with
a GPT-based system objectively ranking outputs, ensuring robust and human-like
assessment, as further confirmed by our user study. Beyond showcased scenarios,
our method is general and could be extended to different types of diffusion
models and guidance functions.
A method using pretrained rectified flow models with periodic guidance successfully transfers appearance and geometric details to 3D assets, outperforming baselines and evaluated using a GPT-based system.
rectified flow model guidance differentiable loss function part-aware losses self-similarity GPT-based system diffusion models