UniF^2ace: Fine-grained Face Understanding and Generation
  with Unified Multimodal Models
Unified multimodal models (UMMs) have emerged as a powerful paradigm in
foundational computer vision research, demonstrating significant potential in
both image understanding and generation. However, existing research in the face
domain primarily focuses on coarse facial attribute understanding,
with limited capacity to handle fine-grained facial attributes and
without addressing generation capabilities. To overcome these limitations, we
propose UniF^2ace, the first UMM tailored specifically for
fine-grained face understanding and generation. In general, we train
UniF^2ace on a self-constructed, specialized dataset utilizing two
mutually beneficial diffusion techniques and a two-level mixture-of-experts
architecture. Specifically, we first build a large-scale facial dataset,
UniF^2ace-130K, which contains 130K image-text pairs with one
million question-answering pairs that span a wide range of facial attributes.
Second, we establish a theoretical connection between discrete diffusion score
matching and masked generative models, optimizing both evidence lower bounds
simultaneously, which significantly improves the model's ability to synthesize
facial details. Finally, we introduce both token-level and sequence-level
mixture-of-experts, enabling efficient fine-grained representation learning for
both understanding and generation tasks. Extensive experiments on
UniF^2ace-130K demonstrate that UniF^2ace outperforms
existing UMMs and generative models, achieving superior performance across both
understanding and generation tasks.
67d0f5ace3c8042929eea946
Junzhe Li
claimed_verified
2025-03-12T08:39:26.197Z
64c860d23a3f428da65ea499
/avatars/f0bcc6ae7e558babe691b6bbf1059c9d.svg
lijunzhe
user
tulvgengenr
67d0f5ace3c8042929eea947
Xuerui Qiu
admin_assigned
2025-03-12T15:38:04.042Z
66d98c2bc9758ee602cb2e22
/avatars/cea03f263d5ed292c4e558871112c61c.svg
Qiu
user
Xuerui123
67d0f5ace3c8042929eea948
Linrui Xu
67d0f5ace3c8042929eea949
Liya Guo
admin_assigned
2025-03-12T15:38:31.340Z
67d106a055363e24d8c4511e
/avatars/ac3bd0d9632799e5ccb8fd96bf739e42.svg
Liya Guo
user
LiyaGuo66
67d0f5ace3c8042929eea94a
Delin Qu
claimed_verified
2025-03-12T08:39:29.349Z
64daecec888b7e9c400f59b5
https://cdn-avatars.huggingface.co/v1/production/uploads/64daecec888b7e9c400f59b5/f4pfOfWk6jYJX-Nf2-qHn.png
Delin Qu
user
delinqu
67d0f5ace3c8042929eea94b
Tingting Long
67d0f5ace3c8042929eea94c
Chun Fan
67d0f5ace3c8042929eea94d
Ming Li
claimed_verified
2025-03-12T14:25:54.054Z
63842a11e58a1678ad7669c1
/avatars/556f82d7074734a5829afaf9147d5493.svg
ming li
user
Frankboy
UniF^2ace: Fine-grained Face Understanding and Generation
  with Unified Multimodal Models
Unified multimodal models (UMMs) have emerged as a powerful paradigm in
foundational computer vision research, demonstrating significant potential in
both image understanding and generation. However, existing research in the face
domain primarily focuses on coarse facial attribute understanding,
with limited capacity to handle fine-grained facial attributes and
without addressing generation capabilities. To overcome these limitations, we
propose UniF^2ace, the first UMM tailored specifically for
fine-grained face understanding and generation. In general, we train
UniF^2ace on a self-constructed, specialized dataset utilizing two
mutually beneficial diffusion techniques and a two-level mixture-of-experts
architecture. Specifically, we first build a large-scale facial dataset,
UniF^2ace-130K, which contains 130K image-text pairs with one
million question-answering pairs that span a wide range of facial attributes.
Second, we establish a theoretical connection between discrete diffusion score
matching and masked generative models, optimizing both evidence lower bounds
simultaneously, which significantly improves the model's ability to synthesize
facial details. Finally, we introduce both token-level and sequence-level
mixture-of-experts, enabling efficient fine-grained representation learning for
both understanding and generation tasks. Extensive experiments on
UniF^2ace-130K demonstrate that UniF^2ace outperforms
existing UMMs and generative models, achieving superior performance across both
understanding and generation tasks.
UniF<sup>2</sup>ace is a unified multimodal model that excels in fine-grained face understanding and generation using a two-level mixture-of-experts and diffusion techniques.
unified multimodal models UMMs UniF<sup>2</sup>ace fine-grained face understanding coarse facial attribute understanding UniF<sup>2</sup>ace-130K diffusion techniques discrete diffusion score matching masked generative models evidence lower bounds mixture-of-experts