FastHMR: Accelerating Human Mesh Recovery via Token and Layer Merging
  with Diffusion Decoding
Recent transformer-based models for 3D Human Mesh Recovery (HMR) have
achieved strong performance but often suffer from high computational cost and
complexity due to deep transformer architectures and redundant tokens. In this
paper, we introduce two HMR-specific merging strategies: Error-Constrained
Layer Merging (ECLM) and Mask-guided Token Merging (Mask-ToMe). ECLM
selectively merges transformer layers that have minimal impact on the Mean Per
Joint Position Error (MPJPE), while Mask-ToMe focuses on merging background
tokens that contribute little to the final prediction. To further address the
potential performance drop caused by merging, we propose a diffusion-based
decoder that incorporates temporal context and leverages pose priors learned
from large-scale motion capture datasets. Experiments across multiple
benchmarks demonstrate that our method achieves up to 2.3x speed-up while
slightly improving performance over the baseline.
68edbe4bde1fee572713a86f
Soroush Mehraban
68edbe4bde1fee572713a870
Andrea Iaboni
68edbe4bde1fee572713a871
Babak Taati
FastHMR: Accelerating Human Mesh Recovery via Token and Layer Merging
  with Diffusion Decoding
Recent transformer-based models for 3D Human Mesh Recovery (HMR) have
achieved strong performance but often suffer from high computational cost and
complexity due to deep transformer architectures and redundant tokens. In this
paper, we introduce two HMR-specific merging strategies: Error-Constrained
Layer Merging (ECLM) and Mask-guided Token Merging (Mask-ToMe). ECLM
selectively merges transformer layers that have minimal impact on the Mean Per
Joint Position Error (MPJPE), while Mask-ToMe focuses on merging background
tokens that contribute little to the final prediction. To further address the
potential performance drop caused by merging, we propose a diffusion-based
decoder that incorporates temporal context and leverages pose priors learned
from large-scale motion capture datasets. Experiments across multiple
benchmarks demonstrate that our method achieves up to 2.3x speed-up while
slightly improving performance over the baseline.
Two merging strategies and a diffusion-based decoder improve 3D Human Mesh Recovery by reducing computational cost and slightly enhancing performance.
transformer-based models 3D Human Mesh Recovery Error-Constrained Layer Merging Mask-guided Token Merging Mean Per Joint Position Error diffusion-based decoder temporal context pose priors motion capture datasets