Understanding Embedding Scaling in Collaborative Filtering
Scaling recommendation models into large recommendation models has become one
of the most widely discussed topics. Recent efforts focus on components beyond
the scaling embedding dimension, as it is believed that scaling embedding may
lead to performance degradation. Although there have been some initial
observations on embedding, the root cause of their non-scalability remains
unclear. Moreover, whether performance degradation occurs across different
types of models and datasets is still an unexplored area. Regarding the effect
of embedding dimensions on performance, we conduct large-scale experiments
across 10 datasets with varying sparsity levels and scales, using 4
representative classical architectures. We surprisingly observe two novel
phenomenon: double-peak and logarithmic. For the former, as the embedding
dimension increases, performance first improves, then declines, rises again,
and eventually drops. For the latter, it exhibits a perfect logarithmic curve.
Our contributions are threefold. First, we discover two novel phenomena when
scaling collaborative filtering models. Second, we gain an understanding of the
underlying causes of the double-peak phenomenon. Lastly, we theoretically
analyze the noise robustness of collaborative filtering models, with results
matching empirical observations.
68d2322a1ca7156988a8ee18
Zhuangzhuang He
claimed_verified
2025-09-24T13:55:09.829Z
6621e88fa11ce46061d25a16
/avatars/4a9a965a2d0f33e2855d2909a3e162bc.svg
Zhuangzhuang He
user
bruno888
68d2322a1ca7156988a8ee19
Zhou Kaiyu
68d2322a1ca7156988a8ee1a
Haoyue Bai
68d2322a1ca7156988a8ee1b
Fengbin Zhu
68d2322a1ca7156988a8ee1c
Yonghui Yang
Understanding Embedding Scaling in Collaborative Filtering
Scaling recommendation models into large recommendation models has become one
of the most widely discussed topics. Recent efforts focus on components beyond
the scaling embedding dimension, as it is believed that scaling embedding may
lead to performance degradation. Although there have been some initial
observations on embedding, the root cause of their non-scalability remains
unclear. Moreover, whether performance degradation occurs across different
types of models and datasets is still an unexplored area. Regarding the effect
of embedding dimensions on performance, we conduct large-scale experiments
across 10 datasets with varying sparsity levels and scales, using 4
representative classical architectures. We surprisingly observe two novel
phenomenon: double-peak and logarithmic. For the former, as the embedding
dimension increases, performance first improves, then declines, rises again,
and eventually drops. For the latter, it exhibits a perfect logarithmic curve.
Our contributions are threefold. First, we discover two novel phenomena when
scaling collaborative filtering models. Second, we gain an understanding of the
underlying causes of the double-peak phenomenon. Lastly, we theoretically
analyze the noise robustness of collaborative filtering models, with results
matching empirical observations.
Large-scale experiments reveal double-peak and logarithmic performance patterns in collaborative filtering models as embedding dimensions scale, and provide theoretical insights into their causes.
collaborative filtering models embedding dimensions double-peak phenomenon logarithmic curve noise robustness