A dynamic parallel method for performance optimization on hybrid CPUs
The AIPC concept is gaining popularity, and more and more hybrid CPUs will be
running AI models on client devices. However, the current AI inference
framework overlooks the imbalanced hardware capability of hybrid CPUs, leading
to low inference performance. To address this issue, we have introduced a
dynamic parallel method for hybrid CPUs, which significantly increases LLM
inference performance by balancing the workload for each core of a hybrid CPU
before the parallel work starts. This method has enabled Neural Speed to
achieve more than 90% (on average) of memory bandwidth on two hybrid Intel
CPUs.
674ffc79f3a3876a956ed67e
Luo Yu
674ffc79f3a3876a956ed67f
Liu Yucheng
674ffc79f3a3876a956ed680
Shen Haihao
claimed_verified
2024-12-04T09:44:17.991Z
60ac3318e3de7c7440abb850
https://cdn-avatars.huggingface.co/v1/production/uploads/60ac3318e3de7c7440abb850/DkMrPBr6Ew_dS_c9kog4e.jpeg
Haihao Shen
user
Haihao
A dynamic parallel method for performance optimization on hybrid CPUs
The AIPC concept is gaining popularity, and more and more hybrid CPUs will be
running AI models on client devices. However, the current AI inference
framework overlooks the imbalanced hardware capability of hybrid CPUs, leading
to low inference performance. To address this issue, we have introduced a
dynamic parallel method for hybrid CPUs, which significantly increases LLM
inference performance by balancing the workload for each core of a hybrid CPU
before the parallel work starts. This method has enabled Neural Speed to
achieve more than 90% (on average) of memory bandwidth on two hybrid Intel
CPUs.
A dynamic parallel method for hybrid CPUs improves LLM inference performance by balancing workload across cores, achieving high memory bandwidth utilization.
dynamic parallel method LLM inference memory bandwidth hybrid CPUs workload balancing