Fully-fused Multi-Layer Perceptrons on Intel Data Center GPUs
This paper presents a SYCL implementation of Multi-Layer Perceptrons (MLPs),
which targets and is optimized for the Intel Data Center GPU Max 1550. To
increase the performance, our implementation minimizes the slow global memory
accesses by maximizing the data reuse within the general register file and the
shared local memory by fusing the operations in each layer of the MLP. We show
with a simple roofline model that this results in a significant increase in the
arithmetic intensity, leading to improved performance, especially for
inference. We compare our approach to a similar CUDA implementation for MLPs
and show that our implementation on the Intel Data Center GPU outperforms the
CUDA implementation on Nvidia's H100 GPU by a factor up to 2.84 in inference
and 1.75 in training. The paper also showcases the efficiency of our SYCL
implementation in three significant areas: Image Compression, Neural Radiance
Fields, and Physics-Informed Machine Learning. In all cases, our implementation
outperforms the off-the-shelf Intel Extension for PyTorch (IPEX) implementation
on the same Intel GPU by up to a factor of 30 and the CUDA PyTorch version on
Nvidia's H100 GPU by up to a factor 19. The code can be found at
https://github.com/intel/tiny-dpcpp-nn.
6603878f527470e01645fe69
Kai Yuan
6603878f527470e01645fe6a
Christoph Bauinger
6603878f527470e01645fe6b
Xiangyi Zhang
admin_assigned
2024-03-27T09:32:01.042Z
6475250c6d4dda6f7c6e9dba
/avatars/ffccf38a96881a7db3dc19954bd55dbe.svg
Xiangyi Zhang
user
Shawnz516
6603878f527470e01645fe6c
Pascal Baehr
6603878f527470e01645fe6d
Matthias Kirchhart
6603878f527470e01645fe6e
Darius Dabert
6603878f527470e01645fe6f
Adrien Tousnakhoff
6603878f527470e01645fe70
Pierre Boudier
6603878f527470e01645fe71
Michael Paulitsch
Fully-fused Multi-Layer Perceptrons on Intel Data Center GPUs
This paper presents a SYCL implementation of Multi-Layer Perceptrons (MLPs),
which targets and is optimized for the Intel Data Center GPU Max 1550. To
increase the performance, our implementation minimizes the slow global memory
accesses by maximizing the data reuse within the general register file and the
shared local memory by fusing the operations in each layer of the MLP. We show
with a simple roofline model that this results in a significant increase in the
arithmetic intensity, leading to improved performance, especially for
inference. We compare our approach to a similar CUDA implementation for MLPs
and show that our implementation on the Intel Data Center GPU outperforms the
CUDA implementation on Nvidia's H100 GPU by a factor up to 2.84 in inference
and 1.75 in training. The paper also showcases the efficiency of our SYCL
implementation in three significant areas: Image Compression, Neural Radiance
Fields, and Physics-Informed Machine Learning. In all cases, our implementation
outperforms the off-the-shelf Intel Extension for PyTorch (IPEX) implementation
on the same Intel GPU by up to a factor of 30 and the CUDA PyTorch version on
Nvidia's H100 GPU by up to a factor 19. The code can be found at
https://github.com/intel/tiny-dpcpp-nn.
A SYCL implementation of MLPs optimized for Intel's Data Center GPU demonstrates superior performance compared to CUDA across various applications, including image compression, neural radiance fields, and physics-informed machine learning.
SYCL Multi-Layer Perceptrons (MLPs) Intel Data Center GPU Max 1550 general register file shared local memory fusion roofline model arithmetic intensity CUDA Nvidia's H100 GPU Image Compression Neural Radiance Fields Physics-Informed Machine Learning Intel Extension for PyTorch (IPEX)