The Impact of Hyperparameters on Large Language Model Inference
  Performance: An Evaluation of vLLM and HuggingFace Pipelines
The recent surge of open-source large language models (LLMs) enables
developers to create AI-based solutions while maintaining control over aspects
such as privacy and compliance, thereby providing governance and ownership of
the model deployment process. To utilize these LLMs, inference engines are
needed. These engines load the model's weights onto available resources, such
as GPUs, and process queries to generate responses. The speed of inference, or
performance, of the LLM, is critical for real-time applications, as it computes
millions or billions of floating point operations per inference. Recently,
advanced inference engines such as vLLM have emerged, incorporating novel
mechanisms such as efficient memory management to achieve state-of-the-art
performance. In this paper, we analyze the performance, particularly the
throughput (tokens generated per unit of time), of 20 LLMs using two inference
libraries: vLLM and HuggingFace's pipelines. We investigate how various
hyperparameters, which developers must configure, influence inference
performance. Our results reveal that throughput landscapes are irregular, with
distinct peaks, highlighting the importance of hyperparameter optimization to
achieve maximum performance. We also show that applying hyperparameter
optimization when upgrading or downgrading the GPU model used for inference can
improve throughput from HuggingFace pipelines by an average of 9.16% and 13.7%,
respectively.
66b1cd82b1c62888cf9595ff
Matias Martinez
claimed_verified
2024-08-06T07:43:50.846Z
63e24ad152b7578dba484031
https://cdn-avatars.huggingface.co/v1/production/uploads/63e24ad152b7578dba484031/N1qX1rJ5cYdiOAQAULAV9.jpeg
Matias Martinez
user
martinezmatias
The Impact of Hyperparameters on Large Language Model Inference
  Performance: An Evaluation of vLLM and HuggingFace Pipelines
The recent surge of open-source large language models (LLMs) enables
developers to create AI-based solutions while maintaining control over aspects
such as privacy and compliance, thereby providing governance and ownership of
the model deployment process. To utilize these LLMs, inference engines are
needed. These engines load the model's weights onto available resources, such
as GPUs, and process queries to generate responses. The speed of inference, or
performance, of the LLM, is critical for real-time applications, as it computes
millions or billions of floating point operations per inference. Recently,
advanced inference engines such as vLLM have emerged, incorporating novel
mechanisms such as efficient memory management to achieve state-of-the-art
performance. In this paper, we analyze the performance, particularly the
throughput (tokens generated per unit of time), of 20 LLMs using two inference
libraries: vLLM and HuggingFace's pipelines. We investigate how various
hyperparameters, which developers must configure, influence inference
performance. Our results reveal that throughput landscapes are irregular, with
distinct peaks, highlighting the importance of hyperparameter optimization to
achieve maximum performance. We also show that applying hyperparameter
optimization when upgrading or downgrading the GPU model used for inference can
improve throughput from HuggingFace pipelines by an average of 9.16% and 13.7%,
respectively.
The performance of large language models (LLMs) is analyzed using vLLM and HuggingFace's pipelines, with a focus on throughput optimization through hyperparameter tuning.
large language models (LLMs) inference engines vLLM efficient memory management throughput hyperparameter optimization floating point operations GPU