Can Large Language Models Understand Context?
Understanding context is key to understanding human language, an ability
which Large Language Models (LLMs) have been increasingly seen to demonstrate
to an impressive extent. However, though the evaluation of LLMs encompasses
various domains within the realm of Natural Language Processing, limited
attention has been paid to probing their linguistic capability of understanding
contextual features. This paper introduces a context understanding benchmark by
adapting existing datasets to suit the evaluation of generative models. This
benchmark comprises of four distinct tasks and nine datasets, all featuring
prompts designed to assess the models' ability to understand context. First, we
evaluate the performance of LLMs under the in-context learning pretraining
scenario. Experimental results indicate that pre-trained dense models struggle
with understanding more nuanced contextual features when compared to
state-of-the-art fine-tuned models. Second, as LLM compression holds growing
significance in both research and real-world applications, we assess the
context understanding of quantized models under in-context-learning settings.
We find that 3-bit post-training quantization leads to varying degrees of
performance reduction on our benchmark. We conduct an extensive analysis of
these scenarios to substantiate our experimental results.
65bc6cec358734fd096bc182
Yilun Zhu
65bc6cec358734fd096bc183
Joel Ruben Antony Moniz
claimed_verified
2025-03-13T08:25:22.272Z
6650e6b9e29125c00a919a4d
/avatars/b0093fa9da51e42168545e83750c3e81.svg
Joel Moniz
user
joelmoniz
65bc6cec358734fd096bc184
Shruti Bhargava
65bc6cec358734fd096bc185
Jiarui Lu
admin_assigned
2024-02-02T11:36:16.850Z
64bad07938953777fe3ed5e9
/avatars/91c07a176d8d51111873751e82a4e0ea.svg
Jiarui Lu
user
jiarui-lu
65bc6cec358734fd096bc186
Dhivya Piraviperumal
65bc6cec358734fd096bc187
Site Li
65bc6cec358734fd096bc188
Yuan Zhang
65bc6cec358734fd096bc189
Hong Yu
65bc6cec358734fd096bc18a
Bo-Hsiang Tseng
Can Large Language Models Understand Context?
Understanding context is key to understanding human language, an ability
which Large Language Models (LLMs) have been increasingly seen to demonstrate
to an impressive extent. However, though the evaluation of LLMs encompasses
various domains within the realm of Natural Language Processing, limited
attention has been paid to probing their linguistic capability of understanding
contextual features. This paper introduces a context understanding benchmark by
adapting existing datasets to suit the evaluation of generative models. This
benchmark comprises of four distinct tasks and nine datasets, all featuring
prompts designed to assess the models' ability to understand context. First, we
evaluate the performance of LLMs under the in-context learning pretraining
scenario. Experimental results indicate that pre-trained dense models struggle
with understanding more nuanced contextual features when compared to
state-of-the-art fine-tuned models. Second, as LLM compression holds growing
significance in both research and real-world applications, we assess the
context understanding of quantized models under in-context-learning settings.
We find that 3-bit post-training quantization leads to varying degrees of
performance reduction on our benchmark. We conduct an extensive analysis of
these scenarios to substantiate our experimental results.
The benchmark evaluates LLMs' context understanding by assessing pre-trained and quantized models across four tasks and nine datasets, highlighting the performance differences between dense and fine-tuned models.
Large Language Models (LLMs) in-context learning pretraining fine-tuning contextual features prompt design benchmark 3-bit post-training quantization