LETS-C: Leveraging Language Embedding for Time Series Classification
Recent advancements in language modeling have shown promising results when
applied to time series data. In particular, fine-tuning pre-trained large
language models (LLMs) for time series classification tasks has achieved
state-of-the-art (SOTA) performance on standard benchmarks. However, these
LLM-based models have a significant drawback due to the large model size, with
the number of trainable parameters in the millions. In this paper, we propose
an alternative approach to leveraging the success of language modeling in the
time series domain. Instead of fine-tuning LLMs, we utilize a language
embedding model to embed time series and then pair the embeddings with a simple
classification head composed of convolutional neural networks (CNN) and
multilayer perceptron (MLP). We conducted extensive experiments on
well-established time series classification benchmark datasets. We demonstrated
LETS-C not only outperforms the current SOTA in classification accuracy but
also offers a lightweight solution, using only 14.5% of the trainable
parameters on average compared to the SOTA model. Our findings suggest that
leveraging language encoders to embed time series data, combined with a simple
yet effective classification head, offers a promising direction for achieving
high-performance time series classification while maintaining a lightweight
model architecture.
668f3348b3991ac0c3f2568b
Rachneet Kaur
extracted_confirmed
2024-07-11T01:23:47.850Z
663c57534edaac49a8afe286
https://cdn-avatars.huggingface.co/v1/production/uploads/663c57534edaac49a8afe286/fUB5uphwK-lqZE6EN2Z8F.jpeg
Rachneet Kaur
user
rachneetkaur
668f3348b3991ac0c3f2568c
Zhen Zeng
668f3348b3991ac0c3f2568d
Tucker Balch
668f3348b3991ac0c3f2568e
Manuela Veloso
LETS-C: Leveraging Language Embedding for Time Series Classification
Recent advancements in language modeling have shown promising results when
applied to time series data. In particular, fine-tuning pre-trained large
language models (LLMs) for time series classification tasks has achieved
state-of-the-art (SOTA) performance on standard benchmarks. However, these
LLM-based models have a significant drawback due to the large model size, with
the number of trainable parameters in the millions. In this paper, we propose
an alternative approach to leveraging the success of language modeling in the
time series domain. Instead of fine-tuning LLMs, we utilize a language
embedding model to embed time series and then pair the embeddings with a simple
classification head composed of convolutional neural networks (CNN) and
multilayer perceptron (MLP). We conducted extensive experiments on
well-established time series classification benchmark datasets. We demonstrated
LETS-C not only outperforms the current SOTA in classification accuracy but
also offers a lightweight solution, using only 14.5% of the trainable
parameters on average compared to the SOTA model. Our findings suggest that
leveraging language encoders to embed time series data, combined with a simple
yet effective classification head, offers a promising direction for achieving
high-performance time series classification while maintaining a lightweight
model architecture.
Using a language embedding model with a convolutional and MLP classification head achieves high accuracy and significantly reduced parameter count in time series classification.
language models pre-trained large language models time series classification language embedding model convolutional neural networks multilayer perceptron time series classification benchmarks