LACoS-BLOOM: Low-rank Adaptation with Contrastive objective on 8 bits
  Siamese-BLOOM
Text embeddings are useful features for several NLP applications, such as
sentence similarity, text clustering, and semantic search. In this paper, we
present a Low-rank Adaptation with a Contrastive objective on top of 8-bit
Siamese-BLOOM, a multilingual large language model optimized to produce
semantically meaningful word embeddings. The innovation is threefold. First, we
cast BLOOM weights to 8-bit values. Second, we fine-tune BLOOM with a scalable
adapter (LoRA) and 8-bit Adam optimizer for sentence similarity classification.
Third, we apply a Siamese architecture on BLOOM model with a contrastive
objective to ease the multi-lingual labeled data scarcity. The experiment
results show the quality of learned embeddings from LACoS-BLOOM is proportional
to the number of model parameters and the amount of unlabeled training data.
With the parameter efficient fine-tuning design, we are able to run BLOOM 7.1
billion parameters end-to-end on a single GPU machine with 32GB memory.
Compared to previous solution Sentence-BERT, we achieve significant improvement
on both English and multi-lingual STS tasks.
645da48bf5760d1530d6e297
Wen-Yu Hua
extracted_pending
2023-05-12T02:29:31.587Z
64091cb4bf3e9a4bb2e30b75
/avatars/96bb4a5abc26684d62f3d16339774c28.svg
Wendy Kung
user
hhuietkung
645da48bf5760d1530d6e298
Brian Williams
645da48bf5760d1530d6e299
Davood Shamsi
LACoS-BLOOM: Low-rank Adaptation with Contrastive objective on 8 bits
  Siamese-BLOOM
Text embeddings are useful features for several NLP applications, such as
sentence similarity, text clustering, and semantic search. In this paper, we
present a Low-rank Adaptation with a Contrastive objective on top of 8-bit
Siamese-BLOOM, a multilingual large language model optimized to produce
semantically meaningful word embeddings. The innovation is threefold. First, we
cast BLOOM weights to 8-bit values. Second, we fine-tune BLOOM with a scalable
adapter (LoRA) and 8-bit Adam optimizer for sentence similarity classification.
Third, we apply a Siamese architecture on BLOOM model with a contrastive
objective to ease the multi-lingual labeled data scarcity. The experiment
results show the quality of learned embeddings from LACoS-BLOOM is proportional
to the number of model parameters and the amount of unlabeled training data.
With the parameter efficient fine-tuning design, we are able to run BLOOM 7.1
billion parameters end-to-end on a single GPU machine with 32GB memory.
Compared to previous solution Sentence-BERT, we achieve significant improvement
on both English and multi-lingual STS tasks.
LACoS-BLOOM, an 8-bit Low-rank Adaptation with Contrastive objective of Siamese-BLOOM, improves text embeddings quality and efficiency in NLP tasks.
Low-rank Adaptation Contrastive objective 8-bit Siamese-BLOOM multilingual large language model semantic word embeddings sentence similarity classification scalable adapter LoRA 8-bit Adam optimizer Siamese architecture multi-lingual labeled data scarcity parameter efficient fine-tuning LACoS-BLOOM Sentence-BERT English and multi-lingual STS tasks