SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal
  Domain
In this paper, we introduce SaulLM-54B and SaulLM-141B, two large language
models (LLMs) tailored for the legal sector. These models, which feature
architectures of 54 billion and 141 billion parameters, respectively, are based
on the Mixtral architecture. The development of SaulLM-54B and SaulLM-141B is
guided by large-scale domain adaptation, divided into three strategies: (1) the
exploitation of continued pretraining involving a base corpus that includes
over 540 billion of legal tokens, (2) the implementation of a specialized legal
instruction-following protocol, and (3) the alignment of model outputs with
human preferences in legal interpretations. The integration of synthetically
generated data in the second and third steps enhances the models' capabilities
in interpreting and processing legal texts, effectively reaching
state-of-the-art performance and outperforming previous open-source models on
LegalBench-Instruct. This work explores the trade-offs involved in
domain-specific adaptation at this scale, offering insights that may inform
future studies on domain adaptation using strong decoder models. Building upon
SaulLM-7B, this study refines the approach to produce an LLM better equipped
for legal tasks. We are releasing base, instruct, and aligned versions on top
of SaulLM-54B and SaulLM-141B under the MIT License to facilitate reuse and
collaborative research.
66a8439359d86126000b54cc
Pierre Colombo
admin_assigned
2024-07-30T08:36:24.366Z
644a900e3a619fe72b14af0f
/avatars/e2d5dac3d92757ed48e37e126a3464a3.svg
Colombo
user
PierreColombo
66a8439359d86126000b54cd
Telmo Pires
66a8439359d86126000b54ce
Malik Boudiaf
admin_assigned
2024-07-30T08:36:48.139Z
639283dd2c0c90e2f31b5500
/avatars/9a3cff6b14fd687bcbe2b78c4c032619.svg
Malik Boudiaf
user
mboudiaf
66a8439359d86126000b54cf
Rui Melo
admin_assigned
2024-07-30T08:36:53.989Z
622f26eb163b4ef6fd7a5361
https://cdn-avatars.huggingface.co/v1/production/uploads/622f26eb163b4ef6fd7a5361/-WYJwPC5QkWecz_XL8w94.jpeg
Rui Melo
user
rufimelo
66a8439359d86126000b54d0
Dominic Culver
admin_assigned
2024-07-30T08:36:59.970Z
6453da2ef2122886b298873f
/avatars/8c79e43e5d76cd8d29db9357180b629d.svg
Dominic Culver
user
CWcx
66a8439359d86126000b54d1
Sofia Morgado
admin_assigned
2024-07-30T08:37:05.378Z
63e76a3ce02ee67e8e4fcdc2
/avatars/ca8668fb2fdafb264291a7585c0ddd6b.svg
Sofia Morgado
user
somorgado
66a8439359d86126000b54d2
Etienne Malaboeuf
66a8439359d86126000b54d3
Gabriel Hautreux
66a8439359d86126000b54d4
Johanne Charpentier
66a8439359d86126000b54d5
Michael Desa
admin_assigned
2024-07-30T08:38:05.700Z
62cee925e7f6014c0e9d7b93
/avatars/39e82af1550d57d06963ad87d42514e5.svg
Michael Desa
user
desa
SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal
  Domain
In this paper, we introduce SaulLM-54B and SaulLM-141B, two large language
models (LLMs) tailored for the legal sector. These models, which feature
architectures of 54 billion and 141 billion parameters, respectively, are based
on the Mixtral architecture. The development of SaulLM-54B and SaulLM-141B is
guided by large-scale domain adaptation, divided into three strategies: (1) the
exploitation of continued pretraining involving a base corpus that includes
over 540 billion of legal tokens, (2) the implementation of a specialized legal
instruction-following protocol, and (3) the alignment of model outputs with
human preferences in legal interpretations. The integration of synthetically
generated data in the second and third steps enhances the models' capabilities
in interpreting and processing legal texts, effectively reaching
state-of-the-art performance and outperforming previous open-source models on
LegalBench-Instruct. This work explores the trade-offs involved in
domain-specific adaptation at this scale, offering insights that may inform
future studies on domain adaptation using strong decoder models. Building upon
SaulLM-7B, this study refines the approach to produce an LLM better equipped
for legal tasks. We are releasing base, instruct, and aligned versions on top
of SaulLM-54B and SaulLM-141B under the MIT License to facilitate reuse and
collaborative research.
Two large legal language models, SaulLM-54B and SaulLM-141B, based on the Mixtral architecture, are introduced for domain-specific adaptation in the legal sector using continued pretraining, specialized protocols, and preference alignment with synthetic data.
Mixtral architecture large language models legal sector domain adaptation continued pretraining legal tokens legal instruction-following protocol human preferences legal interpretations synthetic data LegalBench-Instruct strong decoder models