FlexOlmo: Open Language Models for Flexible Data Use
We introduce FlexOlmo, a new class of language models (LMs) that supports (1)
distributed training without data sharing, where different model parameters are
independently trained on closed datasets, and (2) data-flexible inference,
where these parameters along with their associated data can be flexibly
included or excluded from model inferences with no further training. FlexOlmo
employs a mixture-of-experts (MoE) architecture where each expert is trained
independently on closed datasets and later integrated through a new
domain-informed routing without any joint training. FlexOlmo is trained on
FlexMix, a corpus we curate comprising publicly available datasets alongside
seven domain-specific sets, representing realistic approximations of closed
sets. We evaluate models with up to 37 billion parameters (20 billion active)
on 31 diverse downstream tasks. We show that a general expert trained on public
data can be effectively combined with independently trained experts from other
data owners, leading to an average 41% relative improvement while allowing
users to opt out of certain data based on data licensing or permission
requirements. Our approach also outperforms prior model merging methods by
10.1% on average and surpasses the standard MoE trained without data
restrictions using the same training FLOPs. Altogether, this research presents
a solution for both data owners and researchers in regulated industries with
sensitive or protected data. FlexOlmo enables benefiting from closed data while
respecting data owners' preferences by keeping their data local and supporting
fine-grained control of data access during inference.
686f6171d938c25d68441bca
Weijia Shi
686f6171d938c25d68441bcb
Akshita Bhagia
686f6171d938c25d68441bcc
Kevin Farhat
686f6171d938c25d68441bcd
Niklas Muennighoff
686f6171d938c25d68441bce
Pete Walsh
686f6171d938c25d68441bcf
Jacob Morrison
686f6171d938c25d68441bd0
Dustin Schwenk
686f6171d938c25d68441bd1
Shayne Longpre
686f6171d938c25d68441bd2
Jake Poznanski
686f6171d938c25d68441bd3
Allyson Ettinger
686f6171d938c25d68441bd4
Daogao Liu
686f6171d938c25d68441bd5
Margaret Li
686f6171d938c25d68441bd6
Dirk Groeneveld
686f6171d938c25d68441bd7
Mike Lewis
686f6171d938c25d68441bd8
Wen-tau Yih
686f6171d938c25d68441bd9
Luca Soldaini
686f6171d938c25d68441bda
Kyle Lo
686f6171d938c25d68441bdb
Noah A. Smith
686f6171d938c25d68441bdc
Luke Zettlemoyer
686f6171d938c25d68441bdd
Pang Wei Koh
686f6171d938c25d68441bde
Hannaneh Hajishirzi
686f6171d938c25d68441bdf
Ali Farhadi
686f6171d938c25d68441be0
Sewon Min
FlexOlmo: Open Language Models for Flexible Data Use
We introduce FlexOlmo, a new class of language models (LMs) that supports (1)
distributed training without data sharing, where different model parameters are
independently trained on closed datasets, and (2) data-flexible inference,
where these parameters along with their associated data can be flexibly
included or excluded from model inferences with no further training. FlexOlmo
employs a mixture-of-experts (MoE) architecture where each expert is trained
independently on closed datasets and later integrated through a new
domain-informed routing without any joint training. FlexOlmo is trained on
FlexMix, a corpus we curate comprising publicly available datasets alongside
seven domain-specific sets, representing realistic approximations of closed
sets. We evaluate models with up to 37 billion parameters (20 billion active)
on 31 diverse downstream tasks. We show that a general expert trained on public
data can be effectively combined with independently trained experts from other
data owners, leading to an average 41% relative improvement while allowing
users to opt out of certain data based on data licensing or permission
requirements. Our approach also outperforms prior model merging methods by
10.1% on average and surpasses the standard MoE trained without data
restrictions using the same training FLOPs. Altogether, this research presents
a solution for both data owners and researchers in regulated industries with
sensitive or protected data. FlexOlmo enables benefiting from closed data while
respecting data owners' preferences by keeping their data local and supporting
fine-grained control of data access during inference.
FlexOlmo, a distributed language model using a mixture-of-experts architecture, allows independent training on closed datasets and flexible inference without further training, improving performance while respecting data privacy.
mixture-of-experts domain-informed routing FlexMix closed datasets data-flexible inference distributed training parameter-efficient fine-tuning data licensing data access control