XGen-7B Technical Report
Large Language Models (LLMs) have become ubiquitous across various domains,
transforming the way we interact with information and conduct research.
However, most high-performing LLMs remain confined behind proprietary walls,
hindering scientific progress. Most open-source LLMs, on the other hand, are
limited in their ability to support longer sequence lengths, which is a key
requirement for many tasks that require inference over an input context. To
address this, we have trained XGen, a series of 7B parameter models on up to 8K
sequence length for up to 1.5T tokens. We have also finetuned the XGen models
on public-domain instructional data, creating their instruction-tuned
counterparts (XGen-Inst). We open-source our models for both research
advancements and commercial applications. Our evaluation on standard benchmarks
shows that XGen models achieve comparable or better results when compared with
state-of-the-art open-source LLMs. Our targeted evaluation on long sequence
modeling tasks shows the benefits of our 8K-sequence models over 2K-sequence
open-source LLMs.
64fa7e88bb362cbf2fd3248b
Erik Nijkamp
admin_assigned
2023-09-08T11:55:46.832Z
62472f5139c8c09b3cde0516
/avatars/da865d0e5547fc9dc51b96153e5a8d6d.svg
Erik Nijkamp
user
enijkamp
64fa7e88bb362cbf2fd3248c
Tian Xie
admin_assigned
2023-09-08T11:55:55.808Z
6401323152fb66b80d1da6c0
/avatars/24574113bfe000250196e935e4f7cde2.svg
Tian Xie
user
tianxie-sf
64fa7e88bb362cbf2fd3248d
Hiroaki Hayashi
admin_assigned
2023-09-08T11:56:03.514Z
619aa4e2198a95c9b962252e
https://cdn-avatars.huggingface.co/v1/production/uploads/1648750941188-619aa4e2198a95c9b962252e.jpeg
Hiroaki Hayashi
user
rooa
64fa7e88bb362cbf2fd3248e
Bo Pang
admin_assigned
2023-09-08T11:56:11.852Z
63e08acbf351dc0745749d56
/avatars/8e2d5ce9db5bd8008ac2ad80f6025553.svg
Bo Pang
user
bpucla
64fa7e88bb362cbf2fd3248f
Congying Xia
64fa7e88bb362cbf2fd32490
Chen Xing
64fa7e88bb362cbf2fd32491
Jesse Vig
64fa7e88bb362cbf2fd32492
Semih Yavuz
admin_assigned
2023-09-08T11:57:14.258Z
64f8a68e33fd3f0fcb7810b5
/avatars/cb5ea85e9ed513efaf24a9f36869f98f.svg
Semih Yavuz
user
syavuz
64fa7e88bb362cbf2fd32493
Philippe Laban
admin_assigned
2023-09-08T11:57:21.555Z
610aee2dc30263ce69e0dd9c
/avatars/b6e5bba78425e9c28d1b75ee8a7eaac1.svg
Philippe Laban
user
philippelaban
64fa7e88bb362cbf2fd32494
Ben Krause
64fa7e88bb362cbf2fd32495
Senthil Purushwalkam
admin_assigned
2023-09-08T11:57:33.822Z
637e633366a51ca5d960a9b1
/avatars/e53242e4e36875105be087c46a552e29.svg
Senthil Purushwalkam
user
senthilps8
64fa7e88bb362cbf2fd32496
Tong Niu
64fa7e88bb362cbf2fd32497
Wojciech Kryściński
64fa7e88bb362cbf2fd32498
Lidiya Murakhovs'ka
admin_assigned
2023-09-08T11:58:17.103Z
60551289bf611753d3101522
https://cdn-avatars.huggingface.co/v1/production/uploads/1647111747809-60551289bf611753d3101522.jpeg
Lidiya Murakhovs'ka
user
lidiya
64fa7e88bb362cbf2fd32499
Prafulla Kumar Choubey
admin_assigned
2023-09-08T11:58:43.373Z
64e3ce5a618cd90997ed5973
/avatars/af8709f90478c124aeef9607986f91b4.svg
Prafulla Kumar Choubey
user
prafullachoubey
64fa7e88bb362cbf2fd3249a
Alex Fabbri
admin_assigned
2023-09-08T11:58:59.370Z
626427701ed8d81e47ab6899
/avatars/7c89167744703771e3f163665cdb66cc.svg
Alexander Fabbri
user
alexfabbri
64fa7e88bb362cbf2fd3249b
Ye Liu
64fa7e88bb362cbf2fd3249c
Rui Meng
64fa7e88bb362cbf2fd3249d
Lifu Tu
admin_assigned
2023-09-08T12:00:17.653Z
611d9648dcea95fcbbc5638e
/avatars/e94bf70da30ed5036e2c12a38b6fcad9.svg
Lifu Tu
user
tulifu
64fa7e88bb362cbf2fd3249e
Meghana Bhat
64fa7e88bb362cbf2fd3249f
Chien-Sheng Wu
64fa7e88bb362cbf2fd324a0
Silvio Savarese
64fa7e88bb362cbf2fd324a1
Yingbo Zhou
admin_assigned
2023-09-08T12:01:03.197Z
649bc93758d8b19de0c7785f
/avatars/3ed9473aee23d99f4ee949d3705089ea.svg
Yingbo Zhou
user
yingbozhou
64fa7e88bb362cbf2fd324a2
Shafiq Joty
64fa7e88bb362cbf2fd324a3
Caiming Xiong
admin_assigned
2023-09-08T12:01:15.727Z
649dbcc4e0fff1ed099dc80a
/avatars/c87c273ca628dbcddccbf1ee19b2ce33.svg
Caiming Xiong
user
cxiong
XGen-7B Technical Report
Large Language Models (LLMs) have become ubiquitous across various domains,
transforming the way we interact with information and conduct research.
However, most high-performing LLMs remain confined behind proprietary walls,
hindering scientific progress. Most open-source LLMs, on the other hand, are
limited in their ability to support longer sequence lengths, which is a key
requirement for many tasks that require inference over an input context. To
address this, we have trained XGen, a series of 7B parameter models on up to 8K
sequence length for up to 1.5T tokens. We have also finetuned the XGen models
on public-domain instructional data, creating their instruction-tuned
counterparts (XGen-Inst). We open-source our models for both research
advancements and commercial applications. Our evaluation on standard benchmarks
shows that XGen models achieve comparable or better results when compared with
state-of-the-art open-source LLMs. Our targeted evaluation on long sequence
modeling tasks shows the benefits of our 8K-sequence models over 2K-sequence
open-source LLMs.
XGen, a series of open-source 7B parameter models trained on up to 8K sequence lengths, outperforms other open-source LLMs in long sequence modeling tasks.
Large Language Models sequence lengths fine-tuning instruction-tuned long sequence modeling tasks