How Far Can Camels Go? Exploring the State of Instruction Tuning on Open
  Resources
In this work we explore recent advances in instruction-tuning language models
on a range of open instruction-following datasets. Despite recent claims that
open models can be on par with state-of-the-art proprietary models, these
claims are often accompanied by limited evaluation, making it difficult to
compare models across the board and determine the utility of various resources.
We provide a large set of instruction-tuned models from 6.7B to 65B parameters
in size, trained on 12 instruction datasets ranging from manually curated
(e.g., OpenAssistant) to synthetic and distilled (e.g., Alpaca) and
systematically evaluate them on their factual knowledge, reasoning,
multilinguality, coding, and open-ended instruction following abilities through
a collection of automatic, model-based, and human-based metrics. We further
introduce T\"ulu, our best performing instruction-tuned model suite finetuned
on a combination of high-quality open resources.
  Our experiments show that different instruction-tuning datasets can uncover
or enhance specific skills, while no single dataset (or combination) provides
the best performance across all evaluations. Interestingly, we find that model
and human preference-based evaluations fail to reflect differences in model
capabilities exposed by benchmark-based evaluations, suggesting the need for
the type of systemic evaluation performed in this work. Our evaluations show
that the best model in any given evaluation reaches on average 83% of ChatGPT
performance, and 68% of GPT-4 performance, suggesting that further investment
in building better base models and instruction-tuning data is required to close
the gap. We release our instruction-tuned models, including a fully finetuned
65B T\"ulu, along with our code, data, and evaluation framework at
https://github.com/allenai/open-instruct to facilitate future research.
648270672bc908e2d9e2a0d4
Yizhong Wang
admin_assigned
2023-06-09T16:18:21.440Z
6269be67d1ac0cde592aba29
https://cdn-avatars.huggingface.co/v1/production/uploads/6269be67d1ac0cde592aba29/EP3zZbD7-jWK9ITvHGbnZ.jpeg
Yizhong Wang
user
yizhongw
648270672bc908e2d9e2a0d5
Hamish Ivison
claimed_verified
2023-06-09T15:32:10.401Z
62608fc2ffe8827cb1d89f9f
https://cdn-avatars.huggingface.co/v1/production/uploads/1654027835241-62608fc2ffe8827cb1d89f9f.png
Hamish Ivison
user
hamishivi
648270672bc908e2d9e2a0d6
Pradeep Dasigi
admin_assigned
2023-06-09T16:18:41.250Z
6408fcc93461c51cf735a61e
/avatars/619f3653911d111f046a5a6c30fc8319.svg
Pradeep Dasigi
user
pradeepd
648270672bc908e2d9e2a0d7
Jack Hessel
admin_assigned
2023-06-09T16:19:08.476Z
625498644f4edf771516b2cb
https://cdn-avatars.huggingface.co/v1/production/uploads/1664497251391-625498644f4edf771516b2cb.jpeg
Jack Hessel
user
jmhessel
648270672bc908e2d9e2a0d8
Tushar Khot
admin_assigned
2023-06-09T16:19:27.010Z
62cdf2dbaac2c91c95581830
https://cdn-avatars.huggingface.co/v1/production/uploads/62cdf2dbaac2c91c95581830/N3NuO7j53qkauJOf3kkRS.jpeg
Tushar Khot
user
tusharkhot
648270672bc908e2d9e2a0d9
Khyathi Raghavi Chandu
648270672bc908e2d9e2a0da
David Wadden
admin_assigned
2023-06-09T16:19:55.961Z
5ff4e2a1463be69ae4bd42bd
https://cdn-avatars.huggingface.co/v1/production/uploads/1609884433338-5ff4e2a1463be69ae4bd42bd.jpeg
David Wadden
user
dwadden
648270672bc908e2d9e2a0db
Kelsey MacMillan
admin_assigned
2023-06-09T16:20:21.497Z
64528954c895e6437dc4be98
/avatars/685e3f4bf768a372a30615ec37b0f25b.svg
Kelsey MacMillan
user
kmacmillan
648270672bc908e2d9e2a0dc
Noah A. Smith
648270672bc908e2d9e2a0dd
Iz Beltagy
admin_assigned
2023-06-09T16:20:45.755Z
5ec07694ed25d76864d553f4
https://cdn-avatars.huggingface.co/v1/production/uploads/1589816744885-5ec07694ed25d76864d553f4.jpeg
Iz Beltagy
user
beltagy
648270672bc908e2d9e2a0de
Hannaneh Hajishirzi
How Far Can Camels Go? Exploring the State of Instruction Tuning on Open
  Resources
In this work we explore recent advances in instruction-tuning language models
on a range of open instruction-following datasets. Despite recent claims that
open models can be on par with state-of-the-art proprietary models, these
claims are often accompanied by limited evaluation, making it difficult to
compare models across the board and determine the utility of various resources.
We provide a large set of instruction-tuned models from 6.7B to 65B parameters
in size, trained on 12 instruction datasets ranging from manually curated
(e.g., OpenAssistant) to synthetic and distilled (e.g., Alpaca) and
systematically evaluate them on their factual knowledge, reasoning,
multilinguality, coding, and open-ended instruction following abilities through
a collection of automatic, model-based, and human-based metrics. We further
introduce T\"ulu, our best performing instruction-tuned model suite finetuned
on a combination of high-quality open resources.
  Our experiments show that different instruction-tuning datasets can uncover
or enhance specific skills, while no single dataset (or combination) provides
the best performance across all evaluations. Interestingly, we find that model
and human preference-based evaluations fail to reflect differences in model
capabilities exposed by benchmark-based evaluations, suggesting the need for
the type of systemic evaluation performed in this work. Our evaluations show
that the best model in any given evaluation reaches on average 83% of ChatGPT
performance, and 68% of GPT-4 performance, suggesting that further investment
in building better base models and instruction-tuning data is required to close
the gap. We release our instruction-tuned models, including a fully finetuned
65B T\"ulu, along with our code, data, and evaluation framework at
https://github.com/allenai/open-instruct to facilitate future research.
Various instruction-tuned language models are evaluated across multiple datasets, revealing differences in skill enhancement and suggesting ongoing need for better base models and data.
instruction-tuning language models open instruction-following datasets parameter tuning instruction datasets factual knowledge reasoning multilinguality coding open-ended instruction following automatic metrics model-based metrics human-based metrics T\"ulu benchmark-based evaluations model preference human preference ChatGPT GPT-4