Paloma: A Benchmark for Evaluating Language Model Fit
Language models (LMs) commonly report perplexity on monolithic data held out
from training. Implicitly or explicitly, this data is composed of
domainsx2013varying distributions of language. Rather than assuming
perplexity on one distribution extrapolates to others, Perplexity Analysis for
Language Model Assessment (Paloma), measures LM fit to 585 text domains,
ranging from nytimes.com to r/depression on Reddit. We invite submissions to
our benchmark and organize results by comparability based on compliance with
guidelines such as removal of benchmark contamination from pretraining.
Submissions can also record parameter and training token count to make
comparisons of Pareto efficiency for performance as a function of these
measures of cost. We populate our benchmark with results from 6 baselines
pretrained on popular corpora. In case studies, we demonstrate analyses that
are possible with Paloma, such as finding that pretraining without data beyond
Common Crawl leads to inconsistent fit to many domains.
65812daf1fb6cfa0b48e30e2
Ian Magnusson
admin_assigned
2023-12-19T09:34:09.700Z
62bddd0b1e22ec8427a0f27e
/avatars/c433d0fb2945c70d1998f1e0e98d265f.svg
Ian Magnusson
user
IanMagnusson
65812daf1fb6cfa0b48e30e3
Akshita Bhagia
admin_assigned
2023-12-19T09:34:16.985Z
6282937a9624ab5f75c977e1
https://cdn-avatars.huggingface.co/v1/production/uploads/6282937a9624ab5f75c977e1/ZOysQbGzEzmrN_SaTQgOP.jpeg
Akshita Bhagia
user
akshitab
65812daf1fb6cfa0b48e30e4
Valentin Hofmann
65812daf1fb6cfa0b48e30e5
Luca Soldaini
admin_assigned
2023-12-19T09:34:39.062Z
5f04d8c45d08220171a0ad32
https://cdn-avatars.huggingface.co/v1/production/uploads/5f04d8c45d08220171a0ad32/uXEta6nqBabrUlAOXnS5g.jpeg
Luca Soldaini
user
soldni
65812daf1fb6cfa0b48e30e6
Ananya Harsh Jha
claimed_verified
2023-12-20T10:17:05.811Z
6582bee321aa786b76beba44
https://cdn-avatars.huggingface.co/v1/production/uploads/6582bee321aa786b76beba44/1YqJBHmLxMNRaGtU-lbKh.jpeg
Ananya Harsh Jha
user
ananyahjha93
65812daf1fb6cfa0b48e30e7
Oyvind Tafjord
admin_assigned
2023-12-19T09:35:10.332Z
609b1aeff7c1303cfd1760f3
/avatars/3cb448514885656466886d7c84e12af1.svg
Oyvind Tafjord
user
OyvindTafjord
65812daf1fb6cfa0b48e30e8
Dustin Schwenk
admin_assigned
2023-12-19T09:35:17.065Z
647e791b7f9ad5e44babb965
https://cdn-avatars.huggingface.co/v1/production/uploads/647e791b7f9ad5e44babb965/Vjfy4c3ooDwU24N7mvRdy.jpeg
Dustin Schwenk
user
drschwenk
65812daf1fb6cfa0b48e30e9
Evan Pete Walsh
admin_assigned
2023-12-19T09:35:30.295Z
5eec0b3173936372ea2026d3
https://cdn-avatars.huggingface.co/v1/production/uploads/1623085897467-5eec0b3173936372ea2026d3.jpeg
Evan Pete Walsh
user
epwalsh
65812daf1fb6cfa0b48e30ea
Yanai Elazar
admin_assigned
2023-12-19T09:35:39.162Z
623ca115a795593324c4353f
/avatars/bf11fe728df2786d52ed4d2de12b48d3.svg
Yanai Elazar
user
yanaiela
65812daf1fb6cfa0b48e30eb
Kyle Lo
65812daf1fb6cfa0b48e30ec
Dirk Groeneveld
admin_assigned
2023-12-19T09:36:01.953Z
60369745413a78f892e7339c
https://cdn-avatars.huggingface.co/v1/production/uploads/1636671879171-60369745413a78f892e7339c.png
Dirk Groeneveld
user
dirkgr
65812daf1fb6cfa0b48e30ed
Iz Beltagy
admin_assigned
2023-12-19T09:36:09.411Z
5ec07694ed25d76864d553f4
https://cdn-avatars.huggingface.co/v1/production/uploads/1589816744885-5ec07694ed25d76864d553f4.jpeg
Iz Beltagy
user
beltagy
65812daf1fb6cfa0b48e30ee
Hannaneh Hajishirzi
65812daf1fb6cfa0b48e30ef
Noah A. Smith
65812daf1fb6cfa0b48e30f0
Kyle Richardson
claimed_verified
2024-01-08T21:57:30.483Z
659c6f5e1d398a238152227d
/avatars/dba97fe8cb825102f1eae97104a71f64.svg
Kyle Richardson
user
yakazimir
65812daf1fb6cfa0b48e30f1
Jesse Dodge
admin_assigned
2023-12-19T09:36:53.685Z
6283f38567d336d3e5d5280e
/avatars/d0a54aaec74a90b050e671c191b87a80.svg
Jesse Dodge
user
JesseDodge
Paloma: A Benchmark for Evaluating Language Model Fit
Language models (LMs) commonly report perplexity on monolithic data held out
from training. Implicitly or explicitly, this data is composed of
domainsx2013varying distributions of language. Rather than assuming
perplexity on one distribution extrapolates to others, Perplexity Analysis for
Language Model Assessment (Paloma), measures LM fit to 585 text domains,
ranging from nytimes.com to r/depression on Reddit. We invite submissions to
our benchmark and organize results by comparability based on compliance with
guidelines such as removal of benchmark contamination from pretraining.
Submissions can also record parameter and training token count to make
comparisons of Pareto efficiency for performance as a function of these
measures of cost. We populate our benchmark with results from 6 baselines
pretrained on popular corpora. In case studies, we demonstrate analyses that
are possible with Paloma, such as finding that pretraining without data beyond
Common Crawl leads to inconsistent fit to many domains.
Paloma analyzes the performance of language models across 585 diverse text domains to provide a comprehensive and comparable assessment of their domain fit and cost-effectiveness.
perplexity Perplexity Analysis for Language Model Assessment (Paloma) text domains nytimes.com r/depression benchmark contamination Pareto efficiency parameter count training token count Common Crawl domain fit