Cognitively Inspired Energy-Based World Models
One of the predominant methods for training world models is autoregressive
prediction in the output space of the next element of a sequence. In Natural
Language Processing (NLP), this takes the form of Large Language Models (LLMs)
predicting the next token; in Computer Vision (CV), this takes the form of
autoregressive models predicting the next frame/token/pixel. However, this
approach differs from human cognition in several respects. First, human
predictions about the future actively influence internal cognitive processes.
Second, humans naturally evaluate the plausibility of predictions regarding
future states. Based on this capability, and third, by assessing when
predictions are sufficient, humans allocate a dynamic amount of time to make a
prediction. This adaptive process is analogous to System 2 thinking in
psychology. All these capabilities are fundamental to the success of humans at
high-level reasoning and planning. Therefore, to address the limitations of
traditional autoregressive models lacking these human-like capabilities, we
introduce Energy-Based World Models (EBWM). EBWM involves training an
Energy-Based Model (EBM) to predict the compatibility of a given context and a
predicted future state. In doing so, EBWM enables models to achieve all three
facets of human cognition described. Moreover, we developed a variant of the
traditional autoregressive transformer tailored for Energy-Based models, termed
the Energy-Based Transformer (EBT). Our results demonstrate that EBWM scales
better with data and GPU Hours than traditional autoregressive transformers in
CV, and that EBWM offers promising early scaling in NLP. Consequently, this
approach offers an exciting path toward training future models capable of
System 2 thinking and intelligently searching across state spaces.
666ba21cd3dbd0378e786601
Alexi Gladstone
extracted_confirmed
2024-06-14T01:59:28.888Z
63fbbff41b4b1bd4e7042fd1
/avatars/d186d0859046b35723fe1156d8f68917.svg
Alexi Gladstone
user
alexiglad
666ba21cd3dbd0378e786602
Ganesh Nanduru
admin_assigned
2024-06-14T11:55:39.526Z
64f518ecdf78ce3cc8526639
/avatars/7b6ac7b1ff061f392fccf4af66f831d2.svg
Ganesh Nanduru
user
gnanduru
666ba21cd3dbd0378e786603
Md Mofijul Islam
admin_assigned
2024-06-14T11:55:45.553Z
60d599155a14347cd1f6c6eb
/avatars/c623abb79f6dc48b54f2fb8ba64bb4a4.svg
Md Mofijul Islam
user
mmiakashs
666ba21cd3dbd0378e786604
Aman Chadha
admin_assigned
2024-06-14T11:55:50.871Z
63a4754927f1f64ed7238dac
https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg
Aman Chadha
user
amanchadha
666ba21cd3dbd0378e786605
Jundong Li
666ba21cd3dbd0378e786606
Tariq Iqbal
Cognitively Inspired Energy-Based World Models
One of the predominant methods for training world models is autoregressive
prediction in the output space of the next element of a sequence. In Natural
Language Processing (NLP), this takes the form of Large Language Models (LLMs)
predicting the next token; in Computer Vision (CV), this takes the form of
autoregressive models predicting the next frame/token/pixel. However, this
approach differs from human cognition in several respects. First, human
predictions about the future actively influence internal cognitive processes.
Second, humans naturally evaluate the plausibility of predictions regarding
future states. Based on this capability, and third, by assessing when
predictions are sufficient, humans allocate a dynamic amount of time to make a
prediction. This adaptive process is analogous to System 2 thinking in
psychology. All these capabilities are fundamental to the success of humans at
high-level reasoning and planning. Therefore, to address the limitations of
traditional autoregressive models lacking these human-like capabilities, we
introduce Energy-Based World Models (EBWM). EBWM involves training an
Energy-Based Model (EBM) to predict the compatibility of a given context and a
predicted future state. In doing so, EBWM enables models to achieve all three
facets of human cognition described. Moreover, we developed a variant of the
traditional autoregressive transformer tailored for Energy-Based models, termed
the Energy-Based Transformer (EBT). Our results demonstrate that EBWM scales
better with data and GPU Hours than traditional autoregressive transformers in
CV, and that EBWM offers promising early scaling in NLP. Consequently, this
approach offers an exciting path toward training future models capable of
System 2 thinking and intelligently searching across state spaces.
Energy-Based World Models (EBWM) improve upon traditional autoregressive models by incorporating human-like capabilities such as evaluating prediction plausibility and adapting prediction efforts, achieving better scalability in CV and early promise in NLP.
autoregressive prediction Large Language Models (LLMs) Natural Language Processing (NLP) Computer Vision (CV) Energy-Based Model (EBM) Energy-Based World Models (EBWM) Energy-Based Transformer (EBT) System 2 thinking high-level reasoning plausibility evaluation adaptive prediction