OmniJARVIS: Unified Vision-Language-Action Tokenization Enables
  Open-World Instruction Following Agents
We present OmniJARVIS, a novel Vision-Language-Action (VLA) model for
open-world instruction-following agents in open-world Minecraft. Compared to
prior works that either emit textual goals to separate controllers or produce
the control command directly, OmniJARVIS seeks a different path to ensure both
strong reasoning and efficient decision-making capabilities via unified
tokenization of multimodal interaction data. First, we introduce a
self-supervised approach to learn a behavior encoder that produces discretized
tokens for behavior trajectories tau = {o_0, a_0, dots} and an
imitation learning (IL) policy decoder conditioned on these tokens. These
additional behavior tokens will be augmented to the vocabulary of pretrained
Multimodal Language Models (MLMs). With this encoder, we then pack long-term
multimodal interactions involving task instructions, memories, thoughts,
observations, textual responses, behavior trajectories, etc. into unified token
sequences and model them with autoregressive transformers. Thanks to the
semantically meaningful behavior tokens, the resulting VLA model, OmniJARVIS,
can reason (by producing chain-of-thoughts), plan, answer questions, and act
(by producing behavior tokens for the IL policy decoder). OmniJARVIS
demonstrates excellent performances on a comprehensive collection of atomic,
programmatic, and open-ended tasks in open-world Minecraft. Our analysis
further unveils the crucial design principles in interaction data formation,
unified tokenization, and its scaling potentials.
6683bcf921fcc56c785f6d0a
Zihao Wang
claimed_verified
2024-12-17T08:05:06.581Z
642e8c99c1b0f8e4e76bcaab
https://cdn-avatars.huggingface.co/v1/production/uploads/642e8c99c1b0f8e4e76bcaab/BOs9r0P9KyT9pEba9v0H4.png
Zihao Wang
user
zhwang4ai
6683bcf921fcc56c785f6d0b
Shaofei Cai
claimed_verified
2024-07-03T07:41:07.815Z
6578459d62d3ac1817ed79fe
https://cdn-avatars.huggingface.co/v1/production/uploads/6578459d62d3ac1817ed79fe/AXDJuwLUoEOb4Fj3U0Xxo.jpeg
Shaofei Cai
user
phython96
6683bcf921fcc56c785f6d0c
Zhancun Mu
claimed_verified
2025-04-08T07:01:23.829Z
648f0c64ddc0620d54e199e9
/avatars/ff3fd6e11d451b03f82183e53fc48613.svg
ZhancunMu
user
Zhancun
6683bcf921fcc56c785f6d0d
Haowei Lin
claimed_verified
2025-04-08T07:01:26.143Z
63453f02a05b51f7ded3c579
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/u7p-wTj8zyeWpSFRI8Go0.png
Andy Lin
user
pkuHaowei
6683bcf921fcc56c785f6d0e
Ceyao Zhang
6683bcf921fcc56c785f6d0f
Xuejie Liu
6683bcf921fcc56c785f6d10
Qing Li
claimed_verified
2024-08-03T08:26:12.303Z
6455f8c2c8f569b995d603c9
/avatars/3cb2cad3ab123887c270234bb6a4ca43.svg
Qing Li
user
li-qing
6683bcf921fcc56c785f6d11
Anji Liu
claimed_verified
2024-10-28T15:16:15.361Z
66757955b3882fd587d5f363
/avatars/41fdfa19057ad3ea5ece29b94f163218.svg
Anji Liu
user
anjiliu
6683bcf921fcc56c785f6d12
Xiaojian Ma
admin_assigned
2024-07-03T08:17:50.054Z
60dd0e36a15ddd7d2006d2e9
/avatars/8bd98177a79efbf295be8f6457683297.svg
Xiaojian Ma
user
jeasinema
6683bcf921fcc56c785f6d13
Yitao Liang
claimed_verified
2025-04-09T07:38:35.013Z
64683a5776bb704aa14588b7
/avatars/e532756f52c5b95981470ace41a10556.svg
Yitao Liang
user
YitaoLiang
OmniJARVIS: Unified Vision-Language-Action Tokenization Enables
  Open-World Instruction Following Agents
We present OmniJARVIS, a novel Vision-Language-Action (VLA) model for
open-world instruction-following agents in open-world Minecraft. Compared to
prior works that either emit textual goals to separate controllers or produce
the control command directly, OmniJARVIS seeks a different path to ensure both
strong reasoning and efficient decision-making capabilities via unified
tokenization of multimodal interaction data. First, we introduce a
self-supervised approach to learn a behavior encoder that produces discretized
tokens for behavior trajectories tau = {o_0, a_0, dots} and an
imitation learning (IL) policy decoder conditioned on these tokens. These
additional behavior tokens will be augmented to the vocabulary of pretrained
Multimodal Language Models (MLMs). With this encoder, we then pack long-term
multimodal interactions involving task instructions, memories, thoughts,
observations, textual responses, behavior trajectories, etc. into unified token
sequences and model them with autoregressive transformers. Thanks to the
semantically meaningful behavior tokens, the resulting VLA model, OmniJARVIS,
can reason (by producing chain-of-thoughts), plan, answer questions, and act
(by producing behavior tokens for the IL policy decoder). OmniJARVIS
demonstrates excellent performances on a comprehensive collection of atomic,
programmatic, and open-ended tasks in open-world Minecraft. Our analysis
further unveils the crucial design principles in interaction data formation,
unified tokenization, and its scaling potentials.
OmniJARVIS, a Vision-Language-Action model for Minecraft, uses unified tokenization of multimodal interaction data with a behavior encoder and autoregressive transformers to perform reasoning, planning, question answering, and action execution.
self-supervised learning behavior encoder imitation learning policy decoder Multimodal Language Models unified tokenization autoregressive transformers chain-of-thoughts