PASTA: Pretrained Action-State Transformer Agents
Self-supervised learning has brought about a revolutionary paradigm shift in
various computing domains, including NLP, vision, and biology. Recent
approaches involve pre-training transformer models on vast amounts of unlabeled
data, serving as a starting point for efficiently solving downstream tasks. In
the realm of reinforcement learning, researchers have recently adapted these
approaches by developing models pre-trained on expert trajectories, enabling
them to address a wide range of tasks, from robotics to recommendation systems.
However, existing methods mostly rely on intricate pre-training objectives
tailored to specific downstream applications. This paper presents a
comprehensive investigation of models we refer to as Pretrained Action-State
Transformer Agents (PASTA). Our study uses a unified methodology and covers an
extensive set of general downstream tasks including behavioral cloning, offline
RL, sensor failure robustness, and dynamics change adaptation. Our goal is to
systematically compare various design choices and provide valuable insights to
practitioners for building robust models. Key highlights of our study include
tokenization at the action and state component level, using fundamental
pre-training objectives like next token prediction, training models across
diverse domains simultaneously, and using parameter efficient fine-tuning
(PEFT). The developed models in our study contain fewer than 10 million
parameters and the application of PEFT enables fine-tuning of fewer than 10,000
parameters during downstream adaptation, allowing a broad community to use
these models and reproduce our experiments. We hope that this study will
encourage further research into the use of transformers with first-principles
design choices to represent RL trajectories and contribute to robust policy
learning.
64ba0b7962c0bfc573974118
Raphael Boige
64ba0b7962c0bfc573974119
Yannis Flet-Berliac
admin_assigned
2023-07-21T08:30:09.111Z
64a5623d3e255e5c57dd5efb
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/0ZtKZi2UevQ1AJlw73hq4.jpeg
Yannis Flet-Berliac
user
yfletberliac
64ba0b7962c0bfc57397411a
Arthur Flajolet
admin_assigned
2023-07-21T08:30:18.936Z
64267de5f8868e1ac351095e
https://cdn-avatars.huggingface.co/v1/production/uploads/64267de5f8868e1ac351095e/auPKussoQAlvJtZOtysZD.jpeg
Arthur Flajolet
user
aflajolet
64ba0b7962c0bfc57397411b
Guillaume Richard
admin_assigned
2023-07-21T08:31:07.624Z
6426b768bc4f1d51f546e8c5
/avatars/0cce972b561d050d9fb5569a9d8f3e48.svg
Guillaume Richard
user
GRichard513
64ba0b7962c0bfc57397411c
Thomas Pierrot
admin_assigned
2023-07-21T08:31:21.535Z
63d048c28cf6c8e4d208b072
https://cdn-avatars.huggingface.co/v1/production/uploads/63d048c28cf6c8e4d208b072/sGrp6YITRr1QUtgLWaMKY.jpeg
Thomas Pierrot
user
tpierrot
PASTA: Pretrained Action-State Transformer Agents
Self-supervised learning has brought about a revolutionary paradigm shift in
various computing domains, including NLP, vision, and biology. Recent
approaches involve pre-training transformer models on vast amounts of unlabeled
data, serving as a starting point for efficiently solving downstream tasks. In
the realm of reinforcement learning, researchers have recently adapted these
approaches by developing models pre-trained on expert trajectories, enabling
them to address a wide range of tasks, from robotics to recommendation systems.
However, existing methods mostly rely on intricate pre-training objectives
tailored to specific downstream applications. This paper presents a
comprehensive investigation of models we refer to as Pretrained Action-State
Transformer Agents (PASTA). Our study uses a unified methodology and covers an
extensive set of general downstream tasks including behavioral cloning, offline
RL, sensor failure robustness, and dynamics change adaptation. Our goal is to
systematically compare various design choices and provide valuable insights to
practitioners for building robust models. Key highlights of our study include
tokenization at the action and state component level, using fundamental
pre-training objectives like next token prediction, training models across
diverse domains simultaneously, and using parameter efficient fine-tuning
(PEFT). The developed models in our study contain fewer than 10 million
parameters and the application of PEFT enables fine-tuning of fewer than 10,000
parameters during downstream adaptation, allowing a broad community to use
these models and reproduce our experiments. We hope that this study will
encourage further research into the use of transformers with first-principles
design choices to represent RL trajectories and contribute to robust policy
learning.
Pretrained Action-State Transformer Agents (PASTA) are investigated for a range of reinforcement learning tasks using unified methodologies and parameter-efficient fine-tuning.
self-supervised learning transformer models reinforcement learning expert trajectories behavioral cloning offline RL sensor failure robustness dynamics change adaptation tokenization next token prediction parameter-efficient fine-tuning PEFT