#
Code Example: Datasets, in practice

This section shows you how to work with robotics datasets from Hugging Face using the LeRobotDataset class. We’ll start with simple examples and gradually add complexity, so you can copy and adapt the approach that best fits your project.

The key thing to understand is that any dataset on the Hub that follows LeRobot’s format (with tabular data, visual data, and metadata included) can be loaded with just one line of code.

When working with robotics data, you often need to look at multiple time steps at once rather than single data points. Why? Most robot learning algorithms need to see how things change over time. For example, to pick up an object, a robot might need to see what happened in the last few moments to understand the current situation better. Similarly, many algorithms work better when they can plan several actions ahead rather than just deciding what to do right now.

LeRobotDataset makes this easy with “temporal windowing.” You simply declare which time offsets you want (i.e. current frame plus the two previous ones), and it automatically handles the complexity of getting those frames, even when some might be missing at the beginning or end of an episode.

> Temporal Windows Explained:

> - Observation history: `[-0.2, -0.1, 0.0]` gives you 200ms, 100ms, and current observations
> - Action sequences: `[0.0, 0.1, 0.2]` provides current and next 2 actions (100ms apart)
> - Automatic padding: Missing frames at episode boundaries are handled automatically. The datasets always returns the requested number of frames, and it applies padding where necessary.
> - Mask included: Know which frames are real vs. padded for proper training

Conveniently, by using LeRobotDataset with a PyTorch `DataLoader` one can automatically collate the individual sample dictionaries from the dataset into a single dictionary of batched tensors for downstream training or inference. LeRobotDataset also natively supports streaming mode for datasets. Users can stream data of a large dataset hosted on the Hugging Face Hub, with a one-line change in their implementation. Streaming datasets supports high-performance batch processing (ca. 80-100 it/s, varying on connectivity) and high levels of frames randomization, key features for practical BC algorithms which otherwise may be slow or operating on highly non-i.i.d. data. This feature is designed to improve on accessibility so that large datasets can be processed by users without requiring large amounts of memory and storage.

Here are different ways to set up temporal windows depending on your use case. Skim the options and pick one to start—switching later is just a change to the dictionary.

basic-bc
history-bc
action-chunking

Basic Behavioral Cloning (learn current action from current observation):

    # Simple: current observation → current action
    delta_timestamps = {
        "observation.images.wrist_camera": [0.0],  # Just current frame
        "action": [0.0]  # Just current action
    }

    dataset = LeRobotDataset(
        "lerobot/svla_so101_pickplace",
        delta_timestamps=delta_timestamps
    )

###
Streaming Large Datasets

> When to use streaming:

> - Dataset > available storage - Stream datasets that don’t fit on your disk
> - Experimentation - Quickly try different datasets without downloading
> - Cloud training - Reduce startup time by streaming from Hugging Face Hub
> - Network available - Requires stable internet connection during training

> Performance: Streaming achieves 80-100 it/s with good connectivity! That is (on average) comparable with locally-stored datasets, factoring out initialization overhead.

download
streaming

Download Dataset (faster training, requires storage):

    from lerobot.datasets.lerobot_dataset import LeRobotDataset

    # Downloads dataset to local cache
    dataset = LeRobotDataset("lerobot/svla_so101_pickplace")

    # Fastest access after download
    sample = dataset[100]

##
Training Integration

You can easily integrate regular and streaming datasets with torch data loaders. This makes integrating any LeRobotDataset with your own (`torch`) training loop rather convenient. Because we fetch all frames from the datasets as a tensor, wrapping iterating over a dataset with training is particularly straightforward.

###
PyTorch DataLoader

You can easily integrate regular and streaming datasets with torch data loaders. This makes integrating any LeRobotDataset with your own (`torch`) training loop rather convenient. Because we fetch all frames from the datasets as a tensor, wrapping iterating over a dataset with training is particularly straightforward.

    import torch
    from torch.utils.data import DataLoader
    # Create DataLoader for training
    dataloader = DataLoader(
        dataset,
        batch_size=16,
        shuffle=True,
        num_workers=4
    )

    # Training loop
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

    for batch in dataloader:
        # Move to device
        observations = batch["observation.state"].to(device)
        actions = batch["action"].to(device)
        images = batch["observation.images.wrist_camera"].to(device)

        # Your model training here
        # loss = model(observations, images, actions)
        # loss.backward()
        # optimizer.step()

##
Why This Matters

This simple API hides significant complexity:

- ✅ Multi-modal synchronization - Images and sensors perfectly aligned
- ✅ Efficient storage - Compressed videos, memory-mapped arrays
- ✅ Temporal handling - Easy access to observation/action sequences
- ✅ Scalability - Same code works for small and massive datasets

Compare this to traditional robotics data handling, which often requires:

- Custom parsers for each data format
- Manual synchronization across modalities
- Complex buffering for temporal windows
- Platform-specific loading code

LeRobotDataset standardizes and simplifies all of this!

##
Section Quiz

Test your understanding of LeRobot and its role in robot learning:

###
1. What makes LeRobot different from traditional robotics libraries?

It only works with simulation environments. It provides end-to-end integration across the entire robotics stack with state-of-the-art learning algorithms. It requires expensive industrial robots to function. It only supports classical control methods.

###
2. Which of the following is NOT a key component of LeRobot’s approach?

Unified low-level robot configuration handling Native robotics dataset format (LeRobotDataset) Requiring expert knowledge for each new robot platform State-of-the-art learning algorithms with PyTorch implementations

###
3. What is the main advantage of LeRobot’s optimized inference stack?

It makes training faster on GPUs. It reduces the memory requirements for storing datasets. It decouples action planning from action execution for better real-time performance. It automatically generates training data from robot interactions.

###
4. Which types of robotic platforms does LeRobot support?

Only manipulation robots like robotic arms. Manipulation, locomotion, and whole-body control platforms. Only robots that cost more than $10,000. Only robots manufactured by specific companies.

###
5. What does “end-to-end integration with the robotics stack” mean in the context of LeRobot?

It only handles high-level planning, not low-level control. It covers everything from low-level hardware control to high-level learning algorithms. It requires separate tools for data handling and model training. It only works with specific operating systems.

###
6. What is the primary purpose of the delta_timestamps parameter in LeRobotDataset?

It sets the frame rate for video recording. It defines temporal windows to access observation histories and action sequences. It synchronizes data across different robots. It compresses video data for storage efficiency.

###
7. Which of the following best describes the three main components of LeRobotDataset?

Images, Actions, and Rewards Tabular Data, Visual Data, and Metadata Training, Validation, and Test sets Simulation, Real Robot, and Hybrid data

###
8. What happens when you use StreamingLeRobotDataset instead of LeRobotDataset ?

The data is automatically augmented for better training. The dataset is downloaded faster to your local machine. Data is streamed from the Hugging Face Hub without downloading, saving storage space. The dataset is automatically split into train/validation sets.

###
9. In the context of robot learning, what does “temporal windowing” refer to?

The time it takes to train a robot learning model. Accessing multiple time steps of observations and actions around a given frame. The frequency at which robot sensors collect data. The duration of each robot episode or trajectory.

###
10. What is the main advantage of LeRobotDataset’s approach to storing video data?

Videos are stored in the highest possible quality. Each frame is stored as a separate file for easy access. Multiple episodes are concatenated into larger MP4 files to reduce file system stress. Videos are automatically compressed using AI algorithms.

###
11. Which statement about LeRobotDataset’s compatibility is correct?

It only works with specific robot brands like SO-100. It requires custom code for each new robot platform. It integrates seamlessly with PyTorch DataLoader and Hugging Face ecosystems. It only supports simulation data, not real robot data.

##
References

For a full list of references, check out the tutorial.

- Diffusion Policy: Visuomotor Policy Learning via Action Diffusion (2024)
  Cheng Chi et al.
  This paper introduces diffusion models for robot policy learning and discusses how temporal windowing and action chunking enable smooth visuomotor control.
  arXiv:2303.04137

- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (2023)
  Anthony Brohan et al.
  Demonstrates how vision-language models can be fine-tuned for robotic control, including discussion of temporal context windows and action prediction horizons.
  arXiv:2307.15818

Update on GitHub