\section{Introduction}
Estimating 3D human motion from an egocentric video that records the environment viewed from the first-person perspective is critical to applications in VR/AR, in which the default sensor is a front-facing monocular camera. However, naively learning a mapping between egocentric videos and full-body human motions is challenging for two reasons. First, modeling this complex relationship is difficult: unlike reconstruction motion from third-person videos, the human body is often out of view of an egocentric video. Second, learning this mapping requires a large-scale, diverse dataset containing egocentric videos and corresponding 3D human poses. Such a dataset demands careful instrumentation for data acquisition and this type of dataset does not exist. As such, existing works have only worked on small-scale datasets with limited motion and scene diversity~\cite{yuan20183d, yuan2019ego,luo2021dynamics}.

We introduce a generalized and robust method, \textbf{EgoEgo\xspace}, to estimate full-body human motions from only egocentric video for a wide range of diverse scenarios. Our key idea is to use head motion as an intermediate representation to decompose the problem into two stages: head motion estimation from the input egocentric video and full-body motion estimation from the estimated head motion. For most day-to-day activities, humans have an extraordinary ability to stabilize the head such that it aligns with the center of mass of the body \cite{keshner1988motor}, which makes head motion an excellent feature for full-body motion estimation. More importantly, the decomposition of our method removes the need to learn from paired egocentric videos and human poses, enabling learning from a combination of large-scale, single-modality datasets (e.g\onedot} \def\Eg{E.g\onedot, datasets with egocentric videos or 3D human poses only), which are commonly and readily available.

The first stage, estimating the head pose from an egocentric video, resembles the localization problem. However, applying the state-of-the-art monocular SLAM methods~\cite{teed2021droid} yields unsatisfactory results due to the unknown gravity direction and the scaling difference between the estimated space and the real 3D world. We propose a hybrid solution that leverages SLAM and learned transformer-based models to achieve significantly more accurate head motion estimation from egocentric video.
In the second stage, we generate the full-body motion based on a conditional diffusion model that takes the predicted head pose as input. Finally, to evaluate our method and train other baselines, we build a large-scale synthetic dataset with paired egocentric video and 3D human motions, which can also be useful for future works on visuomotor skill learning and sim-to-real transfer learning.

To summarize, our work made four main contributions. First, we propose a decomposition paradigm, \textbf{EgoEgo}, to decouple the problem of motion estimation from egocentric video into two stages: ego-head pose estimation, and ego-body pose estimation conditioned on the head pose. The decomposition enables us to learn each component separately, eliminating the need for a large-scale dataset with two paired modalities. Second, we develop a hybrid approach for ego-head pose estimation, integrating the results of monocular SLAM and learning. Third, we propose a conditional diffusion model to generate full-body poses conditioned on the head pose. Finally, we contribute a large-scale synthetic dataset with both egocentric videos and 3D human motions as a test bed to benchmark different approaches for this problem and showcase our method outperforms the baselines by a large margin.
\section{Related Work}
\paragraph{Motion Estimation from Third-person Video.}
3D pose estimation from images and videos in third-person view has been extensively studied in recent years. There are mainly two typical categories in this direction. One is to regress joint positions directly from images and videos~\cite{zhou2016sparseness,pavlakos2017coarse,mehta2017vnect,tung2017self}. The other adopts parametric human body model~\cite{SMPL:2015} to estimate body model parameters from images or videos~\cite{kanazawa2018end,kolotouros2019learning,kocabas2021pare,kocabas2021spec,kocabas2020vibe,choi2021beyond,luo20203d}.
And recently, learned motion prior is applied to address the issues of jitters, lack of global trajectory, and missing joints or frames~\cite{yuan2021glamr, rempe2021humor, li2021task}. Moreover, physical constraints are enforced in motion estimation from videos~\cite{yuan2021simpoe,xie2021physics}.
In contrast to third-person videos, where the full body can be seen, body joints are mostly not visible in an egocentric video, which poses a significant challenge for the problem. Although the body joints are unobserved from egocentric views, the visual information of how the environment changes provides a strong signal to infer how the head moves. In this work, we propose to use the head pose as an intermediate representation to bridge the egocentric video and full-body motions.
\vspace{-4mm}
\paragraph{Motion Estimation from Egocentric Video.}
Growing attention is received in pose estimation from egocentric videos. Special hardware like the fisheye camera is deployed to predict full body pose from captured images~\cite{wang2021estimating, tome2019xr,xu2019mo,jiang2021egocentric}. While body joints are usually visible in images captured with a fisheye camera, the distortion of images poses a significant challenge. Jiang et al\onedot~\cite{jiang2017seeing} deploy a standard camera to a human chest and propose an implicit motion graph matching approach to predict full body motions from the input video. You2Me~\cite{ng2020you2me} predicts full body motions by observing the interaction pose of the second-person in the camera view. Towards a goal of estimating and forecasting physically plausible motions from a head-mounted camera, EgoPose~\cite{yuan2019ego,yuan20183d} develop a Deep-RL framework to learn a control policy to estimate current poses and forecast future poses. Follow-up work Kinpoly~\cite{luo2021dynamics} integrates kinematics and dynamics to predict physically plausible motions interacting with known objects. While their method achieves impressive results in their collected dataset, it cannot handle scenes and motions out of their data distribution. This work aims to establish a more generalized and robust framework to infer full-body motions from egocentric video only. To validate the effectiveness in more generalized scenes and motions, we also introduce an approach of synthesizing egocentric video corresponding to mocap data in diverse 3D scenes for quantitative evaluation.
\vspace{-4mm}
\paragraph{Motion Estimation from Sparse Sensors.}
Instead of estimating motion from videos, some work explored reconstructing human motions from sparse sensor input. TransPose~\cite{yi2021transpose} proposes a real-time pipeline to predict full body motions from 6 IMU sensors, including head, torso, left/right arms, and left/right lower legs.
Follow-up work PIP~\cite{yi2022physical} further includes a PD controller on top of the kinematic estimator to introduce physics constraints during motion reconstruction. TIP~\cite{jiang2022transformer} follows the same sensor setting and deploys a transformer-based model to leverage IMU sequential information effectively. Fewer sensors are investigated in LoBSTr~\cite{yang2021lobstr}. Given tracker information from 4 joints (head, left/right hands, torso), they present an RNN-based model to infer lower-body motions from past upper-body joint signals. Recent advances~\cite{dittadi2021full,aliakbarian2022flag,winkler2022questsim,jiang2022avatarposer} further relax the input constraints to head and hand signals only.
In this work, we do not rely on any observations from inertial sensors. Instead, we aim to develop a solution with egocentric video as the only input.

\section{Method}

Our method, EgoEgo\xspace, estimates 3D human motion from a monocular egocentric video sequence. As shown in \fig{fig:overview}, our key idea is to leverage \emph{head motion}: first estimating head motion from egocentric video, and then estimating full body motion from head motion. We show that head motion is an excellent feature for full-body motion estimation and a compact, intermediate representation that reduces the challenge into two much simpler sub-problems. Such a disentanglement also allows us to leverage a large-scale egocentric video dataset with head motion (but no full body motion) in stage one, and a separate 3D human motion dataset (but no egocentric videos) in stage two.

\vspace{-4mm}
\paragraph{Notations.} We denote full body motion as $\bm{X} \in \mathbb{R}^{T \times D}$ and egocentric images captured from a front-facing, head-mounted camera as $\bm{I} \in \mathbb{R}^{T \times h \times w \times 3}$, where $T$ is the sequence length, $D$ is the dimension of the pose state, and $h \times w$ is the size of an image. We introduce head motion $\bm{H} \in \mathbb{R}^{T \times D'}$ as an intermediate representation to bridge the input egocentric video and the output human motions, where $D'$ is the dimension of the head pose.

\subsection{Head Pose Estimation from Egocentric Video}
Estimating the head motion from an egocentric video can be viewed as a camera localization problem. However, we observed three issues that prevent us from directly applying the state-of-the-art SLAM method \cite{teed2021droid} to our problem. First, the gravity direction of the estimated head pose is unknown. Thus, the results cannot be directly fed to the full-body motion estimator since it expects the head pose expressed in a coordinate frame where the gravity direction  is $[0, 0, -1]^T$. Second, the estimated translation by monocular SLAM is not to scale when compared with the distance in the real world. Third, monocular SLAM tends to be less accurate in estimating relative head rotation than relative head translation.

Based on these observations, we propose a hybrid method that leverages SLAM and learned models to achieve more accurate head pose estimation than the state-of-the-art SLAM alone. First, we develop a transformer-based model GravityNet to estimate the gravity direction from the rotation and the translation trajectories computed by SLAM. We rotate the SLAM translation by aligning the estimated gravity direction with the real gravity direction $[0, 0, -1]^{T}$ in the 3D world. Moreover, from the optical flow features extracted from the egocentric video, our method learns a model, HeadNet, to estimate head rotations and translation distance. The predicted translation distance of HeadNet is used to re-scale the translation estimated by SLAM. The predicted head rotation by HeadNet is directly used to replace the rotation estimated by SLAM. Figure \ref{fig:overview} summarizes our process to generate head poses.

\vspace{-4mm}
\paragraph{Monocular SLAM.}
We adopt DROID-SLAM~\cite{teed2021droid} to estimate camera trajectory from egocentric videos. DROID-SLAM~\cite{teed2021droid} is a learning-based method to estimate camera pose trajectory and reconstruct the 3D map of the environment simultaneously. By a design of recurrent iterative updates to camera pose and depth, it showcases superior and more robust results compared to prior SLAM systems~\cite{campos2021orb}. For more details, please refer to ~\cite{teed2021droid}.

\begin{figure*}[t!]
\begin{center}
\vspace{-5mm}
\includegraphics[width=\textwidth]{figures/Figure1.pdf}
\end{center}
\vspace{-6mm}
\caption{Overview of EgoEgo. We take egocentric video as input and predict head poses with a hybrid approach that combines monocular SLAM and learned GravityNet and HeadNet. The predicted head poses are then fed to a conditional diffusion model to generate the full-body poses. }
\label{fig:overview}
\vspace{-1mm}
\end{figure*}

\begin{figure}[t!]
\begin{center}
\vspace{-2mm}
\includegraphics[width=0.48\textwidth]{figures/Figure_diffusion_model.pdf}
\end{center}
\vspace{-6mm}
  \caption{Model architecture of the denoising network in a single step of the reverse diffusion process.}

  \label{fig:full_motion_estimator}
\vspace{-4mm}
\end{figure}

\vspace{-4mm}
\paragraph{Gravity Direction Estimation.}
We introduce GravityNet to predict gravity direction $\bm{g} \in \mathbb{R}^3$ from a sequence of head poses $\bm{\hat{h}}_{1}, \bm{\hat{h}}_{2}, ..., \bm{\hat{h}}_{T}$. The gravity direction $\bm{g}$ is represented by a unit vector. The head poses input $\bm{\hat{h}}_{t}$ consists of a 3D head translation, a head rotation represented by a continuous 6D rotation vector~\cite{zhou2019continuity}, head translation difference, and head rotation difference computed by $\bm{O}_{t-1}^{-1}\bm{O}_{t}$ where $\bm{O}_{t}$ denotes the head rotation matrix at time step $t$. We adopt a transformer-based architecture~\cite{vaswani2017attention} consisting of two self-attention blocks, each of which has a multi-head attention layer followed by a position-wise feed-forward layer. We take the first output of the transformer and feed it to an MLP to predict the gravity direction $\bm{g}$. We train our GravityNet on the large-scale motion capture dataset AMASS~\cite{mahmood2019amass}. However, the motion sequences in AMASS have the correct gravity direction $\bm{g}_c = [0, 0, -1]^T$. To emulate the distribution of the predicted head poses from monocular SLAM, we apply a random scale and a random rotation to the head poses in each AMASS sequence to generate our training data for gravity estimation. $L_1$ loss for the gravity vector is used during training. Based on the prediction of GravityNet, we compute the rotation matrix $\bm{R}_g$ to align the prediction $\bm{g}$ and $\bm{g}_c$. Then we apply $\bm{R}_g$ to the SLAM translation denoted as $\bm{\hat{P}}_1, \bm{\hat{P}}_2, ..., \bm{\hat{P}}_T$ and get $\bm{P}_1, \bm{P}_2, ..., \bm{P}_T$, where $\bm{P}_t = \bm{R}_g\bm{\hat{P}}_t$.

\vspace{-4mm}
\paragraph{Head Pose Estimation.}
We propose HeadNet to predict a sequence of distance $d_1, d_2, ..., d_T$ and head rotations $\bm{R}_1, ..., \bm{R}_T$ from a sequence of optical flow features $\bm{o}_1, ..., \bm{o}_T$. The optical flow features are extracted by a pre-trained ResNet-18~\cite{he2016deep}. We deploy the same model architecture as GravityNet. Since the scale from the monocular SLAM system may not be consistent with the real 3D world where human moves, we use HeadNet to predict the vector norm of the translation difference between consecutive time steps denoted as $d_1, ..., d_T$, where $d_t$ represents a scalar value. For each camera translation sequence produced by monocular SLAM and rotated by aligning gravity direction, given the camera translation trajectory $\bm{P} \in \mathbb{R}^{T \times 3}$, we calculate the distance $d_t^{s}$ between $\bm{P}_{t}$ and $\bm{P}_{t+1}$ as $d_t^{s} = ||\bm{P}_{t+1}-\bm{P}_{t}||_2$. We take the mean value for the sequence of distances as $d^{s} = \frac{1}{T} \sum_{t=1}^{T}d_{t}^{s}$. Similarly, we compute the mean of the predicted distance sequence $d = \frac{1}{T} \sum_{t=1}^{T}d_{t}$. The scale is calculated as $s = \frac{d}{d^{s}}$. We multiply scale $s$ to the predicted translation $\bm{P}$ and use $s\bm{P}$ as our global head translation results.

The network also predicts the head angular velocity, $\bm{\omega}_1, ..., \bm{\omega}_T$, in the head frame. We integrate predicted angular velocity to generate corresponding rotations $\bm{R}_1, ..., \bm{R}_T$. During inference, we assume that the first head orientation is given and integrate the predicted head angular velocity to estimate the subsequent head orientations.

The training loss of the HeadNet is defined as: $\mathcal{L} = \mathcal{L}_{dist} + \mathcal{L}_{vel} + \mathcal{L}_{rot}$. $\mathcal{L}_{dist}$ represents the $L1$ loss for translation distance. $\mathcal{L}_{vel}$ represents the $L1$ loss for angular velocity. $\mathcal{L}_{rot}$ denotes the rotation loss $\mathcal{L}_{rot} = ||\bm{R}_{pred}\bm{R}_{gt}^{T} - \bm{I}||_{1}$ where $\bm{R}_{pred}$ represents the integrated rotation using predicted angular velocity, $\bm{R}_{gt}$ represents the ground truth rotation matrix and $\bm{I}$ represents the identity matrix.

\subsection{Full-Body Pose Estimation from Head Pose}
Predicting full-body pose from head pose is not a one-to-one mapping problem as different full-body motions may have the same head pose.
Thus, we formulate the task using a conditional generative model. Inspired by the recent success of the diffusion model in image generation~\cite{rombach2022high}, we deploy a diffusion model to generate full-body poses conditioned on head poses. We use the formulation proposed in the denoising diffusion probabilistic model (DDPM)~\cite{ho2020denoising}, which has also been applied in some concurrent work~\cite{zhang2022motiondiffuse,kim2022flame,tevet2022human} for motion generation and motion interpolation tasks. We will first introduce our data representation and then detail the conditional diffusion model formulation.

A body pose $\bm{X}_t \in \mathbb{R}^D$ at time $t$ consists of the global joint position ($\mathbb{R}^{J \times 3}$) and global joint rotations ($\mathbb{R}^{J \times 6}$). We adopt the widely used SMPL model~\cite{loper2015smpl} as our skeleton, and the number of joints $J$ is 22. For the convenience of notation in the diffusion model, we use $\bm{x}_n$ to denote a sequence of body poses $\bm{X}_1^n, \bm{X}_2^n, ..., \bm{X}_T^n$ at noise level $n$.

The high-level idea of the diffusion model is to design a forward diffusion process to add Gaussian noises to the original data with a known variance schedule and learn a denoising model to gradually denoise $N$ steps given a sampled $\bm{x}_N$ from a normal distribution to generate $\bm{x}_0$.

Specifically, diffusion models consist of a forward diffusion process and a reverse diffusion process. The forward diffusion process gradually adds Gaussian noise to the original data $\bm{x}_0$. And it is formulated using a Markov chain of $N$ steps as shown in \eqn{eq:forward_diffusion}.
\begin{equation}
\label{eq:forward_diffusion}
    q(\bm{x}_{1:N}|\bm{x}_{0}) := \prod_{n=1}^{N} q(\bm{x}_{n}|\bm{x}_{n-1}).
\end{equation}
Each step is decided by a variance schedule using $\beta_{n}$ and is defined as follows,
\begin{equation}
\label{eq:forward_diffusion_step}
    q(\bm{x}_{n}|\bm{x}_{n-1}) := \mathcal{N}(\bm{x}_{n}; \sqrt{1-\beta_{n}}\bm{x}_{n-1}, \beta_{n}\bm{I}).
\end{equation}
To generate full-body motion conditioned on the head pose, we need to reverse the diffusion process. The reverse process can be approximated as a Markov chain with a learned mean and fixed variance.
\begin{equation}
\label{eq:reverse_diffusion_step}
    p_{\theta}(\bm{x}_{n-1}|\bm{x}_{n}, c) := \mathcal{N}(\bm{x}_{n-1}; \bm{\mu}_{\theta}(\bm{x}_n, n, c), \sigma_{n}^{2}\bm{I}).
\end{equation}
where $\bm{\theta}$ represents the parameters of a neural network, $c$ is the head conditions. The learned mean $\bm{\mu}_{\theta}(\bm{x}_n, n, c)$ (we use $\bm{\mu}_{\theta}$ in the equation for brevity) can be represented as follows where $\alpha_{n}$ and $\bar{\alpha}_{n}$ are fixed parameters, $\hat{\bm{x}}_{\theta}(\bm{x}_{n}, n, c)$ is the prediction of $\bm{x}_0$.
\begin{equation}
\label{eq:reverse_diffusion_step}
     \bm{\mu}_{\theta} = \frac{\sqrt{\alpha_{n}}(1-\bar{\alpha}_{n-1})\bm{x}_{n}+\sqrt{\bar{\alpha}_{n-1}}(1-\alpha_{n})\hat{\bm{x}}_{\theta}(\bm{x}_{n}, n, c)}{1-\bar{\alpha}_{n}}
\end{equation}
Learning the mean can be reparameterized as learning to predict the original data $\bm{x}_0$. The training loss is defined as a reconstruction loss of $\bm{x}_{0}$.
\begin{equation}
\label{eq:loss}
     \mathcal{L} = \mathbb{E}_{\bm{x}_0, n}||\hat{\bm{x}}_{\theta}(\bm{x}_{n}, n, c) - \bm{x}_{0}||_{1}
\end{equation}
As shown in Figure~\ref{fig:full_motion_estimator}, in denoising step $n$, we concatenate head pose condition $\bm{c}_1, ..., \bm{c}_T$ with body pose representation $\bm{X}_{1}^{n}, ..., \bm{X}_{T}^{n}$ at noise level $n$, combined with noise embedding as input to a transformer model, and estimate $\bm{x}_{0}$.

\begin{figure*}[t!]
\begin{center}
\includegraphics[width=\textwidth]{figures/amass_syn_data.pdf}
\end{center}
\vspace{-2mm}
\caption{Illustration of our ARES Dataset. ``Ego-View'' represents the synthetic egocentric images using our proposed data generation pipeline. We also provide a third-person view reference in the second row. The left and right show two sequences from different scenes.}
\label{fig:syn_data}
\vspace{-4mm}
\end{figure*}

\subsection{Synthetic Data Generation}
Our method does not need paired training data. Still, for benchmarking purposes, we develop a way to automatically synthesize a large-scale dataset with various paired egocentric videos and human motions.
\vspace{-4mm}
\paragraph{Generate Motions in 3D Scenes.}
To generate a dataset with both egocentric video and ground truth human motions, we use a large-scale motion capture dataset AMASS~\cite{mahmood2019amass} and a 3D scene dataset Replica~\cite{straub2019replica}. We convert the scene mesh from Replica to the signed distance field (SDF) for the penetration calculation. We divide each sequence of AMASS~\cite{mahmood2019amass} into sub-sequences with 150 frames. For each sub-sequence, based on the semantic annotation provided by Replica~\cite{straub2019replica}, we place the first pose in a random location with the feet in contact with the floor. Then we calculate penetration loss following Wang et al\onedot~\cite{wang2021synthesizing} for each pose in this sequence. We empirically set the threshold to 2 and only keep the poses with penetration loss less than the threshold. Specifically, for human mesh $\bm{M}_t$ at time $t$ represented by a parameterized human model~\cite{SMPL:2015,SMPL-X:2019}, we denote $d_i$ as the signed distance of vertex $i$. The penetration loss is then defined as $L_{pen}^{t} = \sum_{d_i < 0} ||d_{i}||$.

\vspace{-4mm}
\paragraph{Synthesize Realistic Egocentric Images.}

The motion sequences produced by detecting penetration with 3D scenes provide the camera pose trajectories to render synthetic egocentric videos. AI Habitat~\cite{habitat19iccv,szot2021habitat} is a platform for embodied agent research that supports fast rendering given a camera trajectory and a 3D scene. We feed the head pose trajectories to the platform and synthesize realistic images in the egocentric view. We generate 1,664,616 frames with 30 fps, approximately 15 hours of motion in 18 scenes. We name the synthetic dataset AMASS-Replica-Ego-Syn (ARES) and show some examples from our synthetic dataset in Figure~\ref{fig:syn_data}.

\section{Experiments}

\subsection{Datasets and Evaluation Metrics}

\noindent\textbf{AMASS-Replica-Ego-Syn (ARES)} is our synthetic dataset which contains synthetic egocentric videos and ground truth motions. ARES contains about 15 hours of motion across 18 scenes. We remove 5 scenes from training as unseen scenes. The training dataset consists of about 1.2M frames in 13 different scenes. The testing dataset contains 34, 850 frames from 5 unseen scenes.

\noindent\textbf{AMASS}~\cite{mahmood2019amass} is a large-scale motion capture dataset with about 45 hours of diverse motions. We split training and testing data following HuMoR~\cite{rempe2021humor}.

\noindent\textbf{Kinpoly-MoCap}~\cite{luo2021dynamics} consists of egocentric videos captured using a head-mounted camera and corresponding 3D motions captured with motion capture devices. The total duration of the motion is about 80 minutes. Since it uses motion capture devices, the egocentric video is constrained to a single lab scene.

\noindent\textbf{Kinpoly-RealWorld}~\cite{luo2021dynamics} contains paired egocentric videos and head poses captured using iPhone ARKit. Unlike Kinpoly-MoCap which is captured in a lab scene, Kinpoly-RealWorld provides in-the-wild egocentric videos.

\noindent\textbf{GIMO}~\cite{zheng2022gimo} consists of egocentric video, eye gaze, 3D motions, and scanned 3D scenes. This dataset is collected using Hololens, iPhone 12, and IMU-based motion capture suits to study motion prediction tasks guided by eye gaze. We use 15 scenes for training and 4 scenes for testing.

\vspace{-4mm}
\paragraph{Evaluation Metrics.}
\begin{itemize}
   \vspace{-2mm}
    \item \textbf{Head Orientation Error $\left( \mathbf{O}_{head} \right)$} computes the Frobenius norm of the difference between the $3 \times 3$ rotation matrix $||\bm{R}_{pred}\bm{R}_{gt}^{-1} - \bm{I}||_{2}$, where $\bm{R}_{pred}$ is the predicted head rotation matrix and $\bm{R}_{gt}$ is the ground truth head rotation matrix.
    \vspace{-2mm}
    \item \textbf{Head Translation Error $\left( \mathbf{T}_{head} \right)$} is computed by taking the mean Euclidean distance of two trajectories. We use this metric to measure the head joint translation errors in millimeters $\left(mm\right)$.
    \vspace{-2mm}
    \item \textbf{MPJPE} represents mean per-joint position errors in millimeters $\left(mm\right)$.
    \vspace{-2mm}
    \item \textbf{Accel} represents the difference of acceleration between predicted joint positions and ground truth joint positions measured in $\left(mm/s^2\right)$.
    \vspace{-2mm}
    \item \textbf{FS} represents foot skating metric and is computed following NeMF~\cite{he2022nemf}. Specifically, we first project the toe and ankle joints' velocity to the $xy$ plane and compute the $L{1}$ norm of the projected velocities in each step denoted as $v_{t}$. We only accumulate the horizontal translation for those steps that have a height $h_{t}$ lower than a specified threshold $H$. And the metric is calculated as a mean of weighted values $v_{t}(2-2^{\frac{h_{t}}{H}})$ across the sequence and is measured in $\left(mm\right)$.
\end{itemize}

\subsection{Full-Body Pose Estimation from Egocentric Video}
\vspace{-1mm}
\paragraph{Training Data.}
We use the paired egocentric videos and ground truth motions of ARES dataset to train the baseline methods. And we use the paired egocentric videos and head poses provided by ARES, Kinply-RealWorld, and GIMO to train our HeadNet, but do not use the ground truth body motion. Our GravityNet and the conditional diffusion model are both trained on AMASS.
\vspace{-4mm}
\paragraph{Baselines.}
We compare our approach with two baselines PoseReg~\cite{yuan2019ego} and Kinpoly~\cite{luo2021dynamics}. \textbf{PoseReg}~\cite{yuan2019ego} takes a sequence of optical flow features as input and uses an LSTM model to predict the pose state at each time step. The pose state consists of root translation, root orientation, joint rotations, and corresponding velocities including root linear velocity and angular velocities of all the joints. \textbf{Kinpoly-OF}~\cite{luo2021dynamics} proposes a per-step regression model to estimate full body motion from optical flow features. Because our problem only allows for egocentric video as input, we choose the option of Kinpoly which only has optical flow features as input, without relying on ground truth head poses and action labels that depend on additional knowledge.

\begin{table}[t!]
\small
\begin{center}
\footnotesize{
\setlength{\tabcolsep}{5pt}
\begin{tabular}{@{}l||cccccc@{}}
 \toprule
 Method     & $\mathbf{O}_{head}$ & $\mathbf{T}_{head}$ & MPJPE & Accel & FS  \\
        \midrule
        PoseReg~\cite{yuan2019ego} & 0.77 & 354.7 & 147.7 & 127.6 & 87.1 \\
        Kinpoly-OF~\cite{luo2021dynamics} & 0.62 & 323.4 & 141.6 & 7.3 & 4.2  \\
        EgoEgo (ours) & \textbf{0.20} & \textbf{148.0} & \textbf{121.1} & \textbf{6.2} & \textbf{2.7} \\
        \bottomrule
\end{tabular}
}
\end{center}
\vspace{-4mm}
\caption{Full-body motion estimation from egocentric video on ARES.}
    \label{tab:complete_estimation_ares}
\vspace{-2mm}
\end{table}

\begin{table}[t!]
\small
\begin{center}
\footnotesize{
\setlength{\tabcolsep}{5pt}
\begin{tabular}{@{}l||cccccc@{}}
 \toprule
 Method     & $\mathbf{O}_{head}$ & $\mathbf{T}_{head}$ & MPJPE  & Accel & FS  \\
        \midrule
        PoseReg~\cite{yuan2019ego} & 1.05 & 1943.9 & 160.4 & 61.8 & 10.8 \\
        Kinpoly-OF~\cite{luo2021dynamics} & 1.33 & 2475.5 & 230.5 & 16.4 & 15.8  \\
        EgoEgo (ours) & \textbf{0.58} & \textbf{505.1} & \textbf{125.9} & \textbf{8.0} & \textbf{1.6} \\
        \bottomrule
\end{tabular}
}
\end{center}
\vspace{-4mm}
\caption{Full-body motion estimation from egocentric video on Kinpoly-MoCap~\cite{luo2021dynamics}. }
    \label{tab:complete_estimation_kinpoly}
\vspace{-2mm}
\end{table}

\begin{table}[t!]
\small
\begin{center}
\footnotesize{
\setlength{\tabcolsep}{5pt}
\begin{tabular}{@{}l||cccccc@{}}
 \toprule
 Method     & $\mathbf{O}_{head}$ & $\mathbf{T}_{head}$ & MPJPE & Accel & FS \\
        \midrule
        PoseReg~\cite{yuan2019ego} & 1.51 & 1528.6 & 189.3 & 71.5 & 14.2 \\
        Kinpoly-OF~\cite{luo2021dynamics} & 1.52 & 1739.3 & 404.2 & 21.9 & 14.4  \\
        EgoEgo (ours) & \textbf{0.67} & \textbf{356.8} & \textbf{152.1} & \textbf{10.4} & \textbf{1.9} \\
        \bottomrule
\end{tabular}
}
\end{center}
\vspace{-4mm}
\caption{Full-body motion estimation from egocentric video on GIMO~\cite{zheng2022gimo}. }
    \label{tab:complete_estimation_gimo}
\vspace{-4mm}
\end{table}

\vspace{-4mm}
\paragraph{Results.}
We compare the complete pipeline of EgoEgo with baseline methods PoseReg~\cite{yuan2019ego} and Kinpoly-OF~\cite{luo2021dynamics} on ARES, Kinpoly-MoCap and GIMO datasets as shown in Table~\ref{tab:complete_estimation_ares}, Table~\ref{tab:complete_estimation_kinpoly} and Table~\ref{tab:complete_estimation_gimo} respectively. We show that our EgoEgo outperforms all the baselines by a large margin on all three datasets. We show qualitative results in Figure~\ref{fig:qualitative_res}. Our generated motions better preserve the root trajectories. And our approach can also generate more dynamic and realistic motions compared to the baselines.

\begin{table}[!t]
    \centering

    \small
      \begin{tabular}{lcccccccc}
        \toprule

         & \multicolumn{2}{c}{DROID-SLAM~\cite{teed2021droid}} &
        \multicolumn{2}{c}{Ours} \\

        \cmidrule(lr){2-3}\cmidrule(lr){4-5}

        & $\mathbf{O}_{head}$  & $\mathbf{T}_{head}$ & $\mathbf{O}_{head}$  & $\mathbf{T}_{head}$  \\

        \midrule

        ARES & 0.62 & 411.3 & \textbf{0.23} & \textbf{176.5}  \\
        Kinpoly-MoCap & \textbf{0.55} & 1290.8 & 0.58 & \textbf{487.8} \\
        GIMO & \textbf{0.67} & 865.4 & 0.68 & \textbf{304.7} \\
        \bottomrule

      \end{tabular}

     \caption{Head pose estimation on testing datasets.

     }
    \label{tab:head_pose_estimation}
    \vspace{-2mm}
\end{table}

\begin{table}[t!]
\small
\begin{center}
\footnotesize{
\setlength{\tabcolsep}{5pt}
\begin{tabular}{@{}l||cccccc@{}}
 \toprule
 Method     & $\mathbf{O}_{head}$ & $\mathbf{T}_{head}$ & MPJPE & Accel & FS  \\
        \midrule
        AvatarPoser~\cite{jiang2022avatarposer} & 0.19 & \textbf{28.6} & 124.7 & 16.1 & 18.8 \\
        Kinpoly-Head~\cite{luo2021dynamics} & 0.19 & 87.8 & 110.9 & 11.4 & 11.2  \\
        Diffusion Model (ours) & \textbf{0.04} & 36.7 & \textbf{109.0} & \textbf{10.5} & \textbf{4.4} \\
        \bottomrule
\end{tabular}
}
\end{center}
\vspace{-4mm}
\caption{Full-body motion estimation from GT head pose on AMASS testing dataset~\cite{luo2021dynamics}. }
    \label{tab:stage2_estimation_amass}
\vspace{-4mm}
\end{table}

\subsection{Head Pose Estimation from Egocentric Video}
\paragraph{Baselines.}
 We compare our hybrid approach with the prediction results of DROID-SLAM~\cite{teed2021droid}. For a fair comparison, we apply a rotation to the SLAM trajectory by aligning the first predicted head pose of SLAM with the ground truth head pose. We train our GravityNet on AMASS training split. As for HeadNet, we train on ARES, Kinpoly-RealWorld, and GIMO separately for the evaluation of different datasets.

\vspace{-4mm}
\paragraph{Results.}
We evaluate the head pose estimation on the three datasets as shown in Table~\ref{tab:head_pose_estimation}. We show more accurate head rotation prediction results on ARES and comparable results on real-captured data. As the real-captured data is limited in scale (Kinpoly-RealWorld contains 20 minutes of training videos and GIMO contains 30 minutes of training videos), we believe the head rotation prediction can be further improved by future developments of large-scale real-captured datasets with head poses. Overall, our hybrid approach combines the accurate rotation prediction from HeadNet and re-scaled translation of gravity-aligned SLAM results, and produces more accurate head pose estimation results as input to the second stage.

\subsection{Full-Body Pose Estimation from Head Pose}
\paragraph{Baselines.}
We compare our conditional diffusion model for full-body pose estimation with two baselines AvatarPoser~\cite{jiang2022avatarposer} and Kinpoly-Head~\cite{luo2021dynamics}. \textbf{AvatarPoser}~\cite{jiang2022avatarposer} takes both head and hand pose as input to predict full body motion. We remove the hand poses from the input and modified it to a setting with head pose input only. \textbf{Kinpoly-Head}~\cite{luo2021dynamics} is our modified variant of the Kinpoly model that only takes the head pose as input. Both the baselines and our method are trained on the training split of AMASS with high-quality motion capture data.

\vspace{-4mm}
\paragraph{Results.}
We evaluate the baselines and our method on AMASS testing dataset as shown in Table~\ref{tab:stage2_estimation_amass}. Since our model is a generative model, there are multiple plausible predictions from the same head pose input. To compare our method quantitatively, we generate 200 samples for each head pose input and use the one with the smallest MPJPE as our result.

\begin{figure*}[h]
\begin{center}
\vspace{-5mm}
\includegraphics[width=0.95\textwidth]{figures/Figure_last_qualitative_res.pdf}
\end{center}
\vspace{-4mm}
\caption{Qualitative results comparisons. We show the results of two egocentric videos from different scenes.}
\label{fig:qualitative_res}
\vspace{-1mm}
\end{figure*}

\subsection{Ablation Studies}
\paragraph{Effects of Components in Head Pose Estimation.}
We study the effects of each component in head pose estimation in Table~\ref{tab:ablation_head_pose_hybrid}. We showcase that the rotation for aligning gravity direction and the learned scale are both effective to improve head translation results.

\begin{table}[!t]
    \centering

    \small
    \setlength{\tabcolsep}{2pt}
      \begin{tabular}{lcccccccc}
        \toprule

         & \multicolumn{2}{c}{ARES} &
        \multicolumn{2}{c}{Kinpoly-MoCap} &
        \multicolumn{2}{c}{GIMO} \\

        \cmidrule(lr){2-3}\cmidrule(lr){4-5}\cmidrule(lr){6-7}

        & $\mathbf{O}_{head}$  & $\mathbf{T}_{head}$ & $\mathbf{O}_{head}$  & $\mathbf{T}_{head}$ & $\mathbf{O}_{head}$  & $\mathbf{T}_{head}$  \\

        \midrule
        SLAM & 0.62 & 411.33 & \textbf{0.55} & 1290.82 & \textbf{0.67} & 865.41 \\
        SLAM+S & 0.62 & 325.95 & 0.55 & 643.45 & 0.67 & 569.48 \\
        SLAM+S+G & 0.62 & 176.54 & 0.55 & 487.77 & 0.67 & 304.74 \\
        Full model & \textbf{0.23} & \textbf{176.54} & 0.58 & \textbf{487.77} & 0.68 & \textbf{304.74}  \\
        \bottomrule

      \end{tabular}

     \caption{Ablation study for the components in head pose estimation. S represents the scale predicted by HeadNet, and G represents GravityNet.}
    \label{tab:ablation_head_pose_hybrid}
    \vspace{-2mm}
\end{table}

\vspace{-4mm}
\paragraph{Effects of Head Pose in Full-Body Pose Estimation.}
We compare the full-body pose estimation results that take our predicted head poses and the ground truth head poses as input. Table~\ref{tab:ablation_head_pose_gt} shows that the ground truth head poses significantly improve the full body pose estimation, indicating that by developing methods that predict more accurate head pose, the full body pose estimation can be further improved.

\begin{table}[!t]
    \centering

    \small
    \setlength{\tabcolsep}{2pt}
      \begin{tabular}{lcccccc}
        \toprule

         & \multicolumn{2}{c}{ARES} &
        \multicolumn{2}{c}{Kinpoly-MoCap} &
        \multicolumn{2}{c}{GIMO} \\

        \cmidrule(lr){2-3}\cmidrule(lr){4-5}\cmidrule(lr){6-7}

        & EE  & EE w GT & EE  & EE w GT &  EE  & EE w GT  \\

        \midrule
        $\mathbf{O}_{head}$ & 0.20 & \textbf{0.04} & 0.58 & \textbf{0.03} & 0.67 & \textbf{0.06} \\
        $\mathbf{T}_{head}$ & 148.0 & \textbf{29.1} & 505.1 & \textbf{60.7} & 356.8 & \textbf{66.0}  \\
        MPJPE & 121.1 & \textbf{105.6} & 125.9 & \textbf{76.0} & 152.1 & \textbf{125.7} \\
        Accel & \textbf{6.2} & 6.3 & 8.0 & \textbf{7.2} & 10.4 & \textbf{10.2}  \\
        FS & \textbf{2.7} & 3.0 & \textbf{1.6} & 2.5 & 1.9 & \textbf{1.7}  \\
        \bottomrule

      \end{tabular}

     \caption{Ablation study for the effects of head pose to full-body pose estimation. EE represents EgoEgo. GT represents ground truth head poses.}
    \label{tab:ablation_head_pose_gt}
    \vspace{-4mm}
\end{table}

\begin{figure}[t!]
\begin{center}
\vspace{-2mm}
\includegraphics[width=0.45\textwidth]{figures/Figure_human_study.pdf}
\end{center}
\vspace{-6mm}
  \caption{Results of human perceptual studies. The numbers shown in the chart represent \%.}

  \label{fig:human_study}
\vspace{-6mm}
\end{figure}

\subsection{Human Perceptual Study}
We also conduct two human perceptual studies as part of the evaluation. The first is to evaluate the quality of predicted full-body motion from egocentric video, the second is to evaluate the quality of predicted full-body motion from ground truth head poses. In both studies, we compare four types of motion: results from our EgoEgo\xspace and from two baselines, as well as the ground truth. For the first human study, each time, users are presented with two motions and an egocentric video, and asked to select which one is more plausible. For the second human study, users are presented with two motions and asked to select which one looks more natural and realistic. Because there are 10 examples and the two motions can be from four sources, we have 60 questions for each study. Each question was answered by 20 Amazon Mechanical Turk workers.

As shown in Figure~\ref{fig:human_study}(a), for full-body estimation from egocentric video, our results are preferred by 98\% and 69\% of workers when compared to the baselines. Also, when compared with the ground truth, 34\% of the responses prefer our results (note that a perfect output would achieve 50\%), suggesting that people cannot easily distinguish our results from ground truth motions. As shown in Figure~\ref{fig:human_study}(b), for full-body estimation from head poses, our results are preferred by 88\% and 79\% of workers when compared to the baselines.

\section{Conclusion}
We presented a generalized framework to estimate full-body motions from egocentric video. The key is to decompose the problem into two stages. We predicted the head pose from an egocentric video and fed the output from the first stage to estimate full body motions in the second stage. In addition, we developed a hybrid solution to produce more accurate head pose results on top of monocular SLAM. And we also proposed a conditional diffusion model to generate diverse high-quality full-body motions from predicted head poses. To benchmark different methods in a large-scale dataset, we proposed a data generation pipeline to synthesize a large-scale dataset with paired egocentric videos and 3D human motions. We showcased superior results on both the synthetic dataset and the real-captured dataset compared to prior work.

\subsection{#1}\vspace{-4pt}}
\newcommand{\myparagraph}[1]{\noindent\textbf{#1}}
\newcommand{\vspace{-5pt}\item}{\vspace{-5pt}\item}
\newcommand{\mybf}[1]{\vspace{4pt}\noindent\textbf{#1}}
\newcommand{\mycell}[1]{\begin{tabular}[t]{@{}l@{}l}#1\end{tabular}}
\newcommand{\xhdr}[1]{{\noindent \bf #1}}


\section{Introduction}
\label{sec:introduction}

\input{PaperBody/introduction}

\section{Related work}
\label{sec:relatedwork}
\input{PaperBody/relatedwork4}

\section{Proposed Method}
\label{sec:method}
\input{PaperBody/method2}

\section{Experimental Results}
\label{sec:experiment}
\input{PaperBody/experiment3}
\section{Conclusion}
\label{sec:conclusion}
\input{PaperBody/conclusion3}

\section{Introduction}
\label{sec:introduction}

\input{introduction}

\section{Related work}
\label{sec:relatedwork}
\input{relatedwork4}

\section{Proposed Method}
\label{sec:method}
\input{method2}

\section{Experimental Results}
\label{sec:experiment}
\input{experiment3}
\section{Conclusion}
\label{sec:conclusion}
\input{conclusion3}
\section*{Acknowledgments}
This work is supported by National Key Research and Development Program of China for Intergovernmental Cooperation with grant No. 2019YFE0110100, Science and technology innovation team project of Guangdong Meteorological Bureau No.GRMCTD202104, Innovation and Development Project of China Meteorological Administration (Grant No. CXFZ2022J002). In addition, the work is also supported in part by the National Key R\& D Program of China with grant No. 2018YFB1800800, the Basic Research Project No. HZQB-KCZYZ-2021067 of Hetao Shenzhen-HK S \& T Cooperation Zone, by Shenzhen Outstanding Talents Training Fund 202002, by Guangdong Research Projects No. 2017ZT07X152 and No. 2019CX01X104, and by the Guangdong Provincial Key Laboratory of Future Networks of Intelligence (Grant No. 2022B1212010001). Thanks Oisin Mac Aodha for his advices on our work.

\section{Architecture design}

\section{Evaluation Metrics}
To evaluate the performance of our MIMO-VP against previous state-of-the-art works in the deterministic prediction domain,
\textit{i}.\textit{e}.  LMC-Memory \cite{lee2021video},
MotionRNN \cite{wu2021motionrnn},
PhyDNet \cite{guen2020disentangling},
Conv-TT-LSTM \cite{su2020convolutional},
CrevNet \cite{yu2019crevnet},
MIM \cite{wang2019memory} and ConvLSTM \cite{xingjian2015convolutional},
several metrics are used in our experiment,
including Mean Squared Error (MSE),
Mean Absolute Error (MAE),
the Structural Similarity (SSIM),
LPIPS \cite{zhang2018unreasonable} and Peak Signal to Noise Ratio (PSNR). All these metrics are averaged for all predicted frames. Lower MSE, MAE, LPIPS, or higher SSIM, PSNR indicates better performance. In addition, for the Weather dataset, we also take the Critical Success Index (CSI) into account. The CSI score is a significant evaluation metric in the precipitation nowcasting domain, which is calculated as:
\begin{equation}
CSI = \frac{hits}{hits+misses+false alarms},
\end{equation}
where \textit{hits}, \textit{misses} and \textit{falsealarms} denote the number of\textit{ true positive},\textit{ false positive} and \textit{false negative}, respectively.
The higher the CSI score, the better the performance.

\section{Implementation Details}
More implementation details are given in this section.
ReduceLROnPlateau is adopted to reduce the learning rate with a 0.5 decay factor when loss stops decreasing.
For the Moving MNIST dataset, we use 6 transformer blocks in the encoder and 10 transformer blocks in the decoder. While for the other three datasets, we use both 6 blocks in the encoder and decoder modules.
The model channel is set to be $C=128$.
We use 8 heads in our 2DMHA module.
The batch size is set to be 32 for the Moving MNIST dataset and 8 for other datasets.

\begin{figure*}[t]
	\centering
	\includegraphics[width=0.9\linewidth]{pictures/attention.pdf}
	\caption{The attention maps of MIMO-VP, at $4$-th layer with the sum of 8 heads: (a) cross-attention map shows that the generated frames have high attention scores with the distant historical frames; (b) self-attention map indicates the dependency among the future frames;
	(c) an illustration of dependency between the predicted frame 51 and the other frames.
	}
	\label{fig:attention}
\end{figure*}

\section{Additional Experimental Results}

\subsection{\textbf{How to make the best design?}}
We report the experimental results regarding the number of transformer blocks and the number of heads in 2DMHA in \Cref{Tab: model design}.
Generally, a deeper model with more layers of the transformer will bring higher performance.
For the trade-off of prediction accuracy and model efficiency,
we use `6-10' model on Moving MNIST, and `6-6' model on the other datasets.
We note that the results of multiple heads' attention are better than single heads in our experiment because multiple heads could effectively capture complicated long-term temporal cues for video prediction.
Therefore, we chose 8 heads in our MIMO-VP.

\begin{table}[h]

	\centering
	\tabcolsep12pt
	\caption{Quantitative comparison of different designs of MIMO-VP on Moving MNIST dataset. Model `a-b' denotes MIMO-VP with `a' encoder blocks and `b' decoder blocks.}
	\begin{tabular}{l|ccc}
		\hline
		Models   & MSE $\downarrow$ & MAE $ \uparrow$ & SSIM $\uparrow$ \\
		\hline
		\hline
		4-8  & 19.3 & 55.4 & 0.960  \\
		\textbf{6-10} & 17.7 & 51.6 & 0.964 \\
		8-12 & 17.6 & 51.3 & 0.964  \\
		\hline
		1 head & 19.1 & 55.0 & 0.960 \\
		4 heads & 18.0 & 52.4 & 0.963 \\
		\textbf{8 heads} & 17.7 & 51.6 & 0.964 \\
		16 heads & 18.1 & 52.7 & 0.963 \\
		\hline
	\end{tabular}
	\label{Tab: model design}
\end{table}

\subsection{More illustration about the superiority of MIMO}
To investigate how the MIMO is capable of capturing the long-term relationships of sequence and preserving the future frame dependency, we conduct experiments on the Moving mnist dataset by predicting 10 future frames conditioned on $50$ observed frames.
The cross-attention map in \Cref{fig:attention} (a) shows that the predicted frames have stronger attention with the distant input frames (small index of frames),
which indicates that MIMO-VP can leverage the distant historical information of sequence based on the 2DMHA mechanism to facilitate future frame generation.
For example, \Cref{fig:attention} (c) shows that the input frames 4, 5, 6, 7, 28, 29 and 45 are helpful for predicting frame 51.
In addition, the self-attention map in \Cref{fig:attention} (b) shows that each predicted frame has high attention scores with its neighbouring frames, indicating their dependency.
This phenomenon verifies the benefit of MIMO-VP in preserving the dependency among the future frames for accurate prediction.

\subsection{More Visualizations}
We give in \Cref{fig:KITTI}, \Cref{fig:weather} and \Cref{fig:mnist-10} additional visualizations completing Figure 5, 6 and 3 in submission, respectively.
We see that the frames generated by MIMO-VP are clearer than those generated by other comparison methods in \Cref{fig:KITTI}.
In \Cref{fig:weather}, our MIMO-VP achieves higher prediction accuracy than MIM, MotionRNN and PhyDNet, especially in the region of high-intensity radar echos.
In \Cref{fig:mnist-10}, MIMO-VP consistently generates accurate and sharp frames, while the number four becomes distorted when they are predicted by PhyDNet and MotionRNN.

We also provide the long-term prediction comparison results on the Moving mnist dataset, as shown in \Cref{fig:mnist-30}.
We see that numbers four and zero in the top sample of PhyDNet and MotionRNN become seriously distorted after timestep 22 when these two numbers become overlap.
Similarly, numbers six and eight in the bottom sample of PhyDNet and MotionRNN become distorted and blurry after timestep 14.
Surprisingly, our MIMO-VP can consistently predict accurate and clear frames.
These results prove the effectiveness of MIMO-VP in long-term prediction.

\begin{figure*}[!]
	\centering
	\includegraphics[width=0.8\linewidth]{pictures_supp/kitti_new.pdf}
	\caption{Prediction examples on KITTI dataset. We predict 5 future frames conditioned on 5 observed frames.
	}
	\label{fig:KITTI}
\end{figure*}

\begin{figure*}[t]
	\centering
	\includegraphics[width=0.9\linewidth]{pictures_supp/weather_new.pdf}
	\caption{Prediction examples on Weather dataset. We predict 10 future frames conditioned on 10 observed frames.
	}
	\label{fig:weather}
\end{figure*}

\begin{figure*}[t]
	\centering
	\includegraphics[width=0.9\linewidth]{pictures_supp/mnist_new.pdf}
	\caption{Prediction examples on Moving mnist dataset. We predict 10 future frames conditioned on 10 observed frames.
	}
	\label{fig:mnist-10}
\end{figure*}

\begin{figure*}[t]
	\centering
	\includegraphics[width=0.9\linewidth]{pictures_supp/mnist_long.pdf}
	\caption{Prediction examples on Moving mnist dataset. We predict 30 future frames conditioned on 10 observed frames. Our MIMO-VP can correctly predict the movement of numbers zero and four in the top sample and numbers six and eight in the bottom sample, while these numbers become distorted when they are predicted by PhyDNet\cite{guen2020disentangling} and MotionRNN \cite{wu2021motionrnn}.
	}
	\label{fig:mnist-30}
\end{figure*}
\subsection{Demo}
We put a demo in the attachment, which shows the dynamic visualization of different predictions.

\subsection{Dataset Descriptions}
\begin{figure}[t]
	\centering
	\includegraphics[width=0.9\linewidth]{mnist.pdf}
	\caption{Prediction examples on Moving MNIST dataset.
	}
	\label{fig:mnist}
\end{figure}

\begin{figure}[t]
	\centering
	\includegraphics[width=0.9\linewidth]{human.pdf}
	\caption{Prediction examples on Human3.6M dataset.
	}
	\label{fig:human}
\end{figure}
\noindent \textbf{Moving MNIST} \cite{srivastava2015unsupervised} is a standard synthetic dataset for video prediction. We generate training set following \cite{guen2020disentangling},
and adopt the widely used testing set of 10,000 sequences provided by \cite{srivastava2015unsupervised}.

\noindent \textbf{Human3.6M} \cite{ionescu2013human3} dataset is a real-world human pose dataset comprising 17 kinds of human actions and 3.6 million poses. We use subjects S1, S5, S6, S7 and S8 for training, and subjects S9, S11 for testing. We predict $4$ frames in the future conditioned on the $4$ observed frames.

\noindent \textbf{Weather} dataset is a real radar echo dataset for precipitation nowcasting, which is obtained from the local weather bureau. Our training set and testing set contain $9,600$ and $2400$ radar sequences (resolution 128$\times$128$\times$1) respectively.

\noindent \textbf{KITTI} dataset is one of the most popular datasets for autonomous driving and also a benchmark for computer vision algorithms. We predict $5$ frames in the future when taking $5$ observed frames as input.

\noindent \textbf{Implementation Details}
We use Adam \cite{kingma2014adam} optimizer with 0.0005 learning rate and L1$+$L2 loss to train our model.  We implemented the model with PyTorch and conducted experiments on NVIDIA V100 GPUs. Mean Squared Error (MSE), Mean Absolute Error (MAE), the Structural Similarity(SSIM), LPIPS~\cite{zhang2018unreasonable} and Peak Signal to Noise Ratio (PSNR) are evaluated in our work. All these metrics are averaged for all predicted frames. Additionally, we also take Critical Success Index (CSI) into account for Weather dataset. Details are contained in \textit{supplementary materials}.

\subsection{Quantitative and Qualitative Comparison}
\label{sec:IS MIMO beats SISO ?}
\noindent \textbf{Results on Moving MNIST.}
Firstly, we report the results which use 10 frames to predict 10 future frames. As shown in \Cref{Tab: Moving MNIST} and \Cref{fig:mnist}, our model significantly outperforms all the other models. Current RNN-based models get blurry and wrong predictions in this case, and their prediction error grows larger over time. On the contrary, our model can predict high quality frames.
To further explore our performance on long-term prediction, we conduct an experiment by predicting 30 future frames conditioned on 10 input frames. As shown in \Cref{Tab: Moving MNIST}, our method far surpasses the others in predicting 30 frames in terms of the MSE metric. Compared with PhyDNet, our MIMO-VP improves by $46.2\%$ in MSE, which is much larger than $26.9\%$ in 10-10 prediction task.

\begin{table}[t]
	\small
	\centering
	\tabcolsep 9pt
	\scalebox{0.9}{
	\begin{tabular}{l|ccc}
		\hline
		Method   & MAE/100 $\downarrow$ & SSIM $\uparrow$ & PSNR $\uparrow$ \\
		\hline
		\hline
		MIM & 17.8& 0.790 & 20.57  \\
		PhyDNet & 16.2 & 0.901 &-\\
		LMC-Memory  & 17.0 & 0.820 &21.05\\
		MotionRNN  & 14.8& 0.846 & 22.16   \\
		\textbf{MIMO-VP}  & \textbf{10.4} & \textbf{0.941}  & \textbf{24.52} \\
		\hline
	\end{tabular}}
	\caption{Quantitative comparisons on Human3.6M
	dataset.}
	\label{Tab:human}
\end{table}

\begin{table}[t]
\small
\centering
\tabcolsep3pt
  \begin{tabular}{l|cccc}
  \hline
    Method   & MAE/100 $\downarrow$ & SSIM $\uparrow$& LPIPS $\downarrow$ & PSNR $\uparrow$ \\
	\hline
	\hline
	CrevNet & 56.3 & 0.587  & 0.454  & 17.046  \\

		PhyDNet   & 44.9 & 0.674  & 0.403& 19.159 \\
		LMC-Memory  & 44.6 & 0.660  &  0.410  & 18.692\\
		MotionRNN  & 45.0 & 0.652  & 0.384& 18.931 \\
		\textbf{MIMO-VP}  & \textbf{40.9} & \textbf{0.703} & \textbf{0.308}& \textbf{19.616}  \\
     \hline
	\end{tabular}
\caption{Quantitative comparisons on KITTI dataset.}
\label{Tab: kitti}
\end{table}

\begin{table}[!]
\centering
\tabcolsep8pt
  \small
  \scalebox{0.9}{
  \begin{tabular}{l|cccc}
  \hline
    Method  & MAE $\downarrow$ & SSIM $\uparrow$ & CSI-20 $\uparrow$ & CSI-30 $\uparrow$\\
	\hline
	\hline
	MIM  & 402.6 & 0.725& 0.402  & 0.164\\

		PhyDNet & 280.1 & 0.859 & 0.557 & 0.368 \\
		MotionRNN & 336.9 & 0.809 & 0.519 & 0.290 \\
		\textbf{MIMO-VP}  & \textbf{235.6} & \textbf{0.865} & \textbf{0.595} & \textbf{0.400}\\
     \hline
	\end{tabular}}
\caption{Quantitative comparisons on Weather dataset.}
\label{Tab: weather}
\end{table}

\begin{figure}[t]
	\centering
	\includegraphics[width=0.85\linewidth]{kitti.pdf}
	\caption{Prediction examples on KITTI dataset.
	}
	\label{fig:KITTI}
\end{figure}

\begin{figure}[t]
	\centering
	\includegraphics[width=0.85\linewidth]{weather.pdf}
	\caption{Prediction examples on Weather dataset.
	}
	\label{fig:weather}
\end{figure}

\begin{table}[t]
\small
\centering
\tabcolsep3pt
  \begin{tabular}{l|cccc}
  \hline
    Method   & MSE $\downarrow$ & MAE $\downarrow$ & SSIM $\uparrow$ & \#Params (M)\\
	\hline
	\hline
    CrevNet & 22.3 & - & 0.949  &- \\
	\hline
	3D convolution  & 32.8 & 84.4 & -  & 19.8 \\
		ViT & 50.2 & 126.5 & 0.887  & 17.5\\
		Swin-Transformer & 28.8 & 77.2 & 0.940 & 21.8 \\
		SimVP & 23.8& 68.9 & 0.948 & 15.8\\
		\textbf{MIMO-VP} & \textbf{17.7}  & \textbf{51.6} & \textbf{0.964}  & 20.2 \\
     \hline
	\end{tabular}
\caption{Results of different MIMO models on Moving MNIST.}
\label{Tab: MIMO_compare}
\end{table}

\noindent \textbf{Results on Human3.6M.}
\Cref{Tab:human} shows that MIMO-VP achieves the best performance on Human3.6M.
Compared with PhyDNet, MIMO-VP improves by $35.8\%$ in MAE.
In \Cref{fig:human}, complicated background and human motion give much more challenge for predicting accurate and reasonable results. While our MIMO-VP can still product relatively precise future frames comparing with other models.

\noindent \textbf{Results on KITTI.}
As shown in \Cref{Tab: kitti}, our proposed MIMO-VP achieves consistent improvement in all evaluation metrics.
Compared with LMC-Memory, our model improves by $8.3\%$ in MAE.
Quantitative visualization results in \Cref{fig:KITTI} show that our model can preserve much more details of both background and foreground.
For instance, from the right part of last predicted frames, we can find that the results of our MIMO-VP is much similar to GroundTruth while other methods produce obviously blurry frames.

\noindent \textbf{Results on Weather.}
From Table \ref{Tab: weather}, we can find that our MIMO-VP has significantly improvement in CSI scores within different thresholds, which indicates our approach can predict more credible radar echoes for precipitation nowcasting.
The visualization radar echo maps are shown in \Cref{fig:weather}. Different color represents different radar echo intensity (dBZs), \textit{i}.\textit{e}. \ yellow indicates higher intensity than green.
Additionally, the center region of the radar echo is zoomed in and placed below each predicted frame.
We find that our MIMO-VP predicts more correct yellow regions  than other methods, which means MIMO-VP is a more suitable method for precipitation nowcasting.

\noindent \textbf{The comparison of different MIMOs}
Quantitative comparisons are detailed in Table~\ref{Tab: MIMO_compare}, where 3D convolution is a fully convolutional network.
For ViT \cite{dosovitskiy2020image} and Swin-Transformer \cite{liu2021swin}, we replace the classification layer with convolution operation to be consistent with original image size.
We observe that all other MIMO methods can not achieve state-of-the-art performance when compared with the best SISO method--CrevNet \cite{yu2019crevnet} except ours.
To verify the effectiveness of our MIMO-VP, we carry out an ablation study to analyse each key component.

\begin{table}[t]
	\small
	\centering
	\tabcolsep15pt
	\begin{tabular}{l|cc}
		\hline
		Method  & MSE $\downarrow$ & MAE $\downarrow$  \\
		\hline
		\hline
		Without 2DMHA & 32.8 & 84.4  \\
		Without LSB  & 21.2 & 59.2  \\
		MISO-VP  &22.8 & 62.8 \\

		\hline
		\textbf{MIMO-VP}  & \textbf{17.7} & \textbf{51.6}  \\
		\hline
	\end{tabular}
	\caption{Ablation Study for Each Components.}
	\label{Tab: ablation}
\end{table}

\begin{figure}[t]
	\centering
	\begin{subfigure}[b]{0.49\linewidth}
		\centering
		\includegraphics[width=1\linewidth]{long-term.pdf}
		\caption{MSE w.r.t input length}
		\label{figure:long-term}
	\end{subfigure}
	\hfill
	\begin{subfigure}[b]{0.49\linewidth}
		\centering
		\includegraphics[width=1\linewidth]{error.pdf}
		\caption{Frame-wise MSE}
		\label{figure:error accumulation}
	\end{subfigure}
	\caption{(a) Comparison of MIMO-VP and PhyDNet by predicting
10 future frames conditioned on different length of input sequence;
(b) Frame-wise MSE of MIMO-VP and PhyDNet on Moving
MNIST dataset.}
\end{figure}

\begin{figure}[t]
	\centering
	\begin{subfigure}[b]{0.49\linewidth}
		\centering
		\includegraphics[width=1\linewidth]{longterm-mae.pdf}
		\caption{MAE w.r.t Output length}
		\label{figure:long term prediction1}
	\end{subfigure}
	\hfill
	\begin{subfigure}[b]{0.49\linewidth}
		\centering
		\includegraphics[width=1\linewidth]{longterm-ssim.pdf}
		\caption{SSIM w.r.t Output length}
		\label{figure:long term prediction2}
	\end{subfigure}
	\caption{Results of long-term predication on Moving MNIST dataset.}
\end{figure}

\begin{table}[!t]
    \small
	\centering
	\tabcolsep10pt
	\begin{tabular}{l|ccc}
		\hline
		Method  & MSE $\downarrow$ & MAE $\downarrow$ & SSIM $\uparrow$ \\
		\hline
		\hline
		MIMO-VP\_F & 21.6 & 59.4 & 0.955 \\
		\textbf{MIMO-VP} & \textbf{17.7} & \textbf{51.6} & \textbf{0.964} \\
		\hline
	\end{tabular}
	\caption{Comparison of MIMO-VP and its variant on Moving MNIST dataset.}
	\label{Tab: dependncy of future}
\end{table}

\begin{table}[!t]
\small
\centering
\tabcolsep3pt
  \begin{tabular}{l|ccccc}
  \hline
    Models   & PhyDNet & SimVP  & MotionRNN & MIMO-VP\\
	\hline
	\hline
	Time (s) & 0.0076 & 0.0021 & 0.0210 & 0.0050    \\
     \hline
   \end{tabular}
   \caption{Inference time on Moving MNIST.}
\label{Tab: Inference Time}
\end{table}

\subsection{Ablation Study of Each Component}
\label{sec:Which MIMO?}
 Quantitative results on Moving MNIST  dataset are shown in \Cref{Tab: ablation}.
(1) \textbf{The Spatial Dependency of Frames}. We can see that MIMO-VP without 2DMHA results in a large performance degradation in comparison with MIMO-VP, with an increase of 15.1 and 32.8 in terms of MSE and MAE respectively. This phenomenon indicates that
2DMHA mechanism, which is able to capture the global long-term spatiotemporal relationships of sequences, makes an important contribution to the accurate video prediction.
(2) \textbf{The Local Spatio-temporal Block (LSB)}. The results of MIMO-VP without LSB also show inferior to those of MIMO-VP, which verifies that the local spatial-temporal variations of sequence are helpful for prediction.
(3) \textbf{The Multi-Out Decoder}.
MISO-VP denotes a variant of MIMO-VP that adopts the MISO predictive strategy similar to vanilla Transformer.
It infers the next one frame using the predicted frames, in a recursive manner.
Experimental results show that MIMO-VP beats down the MISO-VP with performance improves by $18.0\%$ and $14.0\%$ in MSE and MAE respectively. The reason may be that recursive single-out predictive strategy will trigger the error accumulation problem and overlook the stochastic dependency among the future frames.

\subsection{Comparison of MIMO and SISO}
\label{sec:Why MIMO beats SISO?}
\noindent \textbf{Significance of `MI’: long-term dependency }
Comparison results between MIMO-VP and PhyDNet of predicting 10 future frames on Moving MNIST with different input lengths are shown in \Cref{figure:long-term}.
It is observed that the MSE metric of MIMO-VP drops quickly with the input length increasing from 5 to 50. PhyDNet improves
significantly when the input length increases from 5 to 10,
while improves slightly from 10 to 50. Precisely, the MSE
of MIMO-VP and PhyDNet deceases by $64.3\%$  and $24.4\%$
respectively from 10 input frames to 50 input frames. Those
results demonstrate that our MIMO-VP has a stronger ability
to capture the long-term sptiotemporal dependency of
video sequences for accurate prediction, while PhyDNet, an
RNN-based method, suffers from the long-term dependency
problem because of the limited capacity of memory.

\noindent \textbf{Significance of `MO': free of error accumulation}
\Cref{figure:error accumulation} illustrates the frame-wise MSE of MIMO-VP and PhyDNet, corresponding to the sample in \Cref{fig:mnist}.
We can observe that the curve of PhyDNet increase dramatically over time because PhyDNet suffers from the error accumulation problem.
By contrast, the curve of MIMO-VP increase slightly because the normal predictive error, which is related to the digits overlapping, increases over time.
To futher verify that the error accumulation problem is even worse in long-term case,  we compare the results of predicting 10, 30, 50 and 90 frames on Moving MNIST dataset.
In training phase, all the models are trained by using 10 frames to predict next 10 frames, while in testing phase, 10, 30, 50, and 90 frames are predicted in a recurrent manner.
Experimental results are shown in \Cref{figure:long term prediction1} and \Cref{figure:long term prediction2}, from which we can observe that the error accumulation problems of SISO models (MotionRNN, PhyDNet) is much severe than MIMO models (SimVP, MIMO-VP).
Thus, `MO' models can overcome the error accumulation problem in video prediction.

\noindent \textbf{Future frame dependency }
We compare our MIMO-VP with its variant \textit{MIMO-F}, a baseline which removes the self-attention module in the decoder so that it can not capture all the dependency of frames.
Predictive results are shown in
\Cref{Tab: dependncy of future}, from which we can see MIMO-VP achieves much better performance than \textit{MIMO-F}.  This phenomenon verifies the benefit of MIMO-VP in preserving the dependency among the future frames for accurate prediction.

\noindent \textbf{Speed and accuracy trade-off}
\Cref{Tab: Inference Time} shows the frame-wise inference time, we can observe that our `MO' methods runs much faster than `SO' methods. According to \Cref{figure:long term prediction1}, \Cref{figure:long term prediction2}, we can see that MIMO-VP can achieve state-of-the-art performance with a comparable inference time.

\subsection{Problem Formulation}
Suppose we observe a video sequence $\mathcal{S}_{t-m+1:t} := \{\mathcal{X}_{t-m+1},$ $\mathcal{X}_{t-m+2}, \dots, \mathcal{X}_t \}$,
where $\mathcal{X}_t\in \mathbb{R}^{C_0\times H_0\times W_0}$ represents the frame (channel $C_0$, height $H_0$, width $W_0$) at the current time step $t$.
The goal of the Spatio-temporal sequence prediction is to generate the most likely length-$n$ sequence in the future given the previous length-$m$ sequence:
\begin{equation}
\begin{split}
\mathcal{S}^*_{t+1:t+n} = \mathop{\arg\max}\limits_{\mathcal{S}_{t+1:t+n}} \  p(\mathcal{S}_{t+1:t+m} | \mathcal{S}_{t-m+1:t})
\end{split}
\label{eq: Spatiotemporal Sequence Prediction}
\end{equation}
\noindent \textbf{SISO predictive strategy.}
To solve the problem, the Single-In-Single-Out architectures are widely used. They can naturally preserve the time order logic of the generated frames. Their prediction paradigm can be written as:
\begin{equation}
\begin{split}
\mathcal{X}_{t'+1} = F(\mathcal{X}_{t'}, M_{t'})+\sigma_{t'},
\end{split}
\label{eq: RNN_solve}
\end{equation}
where $F$ denotes the model, $M_{t'}$ is the memory cell that remembers the past historical information. $\sigma_{t'}$ is a Tensor noise term. $\mathcal{X}_{t'}$ is the observed frame ($t'\leq t$) or the generated frame ($t'>t$).
Note that SISO predictive strategy is susceptible to the error accumulation problem.
For simplicity, consider an Auto Regressive with $K$=$1$ and without exogenous inputs model:
\begin{equation}
\begin{split}
\mathcal{X}_{t} = A\mathcal{X}_{t-1}+\sigma_{t}
\end{split}
\label{eq: AutoRegressive}
\end{equation}
where $A$ is a learnable Tensor.
It is easy to have:
\begin{equation}
\begin{split}
\mathcal{X}_{t+n} = A^{t+n-1}\mathcal{X}_{t}+\sum_{k=t}^{t+n-1}A^k\sigma_{t+n-k}+\sigma_{t+n}
\end{split}
\label{eq: error}
\end{equation}
It is clear that the error $R_{t+n}=\sum_{k=t}^{t+n-1}A^k\sigma_{t+n-k}+\sigma_{t+n}$ will accumulate as time step $t+n$ increases, which makes accurate prediction challenging. Besides, the long-term dependency problem inherited from RNNs would further limit the performance of the SISO strategy on video prediction.

\begin{figure}[t]
	\centering
	\includegraphics[width=0.9\linewidth]{framework.jpg}
	\caption{\textbf{Illustration of MIMO-VP architecture.}
	}
	\label{fig:model}
\end{figure}

\noindent \textbf{MIMO predictive strategy.}
There is an alternative way to deploy MIMO predictive strategy:
\begin{equation}
\begin{split}
\mathcal{S}_{t+1:t+n} = F(\mathcal{S}_{t-m+1:t})+ \Sigma_{t+1:t+n},
\end{split}
\label{eq: our_solve}
\end{equation}
where $\Sigma_{t+1:t+n}:=\{\sigma_{t+1},\dots,\sigma_{t+n}\}$ is the normal prediction error. Therefore, the MIMO strategy breaks the error accumulation by synthesising all the frames in parallel. However, it is not sufficient to deduce MIMO models always outperform SISO models. Actually, the single error $\sigma_{t+i}$ of one frame can be extremely high since the MIMO models may miss some important relationships which SISO models naturally capture. Notably, it is simple to present MIMO architectures but it is non-trivial to equip them with a strong ability to model long-term and short-term Spatio-temporal dependencies which are important for complex video data.

\subsection{Our MIMO-VP Architecture}
Encouraging by the great success of Transformer \cite{vaswani2017attention} models, we build up our MIMO architecture specific for video prediction based on the Transformer backbone. Their placeholder embedding method also helps to capture the global dependency of the frames. However, a pure Transformer is a Multi-In-Single-Out architecture and achieves only moderate performance in video prediction since they lack of ability to model complex Spatio-temporal dynamics. There is still a big gap in our goal to tailor it for video data. To tackle the issue, we propose a series of surgeries to enhance the ability of the model to capture spatial structures and local Spatio-temporal dependencies. Our final model is presented in \Cref{fig:model}.

The formulation of our full model can be written as:
\begin{equation}
\mathcal{S}_{t+1:t+n} = \Phi_d(\Phi_e(\mathcal{S}_{t-m+1:t}), \mathcal{T}_{t+1:t+n})
\label{eq: learning procedure}
\end{equation}
where $\Phi_e$ and $\Phi_d$ are the encoder and decoder modules respectively. $\mathcal{T}_{t+1:t+n}$ is a length-$n$ timestep queries. For an input video $\mathcal{S}_{t-m+1:t}\in \mathbb{R}^{m\times C_0\times H_0 \times W_0}$ with $m$ frames, we first decompose each frame into non-overlapping patches and then obtain the sequence-level feature maps $h\in \mathbb{R}^{m\times C \times H\times W}$ after several convolution layers.

Note that, we operate 2D Convolution Multi-Head Attention (MHA) first to generate the query map and paired key-value map of each frame. It benefits the model to simultaneously learn the temporal correlation and preserve the spatial information of sequence. Moreover, different from the existing auto-regressive prediction paradigm of Transformers,
we do not apply the mask operation on the multi-head self-attention in the decoder block. Because our decoder inputs are time step embedding that is already known. In our model, each generated frame is conditioned not only on the past sequence but also on its future frames, which preserves the dependency of future frames. After exploration, we figure out two operations can boost the performance drastically. The details are provided in the following.

\noindent \textbf{Local Spatio-Temporal Block}
We figure out one reason that the vanilla Transformer is limited in performance since they lack the ability to capture the short-term variation of sequences which is crucial for video generation. Therefore, we are motivated to install a local Spatio-temporal block which aims to learn the high order relationship between time step queries $\mathcal{T}$ and their corresponding output sequence $\mathcal{S}$. We find out a simple way to implement this idea by installing this block in the place of the FCN layers of the encoder and decoder. Theoretically, this block can be any neural network which has the ability to capture Spatio-temporal information for video data. While a more complex design potentially will result in more gains, we hope to design this block as simple as possible to validate the importance of local Spatio-temporal clues for video prediction. Inspired by most MIMO frameworks in video domains, we choose a simple 3D Convolution layer which is widely used to capture these clues as our block. Specifically, the block is consisted of two embedding layers with each is a \texttt{3DConv}, \texttt{LayerNorm}, \texttt{SiLU} block. The kernel size of \texttt{3DConv} is $3\times 3\times 3$.

\noindent \textbf{Multi-Out Decoder}
To achieve our goal, we need to rectify the original decoder to enable it to produce several future frames parallelly. The challenge here is how to maintain the order information of in video sequences. Considering the permutation-invariant character of the Transformer architecture, we here focus on the temporal positional encoding of the sequence. Specifically, we define a time step vector $T=[1,2,\dots,m+n]$. Then we embed $T\in \mathbb{R}^{1\times (m+n)}$ to $\hat{T}\in \mathbb{R}^{(m+n)\times C}$ by an embedding layer. Finally, we expanse $\hat{T}$ to $\mathcal{T}\in \mathbb{R}^{(m+n)\times C\times H\times W}$ along the height and width dimensions. Accordingly, the temporal positional encoding of input sequence is $\mathcal{T}_{1:m} \in \mathbb{R}^{m\times C\times H\times W}$. After that, we regard $\mathcal{T}_{m+1:m+n}\in \mathbb{R}^{n\times C\times H\times W}$ as time step queries and input them to the decoder module. In this way, our new decoder is capable of dealing with multi-output scenarios. SISO strategy is somehow a specific case of MIMO strategy when the input length is equal to one. The details of this Multi-Out decoder are illustrated in \textit{supplementary materials}.

\begin{table*}[t]
	\small
	\centering
	\tabcolsep10pt
	\caption{Quantitative comparison of different models on Moving MNIST dataset.}
	\begin{tabular}{l|ccc|ccc}
		\hline
		\multirow{2}{*}{Method} & \multicolumn{3}{c|}{10 - 10}& \multicolumn{3}{c}{10 - 30} \\
		\cline{2-7}
		& MSE  $\downarrow$ & MAE  $\downarrow$ & SSIM $\uparrow$& MSE  $\downarrow$ & MAE  $\downarrow$ & SSIM $\uparrow$\\
		\hline
		\hline
		ConvLSTM \cite{shi2017deep} & 103.3& 182.9 & 0.707 &-&-&-\\
		MIM \cite{wang2019memory} & 44.2 & 101.1 & 0.910 &-&-&- \\
		CrevNet\cite{yu2019crevnet} & 22.3 & - & 0.949 &-&-&- \\ \hdashline
		ConvTTLSTM \cite{su2020convolutional}& 56.0& 92.9& 0.913 & 105.7  & - & 0.840\\
		PhyDNet \cite{guen2020disentangling} & 24.2  & 70.3 & 0.947&58.7 & 144.1 & 0.852 \\
		LMC-Memory \cite{lee2021video} & 41.5  & - & 0.924 &73.2 & - & 0.879\\
		MotionRNN \cite{wu2021motionrnn}& 29.2  & 87.7& 0.933&64.2 & 156.1 & 0.844\\
		SimVP \cite{gao2022simvp}& 23.8 & 68.9 & 0.948 &-&-&- \\
		\hline
		\textbf{MIMO-VP} & \textbf{17.7} & \textbf{51.6} & \textbf{0.964} & \textbf{31.6}& \textbf{79.1}& \textbf{0.934}\\
		\hline
	\end{tabular}
	\label{Tab: Moving MNIST}
\end{table*}
\subsection{Spatiotemporal Sequence Prediction}

Suppose we observe a video sequence $\ \mathcal{S}_{t-m+1:t} := \{\mathcal{X}_{t-m+1},$ $\mathcal{X}_{t-m+2}, \dots, \mathcal{X}_t \}$,
where $\mathcal{X}_t\in \mathbb{R}^{C_0\times H_0\times W_0}$ represents the frame (channel $C_0$, height $H_0$, width $W_0$) at the current timestep $t$.
The goal of the spatiotemporal sequence prediction is to generate the most likely length-$n$ sequence in the future given the previous length-$m$ sequence including the current observation:
\begin{equation}
\begin{split}
\mathcal{S}^*_{t+1:t+n} = \mathop{\arg\max}\limits_{\mathcal{S}_{t+1:t+n}} \  p(\mathcal{S}_{t+1:t+m} | \mathcal{S}_{t-m+1:t})
\end{split}
\label{eq: Spatiotemporal Sequence Prediction}
\end{equation}
\noindent \textbf{SISO predictive strategy.}
To solve the above problem, RNN-based models usually adopt the SISO strategy that outputs a single next frame taking as input the current frame.
The SISO prediction paradigm can be expressed as:
\begin{equation}
\begin{split}
\mathcal{X}_{t'+1} = F(\mathcal{X}_{t'}, M_{t'})+\sigma_{t'},
\end{split}
\label{eq: RNN_solve}
\end{equation}
where $F$ denotes the model, $M_{t'}$ is the memory cell that remembers the past historical information.
$\sigma_{t'}$ is a Tensor noise term.
$\mathcal{X}_{t'}$ is the observed frame ($t'\leq t$) or the generated frame ($t'>t$).
Note that SISO predictive strategy is susceptible to the error accumulation problem.
For simplicity, consider an AutoRegressive with $K$=$1$ and without exogenous inputs model:
\begin{equation}
\begin{split}
\mathcal{X}_{t} = A\mathcal{X}_{t-1}+\sigma_{t}
\end{split}
\label{eq: AutoRegressive}
\end{equation}
where $A$ is a learnable Tensor.
It is easy to have:
\begin{equation}
\begin{split}
\mathcal{X}_{t+n} = A^{t+n-1}\mathcal{X}_{t}+\sum_{k=t}^{t+n-1}A^k\sigma_{t+n-k}+\sigma_{t+n}
\end{split}
\label{eq: error}
\end{equation}
It is clear that the error $R_{t+n}=\sum_{k=t}^{t+n-1}A^k\sigma_{t+n-k}+\sigma_{t+n}$ will accumulate as timestep $t+n$ increases,
which makes accurate prediction challenging.
Besides, the long-term dependency problem inherited from RNNs would further limit the performance of the SISO strategy on video prediction.

\noindent \textbf{MIMO predictive strategy.}
We propose in this paper a novel multiple-in-multiple-out predictive strategy:
\begin{equation}
\begin{split}
\mathcal{S}_{t+1:t+n} = F(\mathcal{S}_{t-m+1:t})+ \Sigma_{t+1:t+n},
\end{split}
\label{eq: our_solve}
\end{equation}
where $\Sigma_{t+1:t+n}:=\{\sigma_{t+1},\dots,\sigma_{t+n}\}$ is the normal prediction error.
Two attractive properties of this MIMO strategy are:
(1) it is free of error accumulation without the recurrent structure, which facilitates accurate prediction by taking as input the confident observed sequence;
(2) it generates the whole sequence $\mathcal{S}_{t+1:t+n}$ in parallel, which improves  accuracy by preserving the dependency between futures.
However, it is non-trivial to apply this strategy for video prediction because it demands the ability to model long-term spatiotemporal dependencies of sequence.

\begin{figure}[t]
	\centering
	\includegraphics[width=0.9\linewidth]{pictures/model.pdf}
	\caption{\textbf{Illustration of Predormer architecture.} It contrains two main components: 1) a 2D convolutional multi-head attention (2DMHA); and 2) a 3D convolutional feed-forward networks (3DFFN). Predormer predicts target video sequences in parallel.
	}
	\label{fig:model}
\end{figure}
\subsection{Predormer Architecture}
We introduce a MIMO video prediction model built upon Transformer \cite{vaswani2017attention}, termed as Predormer, which is an encoder-decoder architecture, as depicted in \Cref{fig:model}.
The building block for Predormer mainly consists of two modules:
a 2D convolutional Multi-Head Attention (2DMHA) that helps to capture the global long-term motion trend of sequence,
and a 3D convolutional Feed-Forward Network (3DFFN) that focuses on modeling the local short-term variation of sequence.
The overall learning procedure of Predormer is expressed as:
\begin{equation}
\mathcal{S}_{t+1:t+n} = \Phi_d(\Phi_e(\mathcal{S}_{t-m+1:t}), \mathcal{T}_{t+1:t+n})
\label{eq: learning procedure}
\end{equation}
where $\Phi_e$ and $\Phi_d$ are the encoder and decoder modules respectively. $\mathcal{T}_{t+1:t+n}$ is a length-$n$ timestep queries. For an input video $\mathcal{S}_{t-m+1:t}\in \mathbb{R}^{m\times C_0\times H_0 \times W_0}$ with $m$ frames, we first decompose each frame into non-overlapping patches and then obtain the sequence-level feature maps $h\in \mathbb{R}^{m\times C \times H\times W}$ after several convolution layers. Details of the subsequent 2DMHA and 3DFFN are as follow.

\noindent \textbf{2DMHA.}
The core issue for pixel-level video prediction is long-term spatiotemporal sequence modeling.
While the vanilla Transformer \cite{vaswani2017attention} has proven powerful for modeling long-range relationships of sequence in machine translation,
it is also important to preserve the spatial information of frames for video prediction.
To compensate for this limitation, we propose to incorporate the convolutional structures into the multi-head attention to simultaneously model the temporal correlation and preserve the spatial information of sequence.
Given a sequence-level feature map $h\in \mathbb{R}^{m\times C \times H\times W}$ (similarly, $h'\in \mathbb{R}^{n\times C \times H\times W}$ in the decoder),
we apply 2D convolutions to generate the query map and paired key-value
map of each frame:
\begin{equation}
	Q, K, V \in \mathbb{R}^{m\times C \times H\times W} = \Theta_{q}(h), \Theta_{k}(h), \Theta_{v}(h)
\end{equation}
where $\Theta_{q}$, $\Theta_{k}$ and $\Theta_{v}$ are $1\times 1$ convolutions.
Then we compute the output feature maps by Scaled Dot-Product Attention as:
\begin{equation}
\textup{Attention}(Q,K,V)=\textup{SoftMax}(\frac{QK^T}{\sqrt{d_k}})V
\end{equation}
where $d_k=C*H*W$, $QK^T\in \mathbb{R}^{m\times m}$ is the temporal attention map of the input sequence.
Finally, the 2DMHA with $h$ heads used in our work can be expressed as:
\begin{equation}
\begin{split}
\textup{2DMHA}(Q,K,V)=\Theta_o(\textup{Concat}(\textup{head}_1,\dots,\textup{head}_h))\\
\textup{where head}_i = \textup{Attention}(\Theta_{q_i}(h), \Theta_{k_i}(h), \Theta_{v_i}(h))
\end{split}
\end{equation}
where $\Theta_{q_i}$, $\Theta_{k_i}$ and $\Theta_{v_i}$ are $1\times 1$ convolutions with output channel of $C/h$.

Note that, in each decoder block, different from the existing auto-regressive prediction paradigm of Transformers,
we do not apply the mask operation on the multi-head self-attention,
because our decoder inputs are timestep encodings that are already known.
In such a manner,
the decoder can generate efficiently multiple future frames within one-step forward inference, so as to avoid error accumulation.
In addition,
each generated frame is conditioned not only on the past sequence but also on its future frames,
which makes Predormer capable of preserving the dependency of future frames.

\noindent \textbf{3DFFN.}
While 2DMHA can model the long-term motion trend of sequences, we find that the short-term variation of sequences is also crucial for video generation.
We, therefore, propose to adopt the 3D convolutions to model the local spatiotemporal relations among frames,
so as to learn the high order relationship between timestep queries $\mathcal{T}$ and their corresponding output sequence $\mathcal{S}$.
The 3DFFN used in our Predormer architecture consists of two embedding layers with each is a \texttt{3DConv}, \texttt{LayerNorm}, \texttt{SiLU} block.
The kernel size of \texttt{3DConv} is $3\times 3\times 3$.

\begin{comment}
Given a feature map $\mathcal{Z}$ output from 2DMHA, the computing of the 3DFFN is:
\begin{equation}
\begin{split}
\hat{\mathcal{Z}} &= \text{SiLU}(\textup{LayerNorm}(\Theta_{z_1}(\mathcal{Z})))\\
\bar{\mathcal{Z}} &= \textup{SiLU}(\textup{LayerNorm}(\Theta_{z_2}(\hat{\mathcal{Z}})))\\
\textup{3DFFN}(\mathcal{Z}) &= \textup{LayerNorm}(\mathcal{Z}+\bar{\mathcal{Z}})
\end{split}
\end{equation}
where $\Theta_{z_1}$ and $\Theta_{z_2}$ are $3\times 3\times 3$ convolutions with output channel of $C$.
\end{comment}

\noindent \textbf{Temporal positional encoding and timestep quires.}
Considering the permutation-invariant character of the Transformer architecture,
we here focus on the temporal positional encoding of the sequence because the convolutional operation is adopted to preserve the structure information of frames.
We define a timestep vector $T=[1,2,\dots,m+n]$.
Then we embed $T\in \mathbb{R}^{1\times (m+n)}$ to $\hat{T}\in \mathbb{R}^{(m+n)\times C}$ by an embedding layer.
Finally, we expanse $\hat{T}$ to $\mathcal{T}\in \mathbb{R}^{(m+n)\times C\times H\times W}$ along the height and width dimensions.
Accordingly, the temporal positional encoding of input sequence is $\mathcal{T}_{1:m} \in \mathbb{R}^{m\times C\times H\times W}$.
And we regard $\mathcal{T}_{m+1:m+n}\in \mathbb{R}^{n\times C\times H\times W}$ as timestep queries and input them to the decoder module.

\subsection{Learning Objectives.}
We adopt two widely used content losses, \textit{i}.\textit{e}., \textbf{MSE loss} and \textbf{MAE loss}, after each decoder layer to guide the model training.
The loss function $\mathcal{L}$ can be expressed as:
\begin{equation}
\mathcal{L} =  \|\mathcal{S}^*-\mathcal{S}\|_F^2+\|\mathcal{S}^*-\mathcal{S}\|_1,
\end{equation}
where $\mathcal{S}^*$ is the generated frame sequence and $\mathcal{S}$ is the corresponding GroundTruth.