\section{Introduction}
	\label{sec:introduction}
	Multi-sensor calibration is one of the most important prerequisites for successful mobile autonomy in the real-world. To have complete $360^{\circ}$ sensing coverage, mobile vehicles often have multiple sensors around the vehicle, and in fact there can be extra sensors for redundancy and safety reasons (see Fig.~\ref{fig:scania-bevda}). This complexity poses a challenge when calibrating the sensors to an acceptable level before perceiving the environment coherently.

	Recent advances in lidar sensing have promoted their usage in the sensing stack of autonomous vehicles including wider Field-of-View (FoV), longer range and higher resolution. Since, lidar sensors can provide range information along with semantics
	\cite{pointnet_plus_plus_qi_2017} they have become a key component in the perception pipeline of autonomous platforms.

	However, significant calibration degradation can occur due to issues such as thermal expansion/contraction of the lidar mount relative to the platform; navigating on rough terrain (as found in a mine) or the general effect of wear and tear over the long run. Hence, it is imperative to be able to re-calibrate, adopt and update the lidar extrinsic parameters \textit{in situ}. Offline calibration or target-based lidar calibration processes \cite{Huang2009, Pandey2010, Gao2010, Choi2016, Zhou2018, Kim2020} can be time-intensive and costly for large mobile platforms. Marker-less extrinsic calibration with state optimization \cite{Kuemmerle2011b}  or
	trajectory optimization \cite{Tsai_Hand_Eye_1989, Taylor_2015, Taylor_2016} are more desirable because of their time efficiency and ease of usage by an end-user.

	\begin{figure}
		\centering
		\includegraphics[width=1\columnwidth]{figures/calibration/scania-bus-coverage.pdf}

		\caption{Illustration of the Field of View (FoV) of two lidars and four cameras positioned around the data collection vehicle. The scale factors of the sensor ranges and the vehicle dimension are only for illustrative purposes. The vehicle base frame $\mathtt{{B}}$, is located at the center of the rear axle. The sensor frames of the 2 lidars are: $\mathtt{L}_{{F}|{R}}$, where $F$ and $R$ are the front right and rear left lidars.}
		\label{fig:scania-bevda}
		\vspace{-5mm}
	\end{figure}

	\subsection{Motivation}

	To create an efficient extrinsic calibration framework suitable for multiple lidars we started with data collected using a Scania bus with similar sensor FoVs to that shown in Fig. \ref{fig:scania-bevda}. In spite of the various available solutions for multi-lidar calibration there were two major challenges. Problem 1) Although the sensors were synchronized with PTP synchronization \cite{vallat2007clock}, the estimated poses from the state estimation which were used for motion based calibration were not necessarily time synchronized. Problem 2) Because of the limited FoV overlap between the front right and the rear left lidar, we could not verify the quality of the estimated calibration as it was not possible to inspect if the point clouds expressed a consistent view of the surroundings.

	Prior work has addressed the time alignment issue by correlating the angular velocity norms of both pose signals \cite{furrer2018evaluation}. Further calibration refinement has been performed based on overlapping FoV \cite{Jiao2019, Jiao2021}. For non-overlapping FoV scenario, calibration verification was handled by matching features in a global map \cite{Carrera2011} in the visual domain.

	In our work, to address Problem 1) we have used dual quaternion (DQ) \cite{daniilidis1996dual} based time alignment of the poses using screw linear interpolation (ScLERP) \cite{kavan2005spherical} as it provides the shortest path between pose estimates on $\ensuremath{\mathrm{SE}(3)}\xspace$ manifold. We have tackled Problem 2) by evaluating the correspondences of selected similar features observed from non-overlapping FoV lidars at different timestamps, analyzed based on vehicle kinematics. We have additionally used cameras, which were placed below the corresponding lidars as shown in Fig. \ref{fig:coordinate-frames} to improve the quality of feature selection from point clouds.

	\subsection{Contribution}
	\label{sec:contribution}

	Our work is motivated by the Hand-Eye  calibration method of Tsai \textit{et al}~\cite{Tsai_Hand_Eye_1989}. Our specific contributions are additional procedures which increase the accuracy and robustness of the extrinsic calibration process as follows:
	\begin{itemize}
		\item A DQ based prior time alignment of the poses based on ScLERP before applying the DQ based Hand-Eye calibration.
		\item An additional verification method using semantic matching to evaluate the extrinsic calibration quality for non-overlapping FoV lidars.
		\item Experimental results and verification using data collected with a Scania bus mounted with the sensor setup as shown in Fig. \ref{fig:coordinate-frames}, with FoV schematics similar to Fig. \ref{fig:scania-bevda}.
	\end{itemize}

	\section{Related Work}
	\label{sec:related-work}

	There are several techniques to obtain the extrinsic calibration parameters of the multi-sensor platform. This section categorizes those methods and briefly describes the relevant work	in each category.

	\subsection{SLAM-based methods}
	Simultaneous Localization and Mapping (SLAM) approaches normally estimate the extrinsic parameters by considering it as a state variable in the pose graph optimization or bundle adjustment optimization in visual SLAM. Kummerle \textit{et al}~\cite{Kuemmerle2011b} extended the SLAM hyper graph formulation to handle calibration and kinematic parameters without prior knowledge of the environment by including wheels radii and sensor position as state variables.
	Jones \textit{et al}~\cite{Jones2011}, proposed a visual inertial SLAM system that localized the sensor positions in the map frame to estimate the calibration parameter between a camera and an inertial measurement unit (IMU). This approach relied heavily on a robust SLAM solver along with a rich motion sequence. Similarly, Carrera \textit{et al}~\cite{Carrera2011} presented a visual SLAM algorithm for multi-camera extrinsic calibration by building a global landmark map from each camera and then fusing the maps by means of feature correspondences using similarity transformation bundle adjustment.
	This approach required driving in a circular path as the basic prerequisite for calibration to ensure the observability of similar features.
	To address this, Heng \textit{et al}~\cite{Heng2013} extended the prior work by incorporating wheel odometry and formulating a least-squares problem. Their method was restricted to ground robots with odometry. In contrast, their recent work \cite{Heng2015} had no odometry requirement and utilized visual SLAM and bundle adjustment using a multi-camera system where each sensor built its own map. Map merging was then performed using a hand-eye calibration method and the results were utilized in pose graph optimization to estimate the extrinsics among sensors.

	\subsection{Appearance-based methods}
	The recent appearance based approaches can be clustered into two categories; first, those using artificial markers or a known calibration target (e.g. a checkerboard), and second automatic geometric feature extraction methods which do not rely on markers. Appearance based methods in general could be seen as registration problems where artificial markers or geometrical features are employed. Gao \textit{et al}~\cite{Gao2010}, for instance, recovered the calibration parameters between multiple lidars by manually observing reflective landmarks in the scene.
	Zhou \textit{et al}~\cite{Zhou2018}, presented a method to calibrate lidar and camera sensor modalities using a checkerboard by combining 3D line and plane features. Similarly, in \cite{Huang2009}, \cite{Pandey2010} and \cite{Kim2020}, the authors proposed lidar to camera calibration by utilizing plane feature registration using a chessboard. Automatic data extraction from sensors makes the usage of artificial markers redundant. Choi et al. \cite{Choi2016} determined the relative extrinsic parameters between 2D-lidars by extracting orthogonal planes and utilizing the co-planar features of the scan points. Geometric features such as points, lines and planes were extracted in \cite{He2013} and matched to calibrate a dual lidar system. Levinson \textit{et al}~\cite{Levinson2013} presented an online camera to lidar calibration optimizer by aligning edge points from camera images and corresponding lidar points. However, for all these methods to work the general assumption was that the features were visible through an overlapping FoV correspondence.

	\subsection{Motion-based methods}
	Incremental motion based approaches recover the calibration parameters using the well-known hand-eye calibration formulation, \cite{Tsai1989}. The calibration was estimated between a gripper (hand) and a camera (eye) by moving the gripper in some trajectory, while exploiting the fact that the transformation between the camera and the gripper was always the same. More recent research has focused on extending the work to multi-sensor modalities by aligning the motion of each individual sensor. Brookshire \textit{et al}~\cite{Brookshire2013} utilized the DQ method \cite{Daniilidis1999} to recover the relative transformation of two camera sensors mounted on a rigid body. The same principle was also applied by Heng \textit{et al}~ in camodocal \cite{Heng2013}. A more generic sensor configuration was proposed by Taylor et al. \cite{Taylor_2015} for multi-modal sensor arrays calibration including 3D-lidar, IMU and camera by motion combination of sensors using hand-eye formulation. They also provided the uncertainty of the calibration. The same authors extended the work to incorporate the sensor's accuracy reading and estimated the timing offsets between the sensors as well \cite{Taylor_2016}. Moreover, overlapping fields of view was applied to further enhance the calibration when applicable. Jiao \textit{et al}~\cite{Jiao2019} exploited the hand-eye calibration of a multi-lidar system by motion alignment to recover an initial estimation of the extrinsic. A refinement process using point-to-plane features was then used to further enhance the	calibration parameters. In their recent work M-LOAM \cite{Jiao2021}, they used similar techniques with edge and planar features for the calibration refinements.

	Both SLAM based approach and motion based approach can be used to solve the extrinsic calibration problem for our system. In our work we adapted a motion based method to solve the extrinsic calibration problem due to the accuracy of the poses estimated via LOAM \cite{zhang2014loam} based approaches. We performed the trajectory alignment as an initialization step similar to \cite{Jiao2019}. With this initialization, we first did the time alignment based on DQ based ScLERP. After that we recovered the individual sensor poses and formulated the problem as a Hand-Eye calibration problem. The Hand-Eye problem was solved using a DQ based closed form solution \cite{Daniilidis1999}. Finally, we developed a verification method for evaluating the quality of the calibration by semantic matching of the chosen lidar features.

	\section{Problem Statement}
	\label{sec:problem-statement}

	\subsection{Reference Frames}
	\begin{figure}
		\centering
		\vspace{2mm}
		\includegraphics[width=1\columnwidth]{figures/calibration/corner-module.png}
		\vspace{-7mm}
		\caption{Reference frames conventions for our vehicle platform. The world frame $\mathtt{W}$ is a fixed frame, while the base frame $\mathtt{{B}}$, as shown in Fig.~ \ref{fig:scania-bevda}, is located at the rear axle center of the vehicle. Each sensor unit contains the two optical frames $\mathtt{C}$, an IMU frame, $\mathtt{I}$, and lidar frame $\mathtt{L}$.}
		\label{fig:coordinate-frames}
		\vspace{-4mm}
	\end{figure}
	We start by describing the necessary notation and reference frames used in our system according to the convention by Furgale~\cite{furgale2014representing}. The reference frames are shown in Fig.~ \ref{fig:coordinate-frames}. The vehicle base frame $\mathtt{{B}}$ is located on the center of the rear-axle of the vehicle. Sensor readings from lidars, cameras and IMUs are represented in their respective sensor frames as $\mathtt{L}_k$, $\mathtt{C}_k$ and $\mathtt{I}_k$ respectively, where $k$ denotes the location of the sensor in the vehicle.

	For example, $\mathtt{L}_{F}$ and $\mathtt{L}_{R}$ represent the front right and rear left lidar frames. In our system we calibrated $\mathtt{L}_{R}$ with respect to $\mathtt{L}_{F}$ and used an known transformation between $\mathtt{{B}}$ and $\mathtt{L}_{F}$ from the vehicle CAD files. After calibration, we converted the extrinsics to vehicle base frame for verification and visualization purpose discussed in Sec. \ref{verification}.

	\subsection{Pose Estimation Frames}

	For pose estimation of individual lidars we computed the relative transformation between subsequent lidar readings in its corresponding sensor frame. Conscutive estimates were available at times $t_{i-1}$ and $t_i$, denoted as ${\mathbf{T}}{_{\mathtt{L}_{i-1}\mathtt{L}_{i}}}$, where $\mathbf{T} = \left[\begin{array}{ll}\mathbf{R_{3\times3}} & \mathbf{t_{3\times1}} \\ \mathbf{0}^{\top} & 1\end{array}\right]\in \ensuremath{\mathrm{SE}(3)}\xspace$ represents a homogeneous transformation consisting of rotation, $\mathbf{R} \in \ensuremath{\SO(3)}\xspace$ and translation $\mathbf{t} \in \mathbb{R}^3$. In our system we denote the iterative lidar pose transforms as ${\mathbf{T}}{_{F_{i-1}F_{i}}}$ and ${\mathbf{T}}{_{R_{i-1}R_{i}}}$. For brevity of discussion, we will denote them as ${\mathbf{T}}{_{F}}$ and ${\mathbf{T}}{_{R}}$ respectively. In future, we would extend this approach to solve the multi-lidar calibration scenario as the data collection vehicle is equipped with more than two lidars, as shown in Fig. \ref{fig:coordinate-frames}.

	\subsection{Problem Formulation}

	\begin{figure}[!t]
		\centering
		\vspace{2mm}
		\includegraphics[width=1\columnwidth]{figures/calibration/hand-eye.eps}
		\caption{Transformation in between the lidars $\mathtt{L}_{F}$ and $\mathtt{L}_{R}$ in between times $t_{i-1}$ and $t_{i}$.}
		\label{fig:hand-eye}
		\vspace{-7mm}
	\end{figure}

	Given a set of point clouds, ${\cal P}_{\mathtt{L}_{k:1(1){\cal K}}}^{t_{i:1(1){\cal N}}}$, from ${\cal K}$ different lidars in a time sequential order $t_1 \dots t_{\cal N}$, in their respective sensor frames, we wish to recover the extrinsics between the lidars as, ${\mathbf{T}}{_{\mathtt{L}_{a}\mathtt{L}_{b}}}$, where $a,b:1(1){\cal K}$ and $a\neq b$. We assume the lidars are fixed rigidly to the vehicle and all the other lidars are calibrated with respect to a reference lidar. The multi-lidar calibration task is formulated in terms of a Hand-eye \cite{Tsai_Hand_Eye_1989} calibration problem, where the extrinsics are recovered by matching the homogeneous transformations of the multiple lidars. For brevity of discussion we will consider two non-overlapping FoV lidars, the front right and rear left, and aim to recover the extrinsics. However, it can be extended to multi-lidar setup as well. With regards to the Hand-Eye method, we can recover the extrinsic calibration in between the lidars, as shown in Fig.~ \ref{fig:hand-eye}, by using the following equation,

	\vspace{-7pt}
	\begin{equation}
	{\mathbf{T}}{_{F}}{\mathbf{T}}{_{FR}} = {\mathbf{T}}{_{FR}}{\mathbf{T}}{_{R}}
	\label{eq:hand-eye}
	\end{equation}

	which can be further decomposed into,
	\vspace{-3pt}
	\begin{equation}
	\begin{aligned}
		\mathbf{R}_{F}\mathbf{R}{_{FR}} &= \mathbf{R}{_{FR}}\mathbf{R}{_R} \\
		(\mathbf{R}{_F} - \eye_{3 \times 3}) \mathbf{t}_{FR} &= \mathbf{R}{_{FR}} \mathbf{t}{_R} - \mathbf{t}{_F}.
		\label{eq:hand-eye-rt}
	\end{aligned}
	\end{equation}

	Multiple solutions including iterative and closed-form methods along with their benefits and drawbacks \cite{shah2012overview} have been discussed to solve the Hand-Eye calibration eq. \ref{eq:hand-eye} and \ref{eq:hand-eye-rt}. The solution methods can be further categorized as separable and simultaneous solutions. The separable solutions solve for $\mathbf{R_{3\times3}}$ and $\mathbf{t_{3\times1}}$ separately, whereas the simultaneous solutions solve them jointly. In our work, we have adapted a simultaneous approach using DQ \cite{Daniilidis1999}.

	\section{Methodology}
	\label{sec:methodology}

	\subsection{Initialization}
	Pose estimation of the individual sensors was computed using the VILENS estimator~\cite{wisth2020vilens}. However, instead of using a feature-based configuration, our configuration fused ICP-based point cloud matching~\cite{Pomerleau12comp} with IMU pre-integration using GTSAM \cite{Forster2017,Dellaert2017} to  estimate the motion of the individual lidars and to correct for motion distortion effects. We used the set of estimated poses $\mathbf{t_{F}}$ and $\mathbf{t_{R}}$ recovered at ${\cal N}$ time iterations to estimate an initial ${\mathbf{T}}{_{FR}}$ by solving the following least-squares optimization,

	\begin{equation}
		\mathbf{R}^*, \mathbf{t}^*  = \argmin_{\mathbf{R}, \mathbf{t}} \|\mathbf{R}\mathbf{t}_{F} + \mathbf{t} - \mathbf{t}_{R}\|^2.
		\label{eq:kabash-alignment}
	\end{equation}
	We used the closed form solution proposed by Umeyama \cite{umeyama1991least} to recover the initial optimal $\mathbf{R}^*$ and $\mathbf{t}^*$ which gives similar results as Horn et al. \cite{horn1987closed} in quaternion space. This initialization was necessary for the DQ based optimizer to generate better results.

	\begin{comment}
	Horn et al. proposed to break the problem into rotation and translation parts. To recover the rotation part in quaternion form the optimization function was rewritten as,
	\begin{equation}
	\begin{aligned}
	&& \mathbf{R}^*  &= \argmin_{\mathbf{R}} \|\mathbf{R}{\cal T}_{\mathtt{L}_F} - {\cal T}_{\mathtt{L}_{R}}\|^2 \\
	&&&= \|{\cal T}^\top_{\mathtt{L}_F}{\cal T}_{\mathtt{L}_F}\| + \|{\cal T}^\top_{\mathtt{L}_R}{\cal T}_{\mathtt{L}_R}\| - 2\argmin_{\mathbf{R}}\left(\mathbf{R}{\cal T}_{\mathtt{L}_F}{\cal T}_{\mathtt{L}_R}^\top\right)\\
	& &&= \argmax_{\mathbf{R}} \left(\mathbf{R}{\cal T}_{\mathtt{L}_F}{\cal T}^\top_{\mathtt{L}_{R}}\right)\\
	&& \mathbf{R}{\cal T}_{\mathtt{L}_F}{\cal T}^\top_{\mathtt{L}_{R}}  &=  \sum_{i=1}^{{\cal N}}\left(q(\mathbf{R}) {q({\mathbf{t}}_{L_F}^{(i)})}^{\prime} q(\mathbf{R})^{*}\right) \cdot {q({\mathbf{t}}_{L_R}^{(i)})}^{\prime} \\
	&&  &=  \sum_{i=1}^{{\cal N}}\left(q(\mathbf{R}) {q({\mathbf{t}}_{L_F}^{(i)})}^{\prime}\right) \cdot  \left( {q({\mathbf{t}}_{L_R}^{(i)})}^{\prime}q(\mathbf{R})\right) \\
	&&  &=  q(\mathbf{R})\sum_{i=1}^{{\cal N}}\left( {q({\mathbf{t}}_{L_F}^{(i)})}^{\prime} {q({\mathbf{t}}_{L_R}^{(i)})}^{\prime}\right)q(\mathbf{R})		\label{eq:horn-alignment}
	\end{aligned}
	\end{equation}
	The above part can be maximized by analyzing the eigenvalues of ${q({\mathbf{t}}_{L_F}^{(i)})}^{\prime}{q({\mathbf{t}}_{L_R}^{(i)})}^{\prime}$, more details are available here \cite[p.~8--11]{jia2008quaternions}. Similar principle is applied in the DQ case as well. \href{https://www.mvtec.com/doc/halcon/13/en/toc_transformations_dualquaternions.html}{DQ visualization wrt $L$ and $M$.}
	\end{comment}

	\subsection{Dual Quaternion Based Hand-Eye Calibration}
	Homogeneous transformation matrices are orthogonal and due to floating point errors, operations on them often result in matrices that need to be re-normalized. Although, this can be achieved using the Gram-Schmidt method, this is however slower than quaternion normalization. DQs are the natural extension to quaternions; because, unlike quaternions, they can handle translation as well in the same compact notion.

	\subsubsection{Preliminaries}
	We review DQ's and its operations briefly; more details are available here \cite[p.~53--62]{mccarthy1990introduction}. DQ has the advantages of a quaternion such as gimbal lock problem and unambiguous representation but with less intensive operations for transformations. A DQ, $\mathbf{q}$, can be written as,

	\begin{equation}
		\mathbf{q}=q_{r}+\varepsilon q_{d},
		\label{eq:dq}
	\end{equation}

	\noindent where the real part $q_{r}$, and the dual part $q_{d}$, are quaternions. $\varepsilon$ is the dual unit, defined by $\varepsilon^2 = 0$ and $\varepsilon \ne 0$. The conjugate of a DQ, $\mathbf{q}$ is:
	\begin{equation}
		\mathbf{q^\star}=q_{r}^\star+\varepsilon q_{d}^\star,
		\label{eq:dq-conjugate}
	\end{equation}
	where, $q^\star = (q_w,\text{-}q_x,\text{-}q_y,\text{-}q_z)$, is quaternion conjugate. Some operations of DQs, $\mathbf{q_1}$ and $\mathbf{q_2}$ are as follows:
	\begin{align}
		\mathbf{q_1} + \mathbf{q_2} &= (q_r^{1}+q_r^{2})+\varepsilon (q_d^{1} + q_d^{2}) \\
		\mathbf{q_1}  \otimes  \mathbf{q_2} &= q_r^{1} \otimes q_r^{2} +\varepsilon (q_r^{1} \otimes q_d^{2} + q_d^{1} \otimes q_r^{2}) \\
		\|\mathbf{q}\|^2 &= \mathbf{q}  \otimes  \mathbf{q}^\star \\
		\mathbf{q^{-1}} &= \mathbf{q}^\star, \quad \text{if } \|\mathbf{q}\|^2 = 1,
		\label{eq:dq-operations}
	\end{align}

	where, $\otimes$ and $\|\mathbf{\cdot}\|^2$ represent quaternion product and norm of a DQ respectively.
	Given a transformation, $\mathbf{T} (\mathbf{R}, \mathbf{t})$ we can convert it to DQ form  as,
	\begin{equation}
		\mathbf{q(\mathbf{T})}=q(\mathbf{R})+\varepsilon \frac{1}{2}q(\mathbf{t})q(\mathbf{R})
		\label{eq:T_to_dq}
	\end{equation}
	where, $q(\mathbf{R})$ is obtained from Euler-Rodrigues formula and $q(\mathbf{t}) = (0,t_x,t_y,t_z)$. Conversely, we can convert a DQ, $\mathbf{q}(q_r, q_d)$ to a homogeneous transformation form as,

	\begin{equation}
		\mathbf{T}(\mathbf{q}) = \left[\begin{array}{cc}\mathbf{R}(q_{r}) & 2 q_{d} \otimes q_{r}^{*} \\ 0^{\top} & 1\end{array}\right]
		\label{eq:dq_to_T}
	\end{equation}
	where, $\mathbf{R}(q_{r})$ is obtained from Rodrigues' rotation formula.

	\subsubsection{Pose alignment}
	Since, the estimated poses are not necessarily time synchronized we aligned them using ScLERP \cite{kavan2005spherical}, which is a generalization of SLERP \cite{shoemake1985animating} before solving the Hand-Eye problem. The interpolation between two quaternions is given by SLERP whereas, the interpolation between two DQs is given by ScLERP as,
	\begin{equation}
	\begin{aligned}
		\text{ScLERP}(n;\mathbf{q_1}, \mathbf{q_2}) &= \mathbf{q_1} \otimes (\mathbf{q_1}^{-1} \otimes \mathbf{q_2})^n \\
	\end{aligned}
	\end{equation}
	where, $n \in [0,1]$ and $\mathbf{q}^n = \cos(\frac{n\theta}{2}) + \mathbf{k} \cdot \sin(\frac{n\theta}{2})$. Here, $\theta$ is the dual angle of rotation around the unit DQ, $\mathbf{k}$ with zero scalar part. This ensures synchronized lidar poses at all timestamps, improving Hand-Eye calibration results.

	\subsubsection{Solving Hand-Eye Calibration}
	To solve the Hand-Eye calibration problem in DQ space, we initially converted the lidar transformations in between subsequent timestamps in DQ formulation. $\mathbf{q_{{\textstyle\mathstrut}F}}$ and $\mathbf{q_{{\textstyle\mathstrut}R}}$ represent the front right and rear left converted transformations in DQ space respectively, whereas, $\mathbf{q}$ represents the extrinsics in DQ formulation. Similar, to eq. \ref{eq:hand-eye}, the screw motion concatenation in DQ space yields,
	\begin{equation}
	\mathbf{q_{{\textstyle\mathstrut}F}} = \mathbf{q} \mathbf{q_{{\textstyle\mathstrut}R}} \mathbf{q^\star}.
	\label{eq:dq_transformation}
	\end{equation}
	From Screw Congruence Theorem \cite{chen1991screw} it has been shown only the vector parts in the quaternion contribute to the derivation of unknown DQ, $\mathbf{q}$, as the scalar parts are equal. The vector part $\vv{q}$, in a quaternion $q$ is, $(0, \vv{q})$. With the derivations shown in \cite{Daniilidis1999}, eq. \ref{eq:dq_transformation} can be re-written as,

	\begin{equation}
	\resizebox{\columnwidth}{!}{$
		\left[\begin{array}{cccc}\vv{q_{r}^\text{F}}-\vv{q_{r}^\text{R}} & {[\vv{q_{r}^\text{F}}+\vv{q_{r}^\text{R}}]_{\times}} & \mathbf{0}_{3 \times 1} & \mathbf{0}_{3 \times 3} \\
			& & & \\
			\vv{q_{d}^\text{F}}-\vv{q_{d}^\text{R}} & {[\vv{q_{d}^\text{F}}+\vv{q_{d}^\text{R}}]_{\times}} & \vv{q_{r}^\text{F}}-\vv{q_{r}^\text{R}} & {[\vv{q_{r}^\text{F}}+\vv{q_{r}^\text{R}}]_{\times}}\end{array}\right]\left[\begin{array}{l}q_r \\\\ q_d\end{array}\right]=0,$
		\label{eq:dq_T_matrix}}
	\end{equation}
	where, $[\:\cdot\:]_{\times}$ represents a skew-symmetric matrix and the first term, $\mathbf{S}$ is a 6$\times$8 matrix and the second term, is a 8$\times$1 vector. Based on the data collected at ${\cal N}$ different timestamps we can construct a 6${\cal N}\times$8 matrix, $\mathbf{M} = (\mathbf{S}^\text{T}_1 \quad \mathbf{S}_2^\text{T} \quad \dots \quad \mathbf{S}^\text{T}_{\cal N})^\text{T}$. However, as this is not a full-rank matrix, the last two right singular vector obtained from SVD of $\mathbf{M}$ spans the nullspace of $\mathbf{M}$, which is used to recover $\mathbf{q}$. We also recorded transformations in non-planar trajectories as mentioned in \cite{Daniilidis1999} for recovering solutions of eq. \ref{eq:dq_T_matrix} and solved it within a certain batch size, instead of applying SVD on the whole 6${\cal N}\times$8 matrix at once.

	\subsection{Calibration Verification}
	\label{verification}

	To verify the accuracy of extrinsics, we associated extracted lidar semantic features between the lidars. The semantic features from lidar were extracted by associating the projected point clouds into the segmented camera images. The image segmentation was performed using a trained Efficient-Net model. We used an unsupervised method to calculate a time offset based on vehicle kinematics which described the observation delay between the lidars, corresponding to similar features as represented in Fig. \ref{fig:lidar-semantic-matching}.

	\begin{figure} [!t]

		\centering
		\includegraphics[width=1\columnwidth]{figures/calibration/lidar-semantic-matching.eps}
		\vspace{-9mm}
		\caption{Semantic feature matching in between $L_F$ and $L_R$ lidars viewed at $\delta t = t_j-t_i$ time apart so that FoVs correspond to similar region. First, lidar $L_F$ detects a set of features at time $t_i$ and $\delta t$ time later $L_R$ observes the previously observed set of features (by $L_F$) at time $t_j$.}
		\vspace{-7mm}
		\label{fig:lidar-semantic-matching}
	\end{figure}
	\subsubsection{Efficient-Net for semantic segmentation}

	The author of EfficientNet \cite{tan2019efficientnet} demonstrated that model scaling can be achieved by carefully balancing network depth, width, and resolution, leading to a better performance with the fixed computation resources. Based on this study, we used a modified version of EfficientNet to semantically segment the camera images. The modified network architecture backbone is shown in Table \ref{tab:efficientNet}, where each row describes a stage $i$ with $\hat{L}_{i}$ layers, with input resolution $\langle\hat{H}_{i}, \hat{W}_{i}\rangle $ and output channels $\hat{C}_{i}$. MBConv layers are mobile inverted bottlenecks from the MobileNetV2 architecture \cite{sandler2018mobilenetv2} where squeeze-and-excitation optimization have also been added from \cite{hu2018squeeze}. The segmentation head consists of convolutional layers and bilinear interpolation to upsample the network output to its original input resolution.

	\begin{table}[!h]
		\normalsize
		\centering
		\vspace{-2mm}
		\resizebox{\columnwidth}{!}{
			\begin{tabular}{c|cccc} \toprule
				Stage & Operator & Resolution & \#Channels & \#Layers \\
				\( i \) & \( \hat{\mathcal{F}}_{i} \) & \( \hat{H}_{i} \times \hat{W}_{i} \) & \( \hat{C}_{i} \) & \( \hat{L}_{i} \) \\
				\midrule
				1 & Conv3x3 & \( 640 \times 1024 \) & 32 & 1 \\
				2 & MBConv1, k3x3 & \( 320 \times 512 \) & 16 & 1 \\
				3 & MBConv6, k3x3 & \( 160 \times 256 \) & 24 & 2 \\
				4 & MBConv6, k5x5 & \( 80 \times 128 \) & 40 & 2 \\
				5 & MBConv6, k3x3 & \( 40 \times 64 \) & 80 & 3 \\
				6 & MBConv6, k5x5 & \( 40 \times 64 \) & 112 & 3 \\
				7 & MBConv6, k5x5 & \( 20 \times 32 \) & 192 & 4 \\
				8 & MBConv6, k3x3 & \( 20 \times 32 \) & 320 & 1 \\
				9 & Segmentation Head & \( 640 \times 1024 \) & 24 & - \\
				\bottomrule
		\end{tabular}}
		\caption{ Modified EfficientNet-B0 baseline network architecture.}
		\label{tab:efficientNet}
		\vspace{-6mm}
	\end{table}

	\subsubsection{Association of semantic features}

	We chose to match curb features between the lidars because the lidar projection error was stable across frames for the curb semantics. First, we sampled an arbitrary number of curb semantic points from the $L_F$ lidar by projecting the points into camera perspective, for a specific duration to extract a segment of the curb. Then based on the calculated time offset obtained from vehicle kinematics, we sampled curb semantics from the $L_R$ lidar. Since the point clouds of both lidars were represented in the base frame, we expected that the same set of features should be observable from both lidars. Finally we calculated the root mean square error (RMSE) between the commonly observed curb features. The association between the feature points was established using tri-linear interpolation on the sorted points. We therefore sampled several curb features using the aforementioned technique and statistically compared RMSE before and after the calibration. Curb extraction and association results are presented in Sec. \ref{semantic_verification}.

	\section{Experimental Results}
	\label{sec:results}

	\subsection{Dataset}
	We collected the data from a Scania autonomous bus platform consisting of 4 corner lidars and 8 cameras. For ground truth pose estimates (GT) we used data from a Novatel GNSS. All the sensing data was synchronized using PTP synchronization.

	\subsection{Results}
	\subsubsection{Pose Estimation and DQ based Hand-Eye calibration}
	We estimate the poses of the $L_F$ and $L_R$ lidars and convert them to the vehicle base frame for comparison. The pose estimates are shown in Fig. \ref{fig:maps-odom} and Fig. \ref{fig:init-calibration} along with the GT for the initial calibration parameters from the vehicle model.

	\begin{figure} [!htbp]

		\centering
		\includegraphics[width=1\columnwidth]{figures/calibration/odom.png}

		\caption{Results of the odometry estimation from $L_F$ (blue) and $L_R$ (magenta) lidars plotted in Google Maps with respect to the GT (green) with initial calibration parameters. The GT was generated using Novatel GNSS.}
		\label{fig:maps-odom}
	\end{figure}

	\begin{figure} [!htbp]
		\vspace{-4mm}
		\centering
		\includegraphics[width=1.\columnwidth]{figures/calibration/traj_z.pdf}
		\vspace{0.01\columnwidth}
		\includegraphics[width=1.\columnwidth]{figures/calibration/traj_rpy.pdf}
		\vspace{-5mm}
		\caption{Pose estimation for $L_F$ and $L_R$ lidars with respect to GT with initial calibration estimation.}
		\label{fig:init-calibration}
		\vspace{-1mm}
	\end{figure}
	The relative pose error (RPE) and absolute pose error (APE) metrics are used to evaluate the local and global consistency respectively of the estimated trajectory of $L_R$ lidar with respect to $L_F$ lidar, before and after calibration with the Hand-Eye method after time alignment. The ScLERP based time alignment increased the consistency in between the DQ poses for the scalar part match which was a pre-condition for deriving eq. \ref{eq:dq_T_matrix}. We re-estimate the poses of the $L_R$ lidar after putting in new calibration parameters.

	RPE compares the difference between the individual relative poses within two timestamps $t_i$ and $t_j$ as, $RPE_{i,j} = ({\mathbf{T}}{_{\mathtt{L}_{F}(t_i)}^{-1}}{\mathbf{T}}{_{\mathtt{L}_{F}(t_j)}})^{-1}({\mathbf{T}}{_{\mathtt{L}_{R}(t_i)}^{-1}}{\mathbf{T}}{_{\mathtt{L}_{R}(t_j)}})$. APE is based on the absolute relative pose between two poses ${\mathbf{T}}{_{\mathtt{L}_{F}(t_i)}}, {\mathbf{T}}{_{\mathtt{L}_{R}(t_i)}} \in \ensuremath{\mathrm{SE}(3)}\xspace$ at timestamp $t_i$ as, $APE_{i} = {\mathbf{T}}{_{\mathtt{L}_{F}(t_i)}^{-1}}{\mathbf{T}}{_{\mathtt{L}_{R}(t_i)}}$.
	\begin{figure} [h!]
		\vspace{-3mm}
		\centering
		\includegraphics[width=1.\columnwidth]{figures/calibration/err_REL_t.pdf}
		\includegraphics[width=1.\columnwidth]{figures/calibration/err_REL_R.pdf}
		\vspace{-5mm}
		\caption{RPE between $L_F$ and $L_R$ lidars before and after calibration.}
		\label{fig:calibration-rpe}
		\vspace{-2mm}
	\end{figure}
	\begin{figure} [h]
		\vspace{-7mm}
		\centering
		\includegraphics[width=1.\columnwidth]{figures/calibration/err_APE_t.pdf}
		\includegraphics[width=1.\columnwidth]{figures/calibration/err_APE_R.pdf}
		\vspace{-8mm}
		\caption{APE between $L_F$ and $L_R$ lidars before and after calibration.}
		\label{fig:calibration-ape}
	\end{figure}
	We compare the RPE, Fig. \ref{fig:calibration-rpe} and APE, Fig. \ref{fig:calibration-ape} between $L_F$ and $L_R$ lidar trajectories before and after calibration, which shows that we recover better calibration estimates in between the $L_F$ and $L_R$ lidar.
	\begin{comment}
	APE-R rmse: 1.7614355386488694
	APE-R std: 0.4827498217074241
	APE-T rmse: 5.684375355794342
	APE-T std: 3.471190785918465
	\end{comment}

	\subsubsection{Semantic verification}
	\label{semantic_verification}
	For semantic verification we train the EfficientNet model on a diversified scenario set (city driving, snowy conditions, dessert area, suburban area). The overall training accuracy is 79.10\% and the mean IoU (Intersection over Union) is 0.495. Whereas, for curb class the accuracy is 67.20\% and the IoU is 0.557. The model performance is correlated with the amount of data it is trained on. We extract the curb points by doing a perspective projection of the lidar points in image space and choosing points closer to the curb pixels within a bound of $\pm$5 pixels.
	\begin{comment}
	\begin{table}[h]
	\normalsize
	\centering
	\resizebox{0.75\columnwidth}{!}{
	\begin{tabular}{l|cc} \toprule
	\multirow{2}{*}{\makecell[l]{Class}} & \multicolumn{2}{c}{\makecell[c]{Validation}}\\
	\cline{2-3}
	& \textbf{Accuracy (\%)} & \textbf{Mean IoU}\\
	\midrule
	Drivable Road & 96.20 & 0.834\\
	Building & 92.80 & 0.797\\
	Car & 91.30 & 0.774\\
	Side Walk & 79.30 & 0.675\\
	Curb & 67.20 & 0.557\\
	Fence & 52.30 & 0.379\\
	\bottomrule
	\end{tabular}}
	\caption{Semantic segmentation results using modified EfficientNet-B0 Network with Training accuracy: $\sim$80\% and Mean IoU: $\sim$50\%.}
	\label{tab:efficientNet-results}
	\vspace{-3mm}
	\end{table}

	\begin{figure} [!htbp]
	\centering
	\includegraphics[width=0.49\columnwidth]{figures/calibration/raw-front.png}
	\includegraphics[width=0.49\columnwidth]{figures/calibration/seg-front.png} \\
	\vspace{0.013\columnwidth}
	\includegraphics[width=0.49\columnwidth]{figures/calibration/raw-back.png}
	\includegraphics[width=0.49\columnwidth]{figures/calibration/seg-back.png}
	\caption{Semantic segmentation results using modified EfficientNet-B0 network architecture as shown in Table \ref{tab:efficientNet}. \textit{Top:} Front camera raw image and its corresponding segmentation. \textit{Bottom:} Results from rear camera.}
	\label{fig:semseg-results}
	\vspace{-3mm}
	\end{figure}
	\end{comment}

	\begin{figure} [!htbp]
		\vspace{1mm}
		\centering
		\includegraphics[width=1\columnwidth]{figures/calibration/pc-projection-curb.png}
		\vspace{-7mm}
		\caption{Lidar point cloud projected into camera perspective with filtered curb semantics, shown as cyan colored points.
		\vspace{-4mm}
		\label{fig:pc-projection}
	\end{figure}

	\begin{figure} [!htbp]
		\vspace{1mm}
		\centering
		\includegraphics[width=1\columnwidth]{figures/calibration/dnn-extrapolation.png}
		\vspace{-7mm}
		\caption{Calibrated lidar point clouds from $L_F$ and $L_R$ lidar with their respective image visuals with corresponding curb features highlighted.
		\vspace{-5mm}
		\label{fig:dnn-extrapolation}
	\end{figure}
	As seen in Fig. \ref{fig:pc-projection}, we extract the curb points from the front lidar. And similar extraction was performed on the rear lidar as well. After calculating the time offset based on vehicle kinematics we could verify for the calibrated scenario the RMSE was statistically lower. A full representation of the calibrated lidar point clouds are shown in Fig. \ref{fig:dnn-extrapolation}.

	\section{Conclusion}
	\label{sec:conclusion}
	We have presented a multi-lidar calibration method that
	helps in automatically determine and verify the quality of the extrinsic calibration parameters between the lidars. We have also shown that aligning the poses with ScLERP and applying the DQ based Hand-Eye calibration improves the quality of the extrinsic calibration parameters. In future, the extrinsics can be used in state vector in SLAM solutions and be recovered online by solving the DQ based Hand-Eye calibration. Furthermore, the verification part of the extrinsics can be improved by calculating the correlation in between the features of individual lidars in a common map reference frame. For the verification purpose we assumed the perspective projection from the lidar to camera did not introduce any errors into the system. This can be improved by extracting semantic features directly from the point cloud by training a supervised model on labeled point cloud data.

	\begin{comment}
	\begin{table}[!htbp]
	\normalsize
	\centering
	\resizebox{\columnwidth}{!}{
	\begin{tabular}{l|cccccc} \toprule
	\multirow{2}{*}{\makecell[l]{Class}} & \multicolumn{2}{c}{\makecell[c]{Training}} & \multicolumn{2}{c}{\makecell[c]{Validation}} & \multicolumn{2}{c}{\makecell[c]{Testing}} \\
	\cline{2-7}
	& \textbf{Accuracy (\%)} & \textbf{Mean IoU} & \textbf{Accuracy (\%)} & \textbf{Mean IoU} & \textbf{Accuracy (\%)} & \textbf{Mean IoU}\\
	\midrule
	Drivable Road & 00.00 & 00.00 & 00.00 & 00.00 & 00.00 & 00.00\\
	Side Walk & 00.00 & 00.00 & 00.00 & 00.00 & 00.00 & 00.00\\
	Curbs & 00.00 & 00.00 & 00.00 & 00.00 & 00.00 & 00.00\\
	Vehicles & 00.00 & 00.00 & 00.00 & 00.00 & 00.00 & 00.00\\
	Buildings & 00.00 & 00.00 & 00.00 & 00.00 & 00.00 & 00.00\\
	Vegetation & 00.00 & 00.00 & 00.00 & 00.00 & 00.00 & 00.00\\
	\bottomrule
	\end{tabular}}
	\caption{Modified EfficientNet-B0 baseline network segmentation results.}
	\label{tab:efficientNet-results}
	\vspace{-6mm}
	\end{table}

	\begin{table}[!h]
	\vspace{3mm}
	\centering
	\begin{tabular}{l|cc}  \toprule
	\multicolumn{3}{c}{\textbf{Absolute Pose Error (APE) -- Translation
	[\si{\metre}] / Rotation [\si{\degree}]}} \\
	\midrule
	\multicolumn{3}{c}{\textit{Before Calibration}} \\ \midrule
	\textbf{Calibration Offset} & \textbf{Kabash} &
	\textbf{DQ + ScLERP}\\
	\midrule
	\multirow{2}{*}{\makecell[l]{0.00 0.00 0.00 \\ 0.00 0.00 0.00 1.00}} &
	\multirow{2}{*}{\makecell[c]{0.00 / 0.00}} & \multirow{2}{*}{\makecell[c]{\textbf{0.00} / \textbf{0.00}}}\\
	& & \\
	\midrule
	\multicolumn{3}{c}{\textit{After Calibration}} \\ \midrule
	\textbf{Calibration Offset} & \textbf{Kabash}& \textbf{DQ + ScLERP}\\
	\midrule
	\multirow{2}{*}{\makecell[l]{0.00 0.00 0.00 \\ 0.00 0.00 0.00 1.00}} &
	\multirow{2}{*}{\makecell[c]{0.00 / 0.00}} & \multirow{2}{*}{\makecell[c]{\textbf{0.00} / \textbf{0.00}}}\\
	& & \\
	\bottomrule
	\end{tabular}
	\caption{}
	\label{tab:rpe}
	\vspace{-6mm}
	\end{table}
	\end{comment}

	\begin{comment}
	We have presented a novel factor graph formulation for state estimation that
	tightly fuses camera, lidar, and IMU measurements. This fusion enables for
	graceful handling of degenerate modes -- blending between lidar-only feature
	tracking and visual tracking (with lidar depth), depending on the constraints
	which each modality can provide in a particular environment. We have
	demonstrated comparable performance to state-of-the-art lidar odometry systems
	in typical conditions and better performance in extreme conditions, such as
	aggressive motions or abrupt light changes.
	Our approach also presents a novel method of jointly optimizing lidar and visual
	features in the same factor graph. This allows for robust estimation in
	difficult environments such as long corridors, open fields, and dark
	environments.
	\end{comment}

	\section{Acknowledgements}
	This research has been conducted as part of development of autonomous transport solutions at Scania, Sweden. It was jointly funded by Swedish Foundation for Strategic Research (SSF) and Scania. The research was also affiliated with WASP.

	\bibliographystyle{./IEEEtran}


\section{Introduction}
Virtual reality (VR) has gained mainstream acceptance with the mass commercialization of VR head-mounted displays (HMDs). Gaming is one of the main uses of VR due to the immersive experience it provides to players. Despite its positive affordances, the adoption of VR games still faces some challenges. In the saturated market of games, it is important to grab the players' attention and retain it from the initial times they try and play the game(s) \cite{b1}. This is where VR does not do so well because playing VR games often causes discomforts from simulator sickness (SS), which can happen within the first interactions \cite{b2, b3}. SS can lead to an unpleasant experience and a negative perception of VR~\cite{b45}.

\begin{figure}[tpb]
  \centerline{\includegraphics[width=\columnwidth]{Figure/conceptual.png}}
  \caption{(a) The conceptual design of \textit{VRCockpit}. The user is surrounded by four planes, which show 2D copies of the view behind them. The users see the 2D views instead of the full 3D renderings behind them. (b and c) Two examples of applying \textit{VRCockpit} in actual games: a car racing game and a first-person shooter game.}
  \label{fig:vrcockpit}
\end{figure}

To help VR users reduce their SS symptoms, various techniques have been proposed and studied \cite{b4, b5, b6, b7, b8, b9, b10, b11, b12, b13}. Mitigation techniques have been shown to be effective in many non-gaming scenarios, but in games they often show inconsistent or even negative results (e.g., \cite{b15, b16}). Games are complex and visually rich environments and players are required to do frequent (and often rapid) changes of viewpoints (e.g., racing and first-person shooter games \cite{b11,b14,b5,b17,b16,b47, b49}). It is challenging to design techniques that are effective in reducing SS and, at the same time, maintaining a suitable level of immersion and gameplay as good as without such techniques. For example, Teleport has been reported to be efficient in SS reduction, but it makes navigation discrete, which is not suitable for action-based games where continuity of scenes is crucial \cite{b14,b18}. Similarly, using rest frames requires adding additional fixed visual elements to the VE for players to look at continuously. This approach could be limiting, as the player should avoid looking away from these visual elements but should have their visual focus on them frequently. Other common techniques are based on blurring parts of the display \cite{b5, b15, b16} or reducing its field-of-view \cite{b8, b16} and, as such, would likely block away certain parts of the VE, making them inaccessible or invisible to players. Other researchers have explored providing first- vs. third-perspective views \cite{b48,b50,b51}. Some results indicate that third-person VR is less likely to make people have SS compared with first-person, but the former is not perceived as immersive. In addition, some games, like first-person shooters, are more enjoyable and natural when played in first-person view.

This research presents a new visual mitigation technique, \textit{VRCockpit}, for VR games that, as our results show, allows players to have the same level of immersion and performance. VRCockpit surrounds the players with four 2D views, one for each cardinal direction, that show 2D copies of the areas of the 3D environment behind them (see Fig.~\ref{fig:vrcockpit}). It does not impose a non-natural way of navigation, add any additional fixed frames, or block parts of the VE from players' view. We ran a user experiment with a car racing game and a second follow-up experiment with a first-person shooter game. In the two experiments, we measured players' performance, immersion, and usability against their SS levels. Our results show that VRCockpit is a simple yet viable, cost-efficient, and effective technique for SS mitigation in VR racing types of games.

In this paper, we make the following two contributions: (1) VRCockpit, a new SS mitigation technique for VR games; (2) two user studies with two different game contexts and their analysis of VRCockpit's effect on immersion, gameplay, and performance. Our findings suggest that VRCockpit can be applied to different types of games, can maintain the same level of immersion (compared to not using it), and can lead to the same level of gameplay performance.

This paper is organized as follows: we first introduce our theoretical background and related literature on existing techniques, how they influenced our research, and some of their relevant findings. We then introduce VRCockpit and its design rationale, followed by two studies. We then present a discussion about the results and possible directions for future work. The paper ends with a conclusion.

\section{Related Work}
\subsection{Simulator Sickness in VR Games}
Simulator sickness (SS) refers to side effects that can occur during or after experiencing VR \cite{b3, b46}. It is common but has not been fully understood so far \cite{b19}. Two of the most widely accepted theories that can potentially explain SS are: the sensory conflict \cite{b20, b21} and the postural instability \cite{b22, b23}. The former attributes SS to a conflict between sensory systems. Specifically, the mismatch between visual inputs and people's vestibular system perception is the main inducement of SS. The postural instability theory posits that a person's inadaptability to the long-term postural instability leads to SS. In addition, the rest frame hypothesis \cite{b24, b25} assumes that some fixed ambient objects can be used as reference points in the human brain for viewing the world. Accordingly, researchers have attempted to use fixed gazing points or other visual objects to reduce SS \cite{b7, b12}.

\subsection{Mitigation Techniques in Games}
Since SS is always a negative factor hindering the adoption of VR games by users, mitigation techniques have been proposed. Locomotion is a very common and necessary activity for moving in a 3D VE and is considered one of the most significant interaction components of VR experiences \cite{b26}. However, locomotion is also related to the severity of SS because it can elicit either sensory conflict or postural instability \cite{b27, b28}. Accordingly, Teleport, one of the most well-known SS mitigation techniques for locomotion \cite{b4}, allows players to jump to discrete destinations without experiencing continuous transitional movement. As such, the level of sensory conflict is minimized and the level of SS is alleviated. Even if Teleport has shown to be effective and has been applied in some commercial games (i.e., Robo Recall\footnote{https://www.epicgames.com/roborecall/en-US/home}), prior research also indicates that it is detrimental to completing VR tasks due to disorientation \cite{b30, b31, b9}. Furthermore, the technique does not allow continuous movement, and therefore may not be quite appropriate in other game contexts such as car racing, sports, or even first-person shooter games \cite{b14}.

There are other types of visual techniques based on the rest frame hypothesis, which tend to add information to VE. \textit{Gaze Point} \cite{b12} is one example, which offers users a central fixed rest frame. As such, users can focus on the reference point to help mitigate SS. Similarly, Clarke et al. \cite{b7} reported that a target reticule in the center of the screen alleviated dizziness and some SS symptoms. Other types of rest frame techniques have been tested in previous studies to varying degrees of success. Cao et al. \cite{b6} introduced a black see-through mental net as the stationary reference. Their results showed that the net can help reduce SS levels, especially discomfort. Besides, a virtual nose embedded in the middle bottom part of the view has been tested in \cite{b32, b33, b34}, and has been shown to have potential in mitigating SS because it provides a consistent sense of stability.

Other visual techniques aim to reduce SS by lessening the visual information that users perceive. For instance, viewpoint snapping \cite{b17} and rotation blurring \cite{b5} intentionally reduce visual information the user can receive during rotational movement in VR environments. As such, the illusion of self-motion is also reduced. Likewise, field of view (FoV) reduction has been studied and has led to positive results \cite{b8, b35, b13, b10}. By limiting the display's FoV, the feeling of self-motion may be reduced, which, according to the sensory conflict theory, can assist prevent sensory conflicts and therefore mitigate SS \cite{b43}. Also, dynamic FoV modification \cite{b13} has been proposed to improve issues of lower presence, enjoyment, and user performance. However, such techniques lead to information loss and hence to lower performance in games \cite{b16}.

\begin{figure*}[htbp]
    \centerline{\includegraphics[width=\linewidth]{Figure/GameEnv.png}}
    \caption{(a) A bird's view of the whole race track in the game. The zoomed-in screenshot shows a coin to be collected and barriers to be avoided. (b) An example of the first perspective of \textit{VRCockpit} in the game. (c) A third perspective of \textit{VRCockpit} in the game. The car's body was made to be invisible.}
    \label{fig:GameEnv}
\end{figure*}

\section{VRCockpit}
VRCockpit (CP for short) is implemented using render texture in Unity3D, a popular game engine that supports VR devices. CP is composed of four 2D frames, one for each of the cardinal directions around the user (i.e., front, back, left, and right; see Fig.~\ref{fig:vrcockpit}). Each frame displays a copy of the area of the 3D VE behind the frame. They are fixed and cannot be rotated and zoomed in. The borders in the figures in this paper are used to highlight the positions of frames, but they were not shown in the game to avoid separating the 2D views from the 3D VE or adding extra visual elements to it. In each direction, the 2D frame occupies nearly 30\% of users' FoV and all of them are of the same size. This was found to be the ideal size by participants in pilot studies, as users can still see the 3D VE around them between the frames and, at the same, allow the frames to display the main content of interest within them. Fig.~\ref{fig:vrcockpit}b and c show how CP looks in an actual gaming scenario. The four frames are stationary in their positions relative to the user, but their content is updated based on the changes in the areas behind them. As such, it would not change the navigation process. Also, unlike existing techniques that blur or block out part of the screen, CP would not lead to information loss in games but can help reduce SS by transferring 3D content into 2D views.

We also explored other design possibilities, like making the frames move in synchronicity with the user's head. However, feedback from our pilot studies' participants showed that this was unsuitable because they would lose the sense of `3Dness' of the environment and could be somewhat disorientating when experiencing frequent changes. Their suggestion was to keep the frames fixed because they could make rapid eye movements (or head movements) to see through the gaps between the frames and the upper and lower areas not covered by the frames, which was helpful in increasing their immersion in the environment. In addition, fixing the frames' location in relation to the user allowed consistency of views, which has been found to be useful in reducing SS symptoms \cite{b6, b8, b11}.

To evaluate the effects of VRCockpit in reducing SS and players' performance in a fast-paced gaming context, we conducted an experiment with a car racing game as the testbed environment. A second follow-up experiment with another game (a first-person shooter) shows that the technique can be applied to other types of games with similar results (described in Section~\ref{section:follow-up.experiment}).

\section{Experiment}
We compared VRCockpit (CP) with the normal version (Normal) of the game with no technique applied to it. The experiment followed a within-subjects design. All participants played both versions of the car racing game.

\subsection{Game Environment}
This was an in-house developed VR first perspective car racing game, consisting of turns and steep slopes for players to traverse through (see Fig.~\ref{fig:GameEnv}a). In addition, there were barriers along the way that players needed to avoid and coins to collect, as shown in Fig.~\ref{fig:GameEnv}a.
Any uncollected coin would disappear and after the players completed a lap, all coins would be refreshed. Players were required to finish two laps within the same game condition. The car speed was set in the range of 0-70 km/hour (or its equivalent in Unity units). Players were not allowed to stop the car halfway unless they crashed onto barriers. There were no ambiguous paths or shortcuts. Consequently, the players would follow roughly the same path. Fig.~\ref{fig:GameEnv}b demonstrates how VRCockpit was implemented in the game. Because the virtual car's body may be an additional rest frame for participants, which would be an undesired confounding factor, the virtual car's body was made invisible (see Fig.~\ref{fig:GameEnv}c).

\subsection{Evaluation Metrics}
After each condition (CP and Normal), two questionnaires were given to the participants. First, a Simulator Sickness Questionnaire (SSQ) \cite{b40} was adapted to measure the sickness level in terms of different symptoms. Each symptom was rated using a 5-point Likert Scale, with 0 as the least severe and 4 as the most severe. The ratings for each symptom were further grouped into four sub-scores: Nausea, Oculomotor, Disorientation, and Total. Second, an Immersive Experience Questionnaire (IEQ) \cite{b41} with the same rating scale as SSQ was used to elicit participants' perceived immersion in the VR game.

In addition, user performance was determined based on objective measurements. During the experiments, we recorded the total time used (Time), the number of car crashes (Crashes), and coins collected (Coins) to reflect user performance.

\subsection{Participants and Apparatus}
We recruited 18 participants (3 females, 15 males) from a local university with an average age of 19.63 ($s.d.=1.50$), ranging from 18 to 22. All volunteers had a normal or corrected-to-normal vision. None of them declared any history of colour blindness or health issues. Thirteen participants (72.22\%) had some experience with VR systems. Two (11.11\%) reported that they felt sick when playing regular car racing games on a PC.

An HTC VIVE Cosmos VR HMD was used to render the virtual environment in this experiment. It was connected to a desktop with 16GB RAM, an Intel Core i7-9700k CPU @ 3.60GHz, a GeForce GTX 2080Ti dedicated GPU. Oculus handheld controllers were used to control the movement of the virtual car

\subsection{Experimental Procedure}
To eliminate the post-exposure effects of VR sickness, we conducted a two-session experiment where participants only played one version of the game in each session, resting for at least one day. The order of the game version was counterbalanced with a Latin square design to minimize the carry-over effects. At the beginning of the first session, we asked participants to complete a pre-experiment questionnaire to collect demographics and previous gaming experience information, followed by an introduction to the rules and controls of the game. Participants then played the first version of the game, and after this, they were asked to complete the two aforementioned questionnaires based on their experience in the game. The second session was conducted at least 24 hours later to give participants enough rest. Participants played the second version of the game and completed the same questionnaires.

\subsection{Hypotheses}
This experiment aims to verify the following hypotheses:
\begin{itemize}
    \item \textbf{H1}. Participants will have lower SS in CP than Normal.
    \item \textbf{H2}. Participants' perceived immersion will not differ significantly between CP and Normal.
    \item \textbf{H3}. Participants' game performance will not differ significantly between CP and Normal.
\end{itemize}

\section{Results}
We report results of inferential statistics with the support of data visualizations. A Shapiro-Wilk test was first conducted to check the normality of the data. For normally distributed data ($p>.05$), we conducted dependent t-tests. We used Wilcoxon signed-rank tests for the data that was not normally distributed. Spearman's correlation test was used to examine relationships.

\subsection{SSQ}
Fig.~\ref{fig:SSQ} shows a summary of the results of the SSQ data. All the data were not distributed normally in the four sub-scales by the Shapiro-Wilk test. After conducting Wilcoxon tests, we found Nausea was significantly lower in CP ($Mdn=1.5$) than in Normal ($Mdn=6.5, Z=-2.103, p=.035$). Disorientation was also significantly lower in CP ($Mdn=0.5$) than in Normal ($Mdn=4.0, Z=-2.196, p=.028$). However, there was no significant difference in Oculomotor ($Z =-.449, p=.653$) and Total ($Z=-1.658, p=.097$).

\begin{figure}[tbp]
  \centerline{\includegraphics[width=\columnwidth]{Figure/SSQ.png}}
  \caption{Mean results of the SSQ data in the experiment according to the four sub-scales. Bars represent 95\% confidence intervals.}
  \label{fig:SSQ}
\end{figure}

\subsection{Subjective Immersion}
The descriptive analysis of participants' perceived immersion is summarized in Table.~\ref{tab:upandimm}. Wilcoxon test did not reveal a significant difference between two conditions ($Z=-.711, p=.477$). Spearman's correlation test did not reveal any significant relationships between immersion level and reported SS in two conditions ($p>0.5$ for all cases).

\subsection{User Performance}
Table.~\ref{tab:upandimm} summarizes the results for Coins, Time, and Crashes. According to the results of normality test, only Coins was normally distributed. Dependent t-tests did not reveal significant difference in Coins ($t(17)=-1.042, p=.312$). In addition, Wilcoxon tests showed no significant difference in Crashes ($Z=-1.481, p=.138$) and Time ($Z=-.022, p=.983$).

We performed Spearman's correlation test to investigate the relationships between participants' user performance and reported SS, and between user performance and perceived immersion. No significant correlations were identified in the CP condition. While in the Normal condition, there were significant negative relationships between Crashes and Nausea ($r_s=-.659, p=.003$), Crashes and Disorientation ($r_s=-.658, p=.003$), Crashes and Total ($r_s=-.629, p=.005$).

\begin{table}[htbp]
  \caption{A summary of the means, standard deviations, and medians of participants' perceived immersion and user performance (represent using $M, s.d., Mdn$, respectively).}
  \begin{center}
  \begin{tabular}{|c|c|c|c|c|}
  \hline
  \textbf{Measure} & \textbf{Condition} & \textbf{\textit{M}} & \textbf{\textit{s.d.}} & \textbf{\textit{Mdn}}\\ \hline
  \multirow{2}{*}{Immersion} & CP & 25.22 & 7.49 & 24.50\\ \cline{2-5}
   & Normal & 27.39 & 8.70 & 27.50\\ \hline
  \multicolumn{5}{|c|}{  } \\ \hline
  \multirow{2}{*}{Time} & CP & 543.00 & 110.70 & 531.26\\ \cline{2-5}
   & Normal & 556.36 & 153.47 & 502.30\\ \hline
  \multirow{2}{*}{Crashes} & CP & 14.28 & 12.55 & 11.00\\ \cline{2-5}
   & Normal & 9.22 & 7.60 & 7.00 \\ \hline
  \multirow{2}{*}{Coins} & CP & 45.67 & 13.82 & 47.00\\ \cline{2-5}
   & Normal & 48.61 & 15.75 & 50.00\\ \hline
  \end{tabular}
  \end{center}
  \label{tab:upandimm}
\end{table}

\section{Discussion}
The results show that VRCockpit led to significantly lower symptoms in Nausea and Disorientation for all participants than without it, indicating that the technique helped typical players feel lower SS in the car racing game. This finding confirms our hypothesis \textbf{H1}. Moreover, most participants felt either better or indifferent to the technique and most of the participants who felt better had considerable improvements. These participants were the ones who scored the highest when not using any form of mitigation technique (i.e., in the Normal condition).

There was no significant difference in participants' perceived immersion. In addition, there was no significant difference in user performance in terms of total time used, the number of car crashes, and coins collected in our racing game. These results indicate that VRCockpit would be less likely to affect immersion and users' gameplay negatively. As such, Hypotheses \textbf{H2} and \textbf{H3} are both confirmed. Based on participants' feedback, VRCockpit worked well in the car racing game because it served as car windows and was closer to real driving situations. A natural combination between the technique and the virtual environment and scenario appears to be helpful for how users perceive the technique.

Interestingly, we found that the number of crashes and reported SS symptoms were negatively correlated in the car racing game where no SS mitigation technique was applied. During gameplay, participants drove at similar speeds and were not required to make any stops, except when crashing onto the barriers. When participants drove smoothly, the surroundings changed rapidly, which led to the accumulation of discomfort. While such accumulation was interrupted when a crash happened. Participants had a short rest (usually a few seconds) and this helped alleviate some symptoms. However, these negative correlations were not found when VRCockpit was used in the game. The main reason was that the technique had reduced the SS symptoms, so the effects of crashes were not significant.

In summary, the results from our experiment suggest that VRCockpit in a car racing game not only had an effect on mitigating SS but did not affect user performance and immersion---this is a positive indication for VRCockpit's use in gaming environments that require continuous movement. The car racing game can be a typical game involving virtual navigation. In other words, players are mainly focusing on the motion control in the VEs during the gameplay. To validate the usefulness of VRCockpit in a more complex game scenario, we ran a follow-up experiment using a First-Person Shooting (FPS) game. An FPS game is generally more dynamic compared to a car racing game, because it involves weapon-based fighting in addition to virtual navigation. Thus, we used an FPS game as an additional testbed and proof-of-concept of VRCockpit.

\begin{figure}[tbp]
  \centerline{\includegraphics[width=\columnwidth]{Figure/FPSGameEnv.png}}
  \caption{Game elements in the FPS game. (a) A first perspective view of VRCockpit in the game. The frames with yellow lines show the left and right views–but were not actually rendered in the application, like in the car racing game. There was another 2D frame behind the user. (b) A third perspective of VRCockpit in the game. (c) A picture of a robot enemy that the players need to seek and destroy.}
  \label{fig:FPSGameEnv}
\end{figure}

\section{Follow-Up Experiment} \label{section:follow-up.experiment}
This follow-up experiment sought to further investigate whether VRCockpit can help players reduce the Simulator Sickness symptoms without affecting their performance and immersion in a more complex and dynamic VR game scenario---an FPS game.

\begin{table*}
  \caption{A summary of descriptive data and inferential results of participants' perceived sickness, perceived immersion, and user performance ($M, s.d., Mdn$ stand for mean, standard deviation, and median, respectively).}
  \begin{center}
  \begin{tabular}{|c|c|c|c|c|c|}
  \hline
  \textbf{Measure} & \textbf{Condition} & \textbf{\textit{M}} & \textbf{\textit{s.d.}} & \textbf{\textit{Mdn}} & \textbf{Wilcoxon tests}\\ \hline
    \multirow{2}{*}{Nausea} & CP & 0.40 & 0.97 & 0.00 & \multirow{2}{*}{$Z=-2.201, p=.028$}\\ \cline{2-5}
   & Normal & 5.00 & 7.54 & 1.50 & \\ \hline
    \multirow{2}{*}{Oculomotor} & CP & 0.70 & 0.82 & 0.50 & \multirow{2}{*}{$Z=-.108, p=.914$}\\ \cline{2-5}
   & Normal & 1.10 & 1.85 & 0.50 & \\ \hline
    \multirow{2}{*}{Disorientation} & CP & 0.00 & 0.00 & 0.00 & \multirow{2}{*}{$Z=-2.226, p=.026$}\\ \cline{2-5}
   & Normal & 2.40 & 3.86 & 1.00 & \\ \hline
   \multirow{2}{*}{Total} & CP & 1.10 & 1.29 & 1.00 & \multirow{2}{*}{$Z=-1.992, p=.046$}\\ \cline{2-5}
   & Normal & 8.50 & 12.93 & 3.00 & \\ \hline
  \multicolumn{6}{|c|}{  } \\ \hline
    \multirow{2}{*}{Immersion} & CP & 23.20 & 11.12 & 23.50 & \multirow{2}{*}{$Z=-1.123, p=.262$}\\ \cline{2-5}
   & Normal & 20.90 & 8.02 & 20.50 & \\ \hline
  \multicolumn{6}{|c|}{  } \\ \hline
    \multirow{2}{*}{Time} & CP & 322.57 & 77.03 & 311.44 & \multirow{2}{*}{$Z=-1.070, p=.285$}\\ \cline{2-5}
   & Normal & 346.85 & 113.13 & 309.32 & \\ \hline
    \multirow{2}{*}{Distance} & CP & 1861.15 & 456.30 & 1884.28 & \multirow{2}{*}{$Z=-.764, p=.445$}\\ \cline{2-5}
   & Normal & 1956.96 & 500.71 & 1775.75 & \\ \hline
    \multirow{2}{*}{Shots \newline Received} & CP & 197.40 & 99.84 & 153.50 & \multirow{2}{*}{$Z=-.051, p=.959$}\\ \cline{2-5}
   & Normal & 201.90 & 122.38 & 162.50 & \\ \hline
  \end{tabular}
  \end{center}
  \label{tab:FPSresults}
\end{table*}

\subsection{Experimental Setup}
The FPS game was built based on the FPS Microgame Template\footnote{https://learn.unity.com/project/fps-template} from Unity Learn. Players needed to navigate quickly and defeat all the enemy robots in the VE while avoiding getting hit by them to finish the game. Fig.~\ref{fig:FPSGameEnv}a and b show how VRCockpit was implemented in the FPS game. The game environment was made up of towering grey walls with no additional visual clues to help with path memory, while there were no confusing or hidden pathways. Furthermore, the maximum ammo of the gun was 5 and it would reload once depleted. This mechanic was designed to motivate players to move frequently and balance shooting and hiding. The adversaries were same-sized robots (see Fig.~\ref{fig:FPSGameEnv}c), which would follow and attack the players once the players were within their range of detection. Each robot could withstand up to 14 bullet shots from the players.

The experimental design and procedure were similar as in the first experiment. Specifically, participants needed to complete a two-session experiment with a normal version of the game (Normal) and a version that incorporated VRCockpit (CP). We used the same SSQ and IEQ for measuring the sickness and immersion level. For participants' performance, we recorded the total time that participants spent in the game (Time), total distance travelled (Distance), and total hits received from robot enemies (Shots Received). We used the same analysis methods and approach as the first experiment.

We recruited 10 participants (6 females, 4 males) in the follow-up experiment. They were aged between 19 to 28 ($M=21.20, s.d.=2.48$). All participants had a normal or normal-to-corrected vision. None of them declared any history of colour blindness or health issues. Seven participants had some experience with VR systems. Six reported that they felt sick when playing regular FPS games on a PC. An Oculus Rift S VR HMD was used to display the virtual environment. Participants used mouse and keyboard to control their movements and shooting behaviors as they lead to better performance in VR FPS games \cite{b42}.

\subsection{Results and Discussion}
Table.~\ref{tab:FPSresults} summarizes all the results of this second experiment. As shown in the table, Nausea, Disorientation, and Total were significantly lower in CP than in Normal. No other significant differences were found. These results were in line with our results in the first experiment. Given that we had a lower number of participants in the follow-up experiment, we could only provisionally conclude that VRCockpit can also help to mitigate SS symptoms in a VR FPS game without lowering players' immersion and gameplay. However, the follow-up experiment and the results we observed can still be regarded as a proof-of-concept providing further evidence that the technique has the potential to be applied to various types of VR games with different difficulties and requiring different types of navigation strategies.

Out of our expectations, although the FPS game is more complex and dynamic compared to the car racing game, participants generally gave lower ratings of the SS symptoms in the FPS game (which can be reflected in the results shown in Fig.~\ref{fig:SSQ} and Table.~\ref{tab:FPSresults}). One possible reason is that, unlike the car racing game, players needed to show more awareness of all sides around them and react quickly if an enemy showed up. In addition, players may be able to stop, move at different speeds in the FPS game, and could find some chances to hide somewhere and wait for reloading, which provided users with some breaks.

Few participants felt it was somewhat unusual and abrupt to have a set of 2D frames surrounding them. One way that the technique could be better integrated into the FPS is to tell participants that they are wearing a virtual helmet with see-through displays. This may help improve their perception of the technique. As mentioned before, the combination between the technique and the virtual environment and scenario should be natural to improve the game experience.

\section{Limitations and Future Work}
Because of different design elements between experiments (e.g., time, controller), we were unable to conduct a between-subjects analysis to explore if game types could influence the technique's effectiveness. These choices were by design and guided by prior research. For example, we opted for the typical dual-hand controller for the VR racing game to be in line with other studies (e.g., \cite{b16,b44}). On the other hand, we used a mouse and keyboard in the FPS game, because \cite{b11} observed advantages in using the mouse and keyboard over a dual hand-based controller in a VR FPS game. Also, it is often the case that time and distance are within a similar range for all participants in car racing games, whereas this may not be the case for FPS games, which could afford a broader range in how each participant could behave during gameplay. Given that VRCockpit performed better in the car racing game, it is plausible that our technique is more suitable when applied in this type of continuous fast-paced context. Further explorations like comparing different speeds, acceleration, and size of the 2D frames will be helpful in assessing its effectiveness and application in other scenarios. In this research, we wanted to validate if VRCockpit is effective in mitigating SS while maintaining suitable performance and immersion levels, which we have achieved to a large extent. Optimization issues are part of our future plans.

We can also conduct a comparative study to contrast our technique with other existing SS mitigation techniques involving a larger number of participants. However, such studies are often non-trivial, as direct comparisons are often limited to the original choice of environment and not easily transferable to other environments because it is generally challenging to replicate the technique to other scenarios outside of the original testbed setting. However, as demonstrated in our results, the trade-offs of the mitigation technique can be game-specific. Finding the best matches between technique(s) and game type(s) represents potentially a rich avenue for future research.

\section{Conclusion}
In this paper, we explored ways to mitigate simulator sickness (SS) in virtual reality (VR) games. We proposed a new visual mitigation technique for head-mounted displays (HMDs), VRCockpit. It surrounds users with four egocentric frames, one for each cardinal direction, that display 2D replicas of the 3D environment behind each frame. We conducted two user studies to test our technique with two different games (a car racing game and a first-person shooter game) and compared it with the baseline condition (with no visual mitigation technique). Our results show that VRCockpit is able to reduce SS for participants in general and still allows them to attain similar levels of immersion and performance in the two games. Our findings show that VRCockpit is a useful SS mitigation technique and does not change the navigation process, add additional visual elements to the gaming environment, or blur/blackout portions of the display.

\section*{Acknowledgment}
The authors thank the participants who joined the study and the reviewers for their insightful comments and helpful suggestions that helped to improve our paper. This work was supported in part
by Xi’an Jiaotong-Liverpool University–Key Special Fund (\#KSF-A-03) and the Future Network Scientific Research Fund (\#FNSRFP-2021-YB-41).