\section{Introduction}
\label{sec:intro}

Mid-air gesture recognition and control has recently attracted increasing research attention in multimedia  applications.
Early works such as Kinect in the Kitchen \cite{panger2012kinect} has explored mid-air gestural control and feedback using a Kinect in cooking scenarios, where common devices have limited displays and touch is less appropriate when cooking.
Recently, many studies \cite{shakeri2017novel,qian2020aladdin} have proposed mid-air interaction methods for driving based on mid-air gesture recognition and control, which can prevent driver distraction caused by conventional physical handling or touch.
However, conventional gesture interaction methods require a physical device as support and focus on solving physical controls and interactions, such as simple music control, device selection, \textit{etc.}
Hence, in recent years deep learning based gesture recognition methods \cite{ur2021dynamic, ahmed2022radar} have been studied to enhance mid-air gesture control and interaction for many applications.
For instance, \cite{ur2021dynamic} proposed a deep learning architecture based on the combination of a 3D Convolutional Neural Network (3D-CNN) and a Long Short-Term Memory (LSTM) network, which takes advantages of spatial-temporal information from 30-frame video sequences.
Though they achieve significant improvements, these methods remain unsatisfactory due to long time sequences dependencies and high-dimensional feature inputs of models.
In addition, the interpretability of the user gesture interaction process is not addressed by most current methods, which is widely regarded as a critical component in real-world gesture control.

\begin{figure*}
  \centering
  \includegraphics[width=1\textwidth]{Gesture_pipline_50k_2.pdf}
  \caption{Pipeline of mid-air gesture control}
  \label{fig:pipeline}
\end{figure*}

To deal with the mentioned problems, we propose a straightforward solutions to combine gesture recognition with low-dimensional embeddings, which uses low-dimensional embeddings to reduce the dimension of features of multiple frames and visualize the gestures.
Low-dimensional feature embeddings \cite{zebari2020comprehensive, gisbrecht2015parametric, mcinnes2018umap}

 are very popular for real-time tasks, where gesture recognition and interaction are temporally connected, due to multiple types of sensors used for data collection.
These embeddings can be linear mappings based on principal component analysis (PCA) \cite{zebari2020comprehensive} and factor analysis (FA) \cite{gisbrecht2015parametric}, and even non-linear mappings based on t-distributed stochastic neighbor embedding (t-SNE) \cite{gisbrecht2015parametric}, uniform manifold approximation and projection (UMAP) \cite{mcinnes2018umap}, and autoencoders -- all creating effective explorations of low-dimensional embeddings.
However, the linear nature of PCA
limits the ability to project the features to a lower dimension without losing information.
t-SNE and UMAP can perform non-linear feature dimensionality reduction, but they cannot reduce the additional costs associated with subsequent classification.
In this work, we design a fully-connected autoencoder to reduce the dimension of the detected hand pose features and to learn a better low-dimensional pose space for interaction.
The autoencoder takes hand pose features as inputs, reduces their dimensionality via an encoder to get low-dimensional latent features, and then reconstructs them using a decoder.
This is an unsupervised process, and by training using a dataset containing numerous gestures, we can obtain generalization ability and expressiveness.
Our proposed autoencoder reduces the gestures to an interactive 2D space, which facilitates visualization of the embedded pose  space mapped with the corresponding music space to provide a more intuitive hand pose based exploration.
The temporal information from video sequences input in the gesture or action recognition model is exploited to further improve the classification accuracy in many studies \cite{herath2017going,ur2021dynamic}.
However, the dependence of longer frame sequences makes them difficult to be truly interactive in real time. And for gesture recognition and control applied to realistic scenarios, interaction response delays tend to annoy users and reduce perceived usability \cite{anderson2011user, ng2012designing}.
In this paper, we propose a simple PointNet-based classification network to recognize the predefined discrete gestures and continuous hand poses by fewer frames (2 frames) in low-dimensional inputs (2 dimensions) from an autoencoder.
Discrete gestures mean that feedback is obtained after the full gesture has been triggered, while continuous gestures obtain the real-time feedback while the gesture is in progress.
Compared to the methods \cite{kim2010wearable,vogiatzidakis2021mid,liao2021smart} that focus only on discrete gesture interaction, our approach handles both discrete and continuous gesture-based interaction scenarios.
Finally, we defined corresponding functions for the different recognized gestures, which include discrete gesture control for music start/stop and continuous hand pose control for real-time musical space exploration, \textit{etc}.
Different from other gestural interaction strategies \cite{lv2014multimodal} and video process methods \cite{neimark2021video}, our proposed pipeline overcomes to some extent the disadvantages of high-dimensional feature input and long sequence dependence, and  implements a continuous hand pose to explore the music space.
Specifically, we map the predicted continuous hand pose to a music space with different properties, where the two-dimensional hand pose space is embedded from the autoencoder encoder.
The low frame dependency of the gesture recognition stage gives our model the advantage of low latency.
Visible user interaction based on the autoencoder encoding gives the user more freedom of choice and exploration, which is not exploited in the literature.

\section{Methodology}
\label{sec:methodology}
Fig. \ref{fig:pipeline} presents a detailed structure of our proposed pipeline for interaction with a smart music speaker. It includes generating a low-dimensional embedding (Section \ref{ssec:dimension reduction}), gesture classification (Section \ref{ssec:Classification}), and Interaction (Section \ref{ssec:Interaction}).

\subsection{Low-dimensional Embedding}
\label{ssec:dimension reduction}
In this work, the main purpose of the low-dimensional embedding is that  the space distribution of low-dimensional hand pose features can be visualized.
In this way, gesture interactions are more understandable and more controllable when interacting with a music space.
Specifically, MediaPipe Hands \cite{lugaresi2019mediapipe,zhang2020mediapipe}, a real-time hand landmarks detection model, is used to extract 3-dimensional (3D) coordinate information of 21 hand landmarks, which saves a significant model space compared to directly using pixel-level image (image size is $480 \times 620$).
Then, an autoencoder based on fully connected layers \cite{rusu2022low} is designed to reduce the dimensionality of the 3D coordinates of the 21 hand landmarks.
The encoder and decoder in autoencoder contain four fully-connected layers respectively, and the layers use LeakyReLU \cite{xu2015empirical} for non-linear activation.
We set the neuron numbers of 4 fully-connected layers in the encoder are 128, 96, 64 and 2, respectively and the reverse in the decoder.

\subsection{Gesture Classification}
\label{ssec:Classification}  %
We employ a highly efficient and effective PointNet \cite{qi2017pointnet} that directly consumes point information, based on multiple linear layers,  to classify the predefined gestures.
PointNet-based classification network reduces the model size and training time by using low-dimensional inputs from the autoencoder, as well as assisting the pose space to obtain better distinguishability in the autoencoder by their jointly learning.
Furthermore, inspired by the popular sequence-based methods \cite{ur2021dynamic, neimark2021video}, we also explore and take advantage of the sequence information from the gesture video.
Specifically, the frame-based sequence features are encoded by the autoencoder as new inputs of the classification network.
Then, we get the corresponding predicted gesture categories from the optimized classification network.
In this work, we explore single-, 2- and 8-frame sequence inputs for the PointNet-based classification, and we chose a 2-frame sequence as our final inputs, based on trading-off classification performance with minimizing time delay.

Our goal is to jointly learn the parameters of the proposed fully-connected layer based autoencoder and PointNet-based classification by minimizing a loss
function over the training set, which employs mean square loss \cite{sara2019image} and cross-entropy loss \cite{zhang2018generalized}, respectively. We employ Adam \cite{kingma2014adam} with 0.001 weight decay and 0.001 learning rate as the optimizer of our joint model.

\subsection{Interaction}
\label{ssec:Interaction}
In this work, we anticipate that users will interact with the smart music speaker through a combination of discrete gestures and continuous hand poses.
We propose a novel, user-friendly strategy for control interaction and exploration in a visible music space, where a discrete gesture (Pinch) is used for activation and the continuous hand pose (Continuous arm open/close) is used for continuous exploration.
And other gestures will be used for other interactions in future works.
Our music data is provided by our industrial partner, Moodagent, including about 55,000 music tracks. Each track is represented by 34 features, including predicted subjective scores for 6 emotion types, 14 genre types, and 14 style types. These features are derived from the track's audio signal using a convolutional neural network to predict human subjective classifications. The music features are embedded by UMAP down to a 2-dimensional music space for human interaction. In this work, we focus on the exploration of a music space with different emotions, including sadness, joy, fear, erotic, anger and tenderness, which the user can interact with via continuous hand pose changes.
To connect the pose space to the music space, we use a physical mapping, \textit{i.e.}, first scaling the music space and the pose space to the same range and computing the cluster centres for each type of music and then computing the distance to the coordinates of the real-time pose in the two-dimensional pose space. The music category with the closest distance to the coordinates of the real-time pose will be highlighted.
We mark different emotions in different colours in the music space. The colours range from light to dark, indicating light to heavy emotional expressions of music.
In this way, users have more freedom and controllability to explore the music space with an entire music database by the continuous mid-air hand pose movement, as shown in Fig. \ref{fig:pipeline}.
In addition, we explore the possibility of using alternative representations, such as quaternion \cite{rieger2004systematic, saxena2009learning}, to make pose spaces more stable

\section{EXPERIMENTS}
\label{sec:experiments}
\subsection{Dataset}
\label{ssec:dataset}
We investigate and design interactive control gestures that conform to human habits for the smart music speaker, including 'continuous arm open' (the arm and hand away from body), 'continuous arm close' (the arm and hand close to body), index finger drawing 'circle clockwise', drawing 'circle counterclockwise', 'pinch' and 'double-pinch'.
In fact, since these gestures are in pairs and opposite directions, we collect 6 gestures in total.
Specifically, using the Intel$^\circledR$ RealSense$^\text{TM}$ LiDAR Camera L515 depth camera (frame rate is 30 fps), we collected 25 clip videos for each gesture for 7 volunteers.
Each video duration varies from 1 to 3 seconds.
To avoid the influence of the background, we choose a white wall about one meter away from the camera as the background and the collected gestures are 40-50 cm away from the camera.
After that, we extract over 60, 000 frames containing gestures from the collected clip videos.
In Table \ref{tab:gesture}, we provide the detailed information of our collected gesture dataset.

\begin{table}[t!]
\scriptsize
\begin{center}
\fontsize{8}{10}\selectfont
\renewcommand\tabcolsep{8.0pt}
\caption{Overview information of our collected Gesture frames.} \label{tab:gesture}
\begin{tabular}{cc|cc|c}
\hline
\multicolumn{1}{c|}{No.} & Gesture                 & Train & Test  & Total \\ \hline
\multicolumn{1}{c|}{1}          & Continuous arm open          & 8793  & 2209  & 11002 \\
\multicolumn{1}{c|}{2}          & Continuous arm close         & 9096  & 1711  & 10807 \\
\multicolumn{1}{c|}{3}          & Circle clockwise        & 8647  & 1873  & 10520 \\
\multicolumn{1}{c|}{4}          & Circle counterclockwise & 8517  & 1943  & 10460 \\
\multicolumn{1}{c|}{5}          & Pinch                   & 7588  & 1593  & 9181  \\
\multicolumn{1}{c|}{6}          & Double pinch            & 6742  & 1513  & 8255  \\ \hline
\multicolumn{2}{c|}{Total}                                & 49383 & 10842 & 60225 \\ \hline
\end{tabular}
\end{center}
\vspace{-2.5em}
\end{table}

\subsection{Experimental Results}
In this section, we first present an empirical finding that highlights the effectiveness of a well-selected fully-connected autoencoder in the proposed pipeline, which will get low-dimensional pose spaces for interactions.
Then the effectiveness of well-designed PointNet-based classifier will be proved.
Finally, the visualization of mid-air hand pose interactions for a smart music speaker will be given.

\textbf{The effectiveness of the autoencoder.}
As shown in Fig. \ref{fig:autoencoder}, the 2D outputs of the different gestures from the encoder in the autoencoder are plotted in a 2D space on the display.
Compared with the widely used UMAP (indicated by (a), (a') and (a")), which has significant effects on dimensionality reduction \cite{mcinnes2018umap} and clustering \cite{becht2018evaluation}, the proposed fully-connected layer based autoencoder (indicated by (b), (b') and (b")) can better distinguish the distribution of different gestures with lower model complexity.
This allows users to more clearly and intuitively see the positions of different gestures in the pose space and the relationship between different gestures.
When hand pose points in the pose space are sufficiently dispersed, subtle hand pose changes will be clearly tracked.
Furthermore, we explore the distributional effects of using different frame sequences (2-frame and 8-frame) on the encoding of the proposed fully-connected autoencoder.
By comparing the visualization results with the first column of Fig. \ref{fig:autoencoder} with single-frame inputs, using time-sequence information can significantly improve inter-class gesture clustering and intra-class dispersion.
As low latency is very important for user interaction, we focus on 2 frames in our subsequent study, which can avoid long time latency caused by longer sequence dependencies in other methods \cite{ur2021dynamic}.

\begin{figure}[t]
  \centering
  \includegraphics[width=1\linewidth]{gesture_space3.pdf}
  \caption{Visualization of different embedding methods of mid-air gestures. (a), (a') and (a") use single-, 2- and 8-frame gesture sequence inputs of UMAP \cite{mcinnes2018umap}, respectively; (b), (b') and (b'') use single-, 2- and 8-frame gesture sequence inputs of our fully-connected autoencoder without classification; (c) and (d) are joint autoencoder and PointNet-based classifier on single- and 2-frame gesture sequences, respectively.
}
  \label{fig:autoencoder}
\end{figure}

\begin{figure}[t]
  \centering
  \includegraphics[width=1\linewidth]{music_space_point_50k_3.pdf}
  \caption{Visualization of hand-pose-controlled track selection in the music spaces of 55,000 music tracks. The yellow \ding{73} indicates the target music position point, the green \faCircle \ indicates the hand pose starting point, and hand-pose-selected movement tracks are in blue. Different colours of points in the space represent music with different emotions, and the darker the colour, the higher the emotional value
}
  \label{fig:music_space}
\end{figure}
\begin{table}[t]
\scriptsize
\begin{center}
\fontsize{9}{12}\selectfont
\renewcommand\tabcolsep{2.0pt}
\caption{Comparisons of Classification models } \label{tab:cls}
\label{tab:tab1_class}
\begin{tabular}{c|c|cccc}
\hline
Method                      &   Autoencoder       & Frames & Precision(\%) & mAP(\%) & time (ms) \\ \hline
\multirow{4}{*}{\rotatebox{90}{\makecell[c]{PointNet \\ based}}}     &  $-$ & 1       &  70.6         & 62.1    &  12.6    \\
                              & $\surd$ & 1      &   71.3        & 61.9    &  2.3    \\
                              &  $\surd$ & 2      &   75.2        & 62.5    &  2.4    \\
                              &  $\surd$ & 8      &   77.2        &  64.9   &   3.0   \\ \hline
\end{tabular}
\end{center}
\end{table}
\textbf{Effects of classification.}
Different from previous approaches, \textit{e.g.} \cite{rusu2022low}, where only clustering is used for gesture reduction and visualization, in this paper we utilize a PointNet-based classification network to recognize the mid-air gestures for interactions and to further guide the visualization of the distribution of pose space.
As shown in the third row of Fig. \ref{fig:autoencoder}, by joint learning with the classification network, the proposed autoencoder can better distinguish the space distribution of inter- and intra-class mid-air gestures. Although this increases the training time, there is no additional cost in the inference process of gesture feature dimensionality reduction and clustering.
In addition, in Table \ref{tab:tab1_class}, we provide the detailed classification results for different frame-based gestural sequence features from the autoencoder.
We also provide inference time for the entire mid-air hand pose interaction process of each sequence.
Compared to the 12.6 ms required by the original PonitNet without autoencoder, the recognition and interactions with autoencoder take 2.3 ms, 2.4 ms and 3 ms on 1-, 2- and 8-frame based sequences, respectively.
It demonstrates that the joint learning of the fully-connected autoencoder and PointNet-based classifier can improve the clustering effect of the autoencoder while also keeping the accuracy of the classification.
Finally, by balancing classification effectiveness and low latency requirements, we finally choose 2-frame sequences as inputs.

\begin{figure}[t]
  \centering
  \includegraphics[width=1\linewidth]{quaternion_50k_2.pdf}
  \caption{The figure (a) shows the low-dimensional embedding of 'continuous arm open' when quaternion is not used, and the figure (b) shows the low-dimensional embedding of `continuous arm open' when quaternion is used. In contrast to figure (a), the use of quaternion in figure (b) brings more stability to the model. For greater visibility, the background Music Space using a subset (2,000 music tracks) of the music dataset is shown in black.
}
  \label{fig:quaternion}
\end{figure}

\textbf{Visualization of user interaction and exploration.}
In order to provide a better experience and understanding when using the smart music player, we  display the selected music position and use the dynamic hand pose continuous control process to map to locations in the music space.
As shown in the top row of Fig. \ref{fig:music_space}, when we specify different target music positions in music spaces, they can all be reached by continuous movement and exploration of mid-air hand pose.
For the Fig. \ref{fig:music_space} (d), (e), we measure the time for an experienced user to reach the specified target point when exploring the pose space for two consecutive times, 4.4 s and 2.1 s, respectively.
This demonstrates that users can learn to explore the music space by continuous dynamic mid-air hand pose control to enhance their understanding of interaction with the music space, and thus reach the goals faster. The three different tracks of hand pose in the Fig. \ref{fig:music_space} (f) proves that users can reach the same target music position with continuous control by different dynamic hand pose, and that the starting position and pose of the hand does not affect the exploration of the target music.
Notably, the delay of the interaction process of a frame-based gesture sequence (2 frames) is about 2.4 ms, as shown in Table \ref{tab:cls}, including the inference of autoencoder and classifier and drawing.

In addition, we find that different experienced users with different distances from the camera usually produce different control results on the interaction with the music space for the same hand pose. To have a more stable interaction with the music space, we further explore the use of quaternion \cite{saxena2009learning} to avoid the effect of hand size, hand position in the camera field, and distance from the camera on gesture embedding.
Fig. \ref{fig:quaternion} compares two experienced users (indicated U1 and U2) of different palm sizes (palm width and palm length) without and with quaternion conversion (as (a) and (b)), respectively, to control the music space at different locations (start position and distance) from the camera using the same gesture.
Specifically, the palm width and length for U1 are 7 cm and 15 cm respectively, and for U2 are 10 cm and 17 cm, where palm width is the distance from the widest part of the palm and palm length is the distance from the root of the palm to the tip of the middle finger. P1 and P2 indicate different starting positions, with a difference of 20 cm. D1 and D2 indicate different distances from the camera, which are 45 cm and 100 cm, respectively.
Compared with (a) and (b) in Fig. \ref{fig:quaternion}, it suggests that the use of quaternion effectively reduces the effects of hand size, hand position and distance from the camera on the low-dimensional embedding of the gestures, thus allowing the interactive system to work more stably.

\section{Conclusion}
\label{sec:Conclusion}
In this work, we study the problem of mid-air hand pose control and visible interaction for a smart music speaker and propose a novel pose space encoding and visualization model by a fully-connected autoencoder joined with a PointNet-based gesture classification network.
Specifically, a new mid-air gesture dataset is collected to train and evaluate the proposed mid-air gesture recognition and control method.
The proposed autoencoder embeds gestures into low-dimensional spaces suitable for visualisation and interaction, which helps to unify the pose space with the music space for interactions.
In addition, the auxiliary classification network further improves the clustering of gestures in pose space and maintains outstanding classification performance.
Moreover, the proposed interaction strategy requires only a few gesture frames (2-frame sequence) of input to get a continuous control that the user can explore,
which contributes to the reduction of interaction latency.

The paper provides an exploratory demonstration of the ability to control and select different areas of the user space via continuous hand pose changes by experienced users.

\bibliographystyle{IEEEbib}


\section{Appendix: Ada Answers}
\label{sec:adareplies}
We present some of the replies from the Ada model on various prompts in \cref{tab:adareplies} to illustrate the frequent, invalid answers of this model. Similar behaviour was observed when querying GPT~2. Both models failed with respect to our translation quality assessment experiment.

\newcommand{\specialcell}[2][c]{%
  \begin{tabular}[#1]{@{}l@{}}#2\end{tabular}}

\begin{table*}[h]
\centering
\begin{tabular}{ll}
\toprule
DA & \specialcell{0 = no meaning preserved\\1 = perfect meaning and grammar\\2 = excellent meaning\\3 = very good meaning\\4 = excellent\\5 = very good\\6 = excellent\\7 = excellent\\8 = excellent\\9 = excellent\\10 = perfect} \\
\midrule
DA & \specialcell{No meaning preserved: 0 \\ Perfect meaning and grammar: 100} \\
\midrule
DA & \specialcell{0 – It does not exist in my language \\ 1 – It does not exist in my text \\2 – It does not exist in my text \\3 – It exists in my text \\4 – It exists in my text\\5 – It exists in my text\\6 – It exists in my text\\7 – It exists in my text\\8 – It exists in my text\\9 – It exists in my text\\10 – It exists in my text} \\
\midrule
DA & This sentence does not have a score.\\
\midrule
DA[noref]  & \specialcell{The source for this translation is English. The translation is 1 out of 100.} \\
\midrule
DA[noref]  & \specialcell{0 = no meaning preserved\\1 = perfect meaning and grammar\\2 = perfect meaning} \\
\midrule
SQM  & \specialcell{Some meaning preserved\\Most meaning preserved and few grammar mistakes} \\
\midrule
SQM[noref]  & \specialcell{No meaning preserved\\Some meaning preserved\\Most meaning preserved and few grammar mistakes} \\

\bottomrule
\end{tabular}
\caption{Answers by the Ada model for various prompts. We observe that SQM prompts are closer to expected outputs than answers to the corresponding DA prompts. Similar behaviour was observed when querying GPT~2.}
\label{tab:adareplies}
\end{table*}
\section{Appendix: Full Results}
\label{sec:fullresults}

Below table lists all GEMBA results we have obtained for this work. Any missing segment-level scores are due to a subset of segments for which we could not obtain a score even after adding randomness.

\begin{table*}[hbt!]
\small
\centering
\input{tables/wmt22_table11.tex}
\label{tab:all_results}
\caption{System-level pairwise accuracy and segment-level Kendall Tau scores.}
\end{table*}

\section{Appendix: Prompt Templates}
\label{sec:prompttemplates}
Below we provide our prompt templates which we use for the experiments described in this paper. Template \textbf{portions in bold face} are used only when a human reference translation is available.

\subsection{DA: Direct Assessment}
Output scores range from $0-100$.\\
 {\footnotesize
    \begin{Verbatim}[commandchars=+\[\]]
    Score the following translation from {source_lang} to {target_lang} +textbf[with respect to]
    +textbf[the human reference] on a continuous scale from 0 to 100, where a score of zero means
    "no meaning preserved" and score of one hundred means "perfect meaning and grammar".

    {source_lang} source: "{source_seg}"
    +textbf[{target_lang} human reference: {reference_seg}]
    {target_lang} translation: "{target_seg}"
    Score:
    \end{Verbatim}
}
\subsection{SQM: Scalar Quality Metrics}
Output scores range from $0-100$.\\
 {\footnotesize
    \begin{Verbatim}[commandchars=+\[\]]
    Score the following translation from {source_lang} to
    {target_lang} +textbf[with respect to the human reference] on a continuous
    scale from 0 to 100 that starts with "No meaning preserved", goes
    through "Some meaning preserved", then "Most meaning preserved and
    few grammar mistakes", up to "Perfect meaning and grammar".

    {source_lang} source: "{source_seg}"
    +textbf[{target_lang} human reference: "{reference_seg}"]
    {target_lang} translation: "{target_seg}"
    Score (0-100):
    \end{Verbatim}
}
\subsection{Stars: One to Five Stars Ranking}
Output scores range from $1-5$. Special care is taken for answers containing non-numerical answers, such as "Three stars", "****", or "1 star".\\
 {\footnotesize
    \begin{Verbatim}[commandchars=+\[\]]
    Score the following translation from {source_lang} to {target_lang}
    +textbf[with respect to the human reference] with one to five stars.
    Where one star means "Nonsense/No meaning preserved",
    two stars mean "Some meaning preserved, but not understandable",
    three stars mean "Some meaning preserved and understandable",
    four stars mean "Most meaning preserved with possibly few grammar mistakes",
    and five stars mean "Perfect meaning and grammar".

    {source_lang} source: "{source_seg}"
    +textbf[{target_lang} human reference: "{reference_seg}"]
    {target_lang} translation: "{target_seg}"
    Stars:
    \end{Verbatim}
}
\subsection{Classes: Quality Class Labels}
Output label one of {\tt "No meaning
    preserved"}, {\tt "Some meaning preserved, but not understandable"}, {\tt "Some meaning preserved and understandable"}, {\tt "Most meaning preserved, minor issues"}, {\tt "Perfect translation"}.\\
 {\footnotesize
    \begin{Verbatim}[commandchars=+\[\]]
    Classify the quality of translation from {source_lang} to {target_lang}
    +textbf[with respect to the human reference] into one of following classes: "No meaning
    preserved", "Some meaning preserved, but not understandable", "Some meaning
    preserved and understandable", "Most meaning preserved, minor issues", "Perfect
    translation".

   {source_lang} source: "{source_seg}"
   +textbf[{target_lang} human reference: "{reference_seg}"]
   {target_lang} translation: "{target_seg}"
   Class:
    \end{Verbatim}
}

\noindent Templates available from: {\small \url{https://github.com/MicrosoftTranslator/GEMBA/gemba/prompt.py}}
\section{Introduction}
One of the interesting properties of large language models (LLMs) such as GPT~\cite{GPT} is their (implicit) support for multilingual Q\&A. Prompting the model in the right way allows us to translate text between languages \parcite{vilar2022prompting}. This is surprising as GPT has not been fine-tuned for the translation task.

\perscite{hendy2023good} show that GPT-enabled translation achieves high quality when applied for the translation of high-resource languages, but still lacks in terms of translation quality for under-represented languages.
Building on this finding---\emph{if the model can translate, it may be able to differentiate good from bad translations}---we apply GPT for the task of translation quality assessment.

In the remainder of this paper, inspired by recent progress on generative, pre-trained large language models (LLMs), we explore how these models can be applied for automated assessment of translation quality. The main research question for this work lies in the question: \emph{Can LLMs be used for effective quality assessment of translations?}

We propose GEMBA, which stands for \emph{GPT Estimation Metric Based Assessment}. The metric evaluates each segment translation in isolation and then averages across all obtained scores for the final, system-level score.

We define and evaluate several prompt variants for zero-shot assessment of translation quality in two modes, either with a human reference translation, as a quality metric, or without one, as a quality estimation task.

We design the main prompts based on the DA+SQM template used for human assessment of translation quality as implemented in the Appraise framework~\cite{federmann-2018-appraise} for WMT22~\cite{kocmi-EtAl:2022:WMT}, building on previous work conducted by \citet{freitag-etal-2021-experts}.

The main contributions of this paper are:

\begin{itemize}
    \item[-] We demonstrate state-of-the-art capabilities of GPT-based translation quality assessment on the latest WMT22 metrics evaluation data (on the system level);
    \item[-] We experiment with four prompt templates, showing that the least constrained template achieves best performance;
    \item[-] We evaluate seven different models of GPT, showing that only GPT~3.5 and larger models are capable of translation quality assessment;
    \item[-] We show that GEMBA is not yet reliable enough on the segment level and, thus, should be applied for system-level evaluation only.
\end{itemize}

\section{The GEMBA Metric}
To assess translation quality via prompting an LLM, the following parameters are needed:

\begin{itemize}
\item[-] prompt variant (from a pre-defined set)\\[-2em]
\item[-] source language name, e.g., ``Chinese''\\[-2em]
\item[-] target language name, e.g., ``English''\\[-2em]
\item[-] source segments $src_{1..N}$\\[-2em]
\item[-] candidate translations $hyp_{1..N}$\\[-2em]
\item[-] optionally, reference translations $ref_{1..N}$
\end{itemize}
We generate a GPT request for every segment, querying as individual zero-shot problems, and then aggregate results. For this initial proof of concept, we leave improvements such as few-shot queries or document-level context to future work.

\subsection{Prompt variants}
\begin{figure*}[htb]
\input{prompt_da}
\caption{The best-performing prompt based on Direct Assessment expecting a score between 0--100. Template \textbf{portions in bold face} are used only when a human reference translation is available.}
\label{fig:prompt_example}
\end{figure*}

We experiment with four distinct prompt types: modeling two scoring and two classification tasks. For the scoring tasks, first, one based on \textbf{direct assessment} (\emph{GEMBA-DA}), second, another based on recent research efforts on \textbf{scalar quality metrics} (\emph{GEMBA-SQM}).\footnote{Although names are based on existing techniques for human assessment, they do not match perfectly.}
As scoring of translation quality may be an unnatural task for an LLM, we also design two classification tasks. The first is based on \textbf{one-to-five stars ranking} (\emph{GEMBA-stars}), which is a style often used when users are asked to review various services or products. The second prompt asks the LLM to label translation quality as one of five discrete \textbf{quality classes} (\emph{GEMBA-classes}).

For each of these four prompt types, we experiment with two modes that differ with respect to the wording of the corresponding query templates which either have access to a human reference or not.
As an example, we show the \emph{GEMBA-DA}{} prompt in \cref{fig:prompt_example}. Based on token count, this is the least constrained prompt template that we experiment with. The complete set of prompt templates is available in \cref{sec:prompttemplates}.
For naming convention, we mark quality estimation metrics (without reference) with the suffix "[noref]".

\subsection{Scoring process}
The expected scores are in $[0, 100]$ for \emph{GEMBA-DA}{} and \emph{GEMBA-SQM}{} prompts, same as for human assessment \parcite{graham-etal-2013-continuous}; for \emph{GEMBA-stars}{} the output ranges from $[1, 5]$ and \emph{GEMBA-classes}{} assigns one of five class labels.

We average segment-level scores to obtain system-level scores. For the \emph{GEMBA-classes}{} metric variant, we assign classes a numerical value $[0-4]$, based on the label, before averaging.

Depending on the GPT model we query, sometimes answers are returned outside these ranges, as text. When we observe such an \emph{invalid} answer, we add randomness and sample more responses, selecting the first answer matching the output range as the final result.

\subsection{GPT models}

We experiment with seven GPT models---\emph{ranging from GPT~2 up to the latest ChatGPT model}---that are described in \cref{tab:gpt_models}.\footnote{\scriptsize\url{https://learn.microsoft.com/en-us/azure/cognitive-services/openai/concepts/models}}
We use the Davinci-003 model as the default model for most experiments and compare the performance of other models in \cref{sec:various_models}.
Specifically, we use these models:

\begin{description}
\item[GPT~2] We use models provided by \citet{radford2019language}, assessing if GPT~2 may be useful for quality assessment---\emph{we find that it is not};\\[-7mm]

\item[Ada] GPT~3. Max request size of 2,048 tokens and training data up to October 2019;\\[-7mm]

\item[Babbage] GPT~3. More capable than Ada;\\[-7mm]

\item[Curie] GPT~3. More capable than Babbage;\\[-7mm]

\item[Davinci-002] GPT~3.5. Max request size of 4,000 tokens and training data up to June 2021. Uses \texttt{\small FeedME} training;\\[-7mm]

\item[Davinci-003] GPT~3.5.1. Uses \texttt{\small PPO} training;\\[-7mm]

\item[ChatGPT] Improved GPT~3.5 model, fine-tuned using Reinforcement Learning from Human Feedback (RLHF).
\end{description}

GPT~3 models are based on \citet{InstructGPT}. Due to API request limits not all combinations of prompt variants and GPT models may have been evaluated for this work.

\begin{table}[t]
\centering
\scriptsize
\begin{tabular}{llll}
\toprule
Model name & Abbrev. & Details \\
\midrule
GPT-2       & ---   & See \citet{radford2019language}; does not work\\
Ada         & ---   & We use \texttt{text-ada-001}; does not work\\
Babbage     & Bab   & We use \texttt{text-babbage-001}  \\
Curie       & Curie & We use \texttt{text-curie-001}  \\
Davinci-002 & Dav2  & We use \texttt{text-davinci-002} \\
Davinci-003 & Dav3  & We use \texttt{text-davinci-003}\\
ChatGPT     & Chat  & We use \texttt{text-chat-davinci-002}\\
\bottomrule
\end{tabular}
\caption{Details of all models used in this work. Models are sorted from oldest to newest which also reflects their respective power. GPT~2 and Ada do not work.}
\label{tab:gpt_models}
\end{table}

\section{Experiments}
To measure the performance of the proposed GEMBA metric, we follow the methodology and use test data provided by the WMT22 Metrics shared task \parcite{freitag-EtAl:2022:WMT} which hosts an annual evaluation of automatic metrics, benchmarking them against human gold labels. Effectively, we compare GEMBA against the best-performing automatic metrics: COMET \parcite{rei-etal-2020-comet, rei-EtAl:2022:WMT1}, BLEURT \parcite{sellam-etal-2020-bleurt}, or the non-public winner MetricX XXL.

\subsection{Test set}
We use the MQM 2022 test set which contains human judgments for the following three translation directions: English into German, English into Russian, and Chinese into English. The test set contains a total of 54 machine translation system outputs or human translations. It contains a total of 106k segments. Translation systems are mainly from participants of the WMT22 General MT shared task \parcite{kocmi-EtAl:2022:WMT}.

The source segments and human reference translations for each language pair contain around 2,000 sentences from four different texts domains: news, social, conversational, and e-commerce.
The gold standard for scoring translation quality is based on human MQM ratings, annotated by professionals who mark individual errors in each translation, as described in \perscite{freitag-etal-2021-experts}.

\subsection{Evaluation methods}
To determine how well automatic metrics correlate with humans, we measure system-level, pairwise accuracy (\emph{accuracy}, \inparcite{kocmi-etal-2021-ship}). For segment-level evaluation, we use Kendall's Tau ($\tau$, \inparcite{freitag-etal-2022-results}).

Here, accuracy is defined as the number of system pairs ranked correctly by the metric with respect to the human ranking divided by the total number of system pair comparisons.

Formally:

\begin{center}
\footnotesize
\[\mbox{Accuracy} = \frac{|\mbox{sign}(\mbox{metric} \Delta)~=~\mbox{sign}(\mbox{human} \Delta)|}{|\mbox{all system pairs}|}\]
\end{center}

The variant of Kendall's Tau used for metric evaluation changed over the years. Initially, \perscite{callison-burch-etal-2011-findings} proposed to use Kendall's Tau-a ignoring human rankings that tied, while penalising ties in automatic metrics.

\begin{center}
\footnotesize
\[\tau = \frac{\mbox{Concordant} - \mbox{Discordant}}{\mbox{Concordant} + \mbox{Discordant}}\]
\end{center}

where Concordant is the set of all human segment comparisons for which a given metric suggests the same order of systems and Discordant is the set of all human comparisons for which a given metric disagrees.

\begin{center}
\footnotesize
\begin{tabular}{ll|ccc}
 & &   \multicolumn{3}{c}{Metric} \\
 & &     $s_1 < s_2$ & $s_1 = s_2$ & $s_1 > s_2$ \\
\midrule
\multirow{3}{*}{\rotatebox[origin=c]{90}{Human}} & $s_1 < s_2$ & Conc & Disc & Disc\\
& $s_1 = s_2$                        & -- & -- & -- \\
& $s_1 > s_2$                     & Disc & Disc & Conc \\
\end{tabular}
\vspace{1em}
\end{center}

This definition was later updated by \perscite{machacek-bojar-2014-results}, who handle ties as a separate group in contrast to Concordant and Discordant.
Metrics shared tasks \perscite{mathur-etal-2020-results} and \perscite{freitag-etal-2021-results} changed this back to the 2011 version. Last year, \perscite{freitag-etal-2022-results} changed it to Kendall's Tau-b, which makes adjustments for ties.
Overall, ties in automatic metrics are rare for non-identical translations but are an issue when a method outputs only a discrete set of scores (as in our case).
Additionally, Kendall's Tau is susceptible to noise in gold pairwise rankings \cite{freitag-etal-2022-results}.

We reproduced all scores reported in the WMT22 Metrics shared task findings paper with the official WMT22 script.\footnote{\scriptsize\url{https://github.com/google-research/mt-metrics-eval}} Reported scores match Table 11 of the WMT22 metrics findings paper \parcite{freitag-EtAl:2022:WMT}.

\section{Results}
We investigate GEMBA's performance for two modes: with a reference translation and without reference translation (in a quality estimation setting).
\cref{tab:allmain} reports pairwise accuracy on the system level, comparing \emph{GEMBA-DA}{} against the best-performing metrics from the WMT22 Metrics shared task~\parcite{freitag-EtAl:2022:WMT}.

\begin{table}[t]
\footnotesize
\centering
\input{tables/wmt22_table11_accuracy_allmain.tex}
\caption{Results for the system-level pairwise accuracy compared to the current automatic metric. Metrics marked as ``[noref]'' do not use a reference translation.}
\label{tab:allmain}
\end{table}

\subsection{Reference-based}
The results in \cref{tab:allmain} show that our reference-based \textbf{GEMBA-Dav3-DA} metric sets a new state of the art. It outperforms all of the other reference-based metrics from the WMT22 Metrics shared task.
The observed level of metric performance is unexpected, especially considering that human labels used as a gold standard in itself are noisy and therefore an accuracy of 100\% is impossible to obtain for an automatic metric.

\subsection{Quality estimation}
\cref{tab:allmain} shows that our reference-less metric \textbf{GEMBA-Dav3-DA[noref]} achieves the highest performance for the quality estimation mode, and strongly outperforms all of the other reference-less metrics. Moreover, it also outperforms all of the other reference-based metrics, performing only slightly worse than \textbf{GEMBA-Dav3-DA}. Again, the observed level of assessment quality is unexpectedly high, highlighting the potential of using LLMs for translation quality assessment tasks.

\subsection{Comparison of GPT models}
\label{sec:various_models}

We compare the performance of various GPT versions as an automatic metric. \cref{tab:all_models} shows results for all models we have experimented with and all prompt variants tested.

We do not show results for GPT-2 or Ada models. Neither of those have produced replies in the specific scoring range and neither seemed to be producing any meaningful replies. We list a couple of their answers in \cref{sec:adareplies}. Based on our experiments, we conclude that they are not powerful enough to understand the zero-shot prompts.

By contrast, Babbage and Curie models appear to understand what type of answer they should produce, but the quality of their scores seems to be close to random guessing. Thus, both Babbage and Curie are useless for translation quality assessment.

The main performance jump occurs for GPT 3.5 and larger models, i.e., Davinci-002, Davinci-003, and ChatGPT. Each of them achieves highly competitive results for all of the prompt variants we have tested. Interestingly, ChatGPT appears to have the lowest quality among these three models. In addition, ChatGPT often replies with a score followed by an explanation of why it has assigned that score.
One possible reason for this lower level of performance may be in the form of the prompt, which wasn't modified to instruct ChatGPT not to generate an explanation. Therefore, it is possible that different prompts may improve the performance of the ChatGPT model.

Unsurprisingly, the best performance is obtained by the most powerful LLM, Davinci-003. This also confirms the findings of \perscite{hendy2023good} who demonstrated superior translation capabilities with this model over all other GPT variants.

\begin{table}[t]
\scriptsize
\centering
\input{tables/all_models.tex}
\caption{Accuracy of the system-level pairwise accuracy for quality estimation methods for most combinations of prompts and different GPT models. The evaluation is based on three language pairs and MQM human labels.}
\label{tab:all_models}
\end{table}

\subsection{Segment-level performance}

All previous results are reported on the system level. We also investigate how well the GEMBA metric performs on the segment level, with respect to the human gold annotations. We present Kendall's Tau results for each language pair separately in \cref{tab:seglevel} (results for all metrics are in \cref{tab:all_results}).

Reference-based \textbf{GEMBA-Dav3-DA} is slightly behind the top-performing metrics but continues to have a high correlation with human judgment. On the other hand, quality estimation \textbf{GEMBA-Dav3-DA [noref]} has significantly lower segment-level performance in contrast to other top-performing metrics. Still, it outperforms string-based metrics (ChrF and BLEU).

The lower performance of a segment-level correlation could be attributed to Kendall's Tau, which penalises ties. Our metric in contrast to other automatic metrics returns a discrete value between 0--100. There is a high probability that two translations will obtain an equal score.

\begin{table}[t]
\scriptsize
\centering
\input{tables/wmt22_seg_kendal_table11}
\caption{Kendall's Tau ($\tau$) segment-level evaluation. Full results are in \cref{tab:all_results}.}
\label{tab:seglevel}
\end{table}

In order to investigate this further, we collect all answers across all systems and all three language pairs and then calculate the frequency of each distinct answer value.

We can notice several interesting observations in \cref{tab:score_distribution}. The DA reference-based prompt generates only multiples of five, reducing the scale to just 16 distinct values. Over three-quarters of all scores are either score 80, 90, or 95. This could reflect the actual quality of the system translations as the underlying systems are provably high-quality.

When we investigate the ``DA[noref]'', we notice that 79.9\% of all scores are of value "95". Despite this fact, the metric still manages to differentiate the systems from each other and outperform all other quality estimation metrics on the system level.

The SQM-based method uses the scale in a more fine-grained manner, producing several values that are not multiples of five, i.e., 68, 87, 92, and 93.

We conjecture that frequent ties and the discrete scale thus may be the reason behind the lower segment-level performance.

\begin{table}[t]
\scriptsize
\centering
\input{tables/score_count.tex}
\caption{Distribution of all distinct segment-level score values for MQM 2022 for model Davinci-003.}
\label{tab:score_distribution}
\end{table}

\subsection{Failure rate}

As we described earlier, LLMs may answer with an invalid answer, for example with a textual answer instead of a score, mostly explaining its decision. When such a situation happens, we iteratively increase the temperature---\emph{adding randomness to the model}---and take the first answer matching the expected score output range.

This adds non-determinism to our evaluation, therefore we investigate how frequently this phenomenon happens. \cref{tab:failrate} shows the number of invalid answers. For almost all combinations of models and prompts, LLMs understand the prompt and provide answers in a valid range with less than 1\% of the answers being invalid.\footnote{Roughly 1,000 answers equal to 1\% of the total volume.}

\begin{table*}[t]
\footnotesize
\centering
\input{tables/failrate.tex}
\caption{Number of invalid answers (full set size 106,758) that needed to be re-prompted with added randomness.}
\label{tab:failrate}
\end{table*}

Processing answers is straightforward as it is usually a stand-alone number. In some occasions, LLMs give a numerical score and continue with a textual explanation, for such cases, we parse only the first number.
A more complex approach needs to be taken for \emph{GEMBA-stars}{} prompts where the model provides different answers which we parse separately. Here are some examples of two-star answers: "2", "two", "**", "★★", "two stars", or "2 stars". For non-English target languages the answer may be produced in the target language, e.g., "一颗星", or "五".
We have not observed attempts to translate output for other prompts.

\section{Conclusion}
We have presented our work on GEMBA, a GPT-based estimation metric-based assessment method. Comparing our metrics to other automated metrics from the WMT22 Metrics shared task we report state-of-the-art performance on the MQM 2022 test set across three language pairs: English to German, English to Russian, and Chinese to English.

We intend to continue research on the application of GPT models for quality assessment. Further research will focus on the switch to few-shot (as opposed to our current zero-shot methodology) as well as model fine-tuning. Both of which promise to increase GEMBA accuracy. Furthermore, modifying prompts to support MQM error-based evaluation or post-editing efforts may lead to further improvements.

GPT-enhanced evaluation metrics may allow us to make progress with respect to document-level evaluation (due to their ability to use much larger context windows). This could be beneficial as there is little research into document-level metrics \parcite{vernikos-EtAl:2022:WMT}.

\section*{Limitations}
While preliminary results indicate that the GEMBA metric performs very well when compared to other automated metrics evaluated as part of the WMT22 Metrics shared task, it is important to note that these results are based on human labels for {\emph{only three language pairs}}. We expect that the metrics performance may suffer for other language pairs, mainly under-resourced languages similar to \perscite{hendy2023good} who show low translation quality for such languages. In addition, GEMBA's state-of-the-art performance only holds for the system level, while segment-level scores still have room for improvement.
Reported results are indicative of the potential performance LLMs could achieve for the translation quality assessment task in the long run.

\section*{Acknowledgments}
This work would not have been possible without the help and support from our friend and colleague, Olivier Nano, who provided access to GPT models via Microsoft Azure -- {\emph{Merci beaucoup, Olivier!}}
The authors are also grateful to Matt Post, Vikas Raunak, Shabnam Sadegharmaki, and the Microsoft Translator research team for fruitful discussions and helpful feedback.


\section{Introduction}
\label{sec:intro}

\def*}\footnotetext{These authors contributed equally to this work{*}\footnotetext{These authors contributed equally to this work}

Head motion is a ubiquitous challenge for magnetic resonance image (MRI) acquisition. It causes a range of image artifacts that introduce bias in downstream analysis~\cite{reuter2015motion,alexander2016subtle,gilmore2021variations, savalia2017motion,pardoe2016motion,rosen2018quantitative},
which persists despite expert quality control~\cite{reuter2015motion, alexander2016subtle}.
While initially explored for clinical cohorts with increased motion levels~\cite{zaca2018method, zukic2022medical, pardoe2016motion, rosen2018quantitative} or induced motion~\cite{kustner2018automatic, reuter2015motion}, less research focuses on motion in studies of healthy, compliant population cohorts~\cite{fantini2018automatic, alexander2016subtle} such as the Rhineland Study~\cite{breteler2014mri, stocker2016big}. Critically, the lack of a sensitive and reliable motion estimation method to quantify subtle motion hinders the inclusion of motion estimates in statistical models to control motion-induced biases.
For example, careful visual inspection of the Rhineland Study dataset, used in this paper, did not detect any cases with clearly visible motion artefacts that would warrant exclusion.
Yet, even in the 75-participant subset reserved for testing, a statistically significant correlation of motion with age can be shown, underlining the need for sensitive estimation and control of head motion in MRI analyses.

In this paper, we propose a method to directly estimate head motion from the acquired MR image. We measure head motion during MRI acquisition via head tracking with a depth camera and establish a ground truth motion score per sequence. This is contrary to the currently established paradigm of predicting discrete motion severity levels established via an expert manual quality control process~\cite{largent2021image, fantini2018automatic, kustner2018automatic, sujit2019automated, ma2020diagnostic, kustner2018automated, stpien2021fusion, zukic2022medical, lei2022artifact}. Expert ratings are limited by their subjectivity to the specific task and human perception~\cite{barrett2015task, prydeperformance, lei2022artifact} hindering their general utility, specifically for compliant, low-motion cohorts. Camera-based motion measurements, on the contrary, are objective and sensitive even for low-motion cohorts, where motion-induced image artifacts are almost invisible.

Since the rise of deep learning, tools have been able to predict expert motion ratings with increasing accuracy~\cite{largent2021image, fantini2018automatic, kustner2018automatic, sujit2019automated, ma2020diagnostic, kustner2018automated, stpien2021fusion, zukic2022medical, lei2022artifact} sometimes addressing previous limitations, for example the subjectivity to the task~\cite{lei2022artifact}.
Currently, the only alternative to the prediction of expert labels is to predict a perceptual image similarity metric between low motion ''base-line'' images and high-motion images, which are retrospectively simulated~\cite{sciarra2022reference}.
This approach replaces the human annotation task by a comparison of images with a perceptual similarity metric (SSIM), which may also suffer from similar limitations as expert labels. Moreover, the methods accuracy on real-world data relies on realistic, high-quality simulation, which also has to be adapted to each acquisition sequence. Meanwhile, our method can be directly re-trained even on different modalities, without any changes. %
Recent work in the field of in-MRI motion tracking enabled highly accurate tracking of head-motion during acquisition with the MRI scanner~\cite{tisdall2012volumetric, andronesi2021motion}, optical cameras~\cite{slipsager2019markerless, pardoe2021estimation} or other devices~\cite{musa2022mri, laustsen2022tracking}. Yet, until tracking devices and methods are deployed to all imaging sites, image-derived motion estimates may help reduce potential motion induced biases -- even retrospectively. %

Our contributions are threefold -- we 1.~introduce -- for the first time -- the estimation of an objective motion score from images of three MRI sequences, 2.~present a deep-learning-based solution, which outperforms DenseNet and state-of-the-art quality/motion estimation methods on a dataset of compliant, low-motion participants, and 3.~quantify motion from respiration and (relaxation) drift. Finally, our method detects the significant, known correlation between predicted motion and age. We will publish our code on \href{https://github.com/Deep-MI/head-motion-from-MRI/}{GitHub}\footnote{\href{https://github.com/Deep-MI/head-motion-from-MRI/}{https://github.com/Deep-MI/head-motion-from-MRI/}}.

\section{Materials \& Methods}
\label{sec:method}

\begin{figure}[t]
    \centering
    \includegraphics[trim={0 .0cm 0 0.26},clip,width=3.2in]{network_scatt.pdf}
    \caption{Plot of ground truth and predicted motion score. Blue \emph{PASS} and orange \emph{WARN} (visible artifacts) test images may be perfectly separated with a threshold of 0.6.}
    \label{fig:net_scatter}
\end{figure}

\subsection{Data acquisition}

The Rhineland study is an ongoing population study recruiting a representative cohort of healthy participants above the age of 30. Our dataset describes a subset (ages 30 to 95 years) of 500 participants (282 female) with T1w, T2w and FLAIR images (at 0.8, 0.8 and \SI{1.0}{\milli\meter} isotropic voxel size, respectively) following a standardized acquisition protocol\footnote{For details on the acquisition of 3D T1w MPRAGE (scan length 6.5 min), 3D T2w (4.6 min) and FLAIR (4.5 min) images, see Lohner et al.~\cite{lohner2022incidental}.}. %
This dataset also includes expert quality-labels of T1w images (\emph{PASS}, \emph{WARN} or \emph{FAIL}), where \emph{WARN} indicates visible and \emph{FAIL} strong artifacts (insufficient for downstream analysis). However, no images are rated as \emph{FAIL} and only 9 images as \emph{WARN}.
The low number of \emph{WARN} and \emph{FAIL} cases is likely founded on high compliance and extensive efforts to reduce head motion during MRI acquisition including tight head padding, scheduled speaking breaks, careful participant instruction, and calming nature scenes shown during the scan.%

To quantify the participant's head motion, a video of depth images showing a portion of their face is collected concurrently to the scan~\cite{slipsager2019markerless}.
Individual frames are aligned with a reference frame resulting in time series of rigid transformations. To obtain a per-sequence, scalar \emph{motion score}, we 1.\ synchronize MRI and depth camera, 2.\ compute Jenkinson's transformation differences~\cite{jenkinson1999measuring} per pair of transformations, and 3.\ extract and average values for the duration of individual sequences. The Jenkinson's transformation difference summarizes rigid transformations by averaging displacements within a spherical head model. The final \emph{motion score} quantifies the average motion in millimeter per second. We randomly split the dataset into training, validation, and evaluation sets of 350, 75 and 75 participants, respectively.

\subsection{Distinguishing motion patterns}
\label{sec:freq}

In addition to the per-sequence motion score, we distinguish between three prominent types of in-scanner head motion corresponding to three frequency bands: %
i) head drift, ii) periodic motion due to breathing, and iii) ``noisy motion'', which we expect is hard to estimate.
To determine global filter thresholds, we estimate upper and lower median respiratory frequency of participants during the T1w sequence from an independent respiration sensor as \SI{0.1}{\hertz} and \SI{0.5}{\hertz}. We apply symmetric Butterworth filters (low-pass, band-pass and high-pass, respectively) to the time series of Jenkinson's transformation differences. From the filtered signals, we aggregate the motion score as before resulting in three frequency-dependent targets.

\subsection{Neural network \& training}

For the estimation of motion scores from 3D MR images, we adopt a fully convolutional neural network (CNN) from brain age estimation~\cite{peng2021accurate}, an established regression task in medical imaging. The lightweight architecture permits training the 3D CNN on a single NVIDIA A100 GPU with batch size two. Instead of directly regressing the motion score, we follow Peng et al.~\cite{peng2021accurate} in their approach and 1.\ generously define the expected range of motion [0, 3.12] mm/s, 2.\ split it into 40 bins of `prototypes', 3.\ for each prototype calculate the probability that the current motion score belongs into the prototype, and 4.\ train the CNN using a Kullback-Leibler loss (Adam optimizer for 500 epochs, approx.\ \SI{10}{\hour} on one A100 GPU). To reconstruct the motion score from the predicted probability distribution, we sum the product of prototype centers and predictions. %
Many standard data augmentation strategies are not suitable for motion estimation, since the re-sampling of images affects the image noise, which is why we avoid interpolation of images completely. Helpful data augmentation, on the other hand, include intensity scaling in the range [0.9,1.1] and random flipping with 30\% probability along all axes.
In our ablation study (Section~\ref{sec:ablation}), we explore different pre-processing operations. A focus on the 8 Least Significant Bits (LSB8) is useful for this task, leaving only an integer representation of the fine image differences. %

\begin{table}[t]
\centering
\begin{tabular}{| c | l | c | c |}
     \hline
     & Method & R\textsuperscript{2} & Spr-$\rho$  \\
     \hline\hline
     \parbox[t]{2mm}{\multirow{4}{*}{\rotatebox[origin=c]{90}{re-trained}}} & Ours& \textbf{0.433} & \textbf{0.584} \\
     \cline{2-4}
     & DenseNet \cite{huang2017densely} & 0.395 & 0.447 \\
     \cline{2-4}
     & SFCN \cite{peng2021accurate} & 0.275 & 0.454 \\
     \cline{2-4}
     & MIQA CNN \cite{zukic2022medical} & -0.273 & 0.192 \\
     \hline\hline
     \parbox[t]{2mm}{\multirow{3}{*}{\rotatebox[origin=c]{90}{QCtools}}} & %
       MIQA MA / QS & - & 0.243 / 0.240\\
     \cline{2-4}
     & MIQA MA / QS\textsuperscript{1} & - & -\textsuperscript{2} / 0.338 \\
     \cline{2-4}
     & AES~\cite{zaca2018method} & - & 0.110 \\
     \hline
\end{tabular}

\caption{Our methods outperforms SOTA approaches in both Spearman's $\rho$ (Spr-$\rho$) and R\textsuperscript{2} scores (only valid for predicted motion scores) when predicting motion scores from T1w images on the test set. MA: motion artefact score, QS: quality score, \textsuperscript{1}images standardized, \textsuperscript{2}no motion detected  %
}
\label{tab:results}
\end{table}

\subsection{Evaluation \& statistical methods}

We evaluate the regression model with the coefficient of determination (R\textsuperscript{2} score -- a measure for the average error) and Spearman's rank correlation coefficient (Spearman's $\rho$ -- a measure for correct ranking). The R\textsuperscript{2} score normalizes the mean squared error to a range of $[-\infty, 1]$, where score \textless 0 indicates a prediction error worse than a constant prediction of the dataset mean and a score of 1 indicates perfect predictions. The Spearman's $\rho$, on the other hand, is defined in the range $[0,1]$. It is not affected by large, absolute errors of outliers and more sensitive to prediction errors, where the sampling of values is denser (i.e.\ more sensitive to small errors on values close together). We also analyze the rank correlation between motion and age using this method. We use the R\textsuperscript{2} score as the primary metric for ablation, and select parameters that have the highest R\textsuperscript{2} score on the validation set in experiments.

\section{Results}

We visualize the performance of our method on the unseen test set in Figure~\ref{fig:net_scatter}, which illustrates good correlation between ground truth measured motion (horizontal axis) and predictions from images (vertical). Perfect predictions would lie on the black line. Our method perfectly separates \emph{PASS} (no artifacts) and \emph{WARN} (mild artifacts) cases. A horizontal separation line at $\approx\SI{0.6}{\milli\meter/\second}$ can be found but no vertical line for the ground truth motion score.

\subsection{Comparison with state-of-the-art motion estimation}
\label{sec:results_general}

To the best of our knowledge, there is currently no competing method to predict measured, in-scanner head motion from MR images. Since quality estimation methods cannot be easily re-trained on our dataset, which has few \emph{WARN} and no \emph{FAIL} labels, we compare with the pre-trained MIQA quality estimator~\cite{zukic2022medical} and Average Edge Strength (AES)~\cite{zaca2018method}, a heuristic known to correlate with motion~\cite{zaca2018method}. Additionally, we compare our method with three deep learning architectures re-trained on our dataset:
i) DenseNet~\cite{huang2017densely}, ii) SFCN~\cite{peng2021accurate}, a CNN for brain age prediction, and iii) the CNN used by MIQA~\cite{zukic2022medical}. Our method, which is an improvement of SFCN (e.g.\ initializer), achieves the best correlation with ground truth motion scores on the unseen test set (Table~\ref{tab:results}). The negative R\textsuperscript{2} of the re-trained MIQA CNN indicates failed generalization. %

The pre-trained MIQA tool aggregates predictions of expert ratings for 9 artifact types including a `motion artifacts' (MA) score into a continuous quality score (QS). We select their probabilities as potential correlates of our motion score.
Since MIQA has been trained on the \SI{1}{\milli\meter} T1w-PREDICT-HD dataset, we test, whether rescaling and resampling images with FastSurfer's~\cite{henschel2020fastsurfer} conform tool to \SI{1}{\milli\meter} reduces domain shift.
We measure a low but significant correlation between QS and motion score as well as between MA and motion score on the native \SI{0.8}{\milli\meter} images. While previous work reported that probabilities of ratings, like those of MIQA, quantify subtle differences in motion, despite binary ground truth labels~\cite{kustner2018automated}, MA probabilities are zero for all conformed images (consistent with the lack of manual \emph{FAIL} labels). QS, on the other hand, significantly correlates with the motion score.
In addition to the deep learning estimators, we evaluate the performance of AES~\cite{zaca2018method}, but do not find significant correlation between AES and the ground truth motion score.

\subsection{Generalization to T2/FLAIR and motion types}

\begin{table}[t]
    \centering
    \begin{tabular}{|l | c | c | c | c ||}
        \hline
        Input & Target &R\textsuperscript{2} & Spr-$\rho$ \\
        \hline\hline
        T1 & motion score &0.433 & \textbf{0.584}\\
        \hline
        T2 & motion score & 0.362 & 0.556 \\
         \hline
        FLAIR & motion score & 0.299 & 0.489 \\
         \hline\hline
        T1 & drift & 0.183 & \textbf{0.637} \\
         \hline
        T1 & breathing band & 0.185 & 0.382 \\
         \hline
        T1 & noisy motion & 0.050 & 0.337 \\
         \hline

    \end{tabular}
    \caption{The evaluations show generalization of our method to predicting the motion score on T2w and FLAIR images and very promising Spr-$\rho$ performance for detection of drift on the test set.}
    \label{tab:generalize}
\end{table}

We test the generalizability of our method to new tasks by predicting the motion score for T2w and FLAIR images, as well as predicting drift and respiratory motion on T1w images. For each task we re-train the network architecture and show the results in Table~\ref{tab:generalize}. While the R2 score is not directly comparable across different tasks, we find good performance by purely re-training the model for these modalities. The Spearman's-$\rho$ in T2w and FLAIR experiments is similar to T1w experiments despite optimization on T1w only.%

To quantify different motion types, we define two distinct aggregates of participant motion: i) slow relaxation-drift over time, and ii) periodic head motion due to breathing. We filter motion estimates as described in Section~\ref{sec:freq} and train our architecture to predict the aggregates (Table~\ref{tab:generalize}). Our method can predict both slow drift (low frequency) motion and respiratory (medium frequency) motion from the T1w images. The aggregate of high frequencies, which are not associated with known motion types (noisy motion), cannot be predicted. %

\begin{table}[t]
\centering
\begin{tabular}{||l | c | c | c | c ||}
     \hline
     Method & Images & R\textsuperscript{2} & Spr-$\rho$ \\ \hline\hline
     CNN MSE loss & T1 (LSB8) & 0.117 & 0.376 \\
     \hline\hline
     CNN ours & T1 (unprocessed) & 0.341 & 0.551 \\ \hline

    CNN ours & T1 (robust scaling) & 0.322 & 0.489 \\ \hline

    CNN ours & T1 (remove head) & 0.369 & 0.525 \\  \hline
    CNN ours & T1 (LSB8) & \textbf{0.393} & 0.491 \\ \hline

\end{tabular}
\caption{Ablation of loss function and image pre-processing on the validation set. LSB8: 8 Least Significant Bits. %
}
\label{tab:ablation}
\end{table}

\subsection{Ablation study}
\label{sec:ablation}

To optimize the parameters for our method, we compare choices for loss and preprocessing on the validation set.
A critical finding of this work is that direct prediction trained using an Mean Squared Error (MSE) loss only achieves mild correlation with the motion score, while using a multi-dimensional probability distribution together with a Kullback-Leibler divergence loss yields a large performance uplift. For this MSE-loss ablation in Table~\ref{tab:ablation} (top), we reduce the last layer to a single output and remove the softmax.

Several measures of image quality have taken advantage of the background signal to determine a reduction in image quality~\cite{esteban2017mriqc, mortamet2009automatic}. %
Consequently, we explore the effects of four image pre-processing operations on the prediction quality in Table \ref{tab:ablation}: No pre-processing, FastSurfer's~\cite{henschel2020fastsurfer} robust scaling (removing 20\% of high intesity voxels), removing the head with FreeSurfer's head segmentation tool~\cite{fischl2012freesurfer, segonne2004hybrid} (the result is just the background) and dropping the Most Significant Bits leaving us with the 8 Least Significant Bits (LSB8). The latter approach, which drops the information about the absolute size of values directly on the images integer representation, surprisingly outperforms other ways of adjusting image intensities. %
This finding was validated in multiple experiments and across additional hypotheses.

\subsection{Correlation with age}

In adult populations, increased head motion in the MR scanner is associated with the increased age of participants~\cite{madan2018age, savalia2017motion}. We can also measure this correlation within the ground truth motion score using Spearman's $\rho$ and find a significant correlation with age on the whole dataset as well as only on the test set (p \textless 0.001). Our predictions also present this correlation with age on the test set (p \textless 0.001) as illustrated in Figure~\ref{fig:age} with the linear fit. %

\begin{figure}[t]
    \centering
    \includegraphics[trim={0 .0cm 0 .26cm},clip,width=3.2in]{age.pdf}
    \caption{Age is significantly correlated with our motion predictions. This reflects the ground truth motion score labels (not shown) and is outlined by a linear fit, displayed as a red line.}
    \label{fig:age}
\end{figure}

\section{Discussion \& conclusion}
We introduce a novel task of in-scanner head motion estimation directly from MR images. The presented deep learning method provides sensitive predictions of motion levels capturing subtle correlations with known confounders such as age -- all in unseen images that pass visual inspection.
Two factors contribute to this sensitivity: predicting the probability distribution of `prototypes' together with the Kullback-Leibler divergence loss, and the input of least-significant-bit (LSB) images. Why LSB images are preferable to raw structural images or their background should be further investigated in future work. %

A well-known limitation of deep learning methods is the limited generalizability to unseen datasets. Large differences in motion levels between cohorts and the chosen acquisition parameters greatly affect the appearance of motion artifacts, hence we expect dedicated training datasets will be required to for re-training and generalization to unseen MR imaging sequences.
Additionally, future work should investigate, whether motion estimation itself is also affected by biases, like age and diseases, which are known to be an indicator of increased motion levels.

Our method ranks images by their motion level better than comparable, state-of-the-art methods for MRI motion and quality estimation from expert ratings. Therefore, it may aid quality control procedures in the identification and exclusion of cases with artifacts, which is also indicated by the clear separation between \emph{PASS} and \emph{WARN} labels in our experiments (Figure~\ref{fig:net_scatter}). However, confirmation on a dataset with more strongly motion-affected cases is required. Additionally, our method transfers well to the prediction of alternative targets for respiratory- and drift motion and from T2w and FLAIR images with comparable accuracy. %

Finally, our method enables an analysis of other, perhaps unknown, correlates of motion as well as the integration of motion scores as a control variable in statistical models. This is particularly valuable in longitudinal cohort studies, like the Rhineland Study, to disentangle the bias of motion effects from other effects such as participant's age and diseases.

\section{Compliance with ethical standards}
\label{sec:ethics}

The Rhineland Study is carried out in accordance with the recommendations of the ICH‐GCP standards. Written informed consent was obtained from all participants in accordance with the Declaration of Helsinki. Approval was granted by the Ethics Committee of University Bonn. %

\section{Acknowledgments}
\label{sec:acknowledgments}

This work was supported by DZNE institutional funds, by the Federal Ministry of Education and Research of Germany (031L0206), and the Helmholtz-AI project DeGen (ZT-I-PF-5-078).
We thank the Rhineland Study group (PI Monique Breteler) for supporting the data acquisition and management.
The authors do not have any conflict of interest.

\printbibliography

\setlength{\FrameRule}{0pt}
\setcounter{table}{0}
\setcounter{figure}{0}
\renewcommand{\thesubsection}{\Alph{subsection}}
\renewcommand\theequation{A.\arabic{equation}}
\renewcommand\thetable{A\arabic{table}}
\renewcommand\thefigure{A\arabic{figure}}

\end{document}