\section{Introduction}
With the great demand of keeping healthy, reducing high pressure from working and staying in shape, fitness activity has become more and more important and popular during the past decades~\cite{aifit}.
According to the statistics\footnote{https://policyadvice.net/insurance/insights/fitness-industry-statistics}, there are over 200,000 fitness clubs and 170 million club members all over the world. More recently, because of the high expense of coaches and out-breaking of COVID-19, increasing people choose to exclude gym membership and do the workout by watching the fitness instructional videos from fitness apps or YouTube channels (\textit{e.g.,} FITAPP, ATHLEAN-X, The Fitness Marshall, \textit{etc.}). Therefore, it is desirable to advance current intelligent vision systems to assist people to perceive, understand and analyze various fitness activities.

In recent years, a variety of datasets have been proposed in the field~\cite{aifit,EC3D,verma2020yoga}, which have provided good benchmarks for preliminary research. However, these datasets might have limited capability to model complex poses, describe a fine-grained activity, and generalize to different scenarios. We present FLAG3D in this paper, a 3D \textbf{F}itness activity dataset with \textbf{LA}n\textbf{G}uage instruction. Figure~\ref{fig:teaser} presents an illustration of our dataset, which contains 180K sequences of 60 complex fitness activities obtained from versatile sources, including a high-tech MoCap system, professional rendering software, and cost-effective smartphones. In particular, FLAG3D advances current related datasets from the following three aspects:

\input{table/dataset_comparison}

\textbf{Highly Accurate and Dense 3D Pose.} For fitness activity, there are various poses within lying, crouching, rolling up, jumping \textit{etc.}, which involve heavy occlusion and large movements. These complex cases bring inevitable obstacles for conventional appearance-based or depth-based sensors to capture the accurate 3D pose. To address this, we set up an advanced MoCap system with 24 VICON cameras~\cite{vicon} and professional MoCap clothes with 77 motion markers to capture the trainers’ detailed and dense 3D pose.

\textbf{Detailed Language Instruction.} Most existing fitness activity datasets merely provide a single action label or phase for each action~\cite{EC3D,verma2020yoga}. However, understanding fitness activities usually requires more detailed descriptions. We collect a series of sentence-level language instructions for describing each fine-grained movement. Introducing language would also facilitate various research regarding emerging multi-modal applications.

\textbf{Diverse Video Resources.}
To advance the research directly into a more general field, we collect versatile videos for FLAG3D. Besides the data captured from the expensive MoCap system, we further provide the synthetic sequences with high realism produced by rendering software and the corresponding SMPL parameters. In addition, FLAG3D also contains videos from natural real-world environments obtained by cost-effective and user-friendly smartphones.

To understand the new challenges in FLAG3D, we evaluate a series of recent advanced approaches and set a benchmark for various tasks, including skeleton-based action recognition, human mesh recovery, and dynamic action generation. Through the experimental results, we find that
1) while the state-of-the-art skeleton-based action recognition methods have attained promising performance with highly accurate MoCap data under the in-domain scenario, the results drop significantly under the out-domain scenario regarding the rendered and natural videos.
2) Current 3D pose and shape estimation approaches easily fail on some poses, such as kneeling and lying, owing to the occlusion. FLAG3D provides accurate ground truth for these situations, which could improve current methods’ performance in addressing challenging postures.
3) Motions generated by state-of-the-art methods appear to be visually plausible and context-aware at the beginning. However, they cannot follow the text description faithfully as time goes on.

To summarize, our contributions are twofold: 1) We present a new dataset named FLAG3D with highly accurate and dense 3D poses, detailed language instruction, and diverse video resources, which could be used for multiple tasks for fitness activity applications. 2) We conduct various empirical studies and in-depth analyses of the proposed FLAG3D, which sheds light on the future research of activity understanding to devote more attention to the generality and the interaction with natural language.
\section{Related Work}
Table~\ref{tab:actionsetcomparison} presents a comparison of our FLAG3D with the related datasets. Here we briefly review the relevant datasets and methods regarding the three tasks we focus on.

\noindent \textbf{Human Action Recognition.}
As the foundation of video understanding, pursuing diverse datasets has never stopped in action recognition. Existing works have explored various modalities in this area, such as RGB videos \cite{kuehne2011hmdb,soomro2012ucf101,carreira2017quo,shao2020finegym}, optical flows \cite{simonyan2014two}, audio waves \cite{xiao2020audiovisual}, and skeletons ;\cite{yan2018spatial}. Among these modalities, skeleton data draws increasing attention because of its robustness to environmental noises and action-focusing nature. During the past few years, various network architectures have been exploited to model the spatio-temporal evolution of the skeleton sequences, such as different variants of RNNs \cite{du2015hierarchical, song2017end, zhang2017view}, CNNs \cite{choutas2018potion, yan2019pa3d, liu2017two, duan2022revisiting} and GCNs\cite{yan2018spatial, shi2019two, li2019actional, liu2020disentangling, chen2021channel, tang2018deep}. In terms of skeletal keypoint, current action recognition datasets can be divided into two classes: one is 2D keypoint datasets \cite{yan2018spatial, duan2022revisiting} extracted by pose estimation methods \cite{cao2017realtime, chen2018cascaded, martinez2017simple, sun2019deep}, and the other is 3D keypoint datasets \cite{shahroudy2016ntu, liu2019ntu, ionescu2013human3, omran2018neural} collected by sensors or other sophisticated equipment. However, most existing datasets are limited to a single domain of natural scenes. FLAG3D dataset takes a different step towards cross-domain action recognition between rendered videos and real-world scenario videos.

\noindent \textbf{Human Mesh Recovery.}
Human mesh recovery obtains well-aligned and physically plausible mesh results that human models can parametrize, such as SMPL\cite{loper2015smpl}, SMPL-X\cite{pavlakos2019expressive}, STAR\cite{osman2020star} and GHUM\cite{xu2020ghum}. Current methods take keypoints\cite{bogo2016keep, pavlakos2019expressive, Zhang2021LightWeight}, images\cite{gu2018ava, guler2019holopose, kolotouros2019learning, li2020hybrik, omran2018neural, pavlakos2018learning}, videos\cite{choi2021beyond, kanazawa2019learning, luo20203d, mehta2020xnect, moon2020i2l, sun2019human} and point clouds\cite{bhatnagar2020combining, hong2021garmentd, jiang2019skeleton, liu2021votehmr, wang2021locally} as inputs to recover the parametric human model under optimization\cite{bogo2016keep, zanfir2018monocular, lassner2017unite} or regression\cite{kanazawa2018end, pavlakos2018learning, omran2018neural, tung2017self, kolotouros2019learning} paradigm. Besides the above input modalities, there are ground-truth SMPL parameters provided by human datasets. They are registered by marker-less multi-view MoCap\cite{peng2021neural, Yu2020HUMBIAL, mehta2017monocular, pavlakos2019expressive, Mehta2018SingleShotM3, Zhang2020ObjectOccludedHS}, or marker/sensor based Mocap\cite{Sigal2009HumanEvaSV, ionescu2013human3, von2018recovering}. SMPL can also be fitted with the rendered human scan in synthesis datasets\cite{Varol2017LearningFS, cai2021playing, patel2021agora, tao2021function4d}. Easily recovered human poses of existing datasets cause the performance of human mesh recovery algorithms not to be fairly evaluated\cite{pang2022benchmarking}, whereas FLAG3D provides human poses with heavy occlusion and large movements. The work most related to ours is HuMMan~\cite{humman}, which contains large-scale and comprehensive multi-modal resources captured in a single MoCap room. In comparison, FLAG3D is complementary with rendered and natural videos, as well as more detailed language instructions to describe the activity.

\noindent \textbf{Human Action Generation.}
In the past several years, various works have utilized multiple forms of information to guide the generation of actions. Among these, one direction \cite{HaoyeCai2017DeepVG, ZhenyiWang2019LearningDS, PingYu2020StructureAwareHG, ChuanGuo2022Action2videoGV, action2motion, MathisPetrovich2021ActionConditioned3H} is to explore the underlying data structure of action sequences based on action catageries. As real-life movements are often accompanied by audio messages, another direction\cite{KentaTakeuchi2017SpeechtoGestureGA, EliShlizerman2017AudioTB, TaoranTang2018DanceWM, HsinYingLee2019DancingTM, RuoziHuang2020DanceRL} is to use the audio and motion timing alignment feature. Translating text descriptions to human motion is an emerging topic. Several works \cite{ChaitanyaAhuja2019Language2PoseNL, MatthiasPlappert2018LearningAB, TatsuroYamada2018PairedRA, AninditaGhosh2021SynthesisOC, petrovich22temos, HumanML3D, GuyTevet2022MotionCLIPEH, AlecRadford2021LearningTV, zhang2022motiondiffuse, tevet2022human, song2020denoising, ho2020denoising} try to match semantic information and high-dimensional features of action sequences to pursue natural motion sequences guided by language. For traditional motion generation tasks, HumanAct12\cite{action2motion}, UESTC\cite{YanliJi2018ALR} and NTU RGB+D\cite{liu2019ntu} are three commonly used benchmarks. However, the above datasets do not provide paired sophisticated semantic labels to the motion sequences. There are not enough motions for the exact text in the language-motion dataset KIT\cite{MatthiasPlappert2016TheKM}. Recently, BABEL\cite{AbhinandaRPunnakkal2021BABELBA} and HumanML3D\cite{HumanML3D} re-annotates AMASS\cite{NaureenMahmood2019AMASSAO} with English language labels. Nonetheless, they focus on simple actions with uncomplicated descriptions. FLAG3D provides long sequences of actions with detailed and professional language instructions.
\section{The FLAG3D Dataset}
\label{sec:dataset}
\subsection{Taxonomy}
The first challenge of constructing FLAG3D is establishing a systematic taxonomy to organize various fitness activities. In previous literature, most existing fitness datasets~\cite{aifit,EC3D,verma2020yoga} mix up all the activities. We present a deeper hierarchical lexicon as shown in Figure~\ref{fig:lexicon}, which contains three levels from roots to leaves, including body part, fitness activity, and language instructions.

(1) \textit{Body Part.}
For the first level, we share our thoughts with HuMMan~\cite{humman} which uses the driving muscles as basic categories. However, numerous fine-grained muscles exist in the human body, and one activity might be driven by different muscles. We follow the suggestions of our fitness training coaches and choose ten parts of the human body with rich muscles as \textit{chest, back, shoulder, arm, neck, abdomen, waist, hip, leg} and \textit{whole body}\footnote{Some activities are driven by muscles of various body parts.}.

(2) \textit{Fitness Activity.} Sixty everyday fitness activities are selected for the second level, linked to the corresponding body parts of the first level. For example, the activity \textit{``Squat With Alternate Knee Lift’’} is associated with the quadriceps femoris muscle of the \textit{``Leg’’}. We include the complete list of the 60 fitness activities in the Appendix.

(3) \textit{Language Instruction.} We composed the third level of the lexicon with a set of language descriptions from the guidance of the training coaches to instruct users to accomplish the fitness activity. As an example shown in Figure~\ref{fig:lexicon}, the fitness activity \textit{``Squat With Alternate Knee Lift’’} is detailed as \textit{``Stand with feet slightly wider than shoulder-width apart, bend your elbows and put your hands in front of your chest. Flex your hips and squat until your thighs are parallel to the ground, and keep your knees in the same direction as your toes when squatting...’’}. There are about 3 sentences and 57 words for each fitness activity on average.

\subsection{Data Collection}
We deploy a     high-precision MoCap equipment in an open lab to capture accurate human motion information. To obtain the rendered videos, we purchase kinds of virtual scenes and character models to make full use of the collected 3D skeleton sequences. In different environments, we record real-world natural videos. We agree with the volunteers and ensure that researchers can use these data. We detail the data collection process below.

\noindent \textbf{Data from MoCap System.}
Figure~\ref{fig:mocap_overview} presents an overview of our MoCap system, which is equipped in a lab of 20 meters long, 8 meters wide, and 7 meters high. The lab uses the high-tech VICON~\cite{vicon} MoCap system to capture the actors’ body part movements through optical motion capture. Cameras used in this system have a maximum resolution of 4096$\times$4096. It is capable of 120fps while maintaining maximum resolution sampling. Ten volunteers perform the actions in the motion capture field. Infrared cameras transmit the high frame rate IR gray-scale images captured over the fiber optic cable to the data switch. They are clock aligned and finally sent to a device with specialized processing software. Through the professional machine, we can monitor the movements of volunteers in various forms, including masks, bones, and marker points. Moreover, we hire professional technicians to perform data restoration and motion redirection based on high-precision original data so that we can ensure the accuracy and diversity of provided 3D motion data. Meanwhile, we ask each performer to wear MoCap clothes with 77 motion markers listed on Table ~\ref{tab:mar} in supplementary materials. Dense marker points are also a safeguard for our 3D motion data. Before performing the activity, we ask them to watch the instructional video and read the language instructions. For each action, eight males and two females will perform three times, each containing over eight repetitions. In total, we have 7200 motion sequences, where
$
7200 = 10 (\textit{people}) \times 3 (\textit{times}) \times 60 (\textit{actions}) \times 4 (\textit{motion redirection})
$.

\begin{figure*}[!t]
\begin{center}
   \includegraphics[width=0.9\linewidth]{img/figure4.png}
\end{center}
\caption{
A process of producing the rendered videos and the SMPL parameters. First, we apply the MoCap data to get the dynamic human poses in virtual scenes and render RGB videos with camera parameters. Then we use this joint information from dynamic human models to recover the human mesh in SMPL format. Finally, we combined the SMPL mesh and RGB videos to display the output.
}
\label{fig:SMPL_recovery}
\end{figure*}

\noindent \textbf{Data from Rendering Software.}
To fully utilize the 3D MoCap data, we use the rendering software Unity3D~\cite{Unity3D} to produce synthetic 2D videos with RGB color. For 2D videos, we purchase realistic scene models in Unity Asset Store~\cite{Unity_Asset}, including indoor and outdoor scenes. As well, we select six camera positions in each scene. Our camera positions are dispersed around the avatar. However, we change parameters such as the focal length of each camera to ensure that the viewfinder fits and that the camera parameters are diverse. Specifically, we are greatly appreciated that Renderpeople\cite{Renderpeople} provides several free character models. We import the skeleton information into the avatar and record the motion of the avatar in all directions. The resolution of the video is 854 × 480, and the frame rate is 30fps. Totally, we have 172,800 videos, where
$
172,800 = 1800 (\textit{mocap sequence}) \times 6 (\textit{camera position}) \times 4 (\textit{avatar}) \times 4 (\textit{virtual scences})
$.

\noindent \textbf{Data from Real-world Environment.}
To obtain versatile data resources and add diverse scenes, we ask 10 extra people to record videos in different real-world scenarios. The recording process is executed using smartphones from the front view and side view simultaneously, to ensure the diversity of shooting angles. As well, before performing the activity, volunteers are asked to watch the instructional video and read the language instructions carefully. In total, we have 7200 videos, where
$
7200 = 10 (\textit{people}) \times 3 (\textit{times}) \times 60 (\textit{actions}) \times 2 (\textit{views}) \times 2 (\textit{scenes})
$.

\subsection{The Body Model}

To facilitate different applications (\textit{e.g.,} human mesh recovery and human action generation), FLAG3D adopts the SMPL\cite{loper2015smpl} parametric model because of its ubiquity and generality in various downstream tasks.

Specifically, the SMPL parameters comprise pose parameters $\theta \in \mathbb{R}^{N\times 72}$, shape parameters $\beta \in \mathbb{R}^{N \times 10}$ and translation parameters $t \in \mathbb{R}^{N \times 3}$, where $N$ is the number of frames for each video.

We obtain the SMPL parameters based on the captured keypoints and an optimization algorithm. In particular, the optimization process is composed of two stages, where the first stage is to get the shape parameter $\beta \in \mathbb{R}^{N \times 10}$, and the second stage is to gain the pose $\theta \in \mathbb{R}^{N\times 72}$ as well as translation parameter $t \in \mathbb{R}^{N \times 3}$.

We denote the $E_s$ and $E_p$ as two objective functions for shape and pose optimization. In the first stage, the objective function is formulated as follows:
\begin{align}
    E_s(\beta)=&\frac{\lambda_1}{N}\sum_{(i,j)\in \mathcal{L}}||\bm{J}_i(\mathbb{M}(\beta))-\bm{J}_j(\mathbb{M}(\beta))-\mathcal{P}(\bm{g}_i-\bm{g}_j)||_2^2
    \nonumber
    \\
    &+
    \lambda_{2} ||\beta||_2^2.
    \nonumber
\end{align}

Here $J_i$ is the joint regressor for joint $i$, $\bm{g}$ is the ground truth skeleton, and $\mathbb{M}$ is the parametric model\cite{loper2015smpl}. $\mathcal{L}$ and $\mathcal{J}$ represent the body limbs and joint sets, respectively. $\mathcal{P}$ is the projection function that projects the $g_i-g_j$ in the direction of $J_i-J_j$. Similarly, the objective function in the second stage is as follows:
\begin{equation}
    E_p(\theta,t)=\lambda_3\frac{1}{N}\sum_{j \in \mathcal{J}}\lambda_{p1}||\bm{J}_j(\mathbb{M}(\theta,t))-\bm{g}_j||_2^2+\lambda_4||\theta||_2^2.
    \nonumber
\end{equation}

In the equations above, different weights of $\lambda_k (k=1,2,3,...) $ are denoted for each loss term (see supplementary materials for details). We adopt the L-BFGS\cite{liu1989limited} method where the search step-length satisfies the strong Wolfe conditions\cite{wolfe1969convergence} for solving this optimization problem because of its memory and time efficiency.

\section{Experiments}
\subsection{Human Action Recognition}
FLAG3D contains both RGB videos and 3D skeleton sequences, maintaining abundant resources for 2D and 3D skeleton-based action recognition. Moreover, FLAG3D also provides videos from different domains, allowing us to evaluate different models’ generalized abilities. We first report human action recognition accuracy with the 3D skeleton data captured from the MoCap system. Then we use the 2D skeleton data extracted from both rendered and real-world videos to test the transferable ability of the models.

 \noindent \textbf{Experiment Setup.}
For the in-domain evaluation, we use the 5040 skeleton sequences of 7 subjects for training, while the other 2160 skeleton sequences of 3 subjects for testing. For the cross-domain evaluation, we follow \cite{duan2022revisiting} to use the Top-Down approach for 2D pose extraction in both rendered and real-world videos. For data selection, we select the rendered videos from font and side view in one scene (7200 $\times$ 3) for training samples and take all 7200 real-world videos for testing. We evaluate five state-of-the-art methods as ST-GCN\cite{yan2018spatial}, 2s-AGCN\cite{shi2019two}, MS-G3D\cite{liu2020disentangling}, CTR-GCN\cite{chen2021channel} and PoseC3D\cite{duan2022revisiting}. We exclude PoseC3D from in-domain experiments since it only supports 2D keypoint input. Table \ref{tab:action} presents the compared results.

\input{table/cross_action_recognition}

\noindent \textbf{Result and Analysis.}
For the in-domain experiments, the Top-1 accuracy of all models is high, which shows that our 3D skeleton data is effective with recently advanced algorithms. Regarding the out-domain experiments, the accuracy drops drastically when transferring the models from rendered to real-world scenarios. On the widely used NTU RGB+D 60\cite{shahroudy2016ntu} and 120 benchmark\cite{liu2019ntu}, Top-1 accuracy achieves and 96.6\% and 89.6\% respectively with PoseC3D\cite{duan2022revisiting}, but 79.9\% only on FLAG3D. Unlike NTU RGB+D, which has a large proportion of daily actions in the indoor environment, FLAG3D focuses more on the classification of fitness actions, which requires more attention to fine-grained action distinctions. As shown in Figure \ref{fig:mean_acc}, the Top-1 accuracy of 2s-AGCN on \textit{``Bent-over Dumbbell Tricep Extension’’} and \textit{``Lying Shoulder Joint Downward Round’’} are inferior compared with other categories, the most confusing classes for \textit{``Bent-over Dumbbell Tricep Extension’’} are \textit{``Bent-over W-shape Stretch’’}, \textit{``Bent-over Y-shape Stretch’’} and \textit{``Right-side Bent-over Tricep Extension With Resistance Band’’}. Figure \ref{fig:cf_cases} shows these actions share sufficient similarities in motion patterns, such as bending over and swinging arms. As for \textit{``Lying Shoulder Joint Downward Round’’}, the counterpart - \textit{``Lying Shoulder Joint Upward Round’’} challenges the model in the aspect of temporal modeling. These categories require the models to focus on fine-grained action differences. Therefore, the FLAG3D dataset can be served as a new benchmark for out-domain and fine-grained action understanding.

\begin{figure*}[!t]
\begin{center}
   \includegraphics[width=0.95\linewidth]{img/figure6.png}
\end{center}
\vspace{-6pt}
\caption{
Examples of SMPL prediction results.
The top activity is prostrating and the bottom one is taking the knee. ROMP-ft (\textit{i.e.,} fine-tuning ROMP on the training set of FLAG3D) improves its ability to correctly estimate complex postures like kneeling.
}
\label{fig:SMPL_Results}
\end{figure*}

\begin{figure}[t]
\begin{center}
\includegraphics[width=\linewidth]{img/figure7.png}
\end{center}
\vspace{-15pt}
\caption{Case study of 2s-AGCN prediction results. The blue boxes are the selected frames of the target category, and the yellow boxes are the confusing categories. From top to bottom are \textit{``Bent-over Dumbbell Tricep Extension''}, \textit{``Right-side Bent-over Tricep Extension With Resistance Band''}, \textit{``Bent-over W-shape Stretch''} and \textit{``Bent-over Y-shape  Stretch''}.}
\label{fig:cf_cases}
\end{figure}

\subsection{Human Mesh Recovery}
FLAG3D provides the SMPL\cite{loper2015smpl} annotations, which are the prevalent ground truth in human mesh recovery. It is available to perform and evaluate popular methods for estimating 3D human poses and shapes. In this section, we first evaluate deep learning-based regression algorithms to verify that our dataset is qualified as a benchmark. Then we use the SMPL\cite{loper2015smpl} annotation data to train ROMP\cite{romp} to improve its performance on our test set.

\noindent \textbf{Experiment Setup.}
For data selection, we choose 300K frames for each scene and view to ensure the diversity of the subset. To ensure continuity and avoid information leakage (\textit{e.g.}, two videos with the same action and human model but different repetitions), the split method is to select the first 20\% videos for each scene and view as the test set. We benchmark three typical methods: VIBE\cite{VIBE}, BEV\cite{BEV}, and ROMP\cite{romp} on FLAG3D using MPJPE and PA-MPJPE metrics. Table \ref{tab:hmr} presents the compared results.

\input{table/human_mesh_recovery}

\noindent \textbf{Result and Analysis.}
These methods without training have achieved unsatisfactory MPJPE and PAMPJPE on the dataset. There are many reasons for this result (see supplementary material for more details). One of the most important reasons is that when the person in the rendered video is kneeling or lying, the task could be challenging because of the occlusion in the visual view. As displayed in Figure \ref{fig:SMPL_Results}, ROMP\cite{romp} interprets kneeling as lying and interprets taking on the knee as squatting on the ground. For the evaluation part, VIBE\cite{VIBE} and ROMP\cite{romp} achieved the best MPJPE and PA-MPJPE, respectively. But these metrics are still high, indicating that there is still much room for improvement of 3D shape estimation methods on the FLAG3D dataset. Therefore, FLAG3D can be served as a new benchmark for 3D pose and shape estimation tasks. Since ROMP\cite{romp} achieved the top-1 PA-MPJPE, we fine-tuned it on FLAG3D with HR-Net\cite{sun2019deep} backbone. We found ROMP\cite{romp} could handle challenging cases and reach better MPJPE and PA-MPJPE after being fine-tuned on our dataset. It indicates that our dataset could benefit 3D pose estimation approach to improve their performance.

\input{table/human_action_generation_lang.tex}

\begin{figure}[t]
\begin{center}
   \includegraphics[width=1\linewidth]{img/figure8.png}
\end{center}
\caption{Qualitative results in FLAG3D. \textbf{(a)} ``Squat With Arm Lift" visualization results in the category-based method. \textbf{(b)} Small Dumbbell Floor Flies: \textit{``Lie flat on your yoga mat, bend your knees, spread your legs shoulder-width apart, and keep your feet firmly planted on the ground. Sink your shoulder blades so that your upper back is flat against the mat..."} At first, the avatar bent the knee correctly, however, it fails to faithfully follow the text description as time goes on. \textbf{(c)} Chest Fly: \textit{``Raise your head, keep your chest out, and tighten your abdominal muscles. Keep your palms forward and open your arms alternately at the same time..."} When comes to the word ``alternately", Guo \textit{et al.}\cite{HumanML3D} fails to capture the semantic information and comes to a standstill. ( Due to the different forms of data organization in different methods. In Guo \textit{et al.}\cite{HumanML3D}, we use skeletons to demonstrate effects. )}
\label{fig:generation_result}
\end{figure}

\subsection{Human Action Generation}
Detailed language instructions and 3D skeleton-based motion sequences with SMPL\cite{loper2015smpl} annotations are included in FLAG3D, which facilitate the application of human action generation. This section reports the results of action-conditioned 3D human motion synthesis under both category-based settings and language-based settings.

\noindent \textbf{Experiment Setup.}
For category-based action generation, we test the results in ACTOR\cite{MathisPetrovich2021ActionConditioned3H}. We first selected skeleton sequences of 5 subjects for training, while the other 5 subjects were for testing. Same as UESTC~\cite{YanliJi2018ALR} in ACTOR~\cite{MathisPetrovich2021ActionConditioned3H}, we use ST-GCN~\cite{yan2018spatial} as the feature extractor. For language-based action generation, we evaluate the methods of Guo \textit{et al.}\cite{HumanML3D} and TEMOS\cite{petrovich22temos}. We use 90\% language-motion pairs for training and 10\% for testing.

\noindent \textbf{Result and Analysis.}
Owing to the different architectural designs of algorithms, we carry over the metrics set of the original paper in each method. Results are shown in Table~\ref{tab:hag}. \textit{For category-based settings}, the FID of FLAG3D is 14.77, and the Multimod index is 6.53. In some cases, ACTOR~\cite{MathisPetrovich2021ActionConditioned3H} provides satisfactory results as shown in Figure~\ref{fig:generation_result} (a). \textit{For language-based settings}, due to the different complexity of actions in FLAG3D, the action durations are relatively long and they do not satisfy a uniform distribution. As shown in Figure~\ref{fig:generation_result} (b), TEMOS~\cite{petrovich22temos} appears to be visually plausible and context-aware at the beginning. However, it fails to follow the text description faithfully as time goes on. More structure about characterization in temporal dependencies should be designed. Guo \textit{et al.}\cite{HumanML3D} designed a list to record keywords such as body parts and movements. So that the model could focus on specific words. FLAG3D is semantically informative and has many professional descriptions. As shown in Figure~\ref{fig:generation_result} (c), Guo \textit{et al.}\cite{HumanML3D} cannot capture the information of the word ``alternately" after the execution of movement ``keep your palms forward". These cases require models of more generalization so that they could suffer from out-of-distribution descriptions.
\section{Future Works and Discussion}
\label{sec:application}

Based on the high-quality and versatile data resources of FLAG3D, some other potential directions could be further explored when different kinds annotations are included. We discuss some of them as below:

\noindent \textbf{Visual Grounding}.
The language instructions of FLAG3D involve the critical steps of specific body parts to accomplish an activity. Grounding these key phases with the corresponding spatial-temporal regions of the untrimmed long video could better bridge the domain gap between linguistic and visual inputs and promote future research on interpretable activity understanding.

\noindent \textbf{Repetitive Action Counting}.
For an untrimmed long video, counting the occurring times of the repetitive actions benefits users for fitness training~\cite{aifit,DBLP:conf/cvpr/HuDZLLG22}. While this task requires more fine-grained level annotations of the temporal boundary, it is desirable to explore unsupervised or semi-supervised learning methods along this direction.

\noindent \textbf{Action Quality Assessment}.
This task aims to assess how well a specific fitness activity is performed and give feedback to users to avoid injury and improve the training effect. Unlike previous works that predict a score for each action~\cite{tang2020uncertainty,xu2022finediving,parmar2017learning}, future effort could be devoted to our FLAG3D by evaluating whether a 3D activity meets the rule described by the language instruction.

\section{Conclusion}
In this paper, we have proposed FLAG3D, a large-scale comprehensive 3D fitness activity dataset which shares the merits over previous datasets from various aspects, including highly accurate skeleton, fine-grained language description, and diverse resources. Both qualitative and quantitative experimental results have shown that FLAG3D poses new challenges for multiple tasks like cross-domain human action recognition, dynamic human mesh recovery, and language-guided human action generation. We hope the FLAG3D will promote in-depth research and more applications on fitness activity analytics for the community.
\section{Comparision With More Datasets}
To make the main body concise and focused, we did not display numerous relevant datasets, especially for some image-based datasets. Thus, in this section, we provide a more comprehensive comparison for FLAG3D in Table \ref{tab:comparision_2}.

\input{table/supp_comparision}
\section{FLAG3D Dataset}
Figure \ref{fig:sup1} displays more examples from our dataset. From left to right, the MoCap data, rendered RGB image, and RGB image with SMPL fitting are shown, respectively. From top to bottom, we demonstrate several frames from FLAG3D Dataset. They represent action \textit{``Svend Press''}, \textit{``Standing Straight-arm Chest Press With Resistance Band ''},\textit{``Jumping Jacks''},\textit{``Kneeling Left Knee Lift''},\textit{``Left-side Knee Raise And Abdominal Muscles Contract ''} and \textit{``Prone Press Up With Torso Rotation''} respectively.

\section{Labeling System}
There are 60 kinds of fitness actions in FLAG3D. Selected activities exercise most parts of our body, including the chest, back, shoulder, arm, neck, abdomen, waist, hip, and leg. For example, there are six kinds of actions: \textit{``Svend Press''}, \textit{``Keeling Push-ups''},\textit{``Standing Straight-arm Chest Press With Resistance Band''}, \textit{``Small Dumbell Floor Flies''}, \textit{``Chest Fly''}, and \textit{`` Push-ups that exercise chest muscles''}. As shown in Table \ref{tab:acc}, we label each action from A001 to A060.
\section{Additional Details of Data Collection}
\subsection{MoCap Marker}
Volunteers wear special MoCap apparel. MoCap costumes have dense markers to ensure that all parts of the body are accurately recorded. Marker numbers for body parts are listed in Table ~\ref{tab:mar}. Totally, there are 77 markers on the volunteer body to ensure that our dataset could provide accurate 3D skeleton information.
\input{table/mocap_marker.tex}
\subsection{Hardware}
We use the VICON\cite{vicon} optical motion capture system to capture the character’s motion, which consists of 24 VICON V16 cameras. Our system works with Vicon’s powerful software platforms to enable a light touch while we are running a capture session. This MoCap system is highly robust, it can perform bump detection while capturing motions. Bump detection means that the system knows when a camera needs recalibrating and auto-heals without operator input. MoCap system’s technology IP advances allow Vantage+ to hit resolution and speed sweet spots.
Specific parameters of the camera are shown in Table ~\ref{tab:vicon}.
\input{table/supp_vicon.tex}

\subsection{Camera Position}
In Figure \ref{fig:ge}, we show the camera settings for rendered videos and natural videos. We put 6 cameras around the avatars to obtain the multi-view videos and Figure \ref{fig:ge}(a) displays the arrangement. And for natural videos, we require volunteers to shoot their videos from two views. One is the front view and another is any front side view as shown in Figure \ref{fig:ge}(b). The red dotted line represents we only need one camera in the side view.
\begin{figure*}[htbp]
\begin{center}
    \includegraphics[width=0.85\linewidth]{img/figure9.png}
\end{center}
\caption{Camera Position Setting. In rendering software, the camera positions are settled as shown in figure (a). The avatar is surrounded by six cameras in a circle to allow for multi-angle video. In the nature scene, we selected two camera positions as shown in figure (b). The red dotted line represents a random selection of one of these perspectives. }
\vspace{-6pt}
\label{fig:ge}
\end{figure*}

\section{Details of Experiments}

\subsection{Action Recoginition}
For in-domain experiments, we employ an Adam\cite{kingma2014adam} optimizer for 30 epochs using a cosine decay learning rate. A batch size of 64 is used. Specifically, the initial learning rate and weight decay are set to 0.1 and 5e-4, respectively. For out-domain experiments, we use a larger batch size of 128. The learning rate, weight decay, optimizer and scheduler are the same as that for in-domain experiments. Unless otherwise mentioned, for all models, we sample 500 frames uniformly from the entire clip and only adopt the joint stream. We take the 10-clip testing results to report our Top-1 accuracy.
\subsection{Human Mesh Recovery}
For the optimization process to obtain the SMPL mesh, the loss weights $\lambda_1$, $\lambda_2$, $\lambda_3$, $\lambda_4$ in the objective function are $1$, $5\times 10^{-3}$, $1$, $1\times 10^{-3}$ respectively. We verify the SMPL parameter registration process using the MPJPE and PA-MPJPE and their values are 56.09 and 27.02 respectively.

We used the pre-trained HR-Net\cite{sun2019hrnet} backbone to fine-tune the ROMP method on FLAG3D Dataset. We employ an Adam\cite{kingma2014adam} optimizer. An initial learning rate of 5e-5 and a batch size of 64 are used. We use the 3D SMPL joints\cite{loper2015smpl}, 2D COCO-17 keypoints\cite{lin2014microsoft} obtained from HR-Net\cite{sun2019hrnet} method and shape parameters to supervise the training.
\section{Analysis of Evaluation}
A qualitative comparison of different methods on FLAG3D is shown in Figure \ref{fig:sup2}. As we can see in this picture when the human is lying or kneeling the result will be dissatisfactory. We have mentioned this phenomenon in the body part. In this section, we will explore more about it. The first question is, \textit{are these results caused by the reason that the person is too tiny in the picture to recognize or the reason that the action is too complex to recover an accurate pose?} The answer is both but the latter more matters. We crop the original picture into 1/2 and 1/4 to make a larger character in the image and then evaluate methods on it. In Figure \ref{fig:sup2}, we showed the inference results from different methods on three cropped pictures. The results indicate that for the original image with the tiny person, the challenge lies in recognizing the right person in the picture. But for the cropped image because the person is no longer tiny, some algorithms could recognize the person in the right location, which is an improvement. Because we find algorithms could not recognize the person in many cases for original pictures with default threshold $0.2$ and we often have to reduce it to let the algorithm work but the side effect is the method will often misrecognize the background as a person or recognize the person which is not in the proper location(see examples in Figure \ref{fig:sup2} row 4). Therefore, for the cropped pictures, the challenge is to understand the right posture in this case. But there is still a large distance to the acceptable result. From Figure \ref{fig:sup2}, BEV\cite{BEV} and ROMP\cite{romp} indeed estimate the lying posture of one body but can not infer the poses of the limbs accurately. Even if the algorithm could recognize the right person in the right location, it is also very challenging to recover some complex activities.

The second question is,\textit{ what kind of action is difficult to recover?} So we explore the effects of different movements on human mesh recovery based on FLAG3D. We compare the results trying to estimate three activities in Figure \ref{fig:sup2}. To control the unrelevant variable, we choose three actions from the same person, same scene and same view. From left to right, the Figure \ref{fig:sup2} displays the results from standing to lying. We find as people’s center of gravity dropping, the task is more challenging because the visible part of the whole body is less and less. From the visual results of the experiment, standing is easy to recover pose than kneeling and kneeling is easy than lying. The character’s size in column 2 and column 3 is approximately the same but the methods’ performance on these are distinct, which confirms that complex activities are the main challenge for human mesh recovery task.

Another reason that these methods can not deal with hard cases is they never learn about these situations because past datasets hardly include these actions about fitting. So our dataset could serve as a difficult and long-term benchmark for human mesh recovery task.
\input{table/supp_label.tex}

\begin{figure*}[htbp]
\begin{center}
   \includegraphics[width=1\linewidth]{img/figure10.png}
\end{center}
\vspace{-6pt}
\caption{
Several frames from FLAG3D Dataset. From left to right we display the MoCap data, Original Rendered RGB Videos, and Rendered RGB Videos with SMPL Mesh fitting results.
}
\label{fig:sup1}
\end{figure*}

\begin{figure*}[hbtp]
\begin{center}
   \includegraphics[width=1\linewidth]{img/figure11.png}
\end{center}
\vspace{-6pt}
\caption{
Results from different methods on three cropped pictures. From top to bottom: Original Picture(row 1), VIBE\cite{VIBE}(row 2), ROMP\cite{romp}(row 3), BEV\cite{BEV} (row 4). From left to right: Original Image, Cropped Image 1(down to a half of the original), Cropped Image 2(down to 1/4 of the original).
}
\label{fig:sup2}
\end{figure*}

\begin{figure*}[hbtp]
\begin{center}
   \includegraphics[width=1\linewidth]{img/figure12.png}
\end{center}
\vspace{-6pt}
\caption{
From left to right the activities are ``\textit{Bent-over W-shape Stretch}'',``\textit{Kneeling Right Arm Raise}'' and ``\textit{Lying Shoulder Joint Upward Round}'' respectively. For actions in column 1, each method could estimate the shape and pose properly. For actions in column 2, there are some biases for recovering the posture. And for actions in column 3, they all can not handle situations.
From top to bottom:Original Picture(row 1), VIBE\cite{VIBE}(row 2), ROMP\cite{romp}(row 3), BEV\cite{BEV} (row 4).
}
\label{fig:sup3}
\end{figure*}


\section{Introduction}
\vspace{-1.5mm}
Optical fiber nonlinearity significantly limits the information rate in current coherent transmission systems. Moreover, with the ever-increasing transmission bandwidth, nonlinearity becomes even more important \cite{xu2021information}. Various digital signal processing (DSP) techniques have been proposed to minimize nonlinear effects \cite{cartledge2017digital}. Due to the universal approximation capability of neural networks (NNs), the NNs have recently been intensively studied for the optical channel post-equalization, because they can approximate the inverse optical channel transfer function with good accuracy and revert the nonlinear distortions.
In particular, recurrent NNs (RNN) based equalizers have shown the best capability in equalizing nonlinear impairments as compared to the feed-forward NN types \cite{Pedro2021,deligiannidis2020compensation, deligiannidis2021performance}. However, since the RNN structure has a feedback loop, it is not easily parallelizable. This characteristic makes it challenging to implement the RNNs in low-complexity hardware for high-speed processing \cite{chang2017hardware}. Indeed, parallelization is the key to achieving low latency and, simultaneously, high throughput required by high-speed optical networks, as parallel computing increases the throughput and reduces the computational time \cite{robey2021parallel}.
To enable parallelization, in this work, we adopt the concept of knowledge distillation (KD) to transform the RNN-based equalizer into a feedforward structure. Generally, KD is a method to transfer the knowledge from a larger model, known as a \emph{teacher} model, to a more compact one (a \emph{student}), which requires fewer computations \cite{hinton2015distilling}. Most of the prior works using KD have focused on classification tasks where the teacher and student have a similar topology. However, KD for a regression task and the cross-architecture KD, as we propose in our work, have only been studied very recently \cite{xu2022contrastive}. In this work, we also reduce the computational complexity (CC) per recovered symbol by recovering a multi-symbol output \cite{sang2022low}.

In this paper, for the first time, we present a workaround to solve the RNN parallelization issue, by using KD. The KD, in our case, is used to transfer the knowledge from a recurrent teacher model which is a bidirectional long-short term memory (biLSTM) coupled with 1D-convolutional NN (1D-CNN), the efficient equalizer model proposed in \cite{freire2022reducing}, to a feedforward student model: a 1D-CNN, shown in Fig.~\ref{fig:parallelization}(a). Furthermore, we point out that together with the realization of a parallelizable structure, we reduce the inference latency of the model obtained via KD by 38\%. Finally, we also observed a slight 0.5~dB reduction in Q-factor at optimum power.

\vspace{-1.5mm}
\section{Parallelization of Recurrent Equalizers via Knowledge Distillation}
\vspace{-1.5mm}
The recurrent nature of biLSTM, which takes into account the input of the current stage and the output of the previous stage, makes it very useful for learning sequential data. Furthermore, the input flowing in both directions further enhances the biLSTM learning. Due to the sequential processing of the double-recurrent setting and, consequently, limited parallelizability, the RNN-based model is not easily implementable for high-speed transmission. On the other hand, 1D-CNNs are feedforward-based NNs where the input temporal sequential batches are processed independently, hence, parallel operations are possible. Fig.~\ref{fig:parallelization}(b) illustrates the recurrent structure at the top with a feedback loop, preventing parallelization, while the feedforward structure at the bottom can process multiple sets of inputs and provide multiple outputs simultaneously. Transforming the model architecture from the biLSTM to 1D-convolutional layers enables parallel computation for the previously proposed recurrent structure-based equalizer (NNE) or the teacher model.
\begin{figure}[htbp]
\centering
  \includegraphics[scale=0.30]{fig_all.pdf}
  \captionof{figure}{(a) KD framework with biLSTM+1D-CNN as a teacher model and dilated 1D-CNN as a student model; (b) Illustration of the parallelizability comparison between a recurrent cell (top)
  and a 1D-CNN (bottom).}
  \label{fig:parallelization}
  \vspace{-6mm}
\end{figure}

Fig.~\ref{fig:parallelization}(a) shows the teacher and student model structures and the KD process. The teacher NN contains a biLSTM layer with 100 hidden units and a linear 1D-CNN layer with two filters and a kernel size of 51. The student model has a dilated convolutional structure, which allows the NN to deal with long-term temporal dependencies and have large receptive fields within only a few layers \cite{oord2016wavenet,xu2022contrastive}. The last 1D-CNN layer of both the teacher and the student has the same parameters. To mimic biLSTM, which learns the input data in forward and backward directions, the student model learns both directions of the training data. The backward direction input means the input sequence (forward direction) in reverse time order. The Bayesian Optimizer (BO) \cite{Pedro2021} is used to optimize the hyperparameter values of the student model. The estimated optimal values are depicted in Fig.~\ref{fig:parallelization}(a). Note that (38, 23, 1) means that the 1D-CNN layer operates with 38 filters, a kernel size of 23, and a dilation rate of 1. The activation function of the dilated 1D-CNN part is LeakyRelu. The KD loss approach (Eq.~(\ref{eq.kdloss})) for the student model training implies a joint loss function that takes into account both the loss between the student's and teacher's predictions and the loss between the student's predictions and ``ground truth'':
\vspace{-1mm}
\begin{equation}\label{eq.kdloss}
\begin{gathered}
L_{\text{KD}}= \alpha L_1(\boldsymbol{\hat{y}}_S,\boldsymbol{\hat{y}}_\text{T}) + (1-\alpha) L_2(\boldsymbol{\hat{y}}_S,\boldsymbol{y}_\text{true}),
    \end{gathered}
\vspace{-1mm}
\end{equation}
where $\boldsymbol{y}_\text{true}$ is the ground-truth labels, $\boldsymbol{\hat{y}}_S$ and $\boldsymbol{\hat{y}}_\text{T}$ represent the student's and the teacher's predictions, respectively, and $\alpha$ is the hyperparameter balancing the contribution of each term to the final loss. The teacher model is pre-trained and used only to create teacher labels. The student model's training with KD is carried out for 1000 epochs, mini-batch size of $2^{10}$; the learning rate and $\alpha$ found by BO are equal to 0.00026 and 0.903, respectively. The dataset is obtained by numerical simulation, assuming the transmission of a single 30~GBd, 64-QAM  dual-polarization channel along $20\times50$~km standard single mode fiber (SSMF) spans. At the end of each fiber span, optical fiber losses are compensated for by an Erbium-Doped Fiber Amplifier (EDFA) with a 4.5~dB noise figure.

Typically, the NNEs proposed in previous works are designed to recover one symbol at a time \cite{Pedro2021}. However, the single-symbol output NNEs can be computationally inefficient, as the weights and biases trained to recover one symbol may still be useful for recovering multiple symbols \cite{sang2022low}.
Therefore, we focus on multi-symbol output equalizers to reduce the CC per recovered symbol. The last layer of our model adopts the 1D-CNN layer containing two filters with a linear activation function to recover both real and imaginary parts of the signal. A set of 221 input symbols is fed to the NN simultaneously, to recover 171 symbols at the output at each inference step.

\vspace{-1.5mm}
\section{Results and Discussion}
\vspace{-1.5mm}
We compare our proposed student model (1D-CNN) using the KD framework with the teacher model (biLSTM +CNN) pre-trained as in \cite{freire2022reducing}, the student model trained from scratch (without KD) with exactly the same settings, and the student model trained from scratch with L2 regularizer \cite{ng2004feature}. The optimum L2 coefficient depends on the launch power. At 2~dBm launch power, the optimum L2 coefficient found by grid search is $10^{-4}$. Fig.~\ref{fig:results2}(a) depicts Q-factor vs. launch power for different types of NNEs. We can observe that the Q-factor performance of the feedforward student model with KD drops by 0.5~dB compared to the recurrent teacher model at its optimal launch power (2~dBm). With KD, the performance of the student model is comparable to that of the teacher model in the linear transmission regime, but the student's performance degrades slightly as the launch power increases. However, when training the student model from scratch without KD, the model suffers from overfitting, resulting in a noticeable degradation of the peak performance by 2.4~dB at its optimal power. Training the student model with the L2 regularizer, which helps enhance the generalization capability, improves the performance of the NN compared to training it from scratch only, but still does not reach a similar performance level as the one achieved when the student model is trained with KD. The performance achieved using digital back propagation (DBP) 1 step/span (STpS) and chromatic dispersion compensation (CDC) are also shown for reference.
\begin{figure*}[ht!]
  \vspace{-4mm}
\begin{subfigure}{.32\textwidth}
    \centering
 \begin{tikzpicture}[scale=0.58]
    \begin{axis} [
        xlabel={Launch power [dBm]},
        ylabel={Q-Factor [dB]},
        grid=both,
        xmin=-3, xmax=5,
    	xtick={-3, ..., 5},
    	ymin=1.7, ymax=11,
        legend style={legend pos=south west, legend cell align=left,fill=white, fill opacity=0.6, draw opacity=1,text opacity=1},
    	grid style={dashed}]
        ]
    \addplot[color=blue, mark=square, very thick]
    coordinates {
    (-3, 7.88)(-2, 8.69)(-1, 9.49)(0, 10.1)(1,10.5)(2,10.66)(3,10.37)(4,9.7)(5,8.6)
    };
    \addlegendentry{Teacher model (biLSTM+CNN)};

    \addplot[color=red, mark=*, very thick]
    coordinates {
    (-3, 7.856)(-2, 8.66)(-1,9.35)(0, 9.88)(1, 10.196)(2, 10.19)(3, 9.6)(4, 8.6)(5, 7.5)
    };
    \addlegendentry{Student model with KD (1D CNN)};
        \addplot[color=green, mark=o, very thick, dash pattern={on 7pt off 2pt}]
    coordinates {

    (-3,7.01)(-2,7.30)(-1, 7.52)(0, 7.87)(1,8.12)(2,8.3)(3, 8.23)(4, 7.61)(5, 6.52)
    };
    \addlegendentry{Student model trained from scrath};

    \addplot[color=cyan, mark=x, very thick, dash pattern={on 7pt off 2pt}]
    coordinates {
    (-3,7.44)(-2, 8.00)(-1, 8.57)(0, 8.95)(1, 9.19)(2,9.04)(3,8.80)(4, 8.01)(5, 6.83)
    };
    \addlegendentry{Student model with L2 reg.};

    \addplot[color=orange, mark=+,very thick, dotted]    coordinates {
    (-3,  7.79)(-2, 8.46)(-1, 8.96)(0, 9.2)(1,9.1)(2,8.6)(3,7.7)(4,6.5)(5,5.1)
    };
    \addlegendentry{DBP 1 STpS};

    \addplot[color=black,mark=triangle, very thick, dotted]    coordinates {
    (-3, 7.4)(-2, 7.7)(-1, 7.8)(0, 7.3)(1,6.6)(2,5.6)(3,4.4)(4,3)(5,1.7)
    };
    \addlegendentry{CDC (Regular DSP)};

    \end{axis}
    \draw[ <->, red] (2, 5.18) -- (2,5.51);
    \draw [dashed, red] (1.7,5.18) -- (5.2,5.18);
    \draw [dashed, red] (1.7,5.51) -- (5.2,5.51);
    \node[text width=1cm, red] at (0.95,5.35)
    {0.5dB};
    \end{tikzpicture}
\end{subfigure}
\hfill
\begin{subfigure}{.31\textwidth}
    \centering
 \begin{tikzpicture}[scale=0.572]
     \begin{axis}[
    ybar=5,
    symbolic x coords={CPU, GPU},
    bar width=0.4cm,
    enlarge x limits=0.4,
        ymin=0.0002,
    xtick=data,
    ymajorgrids = true,
    ylabel style={align=center},
    ylabel = {Inference time per recovered symbol (sec)\\Result obtained on Google Colab} ,
    xlabel = {Inference engine},
    legend style={nodes={scale=1, transform shape},legend pos=north east, legend cell align=left,fill=white, fill opacity=0.6, draw opacity=1,text opacity=1},

    ]
      \addplot[style={blue,fill=blue,mark=none}]
            coordinates {(CPU, 0.000406) (GPU, 0.0003561)};
    \addplot[style={red,fill=red,mark=none}]
          coordinates {(CPU, 0.0003356) (GPU, 0.000221)};
    \draw[ <->, black] (7.5, 135) -- (7.5,206);
    \draw [dashed, black] (-20,135) -- (20,135);
    \draw [dashed, black] (-20,206) -- (20,206);
    \node[text width=1cm, black, thick] at (27,170)
    {17\%};

    \draw[ <->, black] (107.5, 21) -- (107.5,156);
    \draw [dashed, black] (80,21) -- (120,21);
    \draw [dashed, black] (80,156) -- (120,156);
    \node[text width=1cm, black, thick] at (125,83)
    {38\%};

    \draw[>=latex, thick, <->, red] (32, 165) -- (82,83);
        \node[scale=1.5, red, thick] at (47,110)
    {2.3X};
    \legend {Teacher model, Student model with KD};

  \end{axis}
    \end{tikzpicture}

\end{subfigure}\hfill
\includegraphics[scale=0.28]{weight_dis.pdf}
  \put(-445, 112){\textcolor{black}{(a)}}
  \put(-290, 112){\textcolor{black}{(b)}}
  \put(-158, 112){\textcolor{black}{(c)}}
\vspace{-2mm}
  \caption{(a) Q-factor as a function of the launch power for the NNE obtained via KD, compared to the original (teacher) model, CDC, and 1~STpS DBP; (b) Inference time of teacher/student models; (c) Weight distribution of student model trained with different approaches.}
  \label{fig:results2}
  \vspace{-6mm}
\end{figure*}

Although the teacher model leads to slightly better optical performance than the student model with KD, Fig.~\ref{fig:results2}(b) shows a reduction in the inference time per recovered symbol of the student model with KD of 17\% and 38\% when using the CPU and GPU, respectively. Note that both the teacher and the student models recover 171 symbols per inference step. Using a GPU leads to a more significant reduction in latency because of its better parallel-computing ability, which enables it to better exploit the parallelizability of the student feedforward structure.
The parallelization of the proposed feedforward equalizer and its savings in latency are key to the real-time hardware implementation of NNEs.

Now, we study the features associated with the KD-trained model. For this purpose, we also report the weight distribution of the student model trained with different approaches in  Fig.~\ref{fig:results2}(c). Compared to the student model trained from scratch, the student model with KD has a more regularized weight distribution: the weights are more concentrated around zero. This characteristic helps reduce the model's variance and overfitting. The optimal value of $\alpha$ in the KD loss function is 0.903, which means that the student model learns 90.3\% from the teacher labels and the rest comes from the ground-truth labels.
This fact demonstrates the effectiveness of the teacher labels in the student's learning. The teacher constellation/labels depicted in Fig.~\ref{fig:parallelization}(a) show that the teacher also provides helpful information on the noise, whereas this information cannot be encoded in the ground-truth labels (which contain only real values). The weight distribution of the student model with KD and the improvement in Q-factor, compared to the training of the 1D-CNN without KD, both support the concept of using teacher labels as efficient regularizers \cite{hinton2015distilling}.

\vspace{-1.5mm}
\section{Conclusions}
\vspace{-1mm}
In this work, for the first time, the knowledge distillation technique has been proposed as an efficient tool to achieve the parallelizability of the recurrent equalizers: in our study, KD transfers the knowledge from a recurrent-connection-based biLSTM equalizer to a parallelizable feed-forward 1D-CNN. This approach enables the parallelization of signal processing, allowing us to essentially simplify the hardware implementation of NN models. We also show that the proposed feedforward equalizer obtained with KD, results in a reduced signal processing latency by 38\% compared to the original biLSTM model, at the expense of slightly reducing the maximum Q-factor by 0.5~dB.\\

\vspace{-1.5mm}
\footnotesize
\linespread{0.0}
\textbf{Acknowledgements}: This work has received funding from the EU Horizon 2020 program under the Marie Sk\l{}odowska-Curie grant agreement No.~956713 (MENTOR) and 813144 (REAL-NET). SKT acknowledges the support of the EPSRC project TRANSNET (EP/R035342/1).
\normalsize
\linespread{1.0}
\vspace{-4.2mm}