\section{Introduction}
As one of the widely used 3D shape representations, point cloud can be easily obtained through depth cameras or other 3D scanning devices. Due to the limitations of view-angles or occlusions of 3D scanning devices, the raw point clouds are usually sparse and incomplete \cite{wen2020sa}. Therefore, a shape completion/consolidation process is usually required to generate the missing regions of 3D shape for the downstream 3D computer vision applications like classification \cite{han20193dviewgraph,han2019parts,wen2020point2spatialcapsule,han2019seqviews2seqlabels,liu2019sequence,liu2020lrc}, segmentation \cite{Wen2020MM,jiang2020pointgroup} and other visual analysis \cite{yuan2021survey}.

In this paper, we focus on the completion task for 3D objects represented by point clouds, where the missing parts are caused by self-occlusion due to the view angle of scanner. Most of the previous methods formulate the point cloud completion as a point cloud generation problem \cite{dai2017shape,wen2020sa,yuan2018pcn,tchapmi2019topnet}, where an encoder-decoder framework is usually adopted to extract a latent code from the input incomplete point cloud, and decode the extracted latent code into a complete point cloud. Benefiting from the deep learning based point cloud learning methods, the point cloud completion methods along this line have made huge progress in the last few years \cite{wen2020sa,tchapmi2019topnet}. However, the generation of point clouds remains a difficult task using deep neural network, because the unordered nature of point clouds makes the generative model difficult to capture the detailed topology or structure among discrete points \cite{tchapmi2019topnet}. Therefore, the completion quality of point clouds based on generative models is still unsatisfying.

\begin{figure}[!t]
  \centering
  \includegraphics[width=\columnwidth]{teaser.pdf}\vspace{-0.4cm}
  \caption{Illustration of completion by multi-step point cloud deformation. The point moving paths are denoted by grey lines. At each step, the source point cloud is colored by orange and the target point cloud is colored by yellow, respectively.
  }
  \label{fig:searching_illustration}
\end{figure}

To address this issue, in this paper, we propose a novel neural network, named PMP-Net, to formulate the task of point cloud completion from a new perspective. Different from the generative model that directly predicts the coordinations of all points in 3D space, the PMP-Net learns to move the points from the source 3D shape to the target one.
Through the point moving process, the PMP-Net establishes the point-level correspondences between the source point cloud and the target, which captures the detailed topology and structure relationships between the two point clouds. On the other hand, there are various solutions to move points from source to target, which will confuse the network during training.
Therefore, in order to encourage the network to learn a unique arrangement of point moving path, we take the inspiration from the Earth Mover's Distance (EMD) and  propose to regularize a Point-Moving-Path Network (PMP-Net) under the constraint of the total point moving distances (PMDs), which guarantees the uniqueness of path arrangement between the source point cloud and the target one.

\begin{figure}[!t]
  \centering
  \includegraphics[width=\columnwidth]{searching_illustration.pdf}\vspace{-0.1cm}
  \caption{Illustration of path searching with multiple steps under the coarse-to-fine searching radius. The PMP-Net moves point A to point A' by three steps, with each step reducing its searching radius, and looking back to consider the moving history in order to decide the next place to move.
  }
  \label{fig:searching_illustration}
\end{figure}

Moreover, in order to predict the point moving path more accurately, we propose a multi-step path searching strategy to continuously refine the point moving path under multi-scaled searching radius. Specifically, as shown in Figure \ref{fig:searching_illustration}, the path searching is repeated for multiple steps in a coarse-to-fine manner. Each step will take the previously predicted paths into consideration, and then plan its next move path according to the previous paths. To record and aggregate the history information of point moving path, we take the inspiration from Gated Recurrent Unit (GRU) to propose a novel Recurrent Path Aggregation (RPA) module. It can memorize and aggregate the route sequence for each point, and combine the previous information with the current location of point to predict the direction and the length for the next move. By reducing the searching radius step-by-step, PMP-Net can consistently refine a more and more accurate path for each point to move from its original position on the incomplete point cloud to the target position on the complete point cloud.
In all, the main contribution of our work can be summarized as follows.
\begin{itemize}
  \item We propose a novel network for point cloud completion task, named PMP-Net, to move each point from the incomplete shape to the complete one to achieve a high quality point cloud completion. Compared with previous generative completion methods, PMP-Net has the ability to learn more detailed topology and structure relationships between incomplete shapes and complete ones, by learning the point-level correspondence through point moving path prediction.
  \item We propose to learn a unique point moving path arrangement between input and output point clouds, by regularizing the network using the constraint of Earth Mover's Distance. As a result, the network will not be confused by multiple solutions of moving points, and finally predicts a meaningful point-wise correspondence between the source and target point clouds.
  \item We propose to search point moving path with multiple steps in a coarse-to-fine manner. Each step will decide the next move based on the aggregated information from the previous paths and its current location, by using the proposed Recurrent Path Aggregation (RPA) module.
\end{itemize}

\section{Related Work}
The deep learning technology in 3D reconstruction \cite{Han2020TIP,Han2020Sdrwr,Han2020ECCV,Jiang2019SDFDiffDRcvpr,Han2020Sdrwr} and representation learning \cite{9318534,han2019multi,liu2019l2g,han20193d2seqviews} have boosted the research of 3D shape completion, which can be roughly divided into two categories. (1) Traditional 3D shape completion methods \cite{sung2015data,berger2014state,thanh2016field,wei2019local} usually formulate hand-crafted features such as surface smoothness or symmetry axes to infer the missing regions, while some other methods \cite{shao2012interactive,kalogerakis2012probabilistic,martinovic2013bayesian,shen2012structure} consider the aid of large-scale complete 3D shape datasets, and perform searching to find the similar patches to fill the incomplete regions of 3D shapes. (2) Deep learning based methods \cite{zhang2020detail,sarmad2019rl,huang2020pf,hutaoaaai2020}, on the other hand, exploit the powerful representation learning ability to extract geometric features from the incomplete input shapes, and directly infer the complete shape according to the extracted features. Those learnable methods do not require the predefined hand-crafted features in contrast with traditional completion methods, and can better utilize the abundant shape information lying in the large-scale completion datasets. The proposed PMP-Net also belongs to the deep learning based method, where the methods along this line can be further categorized and detailed as below.

\noindent\textbf{Volumetric aided shape completion.}
The representation learning ability of convolutional neural network (CNN) has been widely used in 2D computer vision research, and the studies concerning application of 2D image inpainting have been continuously surging in recent years. A intuitive idea for 3D shape completion can be directly borrowed from the success of 2D CNN in image inpainting research \cite{yu2018generative,yeh2017semantic,liu2018image}, extending it into 3D space. Recently, several volumetric aided shape completion methods, which are based on 3D CNN structure, have been developed. Note that we use the term \emph{``volumetric aided''} to describe this kind of methods, because the 3D voxel is usually not the final output of the network. Instead, the predicted voxel will be further refined and converted into other representations like mesh \cite{dai2017shape} or point cloud \cite{xie2020grnet}, in order to produce more detailed 3D shapes. Therefore, the voxel is more like an intermediate aid to help the completion network infer the complete shape. Notable works along this line like 3D-EPN \cite{dai2017shape} and GRNet \cite{xie2020grnet} have been proposed to reconstruct the complete 3D voxel in a coarse-to-fine manner. They first predict a coarse complete shape using 3D CNN under an encoder-decoder framework, and then refine the output using similar patches selected from a complete shape dataset \cite{dai2017shape} or by further reconstructing the detailed point cloud according to the output voxel \cite{xie2020grnet}. Also, there are some studies that consider purely volumetric data for shape completion task. For example, Han et. al \cite{han2017high} proposed to directly generate the high-resolution 3D volumetric shape, by simultaneously inferring global structure and local geometries to predict the detailed complete shape. Stutz et. al \cite{stutz2018learning} proposed a variational auto-encoder based method to complete the 3D voxel under weak supervision. Despite the fascinating ability of 3D CNN for feature learning, the computational cost which is cubic to the resolution of input voxel data makes it difficult to process fine-grained shapes \cite{wen2020sa}.

\noindent\textbf{Point cloud based shape completion.}
There is a growing attention on the task of point cloud based shape completion \cite{xin2021c4c,tchapmi2019topnet,wen2020sa,hu2019render4completion} in recent years. Since point cloud is a direct output form of many 3D scanning devices, and the storage and process of point clouds require much less computational cost than volumetric data, many recent studies consider to perform direct completion on 3D point clouds. Enlighten from the improvement of point cloud representation learning \cite{qi2017pointnet++,qi2017pointnet}, previous methods like TopNet \cite{tchapmi2019topnet}, PCN \cite{yuan2018pcn} and SA-Net \cite{wen2020sa} formulate the solution as a generative model under an encoder-decoder framework. They adopted encoder like PointNet \cite{qi2017pointnet} or PointNet++ \cite{qi2017pointnet++} to extract the global feature from the incomplete point cloud, and use a decoder to infer the complete point cloud according to the extracted features. Compare to PCN \cite{yuan2018pcn}, TopNet \cite{tchapmi2019topnet} improved the structure of decoder in order to implicitly model and generate point cloud in a rooted tree architecture \cite{tchapmi2019topnet}. SA-Net \cite{wen2020sa} took one step further to preserve and convey the detailed geometric information of incomplete shape into the generation of complete shape through skip-attention mechanism. Other notable work like RL-GAN-Net \cite{sarmad2019rl} and Render4Completion \cite{hu2019render4completion} focused on the framework of adversarial learning to improve the reality and consistency of the generated complete shape. In all, most of the above methods are generative solution for point cloud completion task, and inevitably suffer from the unordered nature of point clouds, which makes it difficult to reconstruct the detailed typology or structure using a generative decoder. Therefore, in order to avoid the problem of predicting unordered data, PMP-Net uses a different way to reconstruct the complete point cloud, which learns to move all points from the initial input instead of directly generating the final point cloud from a latent code.
The idea of PMP-Net is also related to the research of 3D shape deformation \cite{yin2018p2p}, which mainly considered one-step deformation. However, the deformation between the incomplete and complete shapes is more challenging, which requires the inference of totally unknown geometries in missing regions without any other prior information. In contrast, we propose multi-step searching to encourage PMP-Net to infer more detailed geometric information for missing region, along with point moving distance regularization to guarantee the efficiency of multi-step inference.

\begin{figure*}[!t]
  \centering
  \includegraphics[width=\textwidth]{overview.pdf}\vspace{-0.4cm}
  \caption{Illustration of path searching with multiple steps under the coarse-to-fine searching radius. The PMP-Net moves point A to point A' by three steps, with each step reducing its searching radius, and looking back to consider the moving history in order to decide the next place to move.
  }
  \label{fig:overview}
\end{figure*}

\section{Architecture of PMP-Net}

\subsection{Point Displacement Prediction}
\noindent\textbf{Multi-step framework.}
An overview of the proposed PMP-Net is shown in Figure \ref{fig:overview}(a). Given an input point cloud $P=\{\boldsymbol{p}_i\}$ and a target point cloud $P'=\{\boldsymbol{p'}_j\}$. The objective of PMP-Net is to predict a displacement vector set $\Delta P=\{\Delta \boldsymbol{p}_i\}$, which can move each point from $P$ into the position of $P'$ such that $\{(\boldsymbol{p}_i+\Delta \boldsymbol{p}_i)\}=\{\boldsymbol{p'}_j\}$.
PMP-Net moves each point $\boldsymbol{p}_i$ for $K=3$ steps in total. The displacement vector for step $k$ is denoted by $\Delta\boldsymbol{p}^k_i$, so $\Delta \boldsymbol{p}_i=\sum_{k=1}^{3}\Delta\boldsymbol{p}^k_i$. For step $k$, the network takes the deformed point cloud $\{\boldsymbol{p}^{k-1}_i\}=\{\boldsymbol{p}_i+\sum_{j=1}^{k-1}\Delta\boldsymbol{p}^j_i\}$ from the last step $k-1$ as input, and calculates the new displacement vector according to the input point cloud. Therefore, the predicted shape will be consistently refined step-by-step, which finally produces a complete shape with high quality

\noindent\textbf{Displacement vector prediction.}
At step $k$, in order to predict the displacement vector $\Delta\boldsymbol{p}^k_i$ for each point, we first extract per-point features from the point cloud. This is achieved by first adopting the basic framework of PointNet++ \cite{qi2017pointnet++} to extract the global feature of input 3D shape, and then using the feature propagation module to propagate the global feature to each point in 3D space, and finally producing per-point feature $\boldsymbol{h}^{k,l}_i$ for point $\boldsymbol{p}^k_i$. Since our experimental implementation applies three levels of feature propagation to hierarchically produce per-point feature (see Figure \ref{fig:overview}(b)), we use superscript $k$ to denote the step and the subscript $l$ to denote the level in $\boldsymbol{h}^{k,l}_i$.
The per-point feature $\boldsymbol{h}^{k,l}_i$ is then concatenated with a random noise vector $\hat{\boldsymbol{x}}$, which according to \cite{yin2018p2p} can give point tiny disturbances and force it to leave its original place. Then, the final point feature $\boldsymbol{h}^{k,3}_i$ at step k and level 3 is fed into a multi-layer perceptron (MLP) followed by a hyper-tangent activation (tanh), to produce a 3-dimensional vector as the displacement vector $\Delta\boldsymbol{p}^k_i$ for point $\boldsymbol{p}^k_i$ as
\begin{equation}\label{eq:displacement}\small
  \Delta\boldsymbol{p}^k_i = \mathrm{tanh}(\mathrm{MLP}([\boldsymbol{h}^{k,3}_i:\hat{\boldsymbol{x}}])),
\end{equation}
where ``:'' denotes the concatenation operation.

\noindent\textbf{Recurrent information flow between steps.}
The information of previous moves is crucial for network to decide the current move, because the previous paths can be used to infer the location of the final destination for a single point. Moreover, such information can guide the network to find the direction and distance of next move, and prevent it from changing destination during multiple steps of point moving path searching. In order to achieve this target, we propose to use a special RPA unit between each step and each level of feature propagation module, which is used to memorize the information of previous path and to infer the next position of each point. As shown in Figure \ref{fig:overview}(b), the RPA module in (step $k$, level $l$) takes the output $\boldsymbol{f}^{k,l-1}_i$ from the last level i-1 as input, and combines it with the feature $\boldsymbol{h}^{k-1,l}_i$ from the previous step $k-1$ at the same level $l$ to produce the feature of current level $\boldsymbol{h}^{k,l}_i$, denoted as
\begin{equation}\small
  \boldsymbol{h}^{k,l}_i = \mathrm{RPA}(\boldsymbol{f}^{k,l-1}_i, \boldsymbol{h}^{k-1,l}_i).
\end{equation}
The detailed structure of RPA module is described below.

\begin{figure}[!t]
  \centering
  \includegraphics[width=\columnwidth]{rpa_structure.pdf}\vspace{-0.4cm}
  \caption{Detailed structure of RPA module at step $k$, level $l$.
  }
  \label{fig:rpa_structure}
\end{figure}

\subsection{Recurrent Path Aggregation}
The previous paths of point moving can be regarded as the sequential data, where the information of each move should be selectively memorized or forgotten during the process.
Following this idea, we take the inspiration from the recurrent neural network, where we mimic the behavior of gated recurrent unit (GRU) to calculate an update gate $\boldsymbol{z}$ and reset gate $\boldsymbol{r}$ to encode and forget information, which is according to the point feature $\boldsymbol{h}^{k-1,l}_{i}$ from the last step $k-1$ and the point feature $\boldsymbol{f}^{k,l-1}_{i}$ of current step $k$. The calculation of two gates can be formulated as
\begin{equation}\small
  \boldsymbol{z} = \sigma(W_z[\boldsymbol{f}^{k,l-1}_{i}:\boldsymbol{h}^{k-1,l}_{i}]+\boldsymbol{b}_z),
\end{equation}
\begin{equation}\small
  \boldsymbol{r} = \sigma(W_r[\boldsymbol{f}^{k,l-1}_{i}:\boldsymbol{h}^{k-1,l}_{i}]+\boldsymbol{b}_r),
\end{equation}
where ${W_z, W_r}$ are weight matrix and ${\boldsymbol{b}_z,\boldsymbol{b}_z}$ are biases. $\sigma$ is the \emph{sigmoid} activation function, which predicts a value between 0 and 1 to indicate the ratio of information that allowed to pass the gate. ``:'' denotes the concatenation of two features.

Different from the standard GRU, which emphasizes more importance on the preservation of previous information when calculating the output feature $\boldsymbol{h}^{k,l}_{i}$ at current step, in RPA, we address more importance on the preservation of current input information, and propose to calculate the output feature $\boldsymbol{h}^{k,l}_{i}$ as
\begin{equation}\label{eq:hi}\small
  \boldsymbol{h}^{k,l}_{i} = \boldsymbol{z}\odot\boldsymbol{\hat{h}}^{k,l}_{i}+(1-\boldsymbol{z})\odot \boldsymbol{f}^{k,l-1}_{i},
\end{equation}
where $\boldsymbol{\hat{h}}^{k,l}_{i}$ is the intermediate feature of current step. It contains the preserved information from the past, which is calculated according to the current input feature. The formulation of $\boldsymbol{\hat{h}}^{k,l}_{i}$ is given as
\begin{equation}\small
  \boldsymbol{\hat{h}}^{k,l}_{i} = \varphi(W_h[\boldsymbol{r}\odot\boldsymbol{h}^{k-1,l}_{i}: \boldsymbol{f}^{k,l-1}_{i}]+\boldsymbol{b}_h),
\end{equation}
where $\varphi$ is \emph{relu} activation in our implementation.

The reason of fusing $\boldsymbol{\hat{h}}^{k,l}_{i}$ with $\boldsymbol{f}^{k,l-1}_{i}$ instead of $\boldsymbol{h}^{k-1,l}_{i}$ is that, compared with standard unit in RNN unit, the current location of point should have greater influence to the decision of next move. Especially, when RPA module needs to ignore the previous information which is not important in the current decision making, Eq.(\ref{eq:hi}) can easily allow RPA model to forget all history by simply pressing the update gate $\boldsymbol{z}$ to a zero-vector, and thus enables the RPA module fully focus on the information of current input $\boldsymbol{f}^{k,l-1}_{i}$.

\subsection{Optimized Searching for Unique Paths}

\noindent\textbf{Minimizing moving distance.}
As shown in Figure \ref{fig:emd_solution}, the unordered nature of point cloud allows multiple solutions to deform the input shape into the target one, and the direct constraint (e.g. chamfer distance) on the deformed shape and its ground truth cannot guarantee the uniqueness of correspondence established between the input point set and the target point set. Otherwise, the network will be confused by the multiple solutions of point moving, which may lead to the failure of capturing detailed topology and structure relationships between incomplete shapes and complete ones.
In order to establish a unique and meaningful point-wise correspondence between input point cloud and target point cloud, we take the inspiration from Earth Mover's Distance \cite{rubner2000earth}, and propose to train PMP-Net to learn the path arrangement $\phi$ between source and target point clouds under the constraint of total point moving path distance. Specifically, given the source point clouds $\hat{X}=\{\hat{\boldsymbol{x}}_i|i=1,2,3,...,N\}$ and the target point cloud $X=\{\boldsymbol{x}_i|i=1,2,3,...,N\}$, we follow EMD to learn an arrangement $\phi$ which meets the constraint below
\begin{equation}\label{eq:emd}\small
  \mathcal{L}_{\mathrm{EMD}}(\hat{X},X)=\min_{\phi :\hat{X}\rightarrow X}\frac{1}{\hat{X}}\sum_{\hat{\boldsymbol{x}}\in\hat{X}}\| \hat{\boldsymbol{x}}-\phi(\hat{\boldsymbol{x}}) \|,
\end{equation}
In Eq.(\ref{eq:emd}), $\phi$ is considered as a bijection that minimizes the average distance between corresponding points in $\hat{X}$ and $X$.

According to Eq.(\ref{eq:emd}), bijection $\phi$ established by the network should achieve the minimum moving distance to move points from input shape to target shape. However, even if the correspondence between input and target point clouds is unique, there still exist various paths between source and target points, as shown in Figure \ref{fig:mdl}. Therefore, in order to encourage the network to learn an optimal point moving path, we choose to minimize the point moving distance loss ($\mathcal{L}_{\mathrm{PMD}}$), which is the sum of all displacement vector $\{\Delta \boldsymbol{p}_i^k\}$ output by all three steps in PMP-Net.
The \emph{Point Moving Distance} loss is formulated as
\begin{equation}\label{eq:mdl}\small
  \mathcal{L}_{\mathrm{PMD}} = \sum_{k}\sum_{i}\|\Delta \boldsymbol{p}^k_i \|_2.
\end{equation}

\begin{figure}[!t]
  \centering
  \includegraphics[width=\columnwidth]{emd_solution.pdf}\vspace{-0.4cm}
  \caption{Illustration of multiple solutions when deforming input point cloud (green) into target point cloud (red). The PMD-constraint guarantees the uniqueness of point level correspondence (a) between input and target point cloud, and filter out various redundant solutions for moving points (b).
  }
  \label{fig:emd_solution}
\end{figure}

Eq.(\ref{eq:mdl}) is more strict than EMD constraint. It requires not only the overall displacements of all point achieve the shortest distance, but also limits the point moving paths in each step to be the shortest one. Therefore, in each step, the network will be encouraged to search new path following the previous direction, as shown in Figure \ref{fig:mdl}, which will lead to less redundant moving decision and improve the searching efficiency.

\begin{figure}[!t]
  \centering
  \includegraphics[width=\columnwidth]{mdl.pdf}\vspace{-0.4cm}
  \caption{Illustration of the effectiveness of $\mathcal{L}_{PMD}$. By minimizing the point moving distance, the network is encouraged to learn more consistent paths from source to target, which will reduce redundant searching in each step and improve the efficiency.
  }
  \label{fig:mdl}
\end{figure}

\begin{table*}[!t]\small
\centering
\caption{Point cloud completion on Completion3D dataset in terms of per-point L2 Chamfer distance $\times 10^{4}$ (lower is better).}
\begin{tabular}{l|c|cccccccc}
\toprule
Methods &Average  &Plane    &Cabinet  &Car   &Chair   &Lamp   &Couch    &Table    &Watercraft      \\ \midrule
FoldingNet  \cite{yang2018foldingnet}   &19.07  &12.83    &23.01    &14.88    &25.69    &21.79    &21.31    &20.71    &11.51   \\
PCN  \cite{yuan2018pcn}   &18.22 &9.79    &22.70    &12.43    &25.14    &22.72    &20.26    &20.27    &11.73   \\
PointSetVoting \cite{zhang2020point}  &18.18 &6.88    &21.18    &15.78    &22.54    &18.78    &28.39    &19.96    &11.16   \\
AtlasNet \cite{groueix2018atlasnet}   &17.77   &10.36    &23.40    &13.40    &24.16    &20.24    &20.82    &17.52    &11.62   \\
SoftPoolNet \cite{wang2020softpoolnet} &16.15   &5.81   &24.53   &11.35    &23.63    &18.54      &20.34   &16.89  &7.14   \\
TopNet  \cite{tchapmi2019topnet}     &14.25   &7.32   &18.77   &12.88    &19.82    &14.60      &16.29   &14.89  &8.82   \\
SA-Net \cite{wen2020sa}  &11.22   &5.27   &\textbf{14.45}   &\textbf{7.78}    &13.67    &13.53      &14.22   &11.75  &8.84   \\
GRNet \cite{xie2020grnet}  &10.64   &6.13   &16.90   &8.27    &12.23    &10.22      &14.93   &10.08  &5.86   \\
\midrule
PMP-Net(Ours) &\textbf{9.23}  &\textbf{3.99}    &14.70    &8.55    &\textbf{10.21}    &\textbf{9.27}    &\textbf{12.43}    &\textbf{8.51}   &\textbf{5.77} \\
\bottomrule
\end{tabular}
\label{table:completion3D}
\end{table*}

\noindent\textbf{Multi-scaled searching radius.}
PMP-Net searches the point moving path in a coarse-to-fine manner. For each step, PMP-Net reduces the maximum stride to move a point by the power of 10, which is, for step $k$, the displacement $\Delta\boldsymbol{p}^k_i$ calculated in Eq.(\ref{eq:displacement}) is limited to $10^{-k+1}\Delta\boldsymbol{p}^k_i$. This allows the network converges more quickly during training. And also, the reduced searching range will guarantee the network at next step not to overturn its decision made in the previous step, especially for the long range movements. Therefore, it can prevent the network from making redundant decision during path searching process.

\subsection{Training Loss}
The deformed shape is regularized by the complete ground truth point cloud through Chamfer distance (CD) and Earth Mover Distance (EMD). Following the same notations in Eq.(\ref{eq:emd}), the Chamfer distance is defined as:
\begin{equation}\label{eq:cd}\small
  \mathcal{L}_{CD}({X},\hat{X})=\sum_{\boldsymbol{x}\in X}\min_{\hat{\boldsymbol{x}}\in\hat{X}}\|\boldsymbol{x}-\hat{\boldsymbol{x}} \| + \sum_{\hat{\boldsymbol{x}}\in \hat{X}}\min_{\boldsymbol{x}\in X}\|\hat{\boldsymbol{x}}-\boldsymbol{x} \|.
\end{equation}
The total loss for training is then given as
\begin{equation}\small
  \mathcal{L}=\sum_{k}\mathcal{L}_{\mathrm{CD}}(P^k,P')+\mathcal{L}_{\mathrm{PMD}},
\end{equation}
where $P^k$ and $P'$ denote the point cloud output by step $k$ and the target complete point cloud, respectively. Note that finding the optimal $\phi$ is extremely computational expensive. In experiments, we follow the simplified algorithm in \cite{yuan2018pcn} to estimate an approximation of $\phi$.

\begin{figure*}[!t]
  \centering
  \includegraphics[width=0.95\textwidth]{shapenet_vis.pdf}\vspace{-0.2cm}
  \caption{Visualization of point cloud completion results on Completion3D dataset. PMP-Net is compare with other methods in (a), and (b) shows more completion results of PMP-Net.
  }
  \label{fig:shapenet_vis}
\end{figure*}

\section{Experiments}

\subsection{Evaluation on Completion3D Dataset}
\noindent\textbf{Dataset.} We evaluate our PMP-Net on the widely used benchmark of 3D point cloud completion, i.e. \emph{Completion3D} \cite{tchapmi2019topnet}, which is a large-scaled 3D object dataset derived from the ShapeNet dataset. The partial 3D shapes are generated by back-projecting 2.5D depth images from partial views into 3D space
We follow the settings of training/validation/test splits in Completion3D for fair comparison with the other methods.

\noindent\textbf{Evaluation metric.} Following previous studies \cite{tchapmi2019topnet,xie2020grnet,wen2020sa,yuan2018pcn}, we use the per-point L1/L2 Chamfer distance (CD) as the evaluation metric. L1 version is the CD in Eq.(\ref{eq:cd}) averaged by the point number, and L2 version is the CD replacing L1-norm in Eq.(\ref{eq:cd}) by L2-norm.

\noindent\textbf{Quantitative comparison.} The comparison results\footnote{Results are cited from \url{https://completion3d.stanford.edu/results}} of PMP-Net with the other state-of-the-art point cloud completion methods is shown in Table \ref{table:completion3D}, in which the PMP-Net achieves the best performance in terms of average chamfer distance across all categories. Compared with the second best completion method GRNet (in terms of average chamfer distance), PMP-Net achieves better results in 6 out of 8 categories, which proves the better generalization ability of PMP-Net across different shape categories.
As we discussed in Sec.2, GRNet \cite{xie2020grnet} is a voxel aided shape completion method, where the conversion between point clouds and 3D voxel along with applying 3D CNN on voxel data is a time consuming process. Other methods like SA-Net \cite{wen2020sa} in Table \ref{table:completion3D} are typical generative completion methods which are fully based on point clouds, and the nontrivial improvement of PMP-Net over these methods proves the effectiveness of deformation based solution in point cloud completion task.

\begin{table*}[!t]\small
\centering
\caption{Point cloud completion on PCN dataset in terms of per-point L1 Chamfer distance $\times 10^{3}$ (lower is better).}
\begin{tabular}{l|c|cccccccc}
\toprule
Methods &Average  &Plane    &Cabinet  &Car   &Chair   &Lamp   &Couch    &Table    &Watercraft      \\ \midrule
FoldingNet  \cite{yang2018foldingnet}   &14.31  &9.49    &15.80    &12.61    &15.55   &16.41    &15.97    &13.65    &14.99   \\
TopNet  \cite{tchapmi2019topnet}     &12.15   &7.61   &13.31   &10.90    &13.82    &14.44      &14.78   &11.22  &11.12   \\
AtlasNet \cite{groueix2018atlasnet}   &10.85   &6.37    &11.94    &10.10    &12.06    &12.37    &12.99    &10.33    &10.61   \\
PCN  \cite{yuan2018pcn}   &9.64 &5.50    &22.70    &10.63    &8.70    &11.00    &11.34    &11.68    &8.59   \\
GRNet \cite{xie2020grnet}  &8.83   &6.45   &10.37   &9.45    &\textbf{9.41}    &7.96      &\textbf{10.51}   &8.44  &8.04   \\
CDN. \cite{wang2020cascaded}  &\textbf{8.51}   &\textbf{4.79}   &\textbf{9.97}   &\textbf{8.31}    &9.49    &8.94      &10.69   &\textbf{7.81}  &8.05   \\
\midrule
PMP-Net(Ours) &8.66  &5.50    &11.10    &9.62    &9.47    &\textbf{6.89}    &10.74    &8.77   &\textbf{7.19} \\
\bottomrule
\end{tabular}
\label{table:PCN}
\end{table*}

\noindent\textbf{Qualitative comparison.} In Figure \ref{fig:shapenet_vis}(a), we visually compare PMP-Net with the other completion methods on Completion3D dataset, from which we can find that PMP-Net predicts much more accurate complete shapes on various input categories, while the other methods may output some failure cases for certain input shapes. For example, the input \emph{table} in Figure \ref{fig:shapenet_vis}(a) loses the entire desktop, and methods like GRNet and SA-Net almost cannot predict the whole desktop, while other methods like FoldingNet \cite{yang2018foldingnet}, PCN \cite{yuan2018pcn} and TopNet \cite{tchapmi2019topnet} intend to repair the desktop but fail to predict a good shape of it. Moreover, the advantage of deformation based PMP-Net over the generative methods can be well proved by the case of \emph{boat} in Figure \ref{fig:shapenet_vis}(a). Generative methods, especially like PCN and TopNet, successfully learn the overall structure of the input boat, but fail to reconstruct even the residual part of it. On the other hand, the deformation based PMP-Net can directly preserve the input shape by moving small amount of point to perform completion on certain areas, and keep the input shape unchanged. In Figure \ref{fig:shapenet_vis}(b), we visualize some more completion results of PMP-Net.

\noindent\textbf{Qualitative comparison on ScanNet chairs.}
To evaluate the generalization ability of PMP-Net on point cloud completion task, we pre-train PMP-Net on Completion3D dataset and evaluate its performance on the chair instances in ScanNet dataset without finetune, and compare with GRNet (which is the second best method in Table \ref{table:completion3D}, and also pre-trained on Completion3D). The visual comparison is shown in Figure \ref{fig:scannet_vis}. The PMP-Net completes shapes with less noise than GRNet, which benefits from its point moving practice. Because the differences in data distribution between Completion3D and ScanNet will inevitably confuse the network, and the point moving based PMP-Net can simply choose to leave those points in residual part of an object to stay at their own place to preserve a better shape, in contrast, generation based GRNet has to predict new points for both residual and missing part of an object.

\begin{figure}[!t]
  \centering
  \includegraphics[width=\columnwidth]{scannet_chair.pdf}\vspace{-0.4cm}
  \caption{Visual comparison of PMP-Net and GRNet on ScanNet chairs.
  }
  \label{fig:scannet_vis}
\end{figure}

\subsection{Applying to Dense Point Cloud Completion}
We show that PMP-Net learned on sparse point cloud can be directly applied to the dense point cloud completion. Specifically, we keep training PMP-Net on sparse shape with 2,048 points, and reveal its generalization ability by predicting dense complete shape with 16,384 points on PCN dataset \cite{yuan2018pcn}. PCN dataset is derived from ShapeNet dataset, in which each complete shape contains 16,384 points. The partial shapes have various point numbers, so we first down-sample shapes with more than 2,048 points to 2,048, and up-sample shapes with less than 2048 points to 2048 by randomly copying points.
Since PMP-Net learns to move points instead of generating points, it requires the same number of points in incomplete point cloud and complete one. In order to predict complete shape of 16,384 points, we repeat 8 times of prediction for each shape during testing, with each time moving 2,048 points. Note that PMP-Net is still trained on sparse point clouds with 2,048 points, which are sampled from the dense point clouds of PCN dataset.

The comparison in Table \ref{table:PCN} shows that PMP-Net yields a comparable performance to the state-of-the-art method \cite{wang2020cascaded}, and ranks second on PCN dataset. The result of Wang et al.\cite{wang2020cascaded} is cited from its original paper, while the results of other compared methods are all cited from \cite{xie2020grnet}. Note that most generation based methods (like Wang et al. \cite{wang2020cascaded} and GRNet \cite{xie2020grnet} in Table \ref{table:PCN}) specially designed a coarse-to-fine generation process in order to obtain better performance on dense point cloud completion. In contrast, our PMP-Net trained on 2,048 points can directly generate arbitrary number of dense points by simply repeating the point moving process, and still achieves comparable results to the counterpart methods.

\subsection{Model Analysis}
In this subsection, we analyze the influence of different parts in the PMP-Net. By default, we use the same network sittings in all experiments except for the analyzed part. All studies are typically conducted on the validation set of Completion3D dataset under four categories (i.e. plane, car, chair and table) for convenience.

\noindent\textbf{Analysis of RPA module and PMP loss.} We analyze the effectiveness of RPA module by replacing it with other units in PMP-Net. And for PMP loss, we analyze its effectiveness by removing PMP loss from the network. Specifically, we develop six different variations for comparison: (1) \emph{NoPath} is the variation that removes the RPA module from the network; (2) \emph{Add} is the variation that replaces RPA module with element-wise add layer in the network; (3) RNN, (4) LSTM and (5) GRU are variations that replace RPA module with different recurrent unit; (6) \emph{w/o PMP} is the variation that removes PMP loss from the PMP-Net, where only Chamfer distance is used for training.

The shape completion results are shown in Table \ref{table:analysis_rpa}, from which we can find that baseline RPA module achieves the best performance. The worst results yielded by \emph{Add} variation indicate that directly utilizing history information (paths in previous step) without processing will in return degenerate the network performance, compared to \emph{NoPath} variation. The comparison between \emph{RPA} baseline and \emph{GRU} variation proves the effectiveness of our designation of RPA module, which can give more consideration to the information from current step than GRU unit, and help the network to make more precise decision for point moving.

By comparing baseline variation with \emph{w/o PMP} variation, we can find that PMP loss significantly improves the performance of our network, which is in accordance with our opinion that point moving path should be regularized to better capture the detailed topology and structure of 3D shapes.
\begin{table}[!h]\small
\centering
\caption{Analysis of RPA and PMP loss (baseline marked by ``*'').}
\begin{tabular}{lccccc}
\toprule
Unit. &avg.    &plane   &chair  &car  &table  \\ \midrule
NoPath &11.95 &3.55 &8.30  &16.15 &19.79\\
Add  &12.23  &\textbf{3.32}  &8.10  &16.47 &21.05\\
RNN  &12.12  &3.55  &8.14  &16.19 &20.58\\
LSTM  &11.99  &3.79  &8.09  &15.37 &20.72\\
GRU  &11.87  &3.44  &7.85  &15.44 &20.72\\
w/o PMP &13.66 &4.26 &8.21 &16.69 &25.19 \\
\midrule
baseline* &\textbf{11.58}  &3.42  &\textbf{7.87}  &\textbf{15.88} &\textbf{19.15}\\
\bottomrule
\end{tabular}
\label{table:analysis_rpa}
\end{table}

\noindent\textbf{Effect of multi-step path searching.} In Table \ref{table:step_analysis}, we analyze the effect of different steps for point cloud deformation. Specifically, we fix the ratio of searching radius between each step to $10$, and evaluate the performance of PMP-Net under different step sittings. For example, the searching radius for step=4 is set to $\{1.0, 10^{-1}, 10^{-2}, 10^{-3}\}$, and the searching radius for step=2 is set to $\{1.0, 10^{-1}\}$, respectively. From Table \ref{table:step_analysis}, we can find that deforming point cloud by multiple steps effectively improves the completion performance, by comparing the results of step 1, 2 and 3. On the other hand, the comparison between step 3 and step 4 shows that the performance of multi-step path searching will reach its limitation, because too many steps may cause information redundancy in path searching.
\begin{table}[!h]\small
\centering
\caption{The effect of different steps (baseline marked by ``*'').}
\begin{tabular}{lccccc}
\toprule
Steps. &avg.    &plane   &chair  &car  &table  \\ \midrule
1  &12.26  &3.71  &8.27  &15.59 &21.48\\
2  &11.90  &3.47  &7.95  &15.66 &20.53\\
3*  &\textbf{11.58}  &3.42  &\textbf{7.87}  &\textbf{15.88} &\textbf{19.15}\\
4  &11.67  &\textbf{3.39}  &7.91  &15.89 &19.48\\
\bottomrule
\end{tabular}
\label{table:step_analysis}
\end{table}

\noindent\textbf{Analysis of searching radius.} By default, we decrease the searching radius for each step by the ratio of 10. In Table \ref{table:radius_analysis}, we analyze different sittings of searching radius and evaluate their influence to the performance of PMP-Net. We additionally test two different strategies to perform point moving path searching, i.e. the strategy without decreasing searching radius ($[1.0,1.0,1.0]$ for each step), and the strategy with smaller decreasing ratio ($[1.0,0.5,0.25]$). The baseline result is the default sitting of PMP-Net ($[1.0,0.1,0.01]$ for each step).  Table \ref{table:radius_analysis} shows that PMP-Net achieves the worst performance at $[1.0,1.0,1.0]$, which proves the effectiveness of the strategy to decrease searching radius. And when comparing strategy of $[1.0,0.5,0.25]$ with $[1.0,0.1,0.01]$, we can find that decreasing searching radius with larger ratio can improve the model performance, because larger ratio can better prevent the network from overturning the decisions in previous steps. We also note that when the decreasing ratio goes to large, the PMP-Net will approximate the behavior of network with step=1 in Table \ref{table:step_analysis}, which can in return harm the performance of shape completion.
\begin{table}[!h]\small
\centering
\caption{The effect of searching radius (baseline marked by ``*'').}
\resizebox{\columnwidth}{!}{\begin{tabular}{lccccc}
\toprule
Radius. &Avg.    &Plane   &Chair  &Car  &Table  \\ \midrule
$[1.0,1.0,1.0]$  &12.01  &3.61  &8.22  &16.44 &19.79\\
$[1.0,0.5,0.25]$  &11.77  &\textbf{3.36}  &8.01  &15.92 &19.79\\
$[1.0,0.1,0.01]$*  &\textbf{11.58}  &3.42  &\textbf{7.87}  &\textbf{15.88} &\textbf{19.15}\\
\bottomrule
\end{tabular}}
\label{table:radius_analysis}
\end{table}

\noindent\textbf{Visual analysis of point moving path under different radius.}
In Figure \ref{fig:radius_analysis}, we visualize the searching process under different strategies of searching radius in Table \ref{table:step_analysis}. By analyzing the deformation output in step 1, we can find that PMP-Net with a coarse-to-fine searching strategy can learn to predict a better shape at early step, where the output of step 1 in Figure \ref{fig:radius_analysis}(a) is more complete and tidy than the ones in Figure \ref{fig:radius_analysis}(b) and Figure \ref{fig:radius_analysis}(c). Moreover, a better overall shape predicted in the early stage will enable the network focus on refining a better detailed structure of point cloud, which can be concluded from the comparison of step 3 in Figure \ref{fig:radius_analysis}, where the region highlighted by red rectangles in Figure \ref{fig:radius_analysis}(a) is much better than the other two subfigures.
\begin{figure}[!t]
  \centering
  \includegraphics[width=\columnwidth]{radius_analysis.pdf}\vspace{-0.4cm}
  \caption{Illustration of deformation process in each step under different strategies of searching radius.
  }
  \label{fig:radius_analysis}
\end{figure}

\iffalse
\noindent\textbf{Dimension of noise vector.} The noise vector in Eq.(\ref{eq:displacement}) is used to push the points to leave their original place \cite{yin2018p2p}. In this section, we analyze the dimension and the standard deviation of the noise, which may potentially decide the influence of the noise to the points. Because either the dimension or the standard deviation of the noise vector decreases to 0, there will be no disturbance in the network. On the other hand, larger vector dimension or standard deviation will cause larger disturbance in the network. In Table \ref{table:analysis_noiseDim}, we first analyze the influence of dimension of noise vector. By comparing 0-dimension result with others, we can draw conclusion that the disturbance caused by noise vector is important to learn the point deformation. And by analyzing the performance of different length of noise vector, we can find that the influence of vector length is relatively small, compared with the existence of noise vector.

\begin{table}[!h]\small
\centering
\caption{The effect of noise dimension (baseline marked by ``*'').}
\begin{tabular}{lccccc}
\toprule
Dim. &avg.    &plane   &chair  &car  &table  \\ \midrule
0  &14.56  &4.39  &10.48  &19.01 &24.33\\
8  &11.85  &3.28  &7.95  &15.65 &20.50\\
16  &11.68  &3.44  &\textbf{7.86}  &\textbf{15.22} &20.19\\
32*  &\textbf{11.58}  &3.42  &7.87  &15.88 &\textbf{19.15}\\
64  &\textbf{11.58}  &\textbf{3.14}  &7.96  &16.01 &19.17\\
\bottomrule
\end{tabular}
\label{table:analysis_noiseDim}
\end{table}

\noindent\textbf{Standard deviation of noise distribution.} In Table \ref{table:analysis_noiseStd}, we show the completion results of PMP-Net under different standard deviations of noise vector. Similar to the analysis of vector dimension, we can draw conclusion that larger disturbance caused by bigger standard deviation will help the network achieve better completion performance. The influence of noise vector becomes weak when the standard deviation reaches certain threshold (around $10^-1$ according to Table \ref{table:analysis_noiseStd}).
\begin{table}[!h]\small
\centering
\caption{The effect of standard deviation (baseline marked by ``*'').}
\begin{tabular}{lccccc}
\toprule
Stddev.     &Avg.    &Plane   &Chair  &Car  &Table  \\ \midrule
$10^{-2}$   &11.89  &3.32  &8.15  &16.42 &19.58\\
$10^{-1}$   &\textbf{11.56}  &3.58  &7.78  &15.47 &19.41\\
$1.0$*       &11.58  &3.42  &\textbf{7.87}  &15.88 &\textbf{19.15}\\
$10$        &11.62  &\textbf{3.35}  &7.88  &\textbf{15.29} &19.95\\
\bottomrule
\end{tabular}
\label{table:analysis_noiseStd}
\end{table}

\fi

\section{Conclusions}
In this paper, we propose a novel PMP-Net for point cloud completion by multi-step shape deformation. By moving points from source to target point cloud with multiple steps, PMP-Net can consistently refine the detailed structure and topology of the predicted shape, and establish the point-level shape correspondence between the incomplete and the complete shape. In experiments, we show the superioty of PMP-Net by comparing with other methods on the Completion3D benchmark and PCN dataset.

{\small
\bibliographystyle{ieee_fullname}


\section{Introduction}

Deep convolutional neural networks (CNNs)~\cite{simonyan2014very,krizhevsky2012imagenet,he2016deep,szegedy2015going,xie2017aggregated,wang2018deep,li2018deep} have achieved remarkable success in image classification tasks. When trained on large-scale labeled data, deep networks can learn discriminative representations and present great performance. However, it is difficult to collect and annotate datasets for many domains. A good option is to use the labeled data available in other domains for training models, which however often presents a domain shift challenge between the two domains and degrades the test performance. To address this problem, many unsupervised domain adaptation (UDA) methods~\cite{wang2018deepda,ganin2014unsupervised,long2015learning,long2018conditional,saito2017adversarial} have been proposed. UDA aims to improve the generalization performance on unlabeled target domains. However, in reality a few labeled target instances can be available in target domains, and this semi-supervised domain adaptation (SSDA) setting is more common. According to~\cite{saito2019semi}, UDA methods often fail to improve performance compared with just training on the unified data of labeled source and target samples in the semi-supervised setting.

For the task of domain adaptation, the purpose is to improve the generalization performance in the target domain. In the SSDA setting, we have a few labeled target samples, but the number of them is too small to represent the distribution of target unlabeled data. To increase the number of labeled instances in the target domain without incurring annotation cost, one intuitive strategy is to exploit pseudo-labels of the target domain samples produced by a current prediction model. However, the pseudo-labels can often be very noisy and contain many wrong labels, while training with the mislabeled samples can negatively impact the original model. This motivates the straightforward selective pseudo-labeling strategy which selects the most confident pseudo-labels to increase their probability of correctness~\cite{chen2019progressive}. This simple strategy can select more accurate samples but non-necessary the most useful samples for the prediction model. For example, the more confident and accurate samples may be closer to the labeled data and fail to represent the distribution of the target domain. It is more reasonable but difficult to perform selective pseudo-labeling by balancing the accuracy and the representativeness of the selected samples. To address this challenge, in this paper we propose a reinforcement learning based selective pseudo-labeling method. Our strategy is to use deep Q-learning to learn appropriate selection policies with reward functions that reflect both factors of label correctness and data representativeness.

In addition, due to the lack of labeled data in the target domain, the training methods typically have limited capacity in learning discriminative decision boundaries for the target domain. Inspired by the observation that large margin loss functions~\cite{liu2017sphereface,wang2018cosface,deng2019arcface} can help learn discriminative features, we propose a contrastive target margin loss function over the labeled data from the source and target domains . As illustrated in Fig.~\ref{fig:tml}, the target margin loss make the labeled target samples play a bigger role in learning the decision boundaries.

\begin{figure}[htbp]
\vskip .1in
\begin{center}
   \includegraphics[width=0.98\columnwidth]{tml.pdf}
\end{center}
   \caption{An illustration of the proposed target margin loss. In SSDA setting, the number of labeled target samples is much smaller than that of labeled source samples so that the decision boundary is mainly depending on the source domain. Our target margin loss can make the labeled target data more important for the decision boundary so that the decision regions in target domain will be more separated}
\label{fig:tml}
\end{figure}

Overall, the contribution of this paper can be summarized as follows: (1) We propose a novel reinforcement learning framework for selective pseudo-labeling for semi-supervised domain adaptation. (2) We propose a contrastive target margin loss for SSDA that takes both generalization and discrimination into consideration. (3) Extensive experiments are conducted on DomainNet~\cite{peng2019moment}, Office-31~\cite{saenko2010adapting} and Office-Home~\cite{venkateswara2017deep}. The results show that our proposed method achieves the state-of-the-art semi-supervised domain adaptation performance.

\section{Related Work}

\paragraph{Domain Adaptation.}
Domain adaptation~\cite{wang2018deepda,gong2012geodesic,ganin2014unsupervised,sun2016return} aims to generalize models across different domains of different distributions. Most recent methods are focusing on unsupervised domain adaptation (UDA) which has a label-rich source domain and an unlabeled target domain. A widely used way in UDA methods is to add a domain discriminator which classifies whether a sample is drawn from the source or target domain. Then adversarial learning is applied to minimize the distance between the feature distributions in source and target domain. The domain-adversarial neural network (DANN)~\cite{ganin2014unsupervised} proposes the standard domain adversarial architecture and introduces a gradient reversal layer (GRL) for domain confusion loss produced by the discriminator. Inspired by Conditional Generative Adversarial Networks (CGANs)~\cite{goodfellow2014generative}, Long et al.~\cite{long2018conditional} proposes Conditional Domain Adversarial Network (CDAN) by combining the discriminative information conveyed in the classifier predictions into the adversarial adaptation. There are also other methods~\cite{saito2017adversarial,saito2018maximum} encouraging to generate more discriminative features for the target domain. In fact, semi-supervised domain adaptation (SSDA) is a more common setting in real-world datasets in which limited labeled data can be available in target domain. However, it has not been studied extensively, especially in the field of deep learning. A little conventional work~\cite{donahue2013semi,yao2015semi,ao2017fast} has concentrated on this important task. As for deep learning based methods, Saito et al.~\cite{saito2019semi} propose a Minimax Entropy (MME) method by alternately maximizing the conditional entropy of unlabeled target data and minimizing it to optimize the classifier and the feature extractor respectively. Meanwhile, ~\cite{saito2019semi} shows that UDA methods can rarely improve accuracy in SSDA. Therefore, we propose a reinforcement learning based pseudo-labeling method which can effectively exploit the information of target labeled data to generate more discriminative features with distributions closer to the target domain.

\paragraph{Pseudo-Labeling.}
Pseudo-labeling is an effective way to extend label set when the number of labels is limited. As for SSDA, pseudo-labeling can be used for target domain which has little labeled data. There are two strategies for pseudo-labeling without selection, hard labeling~\cite{long2013transfer,zhang2017joint,wang2018visual} and soft labeling~\cite{pei2018multi}. The hard labeling strategy assigns a pseudo-label with only one class predicted by the classifier to each target unlabeled instance. Then the pseudo-labeled target data will be combined with original labeled data to train an improved model. However, due to the weak classifier in the initial stage of training, many samples will be mis-labeled. Using these mis-labeled data for supervised training can cause serious harm to the model. Thus, soft labeling has been employed. Taking account of the confidence of prediction, the soft labeling strategy assigns the conditional probability of each class predicted by the classifier to the target unlabeled data. As for selective pseudo-labeling~\cite{zhang2018collaborative,wang2019unifying,chen2019progressive}, a subset of target unlabeled samples which are most confident in the prediction are selected to be pseudo-labeled. This sample selection strategy can make the pseudo-labels for training more accurate while it also has a limitation that the selected samples are generally too ``easy'' to represent the distribution of target data. In this work we use reinforcement learning to learn appropriate policies for selecting more accurate and representative pseudo-labeled samples.

\paragraph{Reinforcement Learning.}
Reinforcement learning (RL) is a technique which trains an agent to learn policies based on trial and error in a dynamic environment. The training strategy is to maximize the accumulated reward from the environment. No longer limited to its traditional applications in robotics and control, RL has made great progress in many vision tasks, such as action recognition~\cite{yeung2017learning}, person re-identification~\cite{zhang2018multi} and face recognition~\cite{liu2019fair}. Yeung et al.~\cite{yeung2017learning} design an agent for learning to label noisy web data so that they can select right examples for training a classifier. Zhang et al.~\cite{zhang2018multi} propose to choose sufficient data pairs for multi-shot pedestrian re-identification by training an agent. Liu et al.~\cite{liu2019fair} introduce a policy network for adjusting a margin parameter in the loss function to learn more discriminative features from imbalanced face datasets. In this work, we train an agent with deep Q-learning to select more representative and accurate pseudo-labeled data.

\begin{figure*}[htb]
\begin{center}
   \includegraphics[width=0.71\linewidth]{model.pdf}
\end{center}
   \caption{An overview of our method. We use the proposed target margin loss for the base model training. The candidate set consists of several samples pseudo-labeled by the pre-trained base model. Then we train an agent (deep Q-network) by deep Q-learning to select appropriate pseudo-labeled samples and move them to positive set for the improved model training. Specifically, we generate state ${s_t}$ from the candidate set as the input of the agent and get an output action ${a_t}$ to select an pseudo-labeled instance. Then the state transfers to the next state ${s'_t}$ and the reward ${r_t}$ will be calculated after an epoch of the improved model training. As for the training of the agent, we use an experience pool $M$ to collect a series of recordings $\left\{ {\left( {{s_i},{a_i},{r_i},{s'_i}} \right)} \right\}$ defined in section~\ref{sec:dqn} and sample batches of recordings from $M$ to train the deep Q-network}
\label{fig:model}
\end{figure*}

\section{Method}

For semi-supervised domain adaptation, we have a sufficient labeled dataset from the source domain, ${{\cal D}_s} = \left\{ {\left( {{\bf{x}}_i^s,y_i^s} \right)} \right\}_{i = 1}^{{N_s}}$. In the target domain, we only have a limited number of labeled instances ${{\cal D}_t} = \left\{ {\left( {{\bf{x}}_i^t,y_i^t} \right)} \right\}_{i = 1}^{{N_t}}$, but a large set of unlabeled instances ${{\cal D}_u} = \left\{ {\left( {{\bf{x}}_i^u} \right)} \right\}_{i = 1}^{{N_u}}$. The goal is to train a good prediction model on all these available data ${{\cal D}_s}$, ${{\cal D}_t}$ and ${{\cal D}_u}$ and evaluate it on ${{\cal D}_u}$ with the hidden true labels, as described in~\cite{saito2019semi}. In this section, we present a novel selective pseudo-labeling method for semi-supervised domain adaptation. The method is based on a deep Q-learning framework and a target margin loss. The framework of the proposed method is depicted in Fig.~\ref{fig:model}. First, we use the proposed target margin loss to train a CNN consisting of a feature extractor $F$ and a classifier $C$ for a K-class classification problem. Then, we generate pseudo-labels for the unlabeled samples in the target domain based on the trained CNN classifier. Finally, we alternately train an agent with deep Q-learning and use the agent to select pseudo-labeled samples for the CNN training.

\subsection{Target Margin Loss}

Large margin loss functions~\cite{liu2017sphereface,wang2018cosface,deng2019arcface} (based on traditional softmax loss function) effectively make CNN features more discriminative. For semi-supervised domain adaptation, there is a gap in feature distributions between domains and the number of labeled samples in the target domain is much smaller than in the source domain. Therefore, we propose to add a relative angular margin to the loss on the target labeled data, by contrast to the loss on the source labeled data. This can be considered as making the decision region separations more aligned with the target domain feature distribution. Then the proposed target margin loss can be formulated as follows:
\begin{equation}
\label{eq:tml}
\begin{aligned}
{{\cal L}_{tml}} = & - \frac{1}{{{N_s}}}\sum\limits_{i = 1}^{{N_s}} {\log \frac{{{e^{s\cos \theta _{{y_i}}^s}}}}{{\sum\nolimits_{j = 1}^K {{e^{s\cos \theta _j^s}}} }}} \\
& - \frac{1}{{{N_t}}}\sum\limits_{i = 1}^{{N_t}} {\log \frac{{{e^{s\cos \left( {\theta _{{y_i}}^t + m} \right)}}}}{{{e^{s\cos \left( {\theta _{{y_i}}^t + m} \right)}} + \sum\nolimits_{j = 1,j \ne {y_i}}^K {{e^{s\cos \theta _j^t}}} }}}
\end{aligned}
\end{equation}
where $\cos {\theta _j} = \frac{{{\bf{W}}_j^T}}{{\left\| {{{\bf{W}}_j}} \right\|}} \cdot \frac{{{F({\bf{x}}_i)}}}{{\left\| {{F({\bf{x}}_i)}} \right\|}}$ is the cosine similarity between the $i$-th instance feature vector and the $j$-th class weight. The $\cos ({\theta _{{y_i}}} + m)$ can be computed by the addition formula of the trigonometric functions with the values of $\cos {\theta _{{y_i}}}$ and the margin $m$. $s$ is a re-scaling constant.

In addition, the suitable class separation can benefit from taking the unlabeled target domain data into account. To this end, we adopt the entropy loss to cluster the target unlabeled features into the corresponding decision regions, which is written as follows:
\begin{equation}
\label{eq:ent}
{{\cal L}_{ent}} =  - \frac{1}{{{N_u}}}\sum\limits_{i = 1}^{{N_u}} {\sum\limits_{j = 1}^K {p\left( {y = j|{\bf{x}}_i^u} \right)} } \log p\left( {y = j|{\bf{x}}_i^u} \right)
\end{equation}
where $p\left( {y = j|{\bf{x}}_i^u} \right)$ represents the probability of categorizing ${\bf{x}}_i^u$ to class $j$. Note that the samples should be clustered around the representative centers of the corresponding classes to decrease the entropy, resulting in the desired discriminative features.

Combining Eq. (\ref{eq:tml}) and Eq. (\ref{eq:ent}) provides the following formulation of our final semi-supervised loss function:
\begin{equation}
\label{eq:l}
{\cal L} = {{\cal L}_{tml}} + \alpha {{\cal L}_{ent}}
\end{equation}
where $\alpha$ is a hyper-parameter that balances the target margin loss and the entropy loss.

\subsection{Selective Pseudo-Labeling by Reinforcement Learning}
\label{sec:dqn}

Pseudo-labeling target unlabeled data by the CNN can generate many mis-labeled samples which can cause serious harm to the subsequent learning process. To address this issue, selective pseudo-labeling is used in many works, which selects the most confident unlabeled samples for pseudo-labeling. This sample selection strategy however is subject to the problem that the selected samples are generally ``easy'' ones or belong to ``easy'' classes. As a result, the selected samples could not represent the distribution of the target domain well. To address this drawback, we propose to use reinforcement learning to select more representative and accurate pseudo-labeled samples.

We formulate the problem of selecting pseudo-labeled samples as a Markov Decision Process (MDP), described by $\left( {{\cal S},{\cal A},{\cal T},{\cal R}} \right)$ as the states, actions, transitions and rewards respectively, and train an agent to select pseudo-labeled samples. We define a candidate set ${{\cal D}_c}$ which consists of pseudo-labeled samples to be selected and is initially randomly sampled from ${{\cal D}_u}$ with pseudo-labels, and a positive set ${{\cal D}_p}$ which is composed of the selected pseudo-labeled samples and initialized to be empty. During our CNN training, a series of reinforcement learning samples will be generated, which can be represented as $\left\{ {\left( {{s_i},{a_i},{r_i},{s'_i}} \right)} \right\}$. Here, ${s_i} \in {\cal S}$ is the state and ${r_i} \in {\cal R}$ is the reward. ${a_i} \in {\cal A}$ is the action taken by the agent at state ${s_i}$, which is equivalent to selecting a pseudo-labeled sample from the candidate set ${{\cal D}_c}$ and moving it to the positive set ${{\cal D}_p}$. ${s'_i} \in {\cal S}$ represents the next state which the agent turns to through the action ${a_i}$. Then, we alternately train the agent by using the reinforcement learning samples and use the agent to select pseudo-labeled samples for our CNN training.

\paragraph{States.} We consider that the representative ability and accuracy of a pseudo-labeled target instance can be related to three parts, which are itself, the data with labels in ${{\cal D}_t}$ and ${{\cal D}_p}$, and the unlabeled data in ${{\cal D}_u}$. Note that pseudo-labeled instances are selected to make the current distribution of labeled data similar to the distribution of unlabeled data. Therefore, we formulate the state as a concatenation of three vectors, dependent on ${{\cal D}_c}$, ${{\cal D}_t} \cup {{\cal D}_p}$ and ${{\cal D}_u}$, respectively. For the first part, given the candidate set ${{\cal D}_c} = \left\{ {\left( {{\bf{x}}_i^c,\hat y_i^c} \right)} \right\}_{i = 1}^{{N_c}}$, we use a vector $\left[ {F{{({\bf{x}}_i^c)}^T},C{{({\bf{x}}_i^c)}^T}} \right] \in {\mathbb{R}^{d + K}}$ to represent each instance, where $F({{\bf{x}}_i^c})$ denotes the $d$-dimensional feature vector of instance ${\bf{x}}_i^c$ extracted by the feature extractor $F$, $C({{\bf{x}}_i^c})$ denotes the softmax output of the classifier $C$. The entire vector of this part is a flattened concatenation of all the instances in ${{\cal D}_c}$. After taking an action to move an instance from ${{\cal D}_c}$ to ${{\cal D}_p}$, we replace the selected instance with a zero-valued vector. For the second part, we also use a vector $\left[ {F{{({\bf{x}}_i^{tp})}^T},C{{({\bf{x}}_i^{tp})}^T}} \right] \in {\mathbb{R}^{d + K}}$ to represent each instance in ${{\cal D}_t} \cup {{\cal D}_p}$. The difference is that we calculate the average vectors in each class and then concatenate them to a vector which has a dimension of $K \times \left( {d + K} \right)$. The last part is a vector represented by the instances in ${{\cal D}_u}$ with the same operation as the second part. Thus, the state ${s_i}$ is a flattened concatenation of these three parts.

\paragraph{Actions.} We define the action as selecting one instance from the current candidate set ${{\cal D}_c}$. For each state ${s_i}$, the agent takes an action ${a_i}$ to select the ${a_i}$-th instance in ${{\cal D}_c}$ and move it to ${{\cal D}_p}$. The number of actions is equivalent to the number of instances in ${{\cal D}_c}$, i.e., ${N_c}$.

\paragraph{Rewards.} The rewards should reflect whether the actions taken by the agent are appropriate or not. In other words, selecting more representative and accurate pseudo-labeled instances should lead to positive rewards, and vice versa. We first define a metric function to measure the representative ability and accuracy, which can be formulated as follows:
\begin{equation}
\label{eq:phi}
\varphi ({{\bf{x}}_i},{\hat y_i}) = \log {p_c}(y = {\hat y_i}|{{\bf{x}}_i}) + \beta \log {p_f}(y = {\hat y_i}|{{\bf{x}}_i}) + \lambda {\Delta _e}
\end{equation}
where $\beta$ and $\lambda$ are hyper-parameters. ${p_c}(y = {\hat y_i}|{{\bf{x}}_i})$ represents the probability of the pseudo class ${\hat y_i}$ predicted by the classifier. It indicates the confidence of the prediction. As for ${p_f}(y = {\hat y_i}|{{\bf{x}}_i})$, we take advantage of the target labeled data and define it as follows:
\begin{equation}
{p_f}(y = {\hat y_i}|{{\bf{x}}_i}) = \frac{{{e^{s\cos \left\langle {F({{\bf{x}}_i}),{\bf{z}}_{{{\hat y}_i}}^{tp}} \right\rangle }}}}{{\sum\nolimits_{j = 1}^K {{e^{s\cos \left\langle {F({{\bf{x}}_i}),{\bf{z}}_j^{tp}} \right\rangle }}} }}
\end{equation}
where ${\bf{z}}_j^{tp}$ represents the feature center of the $j$-th class in target labeled and current positive set ${{\cal D}_t} \cup {{\cal D}_p}$, which can be formulated as follows:
\begin{equation}
{\bf{z}}_j^{tp} = \frac{{\sum\nolimits_{i = 1}^{{N_t}} {F({{\bf{x}}_i})\mathbb{I}({y_i} = j) + \sum\nolimits_{i = 1}^{{N_p}} {F({{\bf{x}}_i})\mathbb{I}(\hat y_i^p = j)} } }}{{\sum\nolimits_{i = 1}^{{N_t}} {\mathbb{I}({y_i} = j) + \sum\nolimits_{i = 1}^{{N_p}} {\mathbb{I}(\hat y_i^p = j)} } }}
\end{equation}
$\mathbb{I}$ is the indicator function. Therefore, ${p_f}(y = {\hat y_i}|{{\bf{x}}_i})$ is the softmax output of the cosine distance between ${{\bf{x}}_i}$ and the feature center of its pseudo class ${\hat y_i}$ in ${{\cal D}_t} \cup {{\cal D}_p}$.

The first two terms reflect the confidence of the pseudo-label prediction through two aspects, i.e., the output of the classifier and the similarity with the feature center of the pseudo class in target domain. Since the classifier is more dependent on the source domain data, we add the second term to specifically take the distribution of the target domain into consideration so that the metric function can evaluate the accuracy of a target pseudo-labeled sample better. We also add a third term ${\Delta _e}$, which represents the decrease of the entropy of the target unlabeled data and can be formulated as follows:
\begin{equation}
{\Delta _e} = H - H'
\end{equation}
where $H$ and $H'$ represent the entropy at state ${s_i}$ and the next state ${s'_i}$ respectively and can be calculated in the same way asthe ${{\cal L}_{ent}}$ in Eq. (\ref{eq:ent}). In other words, we first calculate $H$ at state ${s_i}$ and then add a pseudo-labeled sample according to the action ${a_i}$ for one-epoch training. After the training, we calculate $H'$ and derive ${\Delta _e}$. In order to make the entropy not affected by ${{\cal L}_{ent}}$, we only use ${{\cal L}_{tml}}$ in Eq. (\ref{eq:tml}) to optimize the model during the one-epoch training. Note that the more representative the selected sample is, the more the entropy will decrease. Therefore, larger ${\Delta _e}$ means stronger representative ability, and vice versa. In addition, we perform a $\log$ operation on the first two terms to keep these three terms at the same scale.

Directly using the metric function as reward can result in very small differences between the rewards of good and bad actions. Hence, we propose to use the final reward function defined as follows:
\begin{equation}
\label{eq:rew}
{r_i} = \left\{ {\begin{array}{*{20}{c}}
{ + 1,\; \varphi ({{\bf{x}}_i},{{\hat y}_i}) > \tau }\\
{ - 1,\; \varphi ({{\bf{x}}_i},{{\hat y}_i}) \le \tau }
\end{array}} \right.
\end{equation}
where $\tau$ is a threshold and we set it to $(1 + \beta )\log (0.9)$ in our experiments. We use this binary reward instead of the metric function in order to provide the agent more explicit guidance.

\paragraph{Deep Q-learning.} We apply deep Q-learning~\cite{mnih2015human} to learn policies for selecting pseudo-labeled instances. For each state and action $({s_i},{a_i})$, the Q function $Q({s_i},{a_i})$ can represent the discounted accumulated rewards for the state and action. Given a reinforcement learning training sample $\left( {{s_i},{a_i},{r_i},{s'_i}} \right)$, the target value of $Q({s_i},{a_i})$ can be calculated as follows:
\begin{equation}
\label{eq:v}
{V_i} = {r_i} + \gamma \mathop {\max }\limits_{{{a'}_i}} Q({s'_i},{a'_i})
\end{equation}
where $\gamma $ is a discount factor to decide the importance of future accumulated reward compared with the current reward. During the training, we iteratively update the deep Q-network by:
\begin{equation}
\label{eq:ql}
\Omega  \leftarrow \Omega  - \varepsilon \sum\limits_i {\frac{{{\rm{d}}Q({s_i},{a_i})}}{{{\rm{d}}\Omega }}(Q({s_i},{a_i}) - {V_i})}
\end{equation}
where $\Omega $ represents the parameters of the Q-network. As for the entire model training, we alternately train the classification network and the Q-network. The details of our training strategy are summarized in Algorithm \ref{alg:dqn}.

\begin{algorithm}[htb]
\caption{Selective pseudo-labeling by deep Q-learning}
\label{alg:dqn}
\begin{algorithmic}[1]
\renewcommand{\algorithmicrequire}{\textbf{Input:}}
\REQUIRE
Source labeled set ${{\cal D}_s}$, target labeled set ${{\cal D}_t}$ and target unlabeled set ${{\cal D}_u}$
\renewcommand{\algorithmicrequire}{\textbf{Output:}}
\REQUIRE
Feature extractor $F$, classifier $C$ and deep Q-network $Q$
\STATE Pre-train $F$ and $C$ with ${{\cal D}_s}$, ${{\cal D}_t}$ and ${{\cal D}_u}$ by optimizing the loss in Eq. (\ref{eq:l});
\STATE Initialize the positive set ${{\cal D}_p} = \emptyset $;
\STATE Initialize the experience pool $M = \emptyset $;
\WHILE{not converge}
   \STATE Assign pseudo-labels to the unlabeled data in ${{\cal D}_u}$ by current $F$ and $C$;
   \STATE Copy the parameters of $F$ and $C$ to $F'$ and $C'$;
   \STATE Initialize ${{\cal D}_c}$ with random pseudo-labeled samples from ${{\cal D}_u}$ and generate the state ${s_0}$;
   \WHILE{${{\cal D}_c} \ne \emptyset $}
      \STATE Get an output action ${a_t}$ using Eq. (\ref{eq:act}) with the current state ${s_t}$ as input;
      \STATE Update ${{\cal D}_c}$ and ${{\cal D}_p}$ by taking the action ${a_t}$;
      \STATE Update $F'$ and $C'$ with ${{\cal D}_s}$, ${{\cal D}_t}$ and ${{\cal D}_p}$ by optimizing the loss in Eq. (\ref{eq:tml});
      \STATE Generate the next state ${s'_t}$;
      \STATE Calculate the reward ${r_t}$ by Eq. (\ref{eq:rew}) with $F'$ and $C'$;
      \STATE Insert the recording $\left( {{s_t},{a_t},{r_t},{s'_t}} \right)$ into $M$;
      \STATE Sample a batch of recordings $\left\{ {\left( {{s_i},{a_i},{r_i},{s'_i}} \right)} \right\}$ from $M$ to update the deep Q-network $Q$ by Eq. (\ref{eq:ql});
      \IF{${r_t} < 0$}
         \STATE Break;
      \ENDIF
   \ENDWHILE
   \STATE Update $F$ and $C$ with ${{\cal D}_s}$, ${{\cal D}_t}$, ${{\cal D}_u}$ and ${{\cal D}_p}$ by optimizing the loss in Eq. (\ref{eq:l}).
\ENDWHILE
\end{algorithmic}
\end{algorithm}

We simply use a three-layer fully connected network as the deep Q-network. Each fully connected layer is followed by a ReLU activation function. When the agent (i.e., the deep Q-network) is called to select a pseudo-labeled instance, it will output an action ${a_t}$ using a policy as follows:
\begin{equation}
\label{eq:act}
{a_t} = \arg {\max _a} \; Q({s_t},a)
\end{equation}
where ${s_t}$ is the current state representation.

\section{Experiments}

\subsection{Datasets and Baselines}

\paragraph{Datasets.}
We perform our experiments on three datasets, DomainNet~\cite{peng2019moment}, Office-31~\cite{saenko2010adapting} and Office-Home~\cite{venkateswara2017deep}. DomainNet is a large-scale domain adaptation benchmark dataset, which contains six domains. Compared with the other two datasets, it has much more data with 345 classes. Office-31 is a widely used dataset for visual domain adaptation, with 31 categories collected from three different domains: Amazon (A), Webcam (W) and DSLR (D). Office-Home is another domain adaptation dataset in office and home settings, which is more difficult than Office-31. It consists of four distinct domains Artistic images (A), Clipart (C), Product images (P) and Real-World images (R). For the SSDA setting and domain adaptation scenarios, we follow the strategies in ~\cite{saito2019semi}. We randomly select three labeled samples per class as the labeled training target samples to form a three-shot SSDA setting. Due to some noisy domains and categories in DomainNet, we pick 4 domains and 126 categories for DomainNet experiments. Following ~\cite{saito2019semi}, we form 7 adaptation scenarios for testing with the 4 domains, Real (R), Clipart (C), Painting (P) and Sketch (S). As for Office-31, we construct 2 scenarios with Amazon as the target domain since Webcam and DSLR do not have enough samples for effective evaluation.

\paragraph{Baselines.}
S+T baseline directly trains a model using the labeled
source data and labeled target data without unlabeled target data. For the UDA methods (DANN~\cite{ganin2014unsupervised}, ADR~\cite{saito2017adversarial}, CDAN~\cite{long2018conditional}) as baselines, we modify their training strategies following ~\cite{saito2019semi} so that the models can be trained with all the labeled source set, labeled target set and unlabeled target set. ENT~\cite{grandvalet2005semi} is a baseline method using standard entropy minimization which trains a model to minimize the entropy of unlabeled target data. MME~\cite{saito2019semi} is a model with the classifier and the feature extractor optimized by maximizing the conditional entropy of unlabeled target data and minimizing it respectively. We also design a baseline TML\_SPL with a selective pseudo-labeling strategy to compare with our reinforcement learning based selective pseudo-labeling method. TML\_SPL uses our target margin loss for the entire training and selects the most confident pseudo-labeled samples with 0.9 as the threshold to assist to train an improved model.

\subsection{Implementation Details}

As for the architecture of the networks, we adopt ResNet-34~\cite{he2016deep} for experiments on DomainNet and VGG-16~\cite{simonyan2014very} for experiments on Office-31 and Office-Home, finetuned from ImageNet pre-trained models~\cite{russakovsky2015imagenet}. All our experiments are implemented in PyTorch~\cite{paszke2017automatic}. We adopt mini-batch SGD with momentum of 0.9 and the learning rate adjusting schedule as ~\cite{ganin2016domain}. The weight decay is set at 0.0005. As for the $s$ and margin $m$ in Eq. (\ref{eq:tml}), a very small $s$ or a very large $m$ can make the model difficult to converge while a very large $s$ or a very small $m$ can make the margin ignored during training. Therefore, we choose an appropriate pair of values 30 and 0.5 for $s$ and $m$ respectively following previous large margin works. As for the trade-off parameters, we set $\alpha$ in Eq. (\ref{eq:l}) at 0.1 to keep the two losses at a similar scale so that we can prevent any loss from being ignored during the training. $\beta$ and $\lambda$ balance the second and the third terms in Eq. (\ref{eq:phi}). The first two terms in Eq. (\ref{eq:phi}) are both logarithmic forms and the third term is the decrease of the entropy. Thus, we set $\beta$ and $\lambda$ at 1 and 0.1 to keep the three terms at a similar scale so that all the terms can make sense.

For the deep Q-learning, we apply the $\varepsilon $-greedy strategy~\cite{mnih2015human} and the experience replay strategy~\cite{lin1992self}. The $\varepsilon $-greedy strategy allows the deep Q-network to randomly select an action with $\varepsilon $ as the probability and $\varepsilon $ changes from 1 to 0 during the training. We use this strategy for the reason that the output by the deep Q-network at early stage does not reflect the reward and the deep Q-network needs more diverse training samples. The experience replay strategy means that we use an experience pool $M$ to collect a series of recordings $\left\{ {\left( {{s_i},{a_i},{r_i},{s'_i}} \right)} \right\}$ and sample batches of the recordings from $M$ to train the deep Q-network. It can make the deep Q-network learn from both current and past information. Our deep Q-network is composed of three fully connected layers, with two hidden layers of 1024 and 512 units respectively. The discount factor $\gamma $ in Eq. (\ref{eq:v}) is set to 0.9.

\setlength{\tabcolsep}{2pt}
\begin{table}[htb]
\begin{center}
\caption{Comparisons for CML and TML using DomainNet based on ResNet-34}
\scalebox{0.8}{
\begin{tabular}{c|ccccccc|c}
\hline
Method&R to C&R to P&P to C&C to S&S to P&R to S&P to R&Mean\\
\hline\hline
CML&66.3&63.8&67.2&58.8&61.0&57.2&71.7&63.7\\
TML (Ours)&\textbf{72.5}&\textbf{71.6}&\textbf{72.9}&\textbf{61.0}&\textbf{67.7}&\textbf{62.8}&\textbf{79.2}&\textbf{69.8}\\
\hline
\end{tabular}
}
\label{tab:cml}
\end{center}
\end{table}
\setlength{\tabcolsep}{5pt}

\begin{figure}[htb]
\begin{center}
   \includegraphics[width=0.93\columnwidth]{vn.pdf}
\end{center}
\vspace{-2ex}
   \caption{Accuracy vs the number of labeled samples per class in target domain}
\label{fig:vn}
\vspace{-1ex}
\end{figure}

\subsection{Validation Experiments}

In order to show the effectiveness of our method, we design several validation experiments. We pick a scenario in DomainNet as an example scenario with Real as the source domain and Clipart as the target domain.

\paragraph{Extending Margin to Source Domain.}
Margin loss can make features more discriminative while our target margin loss only adds a margin on target domain. We extend the margin parameter to source domain to form a complete margin loss (CML) for comparison, which has a similar formulation to the original margin loss and can be written as:
\begin{equation}
{{\cal L}_{cml}} = \frac{1}{{{N_{s + t}}}}\sum\limits_{i = 1}^{{N_{s + t}}} {\log \frac{{{e^{s\cos \left( {\theta _{{y_i}}^t + m} \right)}}}}{{{e^{s\cos \left( {\theta _{{y_i}}^t + m} \right)}} + \sum\nolimits_{j = 1,j \ne {y_i}}^K {{e^{s\cos \theta _j^t}}} }}}
\end{equation}
where the $s + t$ means all the data in labeled source set and labeled target set. The CML method replaces the ${{\cal L}_{tml}}$ in Eq. (\ref{eq:l}) with ${{\cal L}_{cml}}$. The results are shown in Table~\ref{tab:cml}. Our TML method performs much better than CML though the difference between the two methods is quite small. The reason can be that if both the labeled source data and the labeled target data is constrained by the margin then the decision boundary will still be more dependent on the source domain with much more data.

\begin{table*}[htb]
\begin{center}
\caption{Results on the 7 scenarios in DomainNet based on ResNet-34. The top-performing method in each scenario is bold and the best method without pseudo-labeling is underlined.}
\scalebox{0.9}{
\begin{tabular}{c|ccccccc|c}
\hline
Method&R to C&R to P&P to C&C to S&S to P&R to S&P to R&Mean\\
\hline\hline
S+T&60.0&62.2&59.4&55.0&59.5&50.1&73.9&60.0\\
DANN~\cite{ganin2014unsupervised}&59.8&62.8&59.6&55.4&59.9&54.9&72.2&60.7\\
ADR~\cite{saito2017adversarial}&60.7&61.9&60.7&54.4&59.9&51.1&74.2&60.4\\
CDAN~\cite{long2018conditional}&69.0&67.3&68.4&57.8&65.3&59.0&78.5&66.5\\
ENT~\cite{grandvalet2005semi}&71.0&69.2&71.1&60.0&62.1&61.1&78.6&67.6\\
MME~\cite{saito2019semi}&72.2&69.7&71.7&\underline{61.8}&66.8&61.9&78.5&68.9\\
TML (Ours)&\underline{72.5}&\underline{71.6}&\underline{72.9}&61.7&\underline{67.7}&\underline{62.8}&\underline{79.2}&\underline{69.8}\\
\hline
TML\_SPL&73.2&72.2&73.3&62.1&68.5&63.4&80.0&70.4\\
TML\_DQNPL (Ours)&\textbf{75.8}&\textbf{74.5}&\textbf{75.1}&\textbf{64.3}&\textbf{69.7}&\textbf{64.4}&\textbf{82.6}&\textbf{72.3}\\
\hline
\end{tabular}
}
\label{tab:domainnet}
\end{center}
\vspace{-1ex}
\end{table*}

\begin{table*}[htb]
\begin{center}
\caption{Results on the 12 scenarios in Office-Home based on VGG-16. The top-performing method in each scenario is bold and the best methods without pseudo-labeling are underlined.}
\scalebox{0.88}{
\begin{tabular}{c|cccccccccccc|c}
\hline
Method&R to C&R to P&R to A&P to R&P to C&P to A&A to P&A to C&A to R&C to R&C to A&C to P&Mean\\
\hline\hline
S+T&49.6&78.6&63.6&72.7&47.2&55.9&69.4&47.5&73.4&69.7&56.2&70.4&62.9\\
DANN&56.1&77.9&63.7&73.6&52.4&56.3&69.5&50.0&72.3&68.7&56.4&69.8&63.9\\
ADR&49.0&78.1&62.8&73.6&47.8&55.8&69.9&49.3&73.3&69.3&56.3&71.4&63.0\\
CDAN&50.2&80.9&62.1&70.8&45.1&50.3&74.7&46.0&71.4&65.9&52.9&71.2&61.8\\
ENT&48.3&81.6&65.5&76.6&46.8&56.9&73.0&44.8&75.3&72.9&59.1&77.0&64.8\\
MME&\underline{56.9}&82.9&65.7&76.7&53.6&59.2&\underline{75.7}&\underline{54.9}&75.3&72.9&\underline{61.1}&76.3&67.6\\
TML (Ours)&\underline{56.9}&\underline{83.2}&\underline{67.0}&\underline{76.8}&\underline{54.5}&\underline{59.9}&\underline{75.7}&\underline{54.9}&\underline{75.9}&\underline{73.2}&\underline{61.1}&\underline{77.5}&\underline{68.1}\\
\hline
TML\_SPL&55.4&82.1&67.1&76.5&55.3&60.7&75.5&53.0&75.9&73.4&60.4&77.6&67.7\\
TML\_DQNPL (Ours)&\textbf{58.4}&\textbf{84.0}&\textbf{69.1}&\textbf{78.5}&\textbf{56.8}&\textbf{61.7}&\textbf{77.0}&\textbf{55.9}&\textbf{77.1}&\textbf{74.5}&\textbf{61.9}&\textbf{78.8}&\textbf{69.5}\\
\hline
\end{tabular}
}
\label{tab:officehome}
\end{center}
\vspace{-1ex}
\end{table*}

\begin{table}[htb]
\begin{center}
\caption{Results on the 2 scenarios in Office-31 based on VGG-16. The top-performing method in each scenario is bold and the best methods without pseudo-labeling are underlined.}
\scalebox{0.9}{
\begin{tabular}{c|cc|c}
\hline
Method&W to A&D to A&Mean\\
\hline\hline
S+T&73.2&73.3&73.3\\
DANN&75.4&74.6&75.0\\
ADR&73.3&74.1&73.7\\
CDAN&74.4&71.4&72.9\\
ENT&75.4&75.1&75.3\\
MME&76.3&\underline{77.6}&77.0\\
TML (Ours)&\underline{76.6}&\underline{77.6}&\underline{77.1}\\
\hline
TML\_SPL&75.7&77.2&76.5\\
TML\_DQNPL (Ours)&\textbf{77.5}&\textbf{78.8}&\textbf{78.2}\\
\hline
\end{tabular}
}
\label{tab:office}
\end{center}
\vspace{-1ex}
\end{table}

\paragraph{Varying Number of Target Labeled Samples.}
We verify the number of labeled samples in the target domain from 1 to 20 per class to explore the performance of our method in different settings.
As illustrated in Fig.~\ref{fig:vn}, our TML method can outperform MME and ENT in all the settings with varying number of target labeled samples while MME gradually performs worse than the simple ENT baseline as the number increasing. Furthermore, when the target labeled samples are much enough, the confidence based selective pseudo-labeling method TML\_SPL doesn't work well and can even hurt the original model. Our reinforcement learning based selective pseudo-labeling method TML\_DQNPL can always make progress to the base model due to the representative and accurate selected pseudo-labels.

\subsection{Results}

The results of our main experiments on the DomainNet dataset are shown in Table~\ref{tab:domainnet}. As mentioned in ~\cite{saito2019semi}, the UDA methods cannot improve the performance in some cases. Compared with the present state-of-the-art SSDA method MME~\cite{saito2019semi}, our method with only target margin loss (TML) can perform better except for only one case where it performs similarly to MME. On the basis of TML, our final method TML\_DQNPL with deep Q-network for selective pseudo-labeling can outperform the baseline TML\_SPL with selective pseudo-labeling by confidence, which demonstrates that our deep Q-network can help select more representative and accurate pseudo-labels.

The results on Office-Home and Office-31 are shown in Table~\ref{tab:officehome} and Table~\ref{tab:office} respectively. With these small-scale datasets, our TML method also has better performance than MME in most cases and can perform the same as MME in other cases. In addition, we observe that TML\_SPL can hurt the performance in some cases while our TML\_DQNPL also makes a progress. The potential reason can be that adding mis-labeled target samples for training causes more damage to the model when the number of the original training samples is small in small-scale datasets. Therefore, these comparisons can further confirm the effectiveness of our method.

\section{Conclusions}

We propose a novel reinforcement learning based selective pseudo-labeling method for semi-supervised domain adaptation (SSDA). We first design a target margin loss for our base model training, which can make the feature distribution closer to the target domain and improve the discriminative ability. Then we apply deep Q-learning to train an agent to select more representative and accurate pseudo-labeled samples for our improved model training. Our method obtains competitive results on several domain adaptation benchmarks and outperforms the present state-of-the-art methods. In addition, the training of the deep Q-network is unrelated to the architecture of base model and will not change the training strategy of base model so that the proposed pseudo-labeling agent can be combined with other advanced methods to help train improved models in the SSDA setting.