\section{Conclusion}
We present a mixed-supervised HOI detection framework (MX-HOI) which employs two state-of-the-art fully- and weakly-supervised  pipelines. Within this framework, we first introduce a momentum-independent strategy to tackle the adversarial effect of full and weak supervision by separating their gradient history in momentum learning. Second, we introduce an HOI element swapping strategy to harvest hard negatives across images for weakly-labeled data. Unlabeled data can also be leveraged using a “pseudo label” solution where class labels on HOI pairs are provided by the trained mixed-supervised pipeline.  Extensive experiments on the large-scale HICO-DET dataset show that,  with only  30\% fully-labeled data and 70\% weakly-labeled data, our MX-HOI is able to retain 93.3\% accuracy of the setting of 100\% fully-labeled data. Future work will be focused on developing a stronger weakly-supervised HOI detection pipeline to integrate it into our MX-HOI framework. \\

\medskip

\noindent {\textbf{Acknowledgements:}
This work was partially supported by the READ-IT project, funded by the JPI Cultural Heritage   under   the   European   Union   Horizon   2020 R\&I   program   (grant   agreement   No. 699523).
Miaojing Shi was supported by the National Natural Science Foundation of China (NSFC) under Grant No. 61828602. }

\section{Experiments}
\label{sec:experiments}
\subsection{Experimental setup}\label{Sec:experimentsetup}
\para{Datasets.}
Ever since its introduction in \cite{chao2018wacv}, the HICO-DET has become the defacto standard dataset for human object interaction detection. The dataset has a total of 47,776 images: 38,118 (80\%) are used for training and 9,658 (20\%) for testing.
Each image is provided with the $\langle human, object, predicate\rangle$ triplets which include human and object bounding boxes and HOI classes.
It covers 80 object categories and 117 interactions, which result into 600 HOI classes in total. These classes are subdivided into 138 \textit{rare} ones, whose training samples are less than $10$ images; $462$ \textit{non-rare} ones, whose training samples are more than $10$ images.
On average, 1.67 HOI triplets are annotated in each image.  HICO-DET is a much bigger dataset compared to the previous V-COCO dataset~\cite{gupta2015arxiv}.  In line with recent works on HOI detection~\cite{gupta2019iccv,bansal2020aaai}, we evaluate our method on the large-scale HICO-DET to offer comprehensive study and in-depth analysis on it.

\vspace{0.1cm}
\para{Data splitting.
The training images are randomly split with different ratios of weakly- and fully-labeled data (denoted as WS and FS). The default WS/FS ratio is set to {70/30 and 30/70} where 70\% (30\%) data from the training set are weakly-labeled and the rest are fully-labeled. We also evaluate different WS/FS ratios ranging between 100/0 and 0/100 in the experiments. Some more settings regarding unlabeled data and class-split are also presented in the end.

\vspace{0.1cm}
\para{Implementation details and evaluation protocol.}
Following~\cite{chao2018wacv}, the human and object detection results are taken from the top scoring output of a Faster-RCNN pretrained on MS-COCO~\cite{mscoco14}.
Each human is paired with all the objects within an image.
Faster-RCNN produces numerous candidate bounding boxes. For each object, we filter the 30-top performing boxes depending on the detection scores.
For fully-labeled data, ground truth HOI triplets are provided with human/object bounding boxes and their interaction.
For weakly-labeled data, only image-level HOI labels are provided meaning that the real correspondence from a human detection to an object is not given in the image. For unsupervised data, no HOI labels are provided. \miaojing{The network is trained with a mini-batch containing the set of region proposal pairs in two images, which are randomly selected from either the fully-labelled set or the weakly-labeled set. This is done once before the training for efficiency. Two images from the weakly-labeled set are applied with element swapping. }
The learning rate is 1e-3 and 1e-4 for weakly- and fully-labeled data, respectively. We train 40,000 iterations in total.

Evaluation of HOI detection employs the widely used mean average precision (mAP) metric where a prediction is considered correct only if its HOI class label is correct, and its human and object bounding boxes have an Intersection over Union (IoU) larger than $0.5$ with their respective ground truth bounding boxes. {

\subsection{Ablation study}\label{sec:ablation}

In this section, we first justfiy the importance of our proposed new elements MIL and HES in order to enable a meaningful mix-supervised HOI detection. Next, we give the result of MX-HOI generalizing over different WS/FS ratios from 0/100 to 100/0.

\vspace{0.1cm}
\para{Using weak and strong annotations.}
To start with our ablation study, we first train our HOI detector with weak annotations only; next, we train the detector with both weak and strong annotations. We illustrate the mAP on the test set in Fig.~\ref{fig:MIL_plot}: by default 70\% (30\%) data are chosen as weakly-labeled and the rest are as fully-labeled. The results show that without using our proposed MIL (w/o MIL), adding FS data can perform even worse than using WS data only: for example using 70\% of the training data, all weakly-labeled (WS/FS = 70/0), it (\emph{weak only}, grey) yields a mAP of 14.68; when adding fully-labeled data (WS/FS = 70/30), the detector performs an even worse mAP 13.17 (red).
This illustrates well the adversarial effect between the weakly- and fully-labeled data.
In order to tackle the problem, we first tried a sequence training (ST) strategy~\cite{shi2016eccv,liu2019cvpr}, where all fully-labeled data (resp. all weakly-labeled data) are presented in the network with some epochs before the weakly-labeled data (resp. fully-labeled data) are added in. We denote the strategy as w/ ST-F when the fully-labeled data are trained first, or w/ ST-W when the weakly-labeled data are trained first.  The results are improved in this manner but not too much (see Fig~\ref{fig:MIL_plot}). Next, we introduce our momentum-independent learning strategy to specifically enable the mixed-supervised HOI detection.

\vspace{0.1cm}
\para{Momentum-independent learning (MIL).}
To tackle the inconsistency of gradients in the network backpropagation,
we introduce two independent momentum to record the gradient history of full and weak supervision separately as they are computed on different error surfaces and are functionally different in the network. This is conceptualized as momentum-independent learning and is a key element of our MX-HOI pipeline.  Having a look at Fig.~\ref{fig:MIL_plot}, ours (w/ MIL) significantly increases the mAP to {e.g.}\xspace 16.63 for WS/FS = 70/30 and 17.47 for WS/FS = 30/70. This demonstrates the importance of our proposed MIL to enable a meaningful mixed-supervised HOI detection. Some more detailed comparisons between our MIL and other variants on the rare and non-rare classes are shown in Table~\ref{tab:MIL}.

\vspace{0.1cm}

\begin{figure}
    \centering
    \includegraphics[scale=0.35]{figures/o_MIL_mAP_full.eps}
    \vspace{-2mm}
    \caption{Using weak, strong and mixed annotations for HOI detection. MIL: momentum-independent learning. ST-W: sequence training with weakly-labeled data first; ST-F: sequence training with fully-labeled data first;
    }\label{fig:MIL_plot}
\end{figure}

\begin{table}[t]
  	\setlength{\tabcolsep}{1.5pt}
    \centering
    \small
    \begin{tabular}{cccc|ccc}
    \toprule

   \textbf{Method} &   \textbf{WS/FS} &  \textbf{Rare} & \textbf{Non-Rare} & \textbf{WS/FS} & \textbf{Rare} & \textbf{Non-Rare}\\
        \midrule
        weak only & 70/0  & 	11.84	&	15.72 & 30/0 	&	8.68	&	14.11 \\
        w/o MIL  &  70/30  & 		8.88	&	14.45 &  30/70 	&	9.17	&	15.74 \\
        w/ ST-F & 70/30 & 10.41	& 16.69	&  30/70  &	10.52 &	 17.00\\
        w/ ST-W  & 70/30  & 10.17	& 16.71& 30/70 	& 11.03	& 16.43	\\
        with MIL (ours) &  70/30 & 	\textbf{12.36}	&	\textbf{17.91} & 30/70 	&	\textbf{12.79}	&	\textbf{18.80}\\
\bottomrule
    \end{tabular}
   \vspace{-2mm}
 \caption{Ablation of momentum-independent learning (MIL) in MX-HOI on HICO-DET dataset. mAP is reported. }
    \label{tab:MIL}
\end{table}

\begin{figure*}[t]
\begin{center}
\includegraphics[scale=1.5]{./figures/output_w_text_1.eps}
\vspace{-3mm}
\caption{Examples of correct detections (top) and incorrect detections (bottom) with MX-HOI. The classes shown here are Carry-Sheep, Set-Umbrella, Repair-Parking-meter, Make-Vase and Hold-Hairdryer.}
\label{fig:hes}
\vspace{-3mm}
\end{center}
\end{figure*}

\vspace{0.1cm}
\para{HOI element swapping (HES).} Referring to Sec.~\ref{sec:mixproposals}, HOI element swapping is introduced for hard negative harvest on weakly-labeled data. To verify its effectiveness, we ablate it in Table~~\ref{tab:HES} by comparing with MX-HOI without HES. We vary WS/FS from 100/0 to 30/70 and show that on different mixed levels, HES always helps the HOI detection. For instance, when WS/FS = 70/30, MX-HOI yields +0.81\% improvement over MX-HOI (w/o HES).

Additionally, we also apply HES on fully-labeled data, {i.e.}\xspace WS/FS = 0/100, and obtain the mAP 17.04, 13.35, 18.11 on full, rare and non-rare classes, respectively, which actually harms the performance on non-rare classes while helps a bit on rare classes comparing to 17.82, 12.91, and 19.17 in Table~\ref{tab:ratio1}. Rare classes do not have adequate training samples, HES can help provide hard negatives; while for non-rare classes, hard negatives can be directly mined via the given bounding box ground truth. Overall, we did not find HES to be effective for fully-labeled data in general.

\begin{table}[t]
  	\setlength{\tabcolsep}{3.5pt}
    \centering
    \small
    \begin{tabular}{ccccc}
    \toprule

  \textbf{Method}  &   \textbf{WS/FS} &  \textbf{Full} & \textbf{Rare} & \textbf{Non-Rare} \\
        \midrule
        ours (w/o HES) & 100/0  & 15.14 & 10.65  & 16.48 \\
        ours &  100/0  &\textbf{16.14} & \textbf{12.06} & \textbf{17.50}\\
        \midrule
          ours (w/o HES) & 70/30 & 15.82 & 10.39 & 17.41	\\
        ours &  70/30  & 	\textbf{16.63}	&	\textbf{12.36}	& \textbf{17.91}	\\ \midrule
         ours (w/o HES) & 30/70  &	16.73	&	12.00	    &	18.14	\\
        ours &  30/70   & \textbf{	17.47}	&	\textbf{12.79}	& \textbf{18.80}	\\
     \bottomrule
    \end{tabular}
    \vspace{-2mm}
 \caption{Ablation of HOI element swapping (HES) in MX-HOI on HICO-DET dataset. mAP is reported. }
    \label{tab:HES}
 \vspace{-2mm}
\end{table}

\vspace{0.2cm}
\para{WS/FS variations.} We offer the results of MX-HOI with different WS/FS: 100/0, 80/20, 70/30, 50/50, 30/70, 20/80 and 0/100 in Table~\ref{tab:ratio1}. One can see that the performance increases with an increase of FS for both rare and non-rare classes, as more fully-labeled data are added into the training. The overall full mAP increases from 16.14 to 17.82. In Table~\ref{tab:ratio1}, we also show the result of fixing either WS, or FS to 30\% while varying the other: the performance increases along with the training (sub-)set size;
but the improvement margin is bigger when fixing WS and increasing FS compared to fixing FS and increasing WS.
All these results make perfect sense for MX-HOI: adding more labeled data, regardless WS or FS, increases its performance; FS data in general provides more help than WS data.
Examples of MX-HOI with WS/FS=70/30 are given in Fig.~\ref{fig:hes}.

\begin{table}[t]
  	\setlength{\tabcolsep}{3.5pt}
    \centering
    \small
    \begin{tabular}{cccc}
    \toprule

        \textbf{Supervision (WS/FS)} &  \textbf{Full} & \textbf{Rare} & \textbf{Non-Rare} \\
        \midrule
                 100/0  &  16.14 & 12.06 & 17.50 \\
                 80/20 & 16.49 & 12.28 & 17.81 \\

                 70/30  & 16.63 & 12.36 & 17.91 \\
                 50/50  & 17.08 & 12.58 & 18.17 \\

                30/70  & 17.47 & 12.79 & 18.80 \\
                 20/80 &  17.60 &   12.85  & 18.95  \\
                 0/100  & 17.82 & 12.91 & 19.17 \\ \bottomrule
                 30/30  &	16.05	&	11.64	&	17.37\\
                 30/50  & 	16.84	&	11.81	&	18.47\\

                30/70  & 17.47 & 12.79 & 18.80 \\
                 50/30  &	16.34	&	12.04	&	17.69	\\

                70/30  & 16.63 & 12.36 & 17.91 \\
            \bottomrule
    \end{tabular}
    \vspace{-2mm}
    \caption{Different ratios of WS/FS in MX-HOI on HICO-DET dataset (top). Fixing WS (resp. FS) ratio and varying the other (bottom). mAP is reported.}
    \label{tab:ratio1}
    \vspace{-3mm}
\end{table}

\subsection{Comparison to state of the art}

In Table~\ref{tab:sota-res}, we first compare our method to its lower and upper bounds denoted respectively by WS-No-Frills and No-Frills in the following. WS-No-Frills is an adaption of a representative weakly-supervised relationship detection module~\cite{zhang2017iccv} onto the SOTA HOI detection pipeline~\cite{gupta2019iccv}, which produces mAP 15.14, 10.65, and 16.48 on full, rare and non-rare classes. Our MX-HOI under the same setting WS/FS=100/0 improves the result to 16.14, 12.06, 17.50 due to the adoption of HOI element swapping. In fully-supervised setting (WS/FS = 0/100), our MX-HOI also improves the No-Frills~\cite{gupta2019iccv} by +0.6\%, this is attributed to our two-branch softmax~\cite{bilen2016cvpr} in No-Frills \miaojing{(This softmax is applied in subbranches before the final classification head).} Despite MX-HOI is introduced for mix-supervised HOI detection, it also improves the SOTA bounds as side benefits.

In the mixed-supervised setting, MX-HOI retains 93.3~\% accuracy of the SOTA No-Frills by using a mixture of 30\% fully-labeled data and 70\% weakly-labeled data. To compare with it, we implement a naive \emph{multi-stage training pipeline}: it first trains the model on 30\% fully-labeled data, then infers the HOI class probabilities on the human-object pairs in the rest 70\% weakly-labeled images; the HOI triplet with the largest probability is selected as pseudo ground truth for each given HOI label on the image-level in weakly-labeled data. These selected HOI triplets are mixed with existing fully-labeled data to train the network again. The network remains a fully-supervised pipeline in this manner. This process would repeat several rounds until the convergence of the model. We obtain mAP 15.23, 10.63, and 16.61 under the setting of WS/FS = 70/30, which is much lower than our MX-HOI (16.63, 12.36 and 17.91).

We also compare MX-HOI with other recent arts
~\cite{xu2019cvpr,gao2018bmvc,qi2018eccv,ulutan2020vsgnet,wan2019pose,li2019transferable} using 100\% supervision. One can see that with WS/FS = 70/30, MX-HOI performs very close to the SOTA.

\begin{table}[t]
    \centering
    \begin{tabular}{c|c|ccc}
        \textbf{Methods} &\textbf{WS/FS} &  \textbf{Full} & \textbf{Rare} & \textbf{Non-rare} \\ \hline
                WS-No-Frills &  \multirow{2}{*}{100/0} & 15.14 & 10.65  & 16.48 \\
                 \textbf{MX-HOI} & & \textbf{16.14} & \textbf{12.06} & \textbf{17.50} \\
                 \cline{1-5}
                  \textbf{MX-HOI} & 70/30  &  16.63 & 12.36 & 17.91 \\

                   \textbf{MX-HOI} &30/70  & 17.47 & 12.79 & 18.80\\
                 \cline{1-5}
                 \textbf{MX-HOI} & \multirow{6}{*}{0/100}  & \cellcolor{gray!25}{17.82} & \cellcolor{gray!25}{12.91} &\cellcolor{gray!25}{19.17} \\
                 No-Frills \cite{gupta2019iccv} &  & 17.18 & 12.17 & 18.68\\
                {VSGNet} \cite{ulutan2020vsgnet} &  & \textbf{19.80} & \textbf{16.05} & \textbf{20.91}\\
               {PMFNet} \cite{wan2019pose} &  & 17.46 & 15.65 & 18.00\\
               {TIN} \cite{li2019transferable} &  & 17.22 & 13.51 & 18.32\\

                GCN-HOI~\cite{xu2019cvpr} &  & 14.70& 13.26 & 15.13 \\
            GPNN \cite{qi2018eccv} &  & 13.11  & 9.34  & 14.23 \\
            ICAN \cite{gao2018bmvc} & & 12.80 & 8.53 & 14.07 \\

             \hline
    \end{tabular}
     \vspace{-1mm}
    \caption{Comparison with the state-of-the-art methods on HICO-DET test set (mAP).}
    \label{tab:sota-res}
    \vspace{-4mm}
\end{table}

\subsection{More settings}
\para{Unlabeled data} can also be added into the whole framework: we first obtain all possible human-object pairs in an unlabeled image (US) from the detection result. Given the trained model of the mix-supervised pipeline, we can estimate the marginal HOI class probability for every human-object pair in the unlabeled image. If the probability is larger than a threshold ({e.g.}\xspace 0.5), we take the predicted HOI class as the pseudo ground truth for this human-object pair and add it into the network training in the next cycle. The loss function in (\ref{eq:loss}) now includes another term for the unlabeled data, which is formulated similarly to the fully-labeled data with pseudo ground truth. The loss weights among the three terms remain 1.  This process iterates for several cycles until the convergence of the network.

Table~\ref{tab:unlabel} shows the result of WS/FS/US being 30/40/30, where 30\%, 40\% and 30\% percent data are respectively weakly-, fully- and un-supervised (see Sec.~\ref{Sec:experimentsetup}). Results using unlabeled data improves performance when compared to using only WS/FS with 30/40 ratio.

\begin{table}[t]
  	\setlength{\tabcolsep}{3.5pt}
    \centering
    \small
    \begin{tabular}{ccccc}
    \toprule

   &   \textbf{WS/FS/US} &  \textbf{Full} & \textbf{Rare} & \textbf{Non-Rare} \\
        \midrule

        MX-HOI &  30/40/0  & 16.08 & 12.05 & 17.29 \\
          MX-HOI & 30/40/30  & 16.53 & 11.63 & 17.79 \\

     \bottomrule
    \end{tabular}
    \vspace{-2mm}
 \caption{Adding unlabeled data (US) into MX-HOI. }
    \label{tab:unlabel}
\vspace{-3mm}
\end{table}

\vspace{2mm}
\para{Class-split:} Instead of randomly splitting the dataset images for weak and full supervision, we can randomly split the whole HOI classes into 50\% vs. 50\%. Images from the first 50\% classes are trained with full supervision, the second 50\% with weak supervision. If we train two models separately on the two sets, we got mAP 13.3 and 11.6 on the test set of each own part of classes, respectively. If we train one model over the two sets jointly using MX-HOI, the mAP increases to 14.8 and 13.10. Despite the two sets are of different classes, training them together with more data benefit the performance of both in our pipeline.}

\section{Introduction}

\begin{figure}
    \centering
    \includegraphics[scale=0.46]{figures/introduction_R.eps}
    \caption{\small Human-object interaction detection with different levels of supervision. Top: annotation cost increases from image-level ($x$) labels (left) in weakly-supervised learning to region-level ($r$) bounding boxes (middle) and their correspondences (right) in fully-supervised learning. Bottom: our proposed mixed-supervised HOI detection pipeline (middle) enables joint learning of weakly- and fully-supervised HOI detection (left and right).
    }\label{fig:introduction}
    \vspace{-1mm}
\end{figure}

The task of human-object interaction (HOI) detection is defined as a detection of a triplet ${\langle human, verb, object \rangle}$, identifying not only the bounding boxes of human and object but also their interaction~\cite{gkioxari2018detecting,xu2019mm,gupta2019iccv,qi2018eccv,chao2018wacv,zhou2019iccv,delaitre2011nips,desai2010cvprw}. It is derived from visual relationship detection (VRD) of triplet  ${\langle object_1, predicate, object_2 \rangle}$
~\cite{lu2016eccv,gupta2015arxiv,gupta2008eccv,gupta2009pami,ramanathan2015cvpr,atzmon2016arxiv,zheng2019mm,sun2019mm,zhou2019mm}, but present different challenges: the predicates in VRD can be verbs ({e.g.}\xspace ``push"), spatial ({e.g.}\xspace ``on top of"), prepositions ({e.g.}\xspace ``with"), comparative ({e.g.}\xspace ``
taller than"), while in HOI they are mainly verbs. On the other hand, \emph{human-centric} interactions are more diverse and complicated, one person can easily interact with multiple objects in the meantime, {e.g.}\xspace ``person wearing a jacket and riding a bicycle". This makes HOI a much more fine-grained task than VRD.

Intensive attention has been drawn to HOI alongside the introduction of new benchmarks, {i.e.}\xspace V-COCO~\cite{gupta2015arxiv},  HICO-DET~\cite{chao2015iccv}, featured with diverse and numerous human-object interactions. For instance, in HICO-DET, there exist 600 HOI classes and 80 common object classes in total. Despite that recent advances have reported significant improvement~\cite{gupta2019iccv,qi2018eccv,chao2018wacv,zhou2019iccv}, the annotation cost is exponentially increased in these datasets. Given $N$ humans and $M$ objects in an image, the maximum number of HOI is $N \times M$, where we have to scan each of them and provide instance-level annotations ({e.g.}\xspace bounding boxes) for the real ones (see Fig.~\ref{fig:introduction}: top-right). To alleviate this manual labor, we could provide only image-level HOI labels: a set of images are known to contain triplets of a certain HOI class, but the location and correspondence of objects are unknown in images (see Fig.~\ref{fig:introduction}: top-left); note the location of objects are assumed known sometimes (top-middle). Both cases can be conceptualized as weakly-supervised HOI detection.

There are few works that learn interactions from weak supervision. One representative is for VRD~\cite{zhang2017iccv}, which designs a weakly-supervised predicate prediction module inspired from the two-branch parallel structure in~\cite{bilen2016cvpr}. This can be easily adapted to HOI detection, as shown in Fig.~\ref{fig:introduction}: bottom-left.
Nevertheless, in real application, instead of having only one type of labels, we can have a mixture of them: fully-labeled (instance-level), weakly-labeled (image-level), and even unlabeled.

A generalized HOI detection framework for mixed supervision thus becomes necessary.
An intuitive solution would be merging the weakly-supervised and fully-supervised HOI detection as a multi-task job,
which is nevertheless not straightforward: the different quality of annotations between weakly-labeled and fully-labeled data, as well as their imbalanced ratios
should be considered.
Fig.~\ref{fig:MIL_plot} illustrates an example: in the HICO-DET dataset, when adding different amounts of fully-labeled data (in red), results are either only slightly better than or even worse than learning with only weakly-labeled data (in grey). This simple combination, at best, does not exploit the full potential that could be derived from fully-supervised data, at worst, decreases the results obtained with weak supervision.
This is the first challenge that needs to be solved in the mixed-supervised setting. Furthermore, HOI detection is a fine-grained task requiring the classification of similar interactions such as "eating", "drinking", "blowing". To be able to accurately distinguish them, diverse and hard negatives from similar interactions are essential for the network training. Nevertheless, due to the reason that many samples are weakly-labeled, interactions within them can not be clearly discriminated on the region-level; plus, some interaction classes are not even sufficiently collected. This poses another challenge for HOI detection.

\para{Contributions.} We for the first time propose a generalized framework for mixed-supervised HOI detection (MX-HOI):
\begin{compactitem}

\item We integrate two state-of-the-art pipelines \cite{gupta2019iccv} and \cite{zhang2017iccv} for fully- and weakly-supervised HOI detection into a mixed-supervised pipeline.

\item To tackle the multi-task optimization in the mixed pipeline, we introduce a {momentum-independent} learning strategy to tackle the adversarial effect between full and weak supervision, by separating their gradient history in momentum learning.

\item To tackle the annotation insufficiency in the mixed supervision, we introduce an HOI element swapping strategy to specifically harvest hard negatives across images for the weakly-labeled data.
\end{compactitem}
\noindent By conducting our generalized HOI detection framework on the most challenging HICO-DET dataset, we show our method enables HOI detection with a mixed amount of supervision, {e.g.}\xspace with 30\% fully labeled data and 70\% weakly-labeled data, we are able to retain 93.3\% accuracy of the setting of 100\% fully-labeled data. Furthermore, 1) our model improves both the state of the art weakly- and fully-supervised HOI detection methods~\cite{zhang2017iccv,gupta2019iccv} under the same supervision; 2) unlabeled data can also be leveraged into our pipeline following a "pseudo label" solution~\cite{lee2013icmlw}, where we can use the network trained on labeled data to infer labels of HOI pairs on unlabeled data.

\section{Mixed-supervised HOI detection}\label{sec:method}

\subsection{Preliminary}\label{sec:preliminary}
We build our MX-HOI framework on two state of the art HOI works with full supervision~\cite{gupta2019iccv} and weak supervision~\cite{zhang2017iccv}, respectively.
\cite{gupta2019iccv} introduces a no-frills model for HOI detection where they use appearance features from pretrained object detectors, spatial features through box layout, and encode human pose keypoints, as shown in Fig.~\ref{fig:method_overview}: feature extraction. This is a no-frills detection without relying on attention or graph-based message passing~\cite{zheng2019mm,qi2018eccv}. It uses a factorized multi-layer perceptrons (MLPs) and introduces several new training techniques to improve the MLPs: eliminating a train-inference mismatch, rejecting easy negatives using indicator terms, and training with large negative to positive ratios.

\cite{zhang2017iccv} adopts the weakly-supervised object detection pipeline~\cite{bilen2016cvpr} for weakly-supervised predicate prediction (WSPP): it is accomplished via the {element-wise} multiplication of the predicate selection and classification branch (see Fig.~\ref{fig:method_overview}). The predicate score is softmax normalized over all candidate human-object box pairs with respect to a predicate class for the selection branch, and over all possible predicate classes with respect to one human-object pair for the classification branch, respectively. The predicate score in~\cite{zhang2017iccv} is obtained from a position-role sensitive score maps with a pairwise ROI pooling. To integrate it into the no-frills model above, we use the conventional ROI pooling. Predicate scores are predicted from the FC layers of the two branches, which is similar to the original structure in~\cite{bilen2016cvpr}.

\subsection{Overview}\label{sec:overview}
We introduce a mixed-supervised HOI detection framework (MX-HOI) as shown in Fig.~\ref{fig:method_overview}: the input for MX-HOI is region proposals output from a pretrained object detector. We follow the same procedure as in~\cite{gupta2019iccv} to extract both appearance and layout features for the human and object bounding boxes in a pair. Given human-object pairs, their region features are fed into the adapted two-branch predicate prediction structure from~\cite{zhang2017iccv} (Sec.~\ref{sec:preliminary}). The output of the two branches (matrices) are multiplied element-wise to produce one $N \times C$ matrix $P$ over $N$ human-object pairs (in a batch) and $C$ interaction classes. Each element $p_{ij}$ indicates the probability of the $i^\text{th}$ human-object pair having $j^\text{th}$ interaction type. For fully-labeled data, the predicate prediction is optimized on the region-level on matrix $P$, where a corresponding ground truth matrix is associated with each element being 1 or 0 indicating the true or false for the human-object interaction. For weakly-labeled data, the predicate optimization is on the image-level:  $P$ is accumulated over rows ($\sum_i p_{ij}$) to produce a $C$-dimensional vector where each element signifies the probability of the image containing certain HOI class. Similarly, an image-level ground truth vector with elements 1 or 0 is associated.

This is a multi-task optimization defined jointly with full and weak supervision. The learning is not straightforward: the optimization in the two learning manners is different as one focuses on the image-level and the other on the region-level ($x$ and $r$ in Fig.~\ref{fig:introduction}); the error surface toggles between the gradients from weak supervision and full supervision across different batches in the network.
We therefore propose a momentum-independent learning strategy (Sec.~\ref{sec:moment}). Besides, for the weakly-labeled data in mixed-supervised learning, we introduce an HOI element swapping strategy (Sec.~\ref{sec:mixproposals}) to further augment the hard negatives.
Loss function is given in the end (Sec.~\ref{sec:loss}).

\subsection{Momentum-independent Learning}\label{sec:moment}

In the context of mixed-supervised learning, network weights are updated by either weak or full supervision, depending on the samples within the mini-batch.
Most recent gradient descent based optimizers use momentum-based weights update. Let $w_t$ and $\nabla f(w_{t})$ be respectively the weight and the gradient at iteration $t$, and $\alpha$ be the step size, the momentum-based update rule is given by:
\begin{align}  \label{eq:mom_based}
\vspace{-3mm}
    w_t = w_{t-1} - z_t; ~~
    z_t = \beta z_{t-1} + \alpha \nabla f(w_{t-1})
    \vspace{-3mm}
    \end{align}
where $\beta$ is the momentum parameter (usually $\beta \geq 0.9$) and $z_t$ is the momentum, which is dependent on all the previous gradient values.

Using momentum-based gradient descent can however be a problem in the mixed-supervised learning. In the fully-supervised case, the ground truth is directly given on the instance level such that the gradient of the loss function will accordingly backpropagate to the specific regions of the human and object in a pair. In the weakly supervised case, the ground truth is instead only given on the image-level, and predictions on all possible human-object pairs are aggregated together to the image-level for loss computation; at the backpropagation time, the gradient is distributed among all the human-object pairs. As a result, the gradients for full and weak supervision are computed on different error surfaces and are not compatible. Using one momentum to record both will make the network weight optimization toggles between the two sources of gradients across mini-batches. This indeed leads to an adversarial effect of the mixed-supervised learning (see the ablation study in Sec.~\ref{sec:ablation}).

To mitigate this, we propose to bootstrap the mixed-supervised learning with two independent momentum $z_t^{ws}$ and $z_t^{fs}$ to record the gradient history of weak and full supervision separately. $z_t^{ws}$ will be used and updated only with weakly-labeled samples in the mini-batch, while $z_t^{fs}$ will be instead used for the fully-labeled samples. $w_t$ however is remained to be shared in the network such that the weakly- and fully-supervised pipeline are jointly optimized:
\begin{equation}\label{eq:mom_modified}
\begin{split}
\vspace{-5mm}
   w_t = w_{t-1} - z_{t}^{ws} ; \;\;\;\; z_{t}^{ws} = \beta z_{{t-1}}^{ws} + \alpha \nabla f(w_{t-1}) \\
   w_t = w_{t-1} - z_{t}^{fs} ; \;\;\;\; z_{t}^{fs} = \beta z_{t-1}^{fs} + \alpha \nabla f(w_{t-1})
\end{split}
\end{equation}

\subsection{HOI element swapping}\label{sec:mixproposals}

\begin{figure}
    \centering
    \includegraphics[scale=0.5]{figures/HOI2.eps}
    \caption{Illustration of HOI element swapping. Top: Object classes from the two images are different. Bottom: same object class in the two images. Swapped pairs ($h_i$, $o_j$) are negatives due to wrong object class (top) or wrong spatial layout (bottom).}
    \label{fig:Mixup_HO}
    \vspace{-5mm}
\end{figure}

HOI detection is a fine-grained task. To accurately classify similar interactions, diverse and hard negatives are needed. In the fully-supervised case, where region-level ground truth are available, this can be achieved via choosing the false positive class of large confidence score or false positive region of large intersection-over-union (IoU) with ground truth.
While in the weakly-supervised case, where only image-level ground truths are available, conventional manners of finding negatives no longer apply. We instead introduce an HOI element swapping way to collect diverse and hard negatives across images.

Suppose that two images $im_1$ and $im_2$ contain one human $h$ and one object $o$ inside each, respectively.
The standard way to create the candidate human-object pairs is to group ($h_1$, $o_1$) and ($h_2$, $o_2$) within each image (see Fig.~\ref{fig:Mixup_HO}). To further augment negatives, a simple way is to lower the threshold from the RPN to produce more proposals; but this is inefficient as many proposals with low confidence scores are not good detection of humans or objects, and we do not have ground truth bounding boxes to distinguish them in the weakly-supervised setting. Hence, we propose to keep a fair detection of humans and objects within each image and augment the human-object pairs across images by swapping their HOI elements: given ($h_1$, $o_1$) from $im_1$ and ($h_2$, $o_2$) from $im_2$, we mix the human proposal from the $im_1$ with the object proposal from  $im_2$ and vice versa: ($h_1$, $o_2$) and ($h_2$, $o_1$); this gives us two more mixed human-object pairs.
One image may contain more than one human or object. Considering there are $H_1$ humans and $O_1$ objects in $im_1$, $H_2$ humans and $O_2$ objects in $im_2$, selecting all humans and objects from the two images will produce $(H_1 + H_2) \times (O_1 + O_2)$ pairs in total for the two images.
which is far too much. In practice, we remove those easy negatives with low confidence scores such that the number of human-object pairs is kept the same to that of the original number in two images, {i.e.}\xspace  $H_1 \times O_1 + H_2 \times O2$.

By doing this, we can obtain more diverse combinations of HOI pairs, where many swapped HOI pairs coming from two images might look like positive HOI pairs playing the role of hard negatives. For instance, in Fig.~\ref{fig:Mixup_HO}: bottom, the object class from two images is the same in particular, yet the swapped human-object pairs  ($h_1$, $o_2$) and ($h_2$, $o_1$) are still negatives due to the wrong spatial layout.

The augmented human-object pairs as shown in Fig.~\ref{fig:Mixup_HO} can be hard negatives for both images. To efficiently optimize the learning on two images, we propose to aggregate all the human-object pairs from two images to form one image-level  HOI label vector, where the corresponding ground truth is the HOI labels from both images. Apart from efficiency,  another benefit of doing this, compared to optimizing the image separately,  is that the positive human-object pairs from one image could also serve as negatives for the other image if they are of different HOI labels or as positives if they are of the same HOI label.

\subsection{Loss function}\label{sec:loss}
Loss function is defined within each mini-batch depending on the supervision $\mathbb{S}$ of the input samples, which can be full supervision $(\mathbb{S = FS})$ or weak supervision $(\mathbb{S=WS})$:

\begin{equation}\label{eq:loss}
\begin{split}
{\mathcal{L}_{\text{mini-batch}}} & = \sum_{j=1}^{C} \bigg (\mathbf{1}(\mathbb{S = FS}) \frac{1}{N} \sum_{i=1}^{N} BCE(y_{ij},p_{ij})  \\
& + \mathbf{1}(\mathbb{S = WS}) BCE(y_j,p_j)\bigg ) \\
\end{split}
\end{equation}
BCE is the binary cross-entropy; $p_{ij}$ is the probability of $j^\text{th}$ HOI class for the $i^{th}$ human-object pair, where there are $N$ pairs and $C$ classes in total; $p_j$ is the probability of the $j^\text{th}$ HOI class for the given images in the mini-batch. The former is defined for the fully-labeled data with region-level ground truth, while the latter is defined for the weakly-labeled data with image-level ground truth only. In practice, we feed the features of human-object pairs from two images in each mini-batch.
Referring to Sec.~\ref{sec:overview}, the ground truth for fully-labeled data is given in a form of a matrix, where each element $y_{ij}$ indicates whether the $i^{th}$ human-object pair with $j^\text{th}$ HOI class is true or false. $y_{ij} = 1$ if the human and object boxes in the $i^{th}$ pair have an IoU greater than 0.5 with a ground truth box-pair of the $j^\text{th}$ HOI class. The ground truth for the weakly-labeled data
is given in a form of $C$-dimensional vector where its element $y_j = 1$ if the $j^\text{th}$ HOI class occurs in any of the two images.

\section{Related Work}
\para{Visual relationships} were originally used to help improve object localization~\cite{gupta2008eccv,kumar2010cvpr,sadeghi2011cvpr}, action
recognition and pose estimation~\cite{desai2012eccv,ramanathan2015cvpr} or semantic segmentation~\cite{gould2008ijcv}. Relationships that are often modelled between objects include verbs, actions, spatial and prepositions~\cite{sadeghi2011cvpr,yatskar2016cvpr,gupta2015arxiv,gupta2008eccv,gupta2009pami,ramanathan2015cvpr,atzmon2016arxiv,han2018mm,cui2018mm}.
\cite{lu2016eccv} was the first work
to formulate the detection of visual relationships as a separate task. They propose to leverage
language priors from semantic word embedding to finetune the likelihood of a predicted
relationship. Subsequently, many researchers improved and generalized this model~\cite{yu2017iccv,liang2017cvpr,cui2018mm}.
Triplet metric learning is also adopted to optimize the visual feature connections among semantically related objects in~\cite{li2017cvpr,sun2019mm}.
Attention~\cite{han2018mm,zheng2019mm} and spatial locations~\cite{zhou2019mm} are some other additive cues to visual relationship detection.

\para{Human-Object Interaction} is a concept related to visual relationship. The interactions between humans and objects are mainly focused on verbs, and are much more fine grained ({e.g.}\xspace holding, hitting, throwing, touching) than relationships between generic objects. The study of HOI dates back to~\cite{delaitre2011nips,desai2010cvprw,gupta2009pami,prest2011pami,xu2019mm}, when most works were tested on small datasets. This issue was addressed by Chao~{et al.}\xspace~\cite{chao2015iccv} where they introduced a large dataset (HICO-DET) covering 80 common object categories and 600 HOI categories in total. Many recent works report their performance on this dataset and significant improvement has been achieved~\cite{qi2018eccv,chao2018wacv,gupta2019iccv,liuamplifying, gao2020drg,li2020pastanet,wan2019pose,wang2020learning,li2019transferable, ulutan2020vsgnet}. For instance, Qi~{et al.}\xspace~\cite{qi2018eccv} proposed a graph parsing neural network for HOI and was later extended by~\cite{hu2019iccv,xu2019cvpr,zhou2019iccv}; Gupta~{et al.}\xspace~\cite{gupta2019iccv} showed that a simple factorised model with appearance and layout encoding constructed from pretrained object detectors outperforms more sophisticated approaches; \miaojing{additional cues such as language features~\cite{gao2020drg}, parts based features~\cite{li2020pastanet, wan2019pose} are also exploited.} Our MX-HOI is built on the recent advance of \cite{gupta2019iccv}.

\begin{figure*}[t]
    \centering
    \includegraphics[scale = 0.55]{figures/overview2.eps}
    \caption{\small Illustration of our mixed-supervised HOI detection pipeline (MX-HOI). Human and object bounding boxes are obtained via an object detector. Human-object pairs are randomly created within an image and also across another image via the proposed HOI element swapping. HOI detection is realized via a two-branch structure in parallel for interaction classification and selection. Each branch consists of FC layer ($w_c$/$w_s$) for score prediction and softmax layer ($\sigma_c$/$\sigma_s$) for score normalization over rows or columns of the score matrix, respectively. The score matrices in the two branches are of size $N$ (human-object pairs) and $C$ (HOI classes) and are multiplied to produce the final matrix $P$.  Training data with full and weak supervision ($fs$, $ws$) are optimized with region-level and image-level ground truth, respectively. We introduce a momentum-independent strategy to enable the mixed-supervised learning with two momentum $z^{fs}$ and $z^{ws}$. Human-object pairs from two images are optimized in one batch.
    }

    \label{fig:method_overview}
\vspace{-1mm}
\end{figure*}

\para{Weakly-supervised learning} in visual recognition is mostly focused on object detection~\cite{bilen2016cvpr,shi2016eccv,tang2017cvpr,shi2017iccv,tang2018pami,yang2019arxiv}. One seminal work in weakly-supervised object detection (WSOD) is~\cite{bilen2016cvpr}, where they designed a popular two-branch parallel structure followed by~\cite{tang2017cvpr,tang2018pami,yang2019arxiv}. Weakly-supervised relationship detection is more complex than WSOD as we need to detect individual objects for specific relations. Pretrained object detectors are normally assumed in this scenario~\cite{peyre2017iccv,zhang2017iccv,peyre2019iccv}.
Peyre~{et al.}\xspace~\cite{peyre2017iccv} proposed a weakly-supervised discriminative clustering model to learn relations with only image-level labels; later on, they developed another model for transfer by analogy to obtain visual phrases of never seen relations~\cite{peyre2019iccv}.
Zhang~{et al.}\xspace~\cite{zhang2017iccv} adopted the WSOD module in~\cite{bilen2016cvpr} to do weakly-supervised relationship detection and achieved very good results.
In this work, we also adopt the WSOD module following~\cite{zhang2017iccv} and adapt it to be part of our MX-HOI pipeline.

\para{Mixed-supervised learning} normally refers to learning from a mixture of strongly labeled data and weakly-labeled data. For instance, Cinbis~{et al.}\xspace~\cite{cinbis2016pami} considered mixed supervision in object detection where some images are annotated with bounding boxes while some are only with image-level labels. Papandreou~{et al.}\xspace~\cite{papandreou2015iccv} studied the problem for semantic image segmentation from a combination of few strongly labeled (pixel-level annotations) and many weakly labeled (image-level labels or bounding boxes) images. Mixed-supervised learning can also be realized as leveraging an existing dataset of fully-labeled training images of non-target classes during the weakly-supervised learning of a new object category, which is connected to transfer learning, see {e.g.}\xspace \cite{deslaers2012pami,shi2015bmvc,shi2017iccv,yang2020nips}.


\section{Introduction}\label{sec:intro}
Galactic cosmic rays are believed to be accelerated by shockwaves in supernova remnants (SNR) via the diffusive shock acceleration \citep[DSA;][]{1978MNRAS.182..147B,1978ApJ...221L..29B}, and cosmic ray induced non-thermal radiation has been predicted and/or detected from various SNRs \citep[e.g.,][]{1994A&A...285..645A,1994A&A...287..959D}. In general, the classical DSA theory assumes uniform density distribution of the interstellar medium (ISM) surrounding SNRs. However, observational results indicated that non-thermal X-ray and/or gamma-ray bright SNRs are tightly interacting with dense and clumpy gaseous medium such as molecular and atomic clouds \citep[e.g.,][]{2003PASJ...55L..61F,2017ApJ...850...71F,2008A&A...481..401A,2016ApJ...826...34Z,2020ApJ...902...53S}. A key issue at present is how shock-cloud interaction affects radiation processes of the non-thermal radiation, as well as acceleration mechanisms of cosmic rays beyond the DSA.

\begin{figure*}[]
\begin{center}
\includegraphics[width=\linewidth,clip]{rxj1713_alma_fig1_v10.pdf}
\caption{(a) TeV gamma-ray excess map of RX~J1713.7$-$3946 in equational coordinates obtained by H.E.S.S. \citep{2018A&A...612A...6H}. Superposed contours indicate the ISM proton column density \citep{2012ApJ...746...82F}. The contour levels are 6, 7, 8, 10, 12, 14, 16, 18, 20, and $22 \times 10^{21}$ cm$^{-2}$. Green rectangles represents the ALMA observed area (small rectangle) and to be presented area in Figure \ref{fig1}b. The positions of CO peaks C, D, and L are shown. (b) ALMA CO map toward northwest of RX~J1713.7$-$3946. Superposed contours indicate the X-ray intensity obtained with {\it{Chandra}} \citep{2007Natur.449..576U}. The lowest contour and contour intervals are $9.5 \times 10^{-8}$ and $9.5 \times 10^{-9}$~photons~cm$^{-2}$~s$^{-1}$~pixel$^{-1}$, respectively. The colored image in the top left indicates {\it{Chandra}} X-ray map in the same region as shown in Figure \ref{fig1}b.}
\label{fig1}
\end{center}
\end{figure*}%

The core-collapse SNR RX~J1713.7$-$3946 (a.k.a. G347.3$-$0.5) provides the best laboratory to test the effect because of its bright non-thermal X-rays and TeV gamma-rays \citep[e.g.,][]{1997PASJ...49L...7K,2004Natur.432...75A}, in addition to certain interactions with dense gas clouds at the closed distance of 1~kpc \citep[e.g.,][]{2003PASJ...55L..61F,2005ApJ...631..947M,2010ApJ...724...59S,2012MNRAS.422.2230M} and its young age of $\sim$1600~yr \citep[][]{1997A&A...318L..59W,2003PASJ...55L..61F,2016PASJ...68..108T}. The X-rays are dominated by synchrotron radiation up to 120~keV \citep[e.g.,][]{1999ApJ...525..357S,2008ApJ...685..988T,2019ApJ...877...96T,2019MNRAS.489.1828K}. {\it{Chandra}} observations discovered year-scale time variability of X-ray hotspots on the order of 10~arcsec or 0.05~pc widths as well as X-ray filaments of $\sim$0.1--0.2~pc widths, indicating efficient cosmic-ray acceleration with amplified magnetic field \citep{2007Natur.449..576U,2020ApJ...899..102H}. The TeV gamma-ray observations of RX~J1713.7$-$3946 revealed its shell-like morphology with photon energies from 200~GeV to 40~TeV \citep{2004Natur.432...75A,2006A&A...449..223A,2007A&A...464..235A,2018A&A...612A...6H}. Despite a number of efforts to model the broad-band spectra, the origin of gamma-rays---hadronic, leptonic, or a combination of both---was not clearly established because all scenarios could reproduce the observed spectra (see \citeauthor{2018A&A...612A...6H} \citeyear{2018A&A...612A...6H} and references therein).

\begin{figure*}[]
\begin{center}
\includegraphics[width=\linewidth,clip]{rxj1713_alma_fig2_v2.pdf}
\caption{({\it{top panels}}) ALMA CO velocity channel maps toward {RXJ1713NW-East}. Each panel shows CO intensity distribution integrated every 2.0~km~s$^{-1}$ in the velocity range from $-16.0$ to $-10.0$~km~s$^{-1}$. Superposed contours indicate the same X-ray intensity as shown in Figure \ref{fig1}b. Yellow crosses indicate positions of X-ray hot spots identified by \cite{2020ApJ...899..102H}. The typical CO cloudlets, CL1--6, and a CO filament are also indicated. ({\it{bottom panels}}) ALMA CO profiles of CL1--6 (black solid lines). Each spectrum is fitted by a single Gaussian kernel using the least-squares method.}
\label{fig2}
\end{center}
\end{figure*}%

Investigating the interstellar gas associated with RX~J1713.7$-$3946 holds a key to understanding the radiation processes and efficient acceleration of cosmic-rays. \cite{2012ApJ...746...82F} presented a good spatial correspondence between the TeV gamma-rays and ISM proton column density. This provides an essential condition for gamma-rays to be predominantly of hadronic origin. \cite{2010ApJ...724...59S,2013ApJ...778...59S} found limb-brightening of synchrotron X-rays toward shocked molecular clouds with a size of $\sim$3~pc, where the small photon indices of synchrotron X-rays are seen \citep{2015ApJ...799..175S,2018PASJ...70...77O}. The authors proposed a possible scenario that the shock-cloud interaction generates turbulence that enhances magnetic field and synchrotron X-rays on the surface of  shocked clouds. Moreover, the enhanced turbulence and/or magnetic field may re-accelerate electrons into higher energy. This interpretation is also supported by numerical results, and well explains the observed broad-band spectra without bright thermal X-ray{s} \citep[e.g.,][]{2009ApJ...695..825I,2012ApJ...744...71I,2019MNRAS.487.3199C}. However, previous studies could not spatially resolve tiny molecular clouds relating to the X-ray hotspots {and filaments} at 0.05~pc scales which were predicted by \cite{2009ApJ...695..825I}.

We report here on results of ALMA CO observations toward the northwestern shell of RX~J1713.7$-$3946 with a spatial resolution of $\sim$0.02~pc. Our findings for complexes of molecular cloudlets at 0.01~pc provide a new perspective on the ISM surrounding core-collapse SNRs

\section{Results}\label{sec:results}
Figure \ref{fig1}a shows the overall TeV gamma-ray morphology of RX~J1713.7$-$3946 \citep[][]{2018A&A...612A...6H}. The TeV gamma-ray shell shows a good spatial correspondence with the total interstellar proton column density contours \citep{2012ApJ...746...82F}. For the dense star-forming core named as peak C, TeV gamma-ray intensity is reduced respect to its surroundings. Note that the brightest TeV gamma-ray spot is located in the northwestern shell, especially in the intermediate region of peaks D and L. The intercloud region also bright in X-rays with filamentary structures.

\begin{deluxetable*}{lcccccccc}[]
\tablecaption{Physical properties of typical CO cloudlets associated with RX~J1713.7$-$3946}
\tablehead{\multicolumn{1}{c}{Name} & $\alpha_{\mathrm{J2000}}$ & $\delta_{\mathrm{J2000}}$ & $T_{\mathrm{mb}} $ & $V_{\mathrm{peak}}$ & $\Delta V$ & Size & Mass & $n(\mathrm{H_2})$\\
& ($^{\mathrm{h}}$ $^{\mathrm{m}}$ $^{\mathrm{s}}$) & ($^{\circ}$ $\arcmin$ $\arcsec$) & (K) & \scalebox{0.9}[1]{(km $\mathrm{s^{-1}}$)} & \scalebox{0.9}[1]{(km $\mathrm{s^{-1}}$)} & (pc) & ($M_\sun $) & (cm$^{-3}$)\\
\multicolumn{1}{c}{(1)} & (2) & (3) & (4) & (5) & (6) & (7) & (8) & (9)}
\startdata
CL1 ............ & 17 12 02.95 & $-$39 32 20.4 & $10.9 \pm 0.6$ & $-14.20 \pm 0.05$ & $1.94 \pm 0.14$ & 0.08 & 0.37 & $2.6\times10^4$\\
CL2 ............ & 17 12 02.50 & $-$39 32 06.7 &$11.7 \pm 0.2$ & $-14.45 \pm 0.01$ & $1.70 \pm 0.03$ & 0.07 & 0.27 & $3.1 \times 10^4$\\
CL3 ............ & 17 12 06.48 & $-$39 31 14.7 & \phantom{0}$9.6 \pm 0.4$ & $-12.50 \pm 0.02$ & $0.95 \pm 0.04$ & 0.06 & 0.08 & $1.6 \times 10^4$\\
CL4 ............ & 17 11 57.33 & $-$39 32 08.1 & \phantom{0}$4.4 \pm 0.2$ & $-12.32 \pm 0.05$ & $2.79 \pm 0.12$ & 0.10 & 0.33 & $1.2 \times 10^4$\\
CL5 ............ & 17 11 53.17 & $-$39 31 21.3 & $13.5 \pm 0.9$ & $-11.99 \pm 0.05$ & $1.38 \pm 0.11$ & 0.10 & 0.47 & $2.0 \times 10^4$\\
CL6 ............ & 17 11 52.72 & $-$39 33 35.2 & \phantom{0}$6.0 \pm 0.6$ & $-11.49 \pm 0.06$ & $1.09 \pm 0.14$ & 0.07 & 0.08 & $1.0 \times 10^4$\\
\enddata
\tablecomments{Col. (1): Cloudlet name. Cols. (2--9): Observed properties of cloudlets obtained by single Gaussian fitting with $^{12}$CO($J$=1--0) emission line. Cols. (2)--(3): Position of cloudlets. Col. (4): Maximum brightness temperature. Col. (5): Central velocity. Col. (6): FWHM linewidth. Col. (7): Effective diameter of cloudlets defined as $(S / \pi)^{0.5} \times 2$, where $S$ is the surface area of cloudlets surrounded by a contour of the half level of maximum integrated intensity. Col. (8): Mass of cloudlets defined as $m_\mathrm{p} \mu \Omega D^2 \sum_{i} [N_\mathrm{i}(\mathrm{H_\mathrm{2}})]$, where $m_\mathrm{p}$ is the atomic hydrogen mass, $\mu$ is the mean molecular weight, $D$ is the distance to the SNR, $\Omega$ is the solid angle in a spatial pixel, and $N$($\mathrm{H_2}$) is molecular hydrogen column density for each pixel. We used $\mu = 2.8$ and an equation of $N(\mathrm{H_2}) / W(\mathrm{CO}) = 2.0 \times 10^{20}$~(K~km~s$^{-1}$)$^{-1}$~cm$^{-2}$, where $W$(CO) is the CO integrated intensity \citep{1993ApJ...416..587B}. Col. (9): Number density of molecular hydrogen $n(\mathrm{H_2})$.}
\label{tab:mc}
\end{deluxetable*}

Figure \ref{fig1}b shows the $^{12}$CO($J$=1--0) integrated intensity map {obtained using ALMA (see Appendix \ref{subsec:alma} for detailed information of observations and data reduction).} The integration range of $-16$--$-6$~km~s$^{-1}$ is typical radial velocity in the northwestern molecular clouds, which is certainly associated with RX~J1713.7$-$3946 \citep[e.g.,][]{2003PASJ...55L..61F,2012ApJ...746...82F,2005ApJ...631..947M,2010ApJ...724...59S,2013ApJ...778...59S,2015ApJ...799..175S,2012MNRAS.422.2230M}. We newly identified three molecular complexes with a size of $\sim$1~pc toward the southeastern half of the ALMA observed area, in addition to the previously known molecular cloud peak D. The X-ray bright filaments {as shown by contours} are nicely along not only with the three molecular complexes {(hereafter RXJ1713NW-East)}, but also with a part of the molecular cloud peak D {(hereafter RXJ1713NW-West)}. It is noteworthy that intercloud diffuse regions show significantly low intensity (purple colored areas), whereas there are several tiny CO clumps with a size on the order of 0.01~pc. We hereafter refer to the tiny CO clumps as ``molecular cloudlets.''

Figures \ref{fig2}a--\ref{fig2}c show the velocity channel maps of ALMA CO toward {RXJ1713NW-East}. We confirmed that one of the newly identified molecular complexes is consisting of dozens of molecular cloudlets (e.g., CL1--6) and filaments (e.g., Figure \ref{fig2}b, $\sim$0.06~pc width). Almost all the molecular cloudlets and filaments are spatially associated with the X-ray filaments, {but the X-ray minor peaks and X-ray hotspots identified by \cite{2020ApJ...899..102H} are located in the intercloud regions.} The typical spatial separations between X-ray {minor peaks or} hotspots and nearest CO intensity peaks are $\sim$0.05--0.15~pc. Figure \ref{fig2} bottom panels show CO line profiles of the typical molecular cloudlets CL1--6 {which are located near the X-ray filaments and hotspots without contamination}. All the sampled molecular cloudlets are significantly detected, and show the narrow velocity width ($< 2$~km~s$^{-1}$) except for CL4. Although wing-like profiles are seen toward CL1 and CL5, these can be explained by other overlapped molecular cloudlets. The physical properties of molecular cloudlets CL1--6 are summarized in Table \ref{tab:mc}. The typical diameters and masses of molecular cloudlets are $\sim$0.06--0.10~pc and $\sim$0.1--0.5~$M_{\odot}$, respectively. The number densities of molecular cloudlets are a few 10$^4$~cm$^{-3}$, which were obtained using a CO-to-H$_2$ conversion factor (see Table \ref{tab:mc} note).

Figure \ref{fig3} shows the enlarged view of the X-ray filament from which flux time variability was reported by \cite{2007Natur.449..576U}. We compared spatial distributions of CO and X-rays for each observing epoch {(see Appendix \ref{subsec:chandra} for detailes).} {T}wo X-ray hotspots in July 2000 (Figure \ref{fig3}b) and May 2009 (Figure \ref{fig3}f) are significantly detected. The former was previously reported by \cite{2007Natur.449..576U}, whereas the latter is newly identified ($\sim$4$~\sigma$ above the surrounding level). The latter hotspot was excited within three years and disappeared within four months. It should be also noted that these hotspots are located on the intercloud or low-density region, but not in the direction of dense cloudlets.

\begin{figure*}[]
\begin{center}
\includegraphics[width=\linewidth,clip]{rxj1713_alma_fig3_v9.pdf}
\caption{(a) Enlarged view of ALMA CO map toward an X-ray hotspot presented by \cite{2007Natur.449..576U}. (b--h) Sequence of X-ray observations in July 2000, May 2006, May 2009, and September 2009. All X-ray images show the same intensity scales. Superposed contours indicate the CO intensity. Gray shaded areas were eliminated due to low exposure time (see the text).}
\label{fig3}
\end{center}
\end{figure*}%

\section{Discussion}\label{sec:discussion}
\subsection{Origin of highly inhomogeneous density distribution}\label{subsec:windbubble}
We spatially resolved shocked molecular cloudlets and filaments toward the northwestern shell of RX~J1713.7$-$3946. These clumpy structures at 0.01~pc scales coexist with the low-density intercloud medium. We argue that the highly inhomogeneous gas environment provides conclusive evidence for a wind-bubble scenario proposed by \cite{2012ApJ...746...82F} and \cite{2012ApJ...744...71I}. Before the supernova explosion, the high-mass progenitor of RX~J1713.7$-$3946 ejected its outer hydrogen layer as stellar winds over a time scale of several $10^6$ years. The pre-existent intercloud diffuse gas was completely swept up and a low-density wind-bubble with a density of $\sim$0.01--0.1~cm$^{-3}$ was created \citep[e.g.,][]{1977ApJ...218..377W}. On the other hand, molecular cloudlets and filaments can survive wind erosion due to their high density $\sim$10$^4$~cm$^{-3}$. {Moreover, according to synthetic observations of MHD simulation for colliding H{\sc i} clouds, $\sim$0.1 pc clouds could be newly formed by stellar wind compression \citep[e.g.,][]{2018ApJ...860...33F,2018arXiv181102224T}.} After the passage of the supernova shock wave, dense cloudlets and filaments will not be deformed or evaporated owing to short interacting time. In fact, numerical simulations show that a molecular cloud with the size 0.2~pc and density $10^3$~cm$^{-3}$ can survive shock erosion at least 300~years after the passage of shocks \citep[][]{2019MNRAS.487.3199C}. The CO line emission without line-broadening or wing-like profiles also supports this idea.

Note that such inhomogeneous gas density distribution is also expected in other core-collapse SNRs. Further ALMA observations with high special resolution on the order of 0.01~pc are needed for complete understanding the interstellar environment surrounding the SNRs.

\subsection{Magnetic field amplification via shock-cloudlet interactions}\label{subsec:bamplification}
\cite{2007Natur.449..576U} discovered X-ray hotspots in the northwestern shell of RX~J1713.7$-$3946, which show year-scale time variability of X-ray flux with the typical spatial scale of $\sim$0.05~pc. Considering the acceleration and radiative cooling time of cosmic-ray electrons, the authors concluded that the time variability was caused by amplified magnetic field of mG. Most recently, \cite{2020ApJ...899..102H} presented a detailed analysis of 65 X-ray hotspots; about one-third of them showed year-scale time variabilities and eight hotspots showed monthly variabilities with significance of at least 3$\sigma$. The authors proposed that the time variabilities are caused by dense cloud cores with a density of $10^5$--$10^7$~cm$^{-3}$. In the present section, we argue that the observed time variabilities can be understood by the magnetic field amplification through interactions between shockwaves and cloudlets with a density of $\sim$$10^4$~cm$^{-3}$.

We estimate the magnetic field strength toward the newly identified X-ray hotspot, following the method by \cite{2007Natur.449..576U} and \cite{2020ApJ...899..102H}. {The radiative cooling time of electrons $T_\mathrm{synch}$ can be written as;}
\begin{eqnarray}
T_\mathrm{synch} \sim 1.5\:\Bigl(\frac{B}{\mathrm{1~mG}}\Bigr)^{-1.5}\Bigl(\frac{\varepsilon}{\mathrm{1~keV}}\Bigr)^{-0.5}\mathrm{(year)}\;\;\;\;
\label{eq0}
\end{eqnarray}
{where $\varepsilon$ is the photon energy of synchrotron X-rays and $B$ is the magnetic field strength. Considering the short decay time of four months and $\varepsilon = 1$~keV, we can obtain $\sim$3~mG toward the X-ray hotspot. On the other hand, the acceleration time of electrons $T_\mathrm{acc}$ can be expressed in terms of gyro-factor $\eta = (B/\delta B)^2$ and shock velocity $V_\mathrm{s}$ as}
\begin{eqnarray}
T_\mathrm{acc} \sim 1\eta\:\Bigl(\frac{B}{\mathrm{1~mG}}\Bigr)^{-1.5}\Bigl(\frac{\varepsilon}{\mathrm{1~keV}}\Bigr)^{0.5}\Bigl(\frac{V_\mathrm{s}}{3000~\mathrm{km~s^{-1}}}\Bigr)^{-2}\mathrm{(year)}.\;\;\;\;
\label{eq1}
\end{eqnarray}
{Adopting observed values of $\eta$ = 1 \citep{2019ApJ...877...96T} and $V_\mathrm{s} = 3900$~km~s$^{-1}$ \citep{2016PASJ...68..108T}, the magnetic field strength $B$ is estimated to $\sim$300~$\mu$G. Note that it is natural to have different magnetic field in Equations (\ref{eq0}) and (\ref{eq1}), because cosmic-rays do not need to be accelerated in the same location.}

We argue that the strong magnetic field in X-ray hotspots {as well as diffuse X-ray filaments} was generated by shock-cloudlet interactions. According to MHD simulations by \cite{2009ApJ...695..825I,2012ApJ...744...71I}, the shock-cloud interaction generates multiple eddies that enhance magnetic field strength up to mG on the surface of dense H{\sc i} clouds with a density of $\sim$$10^2$~cm$^{-3}$. Subsequent numerical simulation confirmed that the magnetic field is also amplified by interactions between shocks and a molecular clump with a density of $10^3$~cm$^{-3}$ \citep{2019MNRAS.487.3199C}. Note that the most enhanced magnetic field is placed $\sim$0.4~pc away from the shocked cloud by adopting the cloud size of 0.2~pc and the ISM density contrast of $\sim$$10^5$ \citep[see Figure \ref{fig3} in][]{2019MNRAS.487.3199C}. For the northwestern shell of RX~J1713.7$-$3946, the ISM density contrast is estimated to $\sim$$10^5$ considering the electron density in an intercloud region $\sim$0.10--0.13~cm$^{-3}$ \citep{2015ApJ...814...29K} and the cloudlet density $\sim$$10^4$~cm$^{-3}$. The highest magnetic field is therefore expected to be $\sim$0.12--0.20~pc away from the observed cloudlet with sizes of 0.06--0.10~pc, which is roughly consistent with observed separations between the X-ray hotspots{/filaments} and cloudlets\footnote{{Actual spatial separation between cloudlet and the highest magnetic field will be slightly modified considering the shape of shocked cloudlet and projected distance. For the case of Figure~\ref{fig3}, we assumed the cylinder shape of shocked cloud as an extension of spherical cloud assumed by \cite{2019MNRAS.487.3199C}.}}.

The shock-cloudlet interaction may also induce the spectrum modulation of synchrotron X-rays. Previous observational studies also indicate spectral flattening toward the shock interacting region \citep[e.g.,][]{2015ApJ...799..175S,2018PASJ...70...77O,2020ApJ...900L...5T}. Further spatially resolved X-ray spectroscopy based on distributions of molecular clouds reveals such spectrum modulations.

\subsection{Prospects for gamma-ray spectra toward the northwestern shell}\label{subsec:gspectrum}
It is thought that the hadronic gamma-ray shows flat $\nu F_\nu$ spectrum from the pion-creation threshold energy at $\sim$0.1~GeV to the maximum energy achieved by DSA, whereas the leptonic gamma-ray shows hard $\nu F_\nu$ spectrum instead. Since the {\it{Fermi}}-LAT detected the hard spectrum with photon index of 1.5, this was often used to reject the hadronic gamma-ray scenario from RX~J1713.7$-$3946 \citep[e.g.,][]{2011ApJ...734...28A}. On the other hand, the hadronic gamma-ray can be harder spectrum by considering the inhomogeneous gas distribution and energy dependent diffusion of cosmic-ray protons \citep[e.g.,][]{2009MNRAS.396.1629G,2010ApJ...708..965Z,2012ApJ...744...71I,2014MNRAS.445L..70G}. In this section, we argue that hadronic gamma-ray spectrum from the northwestern shell of RX~J1713.7$-$3946 will {possibly} be flatter than spatially combined spectra.

According to \cite{2012ApJ...744...71I}, the penetration depth of a relativistic particle into molecular clouds $l_\mathrm{pd}$ can be described as below;
\begin{eqnarray}
l_\mathrm{pd} =  0.1\; \eta^{0.5}\Bigl(\frac{E}{10\;\mathrm{TeV}}\Bigr)^{0.5}\Bigl(\frac{B}{100\;\mathrm{\mu G}}\Bigr)^{-0.5}
\Bigl(\frac{t}{1000\;\mathrm{yr}}\Bigr)^{0.5}\mathrm{(pc)},\;\;\;\;
\label{eq3}
\end{eqnarray}
where $E$ is the particle energy and {$t$ is the elapsed time passed since the forward shock encountered the molecular cloud.}
The magnetic field will be enhanced not only on the surface of shocked clouds via shock interaction, but also inside the clouds caused by cosmic-ray streaming what is called Bell instability \citep{2004MNRAS.353..550B}. \cite{2019ApJ...872...46I} argued that the Bell instability induced magnetic field amplification prevent further cosmic-ray penetration into shocked clouds by diminishing the diffusion coefficient for the cosmic-ray energy below $\sim$1~TeV. Note that the degree of magnetic field amplification depends on path length from the surface of shocked clouds, roughly corresponding to a radius of shocked clouds.

Figure \ref{fig4} shows $\nu F_\nu$ spectra of hadronic gamma-rays with various cloud radii. According to \cite{2005ApJ...631..947M}, typical radii of molecular clouds associated with the SNR were derived as $\sim$1--2~pc. The densest star-forming cloud peak C in the southwestern shell is a typical example with a featureless morphology and central concentration density gradient with an outer radius of $\sim$1.5~pc \citep{2010ApJ...724...59S}. Therefore, it is natural to be observed hard gamma-ray spectra ($\nu^{0.5}$ or photon index of 1.5) when we combined them for the entire remnant \citep[e.g.,][]{2011ApJ...734...28A}. On the other hand, the brightest gamma-ray peak in the northwestern shell is associated with dozens of molecular cloudlets and filaments with typical radii of 0.03--0.05~pc, suggesting that the flatter gamma-ray spectrum will be expected than the case of 0.08~pc in red curve of Figure \ref{fig4}. {Future} detailed spatial{ly} resolved spectroscopy of gamma-ray using the Cherenkov Telescope Array (CTA) has a potential to {resolve such spatial difference of gamma-ray spectral shape.}

\begin{figure}[]
\begin{center}
\includegraphics[width=\linewidth,clip]{rxj1713_alma_fig4_v4.pdf}
\caption{Synthetic $\nu F_\nu$ gamma-ray spectra expected from shocked gas clouds with radii of 0.08, 0.16, 0.24, 0.40, and 1.60 pc, which were reproduced using numerical results in \cite{2019ApJ...872...46I}. The dashed line indicates $\nu F_\nu \propto \nu^{0.5}$.}
\label{fig4}
\end{center}
\end{figure}%

{In case we do not detect the gamma-ray spectrum flattening in the northwestern shell of RX~J1713.7$-$3946 using CTA, it can be explained  that the magnetic field in the cloudlet surface is more efficiently enhanced than inside the cloud via the shock-cloudlet interaction. Because smaller cloud size induces shorter time scale of magnetic field amplification even if the maximum magnetic field strength does not depend on the cloud size. Alternatively, the diffusion length of cosmic-rays could be shorter than 0.1 pc if the actual elapsed time in Equation (\ref{eq3}) is significantly smaller than 1000~yrs. In fact, CO cloudlets in RXJ1713NW-East have been fully overtaken by the shocks, while the peak~D including RXJ1713NW-West is likely now interacting with the forward shock \citep[e.g.,][]{2013ApJ...778...59S}. In any case, the clumpy cloud distribution interacting with shockwaves is important in understanding the gamma-ray and X-ray spectra from shock-accelerated cosmic-rays in SNRs.}

\section{Summary}\label{sec:summary}
By using ALMA, we have spatially resolved molecular complexes consisting of the 0.01~pc scale cloudlets and filaments interacting with the northwestern shell of RX~J1713.7$-$3946, where the brightest TeV gamma-rays and synchrotron X-ray filaments/hotspots are detected. The molecular cloudlets having the typical radii of $\sim$0.03--0.05~pc and densities of $\sim$10$^4$~cm$^{-3}$ are nicely along with X-ray filaments and hotspots, indicating that magnetic field is amplified through shock-cloudlet interactions. The ISM density contrast is to be $\sim$$10^5$, consistent with a wind-bubble scenario. The small-scale structures or density fluctuations of the ISM may induce spectral modulations both the hadronic gamma-rays and synchrotron X-rays, beyond the standard DSA.

\acknowledgments
This paper makes use of the following ALMA data: ADS/JAO.ALMA\#2017.1.01406.S. {ALMA is a partnership of ESO (representing its member states), NSF (USA) and NINS (Japan), together with NRC (Canada), MOST and ASIAA (Taiwan), and KASI (Republic of Korea), in cooperation with the Republic of Chile. The Joint ALMA Observatory is operated by ESO, AUI/NRAO and NAOJ.} This study was supported by JSPS KAKENHI Grant Numbers JP18H01232 (R.Y.), JP19K14758 (H.S.) and JP19H05075 (H.S.). K. Tokuda was supported by NAOJ ALMA Scientific Research Grant Number of 2016-03B. {We appreciate the anonymous referee for useful comments and suggestions, which helped authors to improve the paper.}\\