text:
\section{Approach}
\vspace{-0.1in}
\label{sec:approach}
We study two important aspects involved in obtaining generalized proposals from a detection model:

(1) {\bf{Data Properties}} such as the granularity of the label space (shown in Fig.~\ref{fig:g1}), and the visual diversity of object classes under each label, required for generalization of proposals. The idea of label granularity and visual diversity is shown in Fig.~\ref{fig:g2}. We investigate how a smaller subset of ``prototypical" object classes in a dataset which is representative of all other classes can be identified.

\begin{figure}[t!]
    \centering
    \begin{subfigure}[t]{0.56\textwidth}
        \centering
        \includegraphics[width=0.99\textwidth]{figures/label_granularity.pdf}
        \caption{Label semantic hierarchy}
        \label{fig:g1}
    \end{subfigure}\hfill
    \begin{subfigure}[t]{0.4\textwidth}
        \centering
        \includegraphics[width=0.99\textwidth]{figures/rui_figs_gran_v2.pdf}
        \caption{Granularity vs. Diversity}
        \label{fig:g2}
    \end{subfigure}
    \caption{We study two important dataset properties needed to train a proposal model: label granularity and visual diversity. (a) Label granularity can be represented by different levels in a semantic hierarchy as shown. (b) The difference between label granularity and visual diversity is illustrated. At the same granularity, we can either have high or low visual diversity as shown}
    \label{fig:gran_visual}
    \vspace{-0.2in}
\end{figure}

(2) {\bf{Modeling Choice}} for leveraging a detector trained on a dataset with seen classes to obtain proposals that generalize to unseen classes.

\subsection{Dataset Properties}
\label{sec:data_prop}
The choice of labels and data used to train the model is crucial for generalization. To study these properties, we assume: (a) classes are organized in a semantic tree and (b) internal nodes do not have any data of their own, that are not categorized into one of its child nodes. In practice, such a hierarchy is either already available (OIV4\xspace) or can be obtained from Wordnet~\cite{wordnet}.  These assumptions help us study the datasets under controlled settings. However, later we explore a way to identify ``prototypical" subsets even when a semantic hierarchy is unavailable.

\subsubsection{Label Space Granularity}
\label{sec:label_space}
As we noted through some examples earlier, it is intuitive that we might not need fine-grained labels to train a good localization model. To quantitatively study the effect of granularity, we construct different datasets with the same set of images and object bounding boxes, but consider classes at different levels of semantic hierarchy (Fig.~\ref{fig:g1}). We then train a model with these datasets and evaluate the generalization ability as a function of label granularity. For instance, for the coarsest root level, we assign all the bounding boxes the same ``object" label and train a detector to distinguish objects from all non-objects. This pertains to the idea of objectness used in weakly supervised algorithms~\cite{uijlings2013selective} and super-class in \cite{singh2018r}. For an intermediate level, we collapse all leaf-labels to their corresponding parent labels at that level to train the model. While a fine-grained label space provides more information, a model trained at this level also attempts to distinguish object classes with similar structure and this could affect generalization. We quantify this trade-off in Sec.~\ref{sec:exp_data}.

\subsubsection{Prototypical classes to capture visual diversity}
\label{sec:proto}
One of the main aims of our work is to see if we can identify a significantly smaller number of classes than the full object-label space, so that bounding boxes from this set of classes are sufficient to train a generalized proposal model. Note that in Sec.~\ref{sec:label_space}, we wanted to study if a small set of coarse labels are sufficient to train a generalized proposal model. However, this does not answer anything about the visual diversity of objects within each sub-category that is required for generalization. As an example (shown in Fig.~\ref{fig:gran_visual}), in order to localize different types of vehicles like ``car" or ``airplane" it might be sufficient to collapse the label for all these objects into a single label named ``vehicle", however dropping all instances of airplane during training will lead to a drop in performance for this class.

To quantitatively study this effect, we introduce the notion of ``prototypical" classes. Given a large set of leaf classes, these are the smallest subset such that a model trained only with instances from them is sufficient to localize objects from the remaining classes. Note that due to the long-tail distribution of real-world data, obtaining images for large number of semantic classes is a tedious task. If a small set of prototypical classes does exist, this makes the data collection process much easier when scaling detection to large number of classes.

\noindent{\bf{Properties: }}We identify the two properties that are required to quantify the prototypicality of a set of classes :

\textit{Sufficient set}: is a set of classes such that training a model only with examples from them should be sufficient to localize objects from all other classes. The most superfluous sufficient set would be the entire set of leaf classes themselves.

\textit{Necessary set}: is a set of classes such that dropping any class from this set will lead to a significant drop in generalization. A simple example would be a very coarse vertical like ``vehicle". Intuitively dropping all vehicles would affect their localization as they do not share localization properties with other classes.

We provide concrete ways to measure both these properties in Sec.~\ref{sec:exp_data}.

\noindent{\bf{Identifying prototypical classes: }}
Given a set of $N$ leaf classes $\mathbb{C}$, we wish to identify a set of $P$ prototypical classes $\mathbb{P} \subset \mathbb{C}$. Intuitively, this is similar to
clustering the classes that have the same localization structure and then choosing a representative class from each cluster. Below, we discuss three approaches:

\noindent(a) \textbf{Oracle visual clustering}: To get an upper bound for choosing the best $P$ prototypical classes, we assume that bounding box annotations for all the $N$ leaf classes are available. We then use these bounding boxes to compute visual similarity between classes. We note that this is not a practical approach, but is crucial to evaluate the effectiveness of proxies we introduce later.

We first train a detection model using the annotations of all the leaf classes. We then measure the visual similarity between two classes $i, j$ as
\vspace{-0.05in}
{\small
\begin{align}
    \label{eq:max_ap}
    S_{ij} = \max \left( \frac{\text{AP}^i(j)}{\text{AP}^j(j)}, \frac{\text{AP}^j(i)}{\text{AP}^i(i)}\right),
\end{align}}where $AP^i(j)$ is the detection average precision (AP) for the $j^{th}$ class when we use the detections corresponding to the $i^{th}$ class as detections of class $j$. $S_{ij}$ is a measure of how well one class can replace another class in localizing it. We then use the resulting similarity measure to hierarchically cluster the classes into $P$ clusters using agglomerative clustering. We then pick the class with the highest number of examples in each cluster to construct the set of prototypical classes. For practical reasons, we use frequency to choose the representative class, since this results in the construction of the largest dataset.

\noindent(b) \textbf{Semantic clustering based on frequency}: Semantic similarity is often viewed as a good proxy for visual similarity as shown through datasets like Imagenet \cite{deng2009imagenet} and OIV4\xspace. Hence, we use the semantic tree to cluster the classes in an hierarchical fashion starting from the leaves. At any given step, we cluster together two leaf classes that share a common parent if they jointly have the lowest number of examples. The algorithm stops when $P$ clusters are left. We then select the most frequent class from each cluster as a prototypical class. Here we assume that apriori we know the frequency of each class in a dataset. This is a very weak assumption, since a rough estimate of class distribution in a dataset can often be obtained even from weak labels like hashtags. This doesn't require any image-level label or bounding boxes and is easy to implement in practice.

\noindent(c) \textbf{Most frequent prototypical subset}: For this baseline, we choose the top $P$ most frequently occurring classes in the dataset as the prototypical classes. Note that unlike the previous approaches, this does not require any knowledge of the semantic hierarchy.

\subsection{Modeling Choice\label{subsec:model}}
\label{sec:model_choice}
Once the dataset is fixed, the next step is to train a detection model. In our work, we explore the use of two models: Faster R-CNN\xspace and RetinaNet\xspace. The observations made in our work should nevertheless generalize to other two-stage and single-stage detection models as well.

In the case of a single-stage network, the detections from a model trained on a source dataset with seen classes can directly be treated as proposals. Their ability to localize novel classes in a target dataset can be evaluated to test generalization. However, for a two-stage network, another natural choice would be to use the Region Proposal Network (RPN) of the model, since it is trained in a class-agnostic fashion and aims to localize all objects in the image. However, as noted by He et al. \cite{he2017mask}, the detection part of the model is better at localizing the object due to more fine-tuned bounding box regression and better background classification. We study this more rigorously, by comparing the generalization of proposals obtained from the detection head as well as RPN. We vary different model parameters to obtain the optimal setting for proposal generalization.

\section{Conclusion}
We studied the ability of detection models trained on a set of seen classes to localize unseen classes. We showed that Faster R-CNN\xspace can be used to obtain better proposals for unseen classes than RetinaNet\xspace, and studied the effect of model choices on generalization of proposals, like class-agnostic bounding box regression and NMS threshold. We quantitatively measured the importance of visual diversity and showed that using a very fine-grained or very coarse label-space can both affect generalization, while a middle-ground approach is best suited. We introduced the idea of prototypical classes that are sufficient and necessary to obtain generalized proposals. We demonstrated different approaches to determine small prototypical subsets for a given dataset. We believe that our work is a step forward towards learning proposals that generalize to a large number of classes and scaling up detection in a more data-efficient way.

\subsection{Dataset Properties}
\label{sec:exp_data}
\noindent {\textbf{Effect of label space granularity: }} OIV4\xspace organizes object classes in a semantic hierarchy with $5$ levels. We directly leverage this hierarchy to measure the effect of label granularity (Fig.~\ref{fig:g1}). We construct a dataset at each level $L_i$ (OIV4-source\xspace-$L_i$) by retaining all the images in OIV4-source\xspace, but relabeling bounding boxes corresponding to leaf labels with their ancestor at $L_i$. We construct 5 datasets, one for each level with the same set of images and bounding boxes.

We report the performance of these models on OIV4-target\xspace in Tab.~\ref{tab:label_gran}. Along with \AR{100/1000}, we also report the detection AP of the weakly supervised detection models trained with the proposals obtained from the corresponding levels. The weakly supervised models are trained on OIV4-target\xspace-train and evaluated on OIV4-target\xspace-test.

\vspace*{-8mm}
\setlength{\tabcolsep}{4pt}
\begin{table}
\begin{center}
\caption{Effect of different label space granularities on the quality of proposal for OIV4\xspace dataset. The number of classes at each level is shown in brackets. Evaluation is done on OIV4-target\xspace-eval dataset. Both AR and weakly supervised detection AP are reported}
\label{tab:label_gran}
\begin{tabular}{cccc}
\hline\noalign{\smallskip}
Source Dataset & AR@100 & AR@1000 & AP (weak)\\
\noalign{\smallskip}
\hline
\noalign{\smallskip}
OIV4-source\xspace-$L_0 (1)$ & 61.7 & 72.0 & 19.5\\
\hline
OIV4-source\xspace-$L_1 (86)$ & 63.4 & 73.0 & 22.6\\
\hline
OIV4-source\xspace-$L_2 (270)$ & 63.7 & 75.2 & 23.1\\
\hline
OIV4-source\xspace-$L_3 (398)$ & 65.2 & 77.2 & 24.3\\
\hline
OIV4-source\xspace-$L_4 (432)$ & 64.2 & 76.1 & 24.0\\
\hline
\end{tabular}
\end{center}
\end{table}
  \vspace{-0.35in}
\setlength{\tabcolsep}{1.4pt}

Some past works like \cite{singh2018r} postulated that one super-class (similar to $L_0$) could be sufficient. However, we observe that both \AR{100} and \AR{1000} increase as we move from $L_0$ to $L_1$ along with a significant gain ($3.1\%$) in AP. This indicates that training with just a binary label yields lower quality proposals compared to training with at least a coarse set of labels at $L_1$. While both AP and \AR{100} increase as the granularity increases from $L_1$ to $L_3$, the difference is fairly small for both metrics ($ < 2\%$ change). However, annotating bounding boxes with labels at $L_1$ ($86$ labels) is significantly cheaper than $L_3$ ($398$ labels). Hence, $L_1$ can be seen as a good trade-off in terms of labelling cost, and training a good model.

\vspace*{0.02in}
\noindent {\textbf{Need for visual and semantic diversity: }}
We noticed that training with coarse labels can yield good proposals. It would be interesting to observe if all or only some of these coarse classes are crucial to build a good proposal model. To study this, we conduct ablation experiments where we train a model with OIV4-source\xspace-train after dropping all images having a specific $L_1$ label and evaluate the proposals on the OIV4-source\xspace-test images belonging to this label in Fig.~\ref{fig:drop_classes_fig}a. We repeat this experiment for a few fine-grained classes at $L_4$ in Fig.~\ref{fig:drop_classes_fig}b.

\begin{figure}[t]
\vspace{-0.1in}
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \includegraphics[width=0.7\textwidth]{figures/drop_coarse.pdf}
        \label{fig:1}
    \end{subfigure}\hfill
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \includegraphics[width=0.7\textwidth]{figures/drop_fine.pdf}
        \label{fig:2}
    \end{subfigure}
    \vspace{-0.1in}
    \caption{Effect of Semantic Diversity, measured by dropping an object class during training and measuring the resulting change in AR for that class: (a) dropping L1 classes and (b) dropping L4 classes}
    \label{fig:drop_classes_fig}
    \vspace{-0.2in}
\end{figure}
We notice that certain coarse classes (like ``clothing" and ``vehicle") experience a huge drop in performance. On the other hand, ``animal" and ``food" are less affected. This can be explained from the fact that, there are many toy-animal images within the coarse label ``toy", similarly ``containers" is a coarse class in OIV4\xspace which is often depicted with food in it. These classes can act as proxies for ``animal" and ``food" respectively. However, ``clothing" and ``vehicle" do not have good proxies. More interestingly, we  make a similar observation for finer classes at $L_4$ like airplanes and helicopters. This suggests that there is a smaller set of objects that have unique localization properties in OIV4\xspace.

\noindent {\textbf{Prototypical classes: }}
Some object classes are similar to others in terms of localization, while there are classes that are unique and need to be included in training. Motivated by this observation, we try to identify a small set of classes called ``prototypical" classes which are both necessary and sufficient to train a generalizable proposal model.

We use the OIV4-source\xspace dataset as before with 432 leaf classes. We use the different approaches outlined in Sec.~\ref{sec:proto} to identify a subset of ``prototypical" classes. Note that among these methods, oracle visual clustering assumes availability of bounding boxes for all classes and serves as an upper bound on how to identify a really good prototypical set. Some sample clusters of classes obtained by this method are shown in Tab.~\ref{tab:visual_clusters}. The remaining methods make weaker assumptions and are more useful in practice. In addition to these methods,we also train models with a set of randomly chosen prototypical classes.

\vspace*{-0.3in}
\setlength{\tabcolsep}{4pt}
\begin{table}
\begin{center}
\caption{Sample clusters obtained by oracle visual clustering for $P=50$. The most frequent class in each cluster chosen as a prototypical class is highlighted}
\label{tab:visual_clusters}
\begin{tabular}{lll}
\hline\noalign{\smallskip}
\scriptsize{\textbf{Woman}, Girl, Doll} & \scriptsize{\textbf{Wheel}, Tire, Bicyclewheel} & \scriptsize{\textbf{Lobster}, Scorpion, Centipede} \\
\hline
\scriptsize{\textbf{Glasses}, Goggles} & \scriptsize{\textbf{Jeans}, Shorts, Miniskirt} & \scriptsize{\textbf{Goose}, Ostrich, Turkey} \\
\hline
\scriptsize{\textbf{Book}, Shelf, Bookcase} & \scriptsize{\textbf{Musicalkeyboard}, Piano} & \scriptsize{\textbf{Swimmingpool}, Bathtub, Jacuzzi} \\
\hline
\scriptsize{\textbf{Man}, Boy, Shirt} & \scriptsize{\textbf{Apple}, Pomegranate, Peach} & \scriptsize{\textbf{Raven}, Woodpecker, Bluejay}\\
\hline
\end{tabular}
\end{center}
\vspace{-0.3in}
\end{table}
\setlength{\tabcolsep}{1.4pt}

We introduce two ways to measure \textit{sufficiency} and \textit{necessity}. From the $432$ classes, once we pick a subset of $P$ prototypical classes, we train a proposal model and evaluate the resulting model on the $50$ target classes in OIV4-target\xspace, to measure \textit{sufficiency} and \textit{necessity}.

\noindent{\textbf{Dataset construction for fair comparison}} We ensure that the total number of images as well as bounding box annotations are kept fixed when we construct datasets for different prototypical subsets. This is important to ensure that proposals trained with different subsets are comparable. Once we chose a set of $P$ prototypical classes, we uniformly sub-sample OIV4-source\xspace images having any of these prototypical classes to get a subset of $920K$ images. And within each subset, we uniformly sub-sample the bounding boxes corresponding to the prototypical classes to retain $5.2M$ bounding boxes. We do not retain any bounding boxes outside the chosen prototypical classes.

\noindent{\textbf{Training with prototypical subsets}}
For a set of prototypical classes and the corresponding dataset, we train a Faster R-CNN\xspace with those classes as labels. We combine the detections as described in Sec.~\ref{sec:model_choice} to obtain proposals.

\noindent{\textbf{Measuring sufficiency of prototypical classes}}
A subset of classes are sufficient, if a proposal model trained with them generalizes as well as a model trained with all classes. We follow this notion and evaluate the proposals obtained from the models trained with different prototypical subsets on OIV4-target\xspace and report the average recall (\AR{100}) in Fig.~\ref{fig:proto_properties}a. Similar trends are observed with \AR{1000} as well (shown in supplementary).

Looking at the proposals obtained from oracle visual clustering, training with less than 25\% of the classes (100) leads to only a drop of $4.8\%$ in \AR{100}, compared to training with images belonging to all object classes. This gap reduces to $0.4\%$ if we train with 50\% (200) of all the classes. This provides an empirical proof for the existence of a significantly smaller number of object classes that are sufficient to train a generalizable proposal model.

Next, we look at the prototypical classes obtained from a more practical approach: semantic clustering. We notice that the proposal model trained with these prototypical classes always outperform other approaches such as choosing a random set of classes or the most frequent set of classes. Further, the performance of this method is only lower by a margin of $3\%$ compared to oracle visual clustering for different value of $P$. Selecting most frequent set of classes as the prototypical subset performs slightly worse than semantic clustering. This shows that semantic clustering can serve as a good way to identify prototypical classes for large taxonomies when the semantic hierarchy is available for the dataset, else the most frequent subset is a weaker alternative.

\begin{figure}[t]
\vspace{-0.1in}
  \begin{minipage}[t]{0.99\textwidth}
    \centering
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \includegraphics[width=1.0\textwidth]{figures/sampling_method_ar100.pdf}

        \label{fig:suff_1}
    \end{subfigure}\hfill

    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \includegraphics[width=\textwidth]{figures/necessity.pdf}

    \label{fig:necessity}
    \end{subfigure}
    \vspace{-0.2in}
    \caption{(a) Average recall \AR{100} for proposals obtained from models trained with varying number of prototypical classes chosen by different methods. We show the average recall on the OIV4-target\xspace dataset with $50$ unseen classes. $P$ denotes the number of prototypical classes. Higher value indicates higher sufficiency. (b) The relative change in AR for target classes by dropping proposals corresponding to the most similar class in the prototypical subset. Higher value indicates lower redundancy in prototypical subset and higher necessity}
    \label{fig:proto_properties}
  \end{minipage}
\vspace{-0.25in}
\end{figure}

\noindent{\textbf{Measuring necessity of prototypical classes}}
A set of classes are considered necessary, if there is no redundancy among the classes in terms of localization properties. For a given class in the set, there should be no equivalent class which can provide similar bounding boxes. We measure this property for a prototypical subset by evaluating the corresponding proposal model on OIV4-target\xspace dataset using the following method. For every target class in OIV4-target\xspace, we measure the relative change in \AR{100} and \AR{1000} by removing proposals corresponding to the most similar class in the prototypical subset (similarity measured by Eq.~\ref{eq:max_ap}). The change in AR would be minimal if there is another class in the prototypical subset which can localize the target class. This measure, averaged over all target classes provides a good estimate of necessity. A high value symbolizes a high degree of necessity, while a low value corresponds to redundancy among the prototypical classes. We plot this for different number of prototypical classes for oracle visual clustering and semantic clustering in Fig.~\ref{fig:proto_properties}b.

We notice that at any given number of prototypical classes, the change in average recall is higher for oracle visual clustering compared to semantic clustering. This demonstrates that visual clustering leads to prototypical classes which are less redundant (and more necessary). As expected, we see the necessity drops, as we increase the number of prototypical classes for both methods. Again, this is expected since redundancy between classes increases with more number of classes. The relative change in \AR{1000} is also seen to be lower than \AR{100}, since when considering a larger number of proposals, we expect more redundancy among the proposals. Finally, for oracle visual clustering as we move from $200$ to $300$ classes, sufficiency changes by a small amount from $73.2$ to $75.9$ ( Fig.~\ref{fig:proto_properties}a), while the necessity drops steeply in Fig.~\ref{fig:proto_properties}b. This suggests that the ideal number of prototypical classes for OIV4\xspace could be around $200$.
\subsection{Modeling Choices}
We first identify the best detection model and settings to extract proposals that generalize to new unseen classes. We then analyze generalization ability under different settings from this model. We reiterate that in order to test generalization, evaluation is done on target classes that have no intersection with the source classes used during training.

\noindent {\textbf{Choice of detection model:}} We compare the ability of a two-stage network (Faster R-CNN\xspace) and a single-stage network (Retinanet) trained on a set of seen classes to generalize to new unseen classes in Fig.~\ref{fig:mod1}. We also show the performance of the respective models on the seen classes as well. The difference in performance between seen and unseen classes can be viewed as the generalization gap for the respective models. Since, in a two-stage model like Faster R-CNN\xspace, the output from the RPN is class-agnostic and can be used as proposals too, we report the performance of the RPN as well.

\noindent {\textbf{Comparison with Region Proposal Network:}}
The results so far where obtained using class specific bounding box regression (which is the standard setting in Faster R-CNN\xspace) for the detection head. Since we want the bounding boxes to generalize to unseen classes, class agnostic regression could be a valid choice too. We study this in Table~\ref{tab:class_agnostic} for OIV4\xspace and COCO\xspace.

First we see that class agnostic regression is comparable, or provides a slightly higher \AR{K} for small values of $K < 100$. However, when we consider more proposals (\AR{1000}), class specific regression provides a significant gain ($4.5\%$ for OIV4\xspace and $7.5\%$ for COCO\xspace). It results in multiple regressed versions (one corresponding to each class) of the same proposal generated from the RPN head. This helps in improving recall at higher values of $K$ but could be less helpful when $K$ is small. This is an important design consideration depending on the value of $K$ needed an application.

In previous experiments, we fixed the the NMS threshold to $0.5$ to generate proposals. However, NMS threshold could have an impact on the quality of the proposals. We study the effect of NMS threshold in Figure~\ref{fig:nms_fig} for OIV4\xspace and COCO\xspace. We again train on OIV4-source\xspace, COCO-source\xspace and test on OIV4-target\xspace, COCO-target\xspace respectively. We report \AR{K} for different values of $K$.

We observe that for smaller number of proposals ($K < 100$), lower thresholds tend to provide the best results. Intuitively, a low threshold can improve spatial coverage of objects since proposals are well spread out. When considering a larger number of proposals ($K > 100$), there are sufficient boxes to ensure spatial coverage, and having some redundancy is helpful. This is witnessed by the drop in $\AR{100,1000}$ at low NMS thresholds.

\begin{figure}[!tbp]
  \centering
  \begin{minipage}[t]{0.46\textwidth}
    \includegraphics[width=\textwidth]{figures/cls_specific_vs_agnostic.pdf}
    \caption{Effect of class agnostic regression vs. class specific regression when using detections as proposals}
    \label{fig:cls_ag}
  \end{minipage}
  \hfill
  \begin{minipage}[t]{0.46\textwidth}
    \includegraphics[width=\textwidth]{figures/oiv4_nms.pdf}
    \caption{Effect of NMS threshold on performance of proposals}
    \label{fig:nms_fig}
  \end{minipage}
  \vspace*{-0.25in}
\end{figure}

In Fig.~\ref{fig:rpn_vs_det}, we compare the performance of the RPN (from Faster R-CNN\xspace) with the detection head of Faster R-CNN\xspace and a single stage network like Retinanet. For combining the detections, we use NMS and class score thresholds of $0.5$ and $0.0$ respectively. To get an upper bound, we combine data from source and target classes to train a detector (denoted by Source: OIV4-all\xspace and Source: COCO-all\xspace in Fig.~\ref{fig:rpn_vs_det}). The performance of the RPN and detector head of a Faster R-CNN\xspace trained with this full dataset is shown in the figures.

\begin{figure}[t]
  \begin{minipage}[t]{0.60\textwidth}
    \centering
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \includegraphics[width=0.99\textwidth]{figures/oiv4_rpn_vs.pdf}
        \caption{OIV4}
        \label{fig:1}
    \end{subfigure}\hfill
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \includegraphics[width=0.99\textwidth]{figures/oiv4_rpn_vs.pdf}
        \caption{COCO}
        \label{fig:2}
    \end{subfigure}
    \caption{Comparing proposals from RPN head and detection head for the OIV4and COCO dataset}
    \label{fig:rpn_vs_det}
  \end{minipage}
  \hfill
  \begin{minipage}[t]{0.35\textwidth}
    \includegraphics[width=\textwidth]{figures/coco_ar_breakdown_v2.pdf}
    \caption{\AR{1000} on COCO-target\xspace is shown for different IoU thresholds for both RPN and Detection heads}
    \label{fig:nms_fig}
  \end{minipage}
\end{figure}

\noindent {\textbf{Weakly supervised detection:}}
A strong practical utility for generalized proposals is that, no bounding box annotations should be needed to train a detection model for new object classes. Hence, we report the performance of a weakly supervised detection model (that does not use bounding box annotations) trained with the proposals in Tab.~\ref{tab:det_map}. The model is trained on OIV4-target\xspace-train and evaluated on OIV4-target\xspace-test. We show results for proposals obtained from training with OIV4-source\xspace as well as OIV4-all\xspace. We see that the performance of the weakly supervised detection model is directly correlated with the generalization ability of the proposals being used, showing the need for good generalized proposals.

\begin{table}[t]
\scriptsize
\centering
\begin{center}
\caption{Comparing performance of proposals generated by RPN head and detection head for weakly supervised detection. We also show the \AR{100} numbers which are seen to be correlated with detection AP. Det. refers to Detection}\label{tab:det_map}
\begin{tabular}{|P{1.3cm}|P{1.0cm}|P{1.0cm}|P{1.0cm}|P{1.0cm}|}
\hline
\multicolumn{5}{|c|}{Target Dataset - OIV4-target\xspace}\\
\hline
 & \multicolumn{2}{c|}{Source: OIV4-source\xspace} & \multicolumn{2}{c|}{Source: OIV4-all\xspace}\\
& Det. AP & \AR{100} & Det. AP & \AR{100} \\\hline
RPN head & 8.7 & 55.0 & 9.6 & 60.4\\
Det. head & \textbf{24.0} & \textbf{69.4} & \textbf{30.8} & \textbf{76.9} \\
\hline

\end{tabular}
\end{center}
\vspace{-0.2in}
\end{table}

\subsection{Modeling Choices}
We first identify the best detection model and setting to extract proposals that generalize to new unseen classes. We then analyze generalization ability under different settings from this model. We reiterate that in order to test generalization, evaluation is done on target classes that have no intersection with the source classes used during training.

\noindent {\textbf{Choice of detection model:}} We compare the generalization ability of a two-stage network (Faster R-CNN\xspace) and a single-stage network (RetinaNet\xspace) in Fig.~\ref{fig:mod1}. Since, in a two-stage model like Faster R-CNN\xspace, the output from the RPN is class-agnostic and can be used as proposals too, we compare the performance of the RPN as well. The models are trained on COCO-source\xspace-train dataset. We report AR@100 on seen classes in the COCO-source\xspace-test dataset, as well as unseen classes in the COCO-target\xspace-test. The difference in performance between seen and unseen classes reflects the generalization gap. We also show an upper-bound performance on COCO-target\xspace-test obtained by models trained on the full training dataset containing both COCO-source\xspace-train and COCO-target\xspace-train.

\begin{figure}[t]
    \begin{subfigure}[t]{0.45\textwidth}
        \centering
        \includegraphics[width=0.99\textwidth]{figures/coco_rpn_vs.pdf}
        \caption{Comparison of detection models}
        \label{fig:mod1}
    \end{subfigure}\hfill
    \begin{subfigure}[t]{0.45\textwidth}
        \centering
        \includegraphics[width=0.99\textwidth]{figures/coco_ar_breakdown_v2.pdf}
        \caption{RPN vs. detection head}
        \label{fig:mod2}
    \end{subfigure}
    \caption{(a) \AR{100} corresponding to different models trained on COCO-source\xspace-train and evaluated on different test splits. Upper-bound corresponds to model trained on full COCO\xspace dataset and evaluated on COCO-target\xspace-test. (b) Average recall of RPN and detection head at different IoU thresholds, for model trained on COCO-source\xspace-train and evaluated on COCO-target\xspace-test}
    \label{fig:rpn_vs_det}
    \vspace{-0.2in}
\end{figure}

We notice that on seen classes, RetinaNet\xspace achieves a lower performance compared to Faster R-CNN\xspace (drop of $2.4\%$). However, the drop is larger for unseen target classes ($3.5\%$), indicating a larger generalization gap for RetinaNet\xspace. One reason for this is that RetinaNet\xspace is more sensitive to missing bounding boxes corresponding to unlabelled unseen classes in the source dataset. Proposals corresponding to unseen object classes that are not annotated in the training data are treated as hard-negatives, due to the use of focal-loss. Hence, the model heavily penalizes proposals corresponding to unannotated bounding boxes, leading to overall drop in AR. Since some seen classes share visual similarity with unseen classes, this sensitivity to missing annotations affects AR for seen classes too. However, this effect is more magnified for unseen target classes. On the other hand, in Faster R-CNN\xspace, only a small number of proposals (less than $512$) which do not intersect with annotated bounding boxes are sampled at random as negatives. The probability that a proposal corresponding to an unseen object class is chosen as a negative is lower, leading to better generalization. Hence, for the rest of the paper, we use Faster R-CNN\xspace as the detection model.

We also notice that the detection head of Faster R-CNN\xspace provides better overall performance \emph{without} sacrificing generalization. This can be attributed to better bounding box regression from the detection head which has additional layers, following the RPN in the model. To investigate this effect, we measure AR at different IoU thresholds for both sets of proposals for the model trained on COCO-source\xspace and evaluated on COCO-target\xspace in Fig.~\ref{fig:mod2}. We see that the difference in \AR{1000} increases drastically at higher values of IoU threshold, and is negligible at a threshold of $0.5$. This implies that the boxes from the detection head are more fine-tuned to exactly localize objects, unlike the RPN.

\noindent {\textbf{Choice of Faster R-CNN\xspace settings:}}
The results so far were obtained using class-specific bounding box regression (which is the standard setting in Faster R-CNN\xspace) for the detection head. Since we want the bounding boxes to generalize to unseen classes, class agnostic regression could be a valid choice too. We study this in Fig.~\ref{fig:cls_ag} for OIV4\xspace and COCO\xspace. We see that class agnostic regression is better for small number of proposals as seen by \AR{10,20,50}. However, when we consider more proposals (\AR{1000}), class specific regression provides a significant gain ($4.5\%$ for OIV4\xspace and $7.5\%$ for COCO\xspace). It results in multiple regressed versions (one corresponding to each class) of the same proposal generated from the RPN. This helps in improving recall at higher number of proposals.

Previously, we fixed the NMS threshold to $0.5$. We study the effect of this threshold in Fig.~\ref{fig:nms_fig}. We train on OIV4-source\xspace, COCO-source\xspace and test on OIV4-target\xspace, COCO-target\xspace respectively. Intuitively, a low threshold can improve spatial coverage of objects by ensuring proposals are spatially well spread out. When considering a larger number of proposals, there are sufficient boxes to ensure spatial coverage, and having some redundancy is helpful. This is witnessed by the steeper drop in \AR{1000} at low NMS thresholds, unlike \AR{100}.

Based on these observations, we use class-specific bounding box regression with an NMS threshold of $0.5$ for rest of the experiments.

\begin{figure}[t]
\vspace{-0.1in}
  \centering
  \begin{minipage}[t]{0.45\textwidth}
    \includegraphics[width=\textwidth]{figures/cls_specific_vs_agnostic.pdf}
    \caption{Effect of class agnostic regression vs. class specific regression}
    \label{fig:cls_ag}
  \end{minipage}
  \hfill
  \begin{minipage}[t]{0.45\textwidth}
    \includegraphics[width=\textwidth]{figures/nms.pdf}
    \caption{Effect of NMS threshold on performance of proposals}
    \label{fig:nms_fig}
  \end{minipage}
  \vspace{-0.21in}
\end{figure}

\begin{table}[h]
\vspace{-0.4in}
\centering
\begin{center}
\caption{Comparing performance of proposals generated by RPN head and detection head for weakly supervised detection. We also show the \AR{100} numbers which are seen to be correlated with detection AP}\label{tab:det_map}
\begin{tabular}{l|c|c|c|c}
\hline
\multicolumn{5}{c}{Target Dataset - OIV4-target\xspace}\\
\hline
 & \multicolumn{2}{c|}{Source: OIV4-source\xspace} & \multicolumn{2}{c}{Source: OIV4-all\xspace}\\
& Det. AP & \AR{100} & Det. AP & \AR{100} \\\hline
Faster R-CNN\xspace RPN & 8.7 & 55.0 & 9.6 & 60.4\\
Faster R-CNN\xspace Detection & \textbf{24.0} & \textbf{69.4} & \textbf{30.8} & \textbf{76.9} \\
\hline
\end{tabular}
\end{center}
\vspace{-0.35in}
\end{table}

\noindent {\textbf{Weakly supervised detection:}}
A strong practical utility for generalized proposals that localize all objects is that, no bounding box annotations should be needed to train a detection model for new object classes. Hence, we measure the effect of better generalized proposals on the performance of a weakly supervised detection model, trained without bounding box annotations. We show results corresponding to the RPN head and detection head of Faster R-CNN\xspace in Tab.~\ref{tab:det_map}. The weakly supervised model is trained on OIV4-target\xspace-train and evaluated on OIV4-target\xspace-test. We also show results for proposals obtained from training with OIV4-source\xspace as well as OIV4-all\xspace (upper-bound). We see that the performance of the weakly supervised detection model is directly correlated with the quality of the proposals being used, showing the need for good generalized proposals.

\section{Experiments}
\label{sec:expts}
We evaluate the ability of the object proposal obtained from detection models learned with different settings in Section~\ref{sec:model_choice} to generalize to new unseen classes. We also explore the effects of label-space granularity and the need for semantic and visual diversity. Finally, we show that a small set of prototypical classes could be used to train an effective proposal model for all classes in the dataset.

\subsection{Experimental Setup}
\noindent \textbf{Source and target splits: } We split each dataset into two parts: (a) {\it{Source dataset}} consisting of a set of seen classes called {\it{source classes}} and (b) {\it{Target dataset}} consisting of a set of unseen classes called {\it{target classes}}. {\it{Target dataset}} is used to evaluate the generalization of proposal models trained with the {\it{Source dataset}}. Since an image can contain both source and target classes, we ensure that such images are not present in the source class dataset. However, there may be a small number of images in the target dataset that contain source classes. We use the following two datasets for our experiments:

(1) {\it{Open Images V4\xspace(OIV4\xspace)~\cite{kuznetsova2018open}}} consists of $600$ classes. We retain only object classes which have more than $100$ training images. This results in a total of $482$ leaf classes. We randomly split all the leaf classes into $432$ source (OIV4-source\xspace dataset) and $50$ target (OIV4-target\xspace dataset) classes. There are also annotations associated only with internal nodes (for example, "animal") and without a specific leaf label (like the type of animal). We remove such annotations and all associated images, since such images cannot be unambiguously assigned to a source or target split. This leaves us with $1.2M$ images with $7.96M$ boxes in the train split and $73k$ images with $361K$ boxes in the test split. For training proposal models, we always use the train split and for evaluation we use the test split. Wherever needed, we explicitly suffix the dataset with "train" and "test" (for example, OIV4-source\xspace-train and OIV4-source\xspace-test).

(2) {\it{COCO\xspace~\cite{coco}}}: We use the 2017 version of the COCO\xspace dataset and randomly split the classes in to $70$ source (COCO-source\xspace dataset) and $10$ target (COCO-target\xspace dataset) classes. For training, we use the train split and for evaluation, we use the $5000$ images from the validation set. Wherever needed, we explicitly suffix the dataset with ``train" and ``test".

Target classes list is provided in the supplementary.

\noindent\textbf{Evaluation metrics: }
We report the standard average recall (\AR{k})~\cite{hosang2015makes} metric to evaluate the quality of proposals. One of the main motivations for building a generalized proposal model is to use the resulting proposals to train detection models for unseen classes with limited or no bounding box annotation. A typical proposal-based supervised detection model RCNN could also be used to evaluate the quality of proposals. However, the application to weakly supervised detection is more compelling since their performance is closely tied to proposals than supervised models which can correct the inaccuracies in proposals due to availability of labelled bounding boxes.  Hence, we implement a weakly supervised detector with the approach used in YOLO9000~\cite{redmon2017yolo9000}\footnote{We chose~\cite{redmon2017yolo9000} due to its simplicity. In practice, we can use other weakly supervised approaches too.}. We report the detection AP (averaged over IoU thresholds ranging from $0.5$ to $0.95$) on the test set of the target dataset. Please see the supplementary material for more details.

\noindent\textbf{Implementation details: }
We fix Imagenet pre-trained ResNet-50 with Feature Pyramid Networks \cite{lin2017feature} as the backbone for all models. We use the Detectron codebase~\cite{girshick2018detectron}. For COCO\xspace, we train the models for $90k$ iterations with an initial learning rate and the decay suggested in \cite{ren2015faster}. For OIV4\xspace, we train the models for $800k$ iterations with an initial learning rate of $0.01$ and cosine learning rate decay. When training the weakly supervised model (\cite{redmon2017yolo9000}), we use the top $100$ proposals in each image to choose pseudo ground truth at every training iteration.
\section{Introduction}
\label{sec:intro}
Object detection systems have shown considerable improvements for the fully supervised setting \cite{ren2015faster,lin2017focal,liu2016ssd} on challenging benchmarks like COCO~\cite{coco} with large number of bounding box annotations for each class. Since annotating bounding boxes is a tedious time consuming process, weakly supervised methods~\cite{Gao_2019_ICCV,arun2019dissimilarity,tang2018pcl} that only use image-level labels are gaining increasing attention as well.

Both approaches typically consider detection as a combination of two tasks: (a) spatial localization of the object in an image using a bounding box (proposals) and (b) classification of the bounding box into the correct category. One way to scale object detection to a large number of classes would be to develop a good class-agnostic localization model that \emph{generalizes} to all classes. We could then train more data efficient detection models, requiring fewer or no bounding box annotations to only solve the classification task. Recent works \cite{uijlings2018revisiting,singh2018r} have also shown the benefit of good proposal models for training object detectors with limited bounding box annotations.

In our work, we study this notion of a generalized proposal model. In other words, how well can an object localization model trained on a small set of classes also localize other unseen classes? Intuitively, to localize a fine-grained vehicle like taxi, it might be sufficient to train a localization model with examples from other vehicles like cars or vans (as shown in Fig.~\ref{fig:pull_fig}). For localization (unlike classification), we may not need any training data for this class. While there have been few works that leverage this intuition for weakly supervised learning~\cite{uijlings2018revisiting}, the extent to which object localization depends on the categories used to train the localization model has not been well quantified and studied in detail. For example, does the above example extend beyond fine-grained categories? Can a model trained on cars localize other vehicles?

\begin{figure}
\centering
\includegraphics[width=0.9\linewidth]{figures/rui_pull}
\caption{Proposal models learned on source vehicle classes can localize unseen target objects which share similar localization structure like ``bus" and ``taxi". However, ``barge" and ``gondola", which are also vehicles will not be localized by this model, due to lack of visual diversity in the source dataset for vehicles}
\label{fig:pull_fig}
\vspace{-0.2in}
\end{figure}

We show that with \emph{careful design considerations}, it is possible to train a good "generalized proposal model". The main contribution of our work is two-fold: a) We formally and systematically study the desired properties of a dataset required to build such a model, and b) We highlight critical modeling and hyper-parameter choices that enable us to leverage existing  frameworks like Faster R-CNN\xspace~\cite{ren2015faster}.

We answer several key questions regarding the \emph{dataset}:

\noindent {\bf{Label Space Granularity: }}We show that annotating bounding boxes with coarse (e.g. vehicle rather than sports car in Fig.~\ref{fig:pull_fig}) labels is sufficient to get good proposals. Getting these coarse-grained annotations are significantly less tedious and more accurate than the fine-grained ones.

\noindent {\bf{Semantic and Visual Diversity: }}We observe that it is crucial to have semantic and visual diversity at both coarse and fine levels of granularity for good generalization. For instance, when training a proposal model with a coarse label "vehicle", we need examples of different types of vehicles like "car" and "boats", even if the examples are only annotated as ``vehicle" as shown in Fig.~\ref{fig:pull_fig}. But are all classes within a coarse category equally important? This question is important to answer since sourcing data from a large number of fine-grained categories is very challenging.
To this end, we introduce the concept of {\it{prototypical classes}} as a subset of leaf classes in a semantic hierarchy that are sufficient and necessary to construct a dataset to train a good generalized proposal model. We show that it is indeed possible to construct a small prototypical set. This has positive implications for large taxonomies, since it is sufficient to annotate examples only for the prototypical classes. We also study different empirical ways of identifying these classes.

We also propose key \emph{modeling} choices that are critical for leveraging state-of-the-art detectors like Faster R-CNN\xspace to obtain generalized proposals. We observe that the detections from a Faster R-CNN\xspace model serve as better generalized proposals than the outputs from its Region Proposal Network (RPN). We also show the effect of NMS threshold and class agnostic regression in generating good proposals.

On OIV4\xspace \cite{kuznetsova2018open} dataset, we show a significant improvement of $9\%-13\%$ in the average recall (\AR{k}) of proposals, when using detections rather than RPN outputs for generalized proposals. We observe that coarse-grained labels are sufficient to train a good proposal model that generalizes to all the OIV4\xspace classes. Finally, compared to training with all the seen objects, we show that training with less than $25\%$ of the object classes chosen appropriately leads to only a drop of $4.3\%$ in \AR{100}, while training with $50\%$ of such classes leads to a negligible drop of $0.9\%$.
\section{Introduction}
\label{sec:intro}
Object detection systems have shown considerable improvements for fully supervised settings \cite{ren2015faster,lin2017focal,liu2016ssd,redmon2017yolo9000,dai2016r}, as well as weakly supervised settings~\cite{Gao_2019_ICCV,arun2019dissimilarity,tang2018pcl} that only use image-level labels. Both approaches typically consider detection as a combination of two tasks: (a) spatial localization of the objects using proposals and (b) classification of the proposals into correct classes.
A generalized proposal model that localizes all classes can help in scaling object detection. This could lead to the use of fewer or no bounding box annotations to only solve the classification task and development of more sophisticated classifiers, as explored in works like \cite{uijlings2018revisiting,singh2018r}.

Many detection models \cite{ren2015faster,lin2017focal} have been developed in recent years, which can be used to obtain high quality object proposals. However, an equally important aspect that determines the generalization ability of proposals is \emph{the dataset} used to train these models. As illustrated in Fig.~\ref{fig:pull_fig}, the objects and class labels in a dataset significantly impact the ability to generalize to new classes. Intuitively, to localize a fine-grained vehicle like taxi in a target dataset, it might be sufficient to train a localization model with other vehicles like cars or vans in the source dataset. For localization (unlike classification), we may not need any training data for this class. On the other hand, training with these classes will not help in localizing other vehicles like boat.

While few works leverage this intuition for weakly supervised learning~\cite{uijlings2018revisiting}, the extent to which object localization depends on the categories used to train the model has not been well quantified and studied in detail. Towards this end, we define ``generalization" as the ability of a model to localize (not classify) objects not annotated in the training dataset. In our work, we answer the question: \emph{What kind of dataset is best suited to train a model that generalizes even to unseen object classes?}

We further study the ability of popular detection models like Faster R-CNN\xspace \cite{ren2015faster} and RetinaNet\xspace \cite{lin2017focal} to generate proposals that generalize to unseen classes. These networks are designed to improve the detection quality for the small set of seen classes in the training dataset. We carefully study these design choices and provide a way to obtain proposals that generalize to a larger set of unseen classes.

\begin{figure}[t!]
\centering
\includegraphics[width=0.95\linewidth]{figures/rui_pull2}
\caption{Proposal models learned on seen vehicle classes can localize unseen classes which share similar localization structure like ``bus" and ``taxi". However, ``barge" and ``gondola", which are also vehicles will not be precisely localized by this model, due to lack of visual diversity in the training dataset for vehicles}
\label{fig:pull_fig}
\vspace{-0.2in}
\end{figure}

We answer several questions about dataset properties and modeling choices required for generalized proposals:
\begin{itemize}

    \item \textbf{What are the properties of object classes to ensure generalization of proposals from a model?} First, we show that it is crucial to have visual diversity to obtain generalized proposals. We need examples of different vehicles like ``car" and ``boats", even if the examples are only labelled as ``vehicle". Further, we hypothesize the existence of {\it{prototypical classes}} as a subset of leaf classes in a semantic hierarchy that are sufficient and necessary to construct a dataset to train a model for proposal generalization. We define new quantitative metrics to measure these properties for any set of classes and show that it is possible to construct a small prototypical set of object classes. This has positive implications for large taxonomies, since it is sufficient to annotate examples only for the prototypical classes.

    \item \textbf{Does the label-granularity of the dataset affect generalization? If so, what is the coarsest granularity that can be used?} Coarse-grained labels (``vehicles" instead of ``taxis") are significantly less tedious to annotate and more accurate than fine-grained labels. Past works like RFCNN-3000 \cite{singh2018r} argued that a single super class might be sufficient to obtain good proposals. However, we show that there is a trade-off between using very few coarse classes and large-number of fine-grained classes, and a middle-ground approach leads to best generalization.

    \item \textbf{What are the \emph{modeling} choices that are critical for leveraging state-of-the-art detectors to obtain generalized proposals?} We show that: (a) detections from two-stage networks like Faster R-CNN\xspace are better for obtaining generalized proposals than a single-stage network like RetinaNet\xspace, (b) while class-specific bounding box regression is typically used in Faster R-CNN\xspace, it is beneficial only when considering larger number of proposals (average recall AR@1000) and class-agnostic regression is better when considering fewer proposals (AR@100) and (c) choice of NMS threshold is dependent on the number of proposals being considered (AR@100 or AR@1000).

\end{itemize}

On OIV4\xspace \cite{kuznetsova2018open}, we show that compared to training with all the object classes, using a prototypical subset of $25\%$ of the object classes only leads to a drop of $4.3\%$ in average recall (AR@100), while training with $50\%$ of such classes leads to a negligible drop of $0.9\%$.
We also show how the detections from Faster R-CNN\xspace can be fused to obtain high quality proposals that have $10\%$ absolute gain in AR@100 compared to the class-agnostic proposals of the RPN from the same network and $3.5\%$ better than RetinaNet\xspace. To stress the practical importance of generalized proposals, we also show that generalization ability is directly correlated with the performance of weakly supervised detection models.

\subsection{Modeling Choices}
We first identify the best way to extract proposals that generalize to new unseen classes, from a detection model. We analyze bounding boxes obtained under different settings from a detection model.  We reiterate that evaluation is done on target classes that have no intersection with the source classes used during training.

\noindent {\textbf{Proposals from detection model:}} As we showed in Sec.~\ref{subsec:model}, bounding boxes from a detection model can be combined in a class-agnostic fashion to obtain proposals. However, in a two-stage model like Faster R-CNN\xspace, the output from the RPN is class-agnostic and can be used as proposals too. In Fig.~\ref{fig:rpn_vs_det}, we compare the performance of the RPN (from Faster R-CNN\xspace) with the detection head of Faster R-CNN\xspace and a single stage network like Retinanet. For combining the detections, we use NMS and class score thresholds of $0.5$ and $0.0$ respectively. To get an upper bound, we combine data from source and target classes to train a detector (denoted by Source: OIV4-all\xspace and Source: COCO-all\xspace in Fig.~\ref{fig:rpn_vs_det}). The performance of the RPN and detector head of a Faster R-CNN\xspace trained with this full dataset is shown in the figures.

\begin{figure}[t]
  \begin{minipage}[t]{0.60\textwidth}
    \centering
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \includegraphics[width=0.99\textwidth]{figures/oiv4_rpn_vs.pdf}
        \caption{OIV4}
        \label{fig:1}
    \end{subfigure}\hfill
    \begin{subfigure}[t]{0.49\textwidth}
        \centering
        \includegraphics[width=0.99\textwidth]{figures/oiv4_rpn_vs.pdf}
        \caption{COCO}
        \label{fig:2}
    \end{subfigure}
    \caption{Comparing proposals from RPN head and detection head for the OIV4and COCO dataset}
    \label{fig:rpn_vs_det}
  \end{minipage}
  \hfill
  \begin{minipage}[t]{0.35\textwidth}
    \includegraphics[width=\textwidth]{figures/coco_ar_breakdown.pdf}
    \caption{\AR{1000} on COCO-target\xspace is shown for different IoU thresholds for both RPN and Detection heads}
    \label{fig:nms_fig}
  \end{minipage}
\end{figure}

We observe that detection heads from Faster R-CNN\xspace and Retinanet, both significantly outperform the RPN head on both datasets (OIV4-source\xspace and COCO-source\xspace columns) for unseen classes. This shows that the boxes from the detection head generalize to unseen classes in the target dataset better. We also note that Retinanet and Faster R-CNN\xspace yield comparable performances. Hence, we expect the observations made with Faster R-CNN\xspace to transfer to single stage networks like Retinanet and we fix Faster R-CNN\xspace as the base model for all analysis from now on, unless specified otherwise.

The superior performance of the detection head can be attributed to better bounding box regression from the detection head which has additional layers, following the RPN in the model. To investigate this effect, we measure \AR{1000} at different IoU thresholds for both sets of proposals for the model trained on COCO-source\xspace and evaluated on COCO-target\xspace in Fig.~\ref{fig:nms_fig}. We see that the difference increases drastically at higher values of IoU threshold, and is negligible at a threshold of $0.5$. This implies that the boxes from the detection head are more fine-tuned to exactly localize the objects.

\noindent {\textbf{Weakly supervised detection:}}
A strong practical utility for developing generalized proposals that localize all objects is that, no bounding box annotations should be needed to train a detection model for new object classes. Hence, we also report the performance of a weakly supervised detection model (that does not need any bounding box annotations) trained with the proposals in Table~\ref{tab:det_map}. The weakly supervised model is trained on OIV4-target\xspace-train and evaluated on OIV4-target\xspace-test. We show results for proposals obtained from training with OIV4-source\xspace as well as OIV4-all\xspace. We see that the performance of the weakly supervised detection model is directly correlated with the generalization ability of the proposals being used. The better proposals obtained from the detection head lead to higher AP, showing the need for good generalized proposals.

\begin{table}[t]
\scriptsize
\centering
\begin{center}
\caption{Comparing performance of proposals generated by RPN head and detection head for weakly supervised detection. We also show the \AR{100} numbers which are seen to be correlated with detection AP. Det. refers to Detection}\label{tab:det_map}
\begin{tabular}{|P{1.3cm}|P{1.0cm}|P{1.0cm}|P{1.0cm}|P{1.0cm}|}
\hline
\multicolumn{5}{|c|}{Target Dataset - OIV4-target\xspace}\\
\hline
 & \multicolumn{2}{c|}{Source: OIV4-source\xspace} & \multicolumn{2}{c|}{Source: OIV4-all\xspace}\\
& Det. AP & \AR{100} & Det. AP & \AR{100} \\\hline
RPN head & 8.7 & 55.0 & 9.6 & 60.4\\
Det. head & \textbf{24.0} & \textbf{69.4} & \textbf{30.8} & \textbf{76.9} \\
\hline

\end{tabular}
\end{center}
\vspace{-0.2in}
\end{table}

\noindent {\textbf{Class specific regression and NMS threshold:}}
The results so far where obtained using class specific bounding box regression (which is the standard setting in Faster R-CNN\xspace) for the detection head. Since we want the bounding boxes to generalize to unseen classes, class agnostic regression could be a valid choice too. We study this in Table~\ref{tab:class_agnostic} for OIV4\xspace and COCO\xspace.

First we see that class agnostic regression is comparable, or provides a slightly higher \AR{K} for small values of $K < 100$. However, when we consider more proposals (\AR{1000}), class specific regression provides a significant gain ($4.5\%$ for OIV4\xspace and $7.5\%$ for COCO\xspace). It results in multiple regressed versions (one corresponding to each class) of the same proposal generated from the RPN head. This helps in improving recall at higher values of $K$ but could be less helpful when $K$ is small. This is an important design consideration depending on the value of $K$ needed an application.

In previous experiments, we fixed the the NMS threshold to $0.5$ to generate proposals. However, NMS threshold could have an impact on the quality of the proposals. We study the effect of NMS threshold in Figure~\ref{fig:nms_fig} for OIV4\xspace and COCO\xspace. We again train on OIV4-source\xspace, COCO-source\xspace and test on OIV4-target\xspace, COCO-target\xspace respectively. We report \AR{K} for different values of $K$.

We observe that for smaller number of proposals ($K < 100$), lower thresholds tend to provide the best results. Intuitively, a low threshold can improve spatial coverage of objects since proposals are well spread out. When considering a larger number of proposals ($K > 100$), there are sufficient boxes to ensure spatial coverage, and having some redundancy is helpful. This is witnessed by the drop in $\AR{100,1000}$ at low NMS thresholds.

\begin{figure}[!tbp]
  \centering
  \begin{minipage}[t]{0.49\textwidth}
    \includegraphics[width=\textwidth]{figures/cls_specific_vs_agnostic.pdf}
    \caption{Effect of class agnostic regression vs. class specific regression when using detections as proposals}
    \label{fig:cls_ag}
  \end{minipage}
  \hfill
  \begin{minipage}[t]{0.49\textwidth}
    \includegraphics[width=\textwidth]{figures/oiv4_nms.pdf}
    \caption{Effect of NMS threshold on performance of proposals}
    \label{fig:nms_fig}
  \end{minipage}
\end{figure}

\section{Related Work}
\label{sec:relwork}
\noindent \textbf{Generalizing localization across multiple classes: }The idea of different object classes sharing the same structure has been exploited in building detection models for a long time\cite{felzenszwalb2009object,novotny2016have,ott2011shared,salakhutdinov2011learning,torralba2004sharing}. More recently, \cite{dai2016r,ren2015faster} also have a dedicated proposal network for object localization. However these works do not measure the transferability of proposals trained on one set of classes to another.

Uijlings \textit{et al.} \cite{uijlings2018revisiting} tried to transfer information from coarse source classes to fine-grained target classes that share similar localization properties. They showed that this can help weakly supervised detection for the target classes. LSDA \cite{hoffman2014lsda} transformed classifiers into detectors by sharing knowledge between classes. Multiple works \cite{tang2016large,hoffman2016large,rochan2015weakly,guillaumin2012large} showed the benefit of sharing localization information between similar classes to improve semi supervised and weakly supervised detection.
Yang \textit{et al.} \cite{yang2019detecting} trained a large-scale detection model following similar principles. Singh \textit{et al.} \cite{singh2018r} showed that even a detector trained with one class can localize objects of different classes sufficiently well due to commonality between classes. We generalize this idea further. There has also been work on learning models \cite{yang2019detecting,redmon2017yolo9000,gao2019note} with a combination of bounding boxes for certain classes and only class labels for others. They inherently leverage the idea that localization can generalize across multiple classes. We provide systematic ways to quantify and measure this property for proposal models.

\noindent \textbf{Object proposal generation models:}
There have been many seminal works on generating class-agnostic object proposals  \cite{uijlings2013selective,zitnick2014edge,pont2016multiscale,krahenbuhl2014geodesic}. A comprehensive study of different methods can be found in \cite{hosang2015makes} and a study of proposal evaluation metrics can be found in \cite{chavali2016object}. Proposal models have also been trained with dedicated architectures and objectives in \cite{pinheiro2015learning,kuo2015deepbox,szegedy2014scalable}.
In our work, we leverage standard models like Faster R-CNN\xspace and focus on the dataset properties required to achieve generalization with this model.
\section{List of OIV4 classes}
\noindent\textbf{OIV4-Target50}:
\textit{Book, Bookcase, Bull, Cabbage, Camera, Centipede, Coat, Coffee cup, Computer keyboard, Desk, Dog, Doughnut, Egg, Falcon, Flowerpot, Fork, Goldfish, Hippopotamus, Hot dog, House, Juice, Kitchen knife, Knife, Lavender, Missile, Mobile phone, Motorcycle, Mug, Orange, Palm tree, Penguin, Piano, Picture frame, Porch, Rabbit, Refrigerator, Remote control, Rifle, Seahorse, Skateboard, Strawberry, Sunglasses, Swimming pool, Sword, Tap, Tripod, Truck, Washing machine, Wine glass, Zebra}

\vspace*{0.05in}
\noindent\textbf{OIV4-Source432}:
\textit{Accordion, Adhesive tape, Airplane, Alarm clock, Alpaca, Ambulance, Ant, Antelope, Apple, Artichoke, Asparagus, Axe, Backpack, Bagel, Balance beam, Balloon, Banana, Banjo, Barge, Barrel, Baseball bat, Baseball glove, Bat, Bathroom cabinet, Bathtub, Beaker, Bee, Beehive, Beer, Bell pepper, Belt, Bench, Bicycle, Bicycle helmet, Bicycle wheel, Bidet, Billboard, Billiard table, Binoculars, Blender, Blue jay, Boot, Bottle, Bow and arrow, Bowl, Bowling equipment, Box, Boy, Brassiere, Bread, Briefcase, Broccoli, Bronze sculpture, Brown bear, Burrito, Bus, Bust, Butterfly, Cabinetry, Cake, Cake stand, Calculator, Camel, Canary, Candle, Candy, Cannon, Canoe, Cantaloupe, Carrot, Cart, Castle, Cat, Cat furniture, Caterpillar, Cattle, Ceiling fan, Cello, Chainsaw, Chair, Cheese, Cheetah, Chest of drawers, Chicken, Chopsticks, Christmas tree, Closet, Cocktail, Coconut, Coffee, Coffee table, Coffeemaker, Coin, Common fig, Computer monitor, Computer mouse, Convenience store, Cookie, Corded phone, Countertop, Cowboy hat, Crab, Cream, Cricket ball, Crocodile, Croissant, Crown, Crutch, Cucumber, Cupboard, Curtain, Cutting board, Dagger, Deer, Diaper, Dice, Digital clock, Dinosaur, Dog bed, Doll, Dolphin, Door, Door handle, Dragonfly, Drawer, Dress, Drill, Drinking straw, Drum, Duck, Dumbbell, Eagle, Earrings, Elephant, Envelope, Fedora, Filing cabinet, Fire hydrant, Fireplace, Flag, Flute, Flying disc, Food processor, Football, Football helmet, Fountain, Fox, French fries, Frog, Frying pan, Gas stove, Giraffe, Girl, Glasses, Glove, Goat, Goggles, Golf ball, Golf cart, Gondola, Goose, Grape, Grapefruit, Guacamole, Guitar, Hamburger, Hammer, Hamster, Handbag, Handgun, Harbor seal, Harp, Harpsichord, Headphones, Hedgehog, Helicopter, High heels, Hiking equipment, Honeycomb, Horn, Horse, Houseplant, Human arm, Human beard, Human body, Human ear, Human eye, Human face, Human foot, Human hair, Human hand, Human head, Human leg, Human mouth, Human nose, Ice cream, Infant bed, Ipod, Isopod, Jacket, Jacuzzi, Jaguar, Jeans, Jellyfish, Jet ski, Jug, Kangaroo, Kettle, Kitchen and dining room table, Kite, Koala, Ladder, Ladybug, Lamp, Lantern, Laptop, Lemon, Leopard, Lifejacket, Light bulb, Lighthouse, Lily, Limousine, Lion, Lipstick, Lizard, Lobster, Loveseat, Lynx, Magpie, Man, Mango, Maple, Mechanical fan, Microphone, Microwave oven, Milk, Miniskirt, Mirror, Mixer, Mixing bowl, Monkey, Mouse, Muffin, Mule, Mushroom, Musical keyboard, Nail, Necklace, Nightstand, Oboe, Office building, Organ, Ostrich, Otter, Oven, Owl, Oyster, Paddle, Pancake, Panda, Paper towel, Parachute, Parking meter, Parrot, Pasta, Peach, Pear, Pen, Pencil case, Perfume, Picnic basket, Pig, Pillow, Pineapple, Pitcher, Pizza, Plastic bag, Plate, Platter, Polar bear, Pomegranate, Popcorn, Porcupine, Poster, Potato, Power plugs and sockets, Pretzel, Printer, Pumpkin, Punching bag, Raccoon, Radish, Ratchet, Raven, Rays and skates, Red panda, Rhinoceros, Rocket, Roller skates, Rose, Rugby ball, Ruler, Salad, Salt and pepper shakers, Sandal, Saucer, Saxophone, Scale, Scarf, Scissors, Scoreboard, Scorpion, Sea lion, Sea turtle, Seafood, Seat belt, Segway, Serving tray, Sewing machine, Shark, Sheep, Shelf, Shirt, Shorts, Shotgun, Shower, Shrimp, Sink, Ski, Skull, Skyscraper, Slow cooker, Snail, Snake, Snowboard, Snowman, Snowmobile, Snowplow, Sock, Sofa bed, Sombrero, Sparrow, Spice rack, Spider, Spoon, Sports uniform, Squid, Squirrel, Stairs, Starfish, Stationary bicycle, Stool, Stop sign, Street light, Stretcher, Studio couch, Submarine sandwich, Suit, Suitcase, Sun hat, Sunflower, Surfboard, Sushi, Swan, Swim cap, Swimwear, Syringe, Table tennis racket, Tablet computer, Taco, Tank, Tart, Taxi, Tea, Teapot, Teddy bear, Television, Tennis ball, Tennis racket, Tent, Tiara, Tick, Tie, Tiger, Tin can, Tire, Toilet, Toilet paper, Tomato, Toothbrush, Tortoise, Towel, Tower, Traffic light, Train, Training bench, Treadmill, Tree house, Trombone, Trumpet, Turkey, Umbrella, Unicycle, Van, Vase, Vehicle registration plate, Violin, Volleyball, Waffle, Wall clock, Wardrobe, Waste container, Watch, Watermelon, Whale, Wheel, Wheelchair, Whisk, Whiteboard, Willow, Window, Window blind, Wine, Wine rack, Wok, Woman, Wood-burning stove, Woodpecker, Worm, Wrench, Zucchini}

\section{List of COCO classes}
\noindent\textbf{COCO-Target10}:
\textit{apple, car, cat, clock, dining table, fire hydrant, giraffe, hair drier, handbag, truck}

\vspace*{0.05in}
\noindent\textbf{COCO-Source70}:
\textit{airplane, backpack, banana, baseball bat, baseball glove, bear, bed, bench, bicycle, bird, boat, book, bottle, bowl, broccoli, bus, cake, carrot, cell phone, chair, couch, cow, cup, dog, donut, elephant, fork, frisbee, horse, hot dog, keyboard, kite, knife, laptop, microwave, motorcycle, mouse, orange, oven, parking meter, person, pizza, potted plant, refrigerator, remote, sandwich, scissors, sheep, sink, skateboard, skis, snowboard, spoon, sports ball, stop sign, suitcase, surfboard, teddy bear, tennis racket, tie, toaster, toilet, toothbrush, traffic light, train, tv, umbrella, vase, wine glass, zebra}

\section{Training weakly supervised model}
We train a weakly supervised detection model based on Fast R-CNN \cite{girshick2015fast}, with the same method as used in YOLO9000\cite{redmon2017yolo9000}. During training, the Fast R-CNN model assigns scores to all the proposals in the image. We then choose the highest scoring proposal (from the top $100$ proposals in the image) for each weakly labelled class in the image and use it as the psuedo ground-truth bounding box to perform back propagation. Additionally all proposals which have an overlap greater than $0.7$ with this bounding box are also treated as psuedo ground-truth, while all bounding boxes with an overlap greater than $0.5$ and less than $0.7$ are not used in backpropagation. We train the model for $90000$ iterations.

We use $500$ proposals in each image for both training and evaluation. However, we only use the top $100$ proposals when choosing psuedo ground-truth during training. This allows the model to choose positive bounding boxes from the most confident proposals in the image, while use other proposals as background proposals for training the detection model.

\section{Sufficiency of prototypical classes}
\begin{figure}[h]
\centering
\includegraphics[width=0.55\linewidth]{figures/sampling_method_ar1000.pdf}
\caption{Average recall \AR{1000} for proposals obtained from models trained with varying number of prototypical classes chosen by different methods. We show the average recall on the OIV4-target\xspace dataset with $50$ unseen classes. $P$ denotes the number of prototypical classes. Higher value indicates higher sufficiency.}
\label{fig:proto_suff_ar1000}
\vspace{-0.2in}
\end{figure}

{\small
\bibliographystyle{splncs04}

redpajama_set_name: RedPajamaArXiv


text:
A leaked image seems to confirm iPhone 13 instead of 12S, and there are uncomfortable rumours about the price
By Carrie Marshall published 28 August 21
The iPhone 13 official name appears to have been leaked, and it's going to surprise anyone who expected it to be called Tiny Tim's Tick-Tock-A-Doodle-Phone: the successor to the iPhone 12 is going to be called… the iPhone 13.
I'll pause for a moment while you pick yourself up from the floor.
Obvious sarcasm aside, it is very slightly surprising news because of course the number 13 has negative connotations for some people: it's why some buildings don't have a 13th floor and why some airplanes don't have a 13th row. So there was a lot of speculation that the iPhone numbering system would take a little iPhone XR/XS-style jump away from unlucky thirteen to be called something else, such as the iPhone 12S. Apparently not: via MacRumors we can see the words iPhone 13 printed very clearly on what looks awfully like proper Apple packaging, which you'd expect to be in production by now.
But the iPhone 13 might have another surprise up its sleeve, and that one might annoy you.
Will the iPhone 13 be unlucky for some?
There's a chance that the iPhone 13 is going to be more expensive than the iPhone 12. According to electronics news site DigiTimes, the ongoing global semiconductor shortage is even starting to affect the mighty Apple. That contradicts previous reports that Apple intended to keep the prices identical to those of the iPhone 12, but the laws of supply and demand have started to make component costs soar, and it's possible Apple has been more affected by this than expected.
Apple has tight control over its manufacturing partners but ultimately it's still reliant on components made by other organisations. That's one of the reasons Apple tends to buy so many companies: it prefers to control the technologies that its products rely on. But even Apple can't own everything.
Apple already raised the price of the iPhone 12 compared to the iPhone 11, which T3's own Matt Bolton griped about in his iPhone 12 review. If we had to gamble, we'd say Apple will probably try to absorb any price increase, and probably won't be too badly affected because of how it control its supplies. But that's just a guess – we'll find out soon enough.
Carrie Marshall
Writer, musician and broadcaster Carrie Marshall has been covering technology since 1998 and is particularly interested in how tech can help us live our best lives. Her CV is a who's who of magazines, newspapers, websites and radio programmes ranging from T3, Techradar and MacFormat to the BBC, Sunday Post and People's Friend. Carrie has written thirteen books, ghost-wrote two more and co-wrote another seven books and a Radio 2 documentary series. When she's not scribbling, she's the singer in Glaswegian rock band HAVR (havrmusic.com).
5 luxury soap brands you need to know about
Our rundown of the best quality soap bars from some of the most premium brands
By Lee Bell • Published 18 January 22
PS5 restock: GAME live with massive stock drop – Spider-Man, Ratchet, FIFA 22, and more
Get a console in time for Uncharted: Legacy of Thieves Collection
By Matthew Forde • Published 18 January 22

redpajama_set_name: RedPajamaCommonCrawl


text:
Research course
University of Southampton · School of Humanities
First or upper second class honours degree and normally Masters at Merit level, or an equivalent standard in other qualifications approved by the University in History or a cognate discipline.
January, September
Studying history can help us define a path for our future society; it helps us to discover where we came from and how our civilisation evolved. Become an expert in your chosen field of research through the PhD History Research Programme. Take a step back in time and research history on a global scale. Develop your own understanding and work with specialists in this field, while networking with fellow like-minded students.
Staff have strong interests in gender history, the history of identities, and the study of history and memory of migration, and offer supervision in a range of subject areas, including interdisciplinary research projects with Humanities colleagues.
The programme is also available via distance learning, which is designed for people who wish to study without travelling to Southampton.
Distance learningis available for this qualification
pgapply.fh@southampton.ac.uk
Similar courses from University of Southampton
Medieval and Renaissance Culture
Applied Linguistics (Research Methodology)
Applied Linguistics for Language Teaching


text:
Vision and Principles
Country Food
Store-Bought Food
Local Food Production
Programs and Community Initiative
Strategy and
The Role of the Coalition
Core Recipe Project
Country Food Guidelines
2013 Pilot Nunavut Food Price Survey
2014 Nunavut Food Price Survey
Nunavut Food Security Strategy and Action Plan 2014-16
Annual Coalition Meeting Outcomes
The Nunavut Food Security Coalition held its annual face-to-face meeting in Iqaluit from March 17-19, 2015.
1. Agreed to a Terms of Reference
The Terms of Reference will clarify the mandate, roles, and responsibilities of the Coalition; guide decision-making regarding membership; and describe how Coalition members will work together.
The Coalition will make the Terms of Reference publicly available once all Coalition members have had the chance to sign it.
2. Agreed to a Coalition position on short-term food relief efforts
The Coalition acknowledges that long-term changes are required to address the root causes of food insecurity, but that short-term food relief efforts are needed in the meantime. The Coalition believes that efforts to support food relief in Nunavut should reflect the Nunavut context by aligning with the Nunavut Food Security Strategy. The Coalition feels as though external support should be directed to community-based efforts, and is committed to developing resources to facilitate relationship building.
The Coalition will provide a publicly available position after the content is approved at the next monthly Coalition meeting.
3. Agreed to work together to advocate for improvements to the Nutrition North Canada program
Coalition members shared their perspectives on the Nutrition North Canada program.
The Coalition will publish the Niqittiavak Committee's report on the Nutrition North Canada program, which offers recommendations for enhancing the program to better suit the needs of Nunavummiut.
The Coalition will create a working group to review the report recommendations and identify emerging issues, and will engage with the Nutrition North Canada Advisory Board on these recommendations and issues whenever possible.
4. Participated in the Public Policy Forum's Iqaluit Roundtable on Food Insecurity in Northern Canada
The Coalition discussed what lessons can be learned from the implementation of the Nunavut Food Security Strategy, what initiatives have proven most successful, what gaps remain to be addressed, and what should be the key areas of focus in aligning future collaboration.
Coalition representatives will participate in the Public Policy Forum's Ottawa Roundtable on April 20, 2015.
Nunavut Food Security Coalition © 2016


Over 500 features being described produce a detailed definition of target groups.
Our Marktanalyst package enables the visualisation of various features – even across areas.
Our experts know the strengths and weaknesses of each feature during use and are pleased to advise you.
trendsetter and innovator for successful geomarketing.
Competence and the long-standing experience of working.