\section{Introduction}

In the era of big data, similarity search, also known as approximate nearest neighbor search, plays a pivotal role in modern information retrieval systems, like content-based image and document search, multimedia retrieval and plagiarism detection \cite{lew2006content}, etc. If the search is carried out directly in the original real-valued feature space, the cost would be extremely high in both storage and computation due to the huge amount of data items. On the contrary, by representing every data item with a compact binary code that preserves the similarity information of items, the hashing technique can significantly reduce the memory footprint and increase the search efficiency by working in the binary Hamming space, and thus has attracted significant attention in recent years.

Existing hashing methods can be roughly categorized into supervised and unsupervised groups. Although supervised hashing generally performs better due to the availability of label information during training, unsupervised hashing is more favourable in practice. Currently, many of competitive unsupervised hashing methods are established based on the goal of satisfying data reconstruction. For instances, in \cite{do2016learning}, binary codes are found by solving a discrete optimization problem on the error between the reconstructed and original images. Later, to reduce the computational complexity, variational auto-encoders (VAE) are trained to directly output hashing codes \cite{DaiGKHS17,ShenLS19}, where the decoder is enforced to reconstruct the original images from the binary codes. Recently, the encoder-decoder structure is used in conjunction with the graph neural network to better exploit the similarity information in the original data \cite{ShenQCY00S020}. A similar idea is also explored under the generative adversarial network in \cite{SongHGXHS18}. It can be seen that what these methods really do is to force the codes to retain as much information of original data as possible through the reconstruction constraint. However, in the task of hashing, the objective is not to retain all information of the original data, but to preserve the distinctive similarity information. For example, the background of an image is known to be unuseful for similarity search. But if the reconstruction criterion is employed, the model may spend lots of effort on reconstructing the background, while ignoring the preservation of more meaningful similarity information.

Recently, a non-reconstruction-based unsupervised representation learning framework ({\textit{i.e.}}, contrastive learning) is proposed \cite{ChenK0H20}, which learns real-valued representations by maximizing the agreement between different views of the same image. It has been shown that the method is able to produce semantic representations that are comparable to those obtained under supervised paradigms. Inspired by the tremendous success of contrastive learning, in this paper, we propose to learn the binary codes under the non-reconstruction-based contrastive learning framework. However, due to the continuous characteristic of learned representations and the mismatch of objectives, the standard contrastive learning does not work at its best on the task of hashing. Therefore, we propose to adapt the framework by modifying its original structure and introducing a probabilistic binary representation layer into the model. In this way, we can not only bridge objective mismatching, but also train the binary discrete model in an end-to-end manner, with the help from recent advances on gradient estimators for functions involving discrete random variables \cite{BengioLC13,MaddisonMT17}. Furthermore, we establish a connection between the proposed probabilistic hashing method and mutual information, and show that the proposed contrastive-learning-based hashing method can be considered under the broader information bottleneck (IB) principle \cite{tishby2000information}. Under this perspective, a more general probabilistic hashing model is obtained, which not only minimizes the contrastive loss but also seeks to reduce the mutual information between the codes and original input data. Extensive experiments are conducted on three benchmark image datasets to evaluate the performance of the proposed hashing methods. The experimental results demonstrate that the contrastive learning and information bottleneck can both contribute significantly to the improvement of retrieval performance, outperforming the state of the art (SOTA) by significant margins on all three considered datasets.

\section{Related Work}

\paragraph{Unsupervised hashing}
Many of unsupervised hashing methods are established on the generative architectures. Roughly, they can be divided into two categories. On the one hand, some of them adopt the encoder-decoder architecture \cite{DaiGKHS17,zheng2020generative,dong2019document,ShenQCY00S020,ShenLS19} and seek to reconstruct the original images or texts. For example, SGH \cite{DaiGKHS17} proposes a novel generative approach to learn stochastic binary hashing codes through the minimum description length principle. TBH \cite{ShenQCY00S020} puts forward a variant of Wasserstein Auto-encoder with the code-driven adjacency graph to guide the image reconstruction process. On the other hand, some models employ generative adversarial nets to implicitly maximize reconstruction likelihood through the discriminator \cite{SongHGXHS18,DizajiZSYDH18,ZiebaSET18}. However, by reconstructing images, these methods could introduce overload information into the hashing code and therefore hurt the model's performance. In addition to the reconstruction-based methods, there are also some hashing models that construct the semantic structure based on the similarity graph constructed in the original high-dimensional feature space. For instance, DistillHash \cite{YangLDLT19} addresses the absence of supervisory signals by distilling data pairs with confident semantic similarity relationships. SSDH \cite{YangDLLT18} constructs the semantic structure through two half Gaussian distributions. Among these hashing methods, DeepBit \cite{LinLCZ16} and UTH \cite{HuangXZW17} are somewhat similar to our proposed model, both of which partially consider different views of images to optimize the hashing codes. However, both methods lack a carefully and theoretically designed framework, leading to poor performance.

\paragraph{Contrastive Learning}

Recently, contrastive learning has gained great success in unsupervised representation learning domains. SimCLR \cite{ChenK0H20} proposes a simple self-supervised learning network without requiring specialized architectures or a memory bank, but still achieves excellent performance on ImageNet. MoCo \cite{He0WXG20} builds a dynamic and consistent dictionary preserving the candidate keys to enlarge the size of negative samples. BYOL \cite{GrillSATRBDPGAP20} abandons negative pairs by introducing the online and target networks.

\paragraph{Information Bottleneck}
The original information bottleneck (IB) work \cite{tishby2000information} provides a novel principle for representation learning, claiming that a good representation should retain useful information to predict the labels while eliminating superfluous information about the original sample. Recently, in \cite{AlemiFD017}, a variational approximation to the IB method is proposed. \cite{Federici0FKA20} extends the IB method to the representation learning under the multi-view unsupervised setting.

\section{Hashing via Contrastive Learning}

In this section, we will first give a brief introduction to the contrastive learning, and then present how to adapt it to the task of hashing.

\subsection{Contrastive Learning}
Given a minibatch of images $x^{(k)}$ for $i=1,2, \cdots, N$, the contrastive learning first transforms each image into two views $v_1^{(k)}$ and $v_2^{(k)}$ by applying a combination of different transformations (\textit{e.g.}, cropping, color distortion, rotation, etc) to the image $x^{(k)}$. After that, the views are fed into an encoder network $f_\theta(\cdot)$ to produce continuous representations
\begin{equation}
z_i^{(k)} = f_\theta(v_i^{(k)}),   \quad i=1 \;\text{or} \;2.
\end{equation}
Since $z_1^{(k)}$ and $z_2^{(k)}$ are derived from different views of the same image $x^{(k)}$, they should mostly contain the same semantic information. The contrastive learning framework achieves this goal by first projecting $z_i^{(k)}$ into a new latent space with a projection head
\begin{equation}
h_i^{(k)} = g_\phi(z_i^{(k)})
\end{equation}
and then minimizing the contrastive loss\footnote{In this paper, we adopt NT-Xent Loss \cite{ChenK0H20} as the contrastive loss function.} on the projected vectors $h_i^{(k)}$ as
\begin{equation}
L_{cl} = \frac{1}{N} \sum_{k=1}^N \left(\ell_1^{(k)} + \ell_2^{(k)}\right),
\end{equation}
where
\begin{equation}
\ell_1^{(k)} \! \triangleq -\log \frac{e^{{sim(h_1^{(k)},h_2^{(k)})}/{\tau}}}
{e^{{sim(h_1^{(k)},h_2^{(k)})}/{\tau}}   \! + \! \sum\limits_{i, n \neq k}e^{{sim(h_1^{(k)},h_i^{(n)})}/{\tau}}},
\label{fml:singleloss}
\end{equation}
and $sim(h_1, h_2) \triangleq  \frac{h_1^Th_2}{{\left\|{h_1}\right\| \left\|h_2\right\|}}$ means the cosine similarity; and $\tau$ denotes a temperature parameter controlling the concentration level of the distribution \cite{HintonVD15}; and $\ell^{(k)}_2$ can be defined similarly by concentrating on $h_2^{(k)}$. In practice, the projection head $g_\phi(\cdot)$ is constituted by an one-layer neural network. After training, for a given image $x^{(k)}$, we can obtain its representation by feeding it into the encoder network
\begin{equation}
r^{(k)} = f_\theta(x^{(k)}).
\end{equation}
Note that $r^{(k)}$ is different from $z^{(k)}_1$ and $z^{(k)}_2$ since $r^{(k)}$ is extracted from the original image $x^{(k)}$ rather than its views. The representations are then used for downstream applications.

\subsection{Adapting Contrastive Learning to Hashing}
\label{AdaptingCL2Hash}

To obtain the binary hashing codes, the simplest way is to set a threshold on every dimension of the latent representation to binarize the continuous representation like
\begin{equation}
[b^{(k)}]_d =
\begin{cases}
0,&  [r^{(k)}]_d < [c]_d\\
1,&  [r^{(k)}]_d \geq [c]_d
\end{cases}
\end{equation}
where $[\cdot]_d$ denotes the $d$-th element of a vector; and $c$ is the threshold vector. There are many different ways to decide the threshold values. For example, we can set $[c]_d$ to be $0$ or the median value of all representations on the $d$-th dimension to balance the number of $0$'s and $1$'s \cite{BalujaC08}. Although it is simple, two issues hinder this method from releasing its greatest potential. {1) \emph{Objective Mismatch}}: The primary goal of contrastive learning is to extract representations for downstream {\emph{discriminative}} tasks, like classification or classification with very few labels. However, the goal of hashing is to preserve the semantic similarity information in the binary representations so that similar items can be retrieved quickly simply according to their Hamming distances. {2) \emph{Separate Training}}: To obtain the binary codes from the representations of original contrastive learning, an extra step is required to binarize the real-valued representations. Due to the non-differentiability of binarization, the operation can not be incorporated for joint training of the entire model.

For the objective mismatch issue, by noticing that the contrastive loss is actually defined on the similarity metric, we can simply drop the projection head and apply the contrastive loss on the binary representations directly. For the issue of joint training, we follow the approach in \cite{shen2018nash} by introducing a probabilistic binary representation layer into the model. Specifically, given a view $v_i^{(k)}$ from the $k$-th image $x^{(k)}$, we first compute the probability
\begin{equation}
p_i^{(k)}  = \sigma(z_i^{(k)}),
\end{equation}
where $z_i^{(k)} = f_\theta(v_i^{(k)})$ is the continuous representation; and $\sigma$ denotes the sigmoid function and is applied to its argument in an element-wise way. Then, the binary codes are generated by sampling from the multivariate Bernoulli distribution as
 \begin{equation}
b^{(k)}_i \sim Bernoulli(p_i^{(k)}),
\end{equation}
where the $d$-th element $[b^{(k)}_i]_d$ is generated according to the corresponding probability $[p^{(k)}_i]_d$. Since the binary representations $b_i^{(k)}$ are probabilistic, to have them preserve as much similarity information as possible, we can minimize the expected contrastive loss
\begin{equation}
\bar L_{cl} = \frac{1}{N} \sum_{k=1}^N\left(\bar \ell_1^{(k)} + \bar \ell_2^{(k)} \right),
\end{equation}
where
\begin{equation} \label{contrastive_loss_binary}
\bar \ell_{1}^{(k)} \!=\! - {\mathbb{E}} \!\!\left[\!\log \! \frac{e^{{sim(b_1^{(k)},b_2^{(k)})}/{\tau}}}
{e^{{sim(b_1^{(k)}, b_2^{(k)})}/{\tau}}   \!\! + \!\!\!\! \sum\limits_{i, n \neq k} \!\! e^{{sim(b_1^{(k)}, b_i^{(n)})}/{\tau}}}\right];
\end{equation}
$b_i^{(k)} \sim Bernoulli\left(\sigma(f_\theta(v_i^{(k)}))\right)$ and the expectation ${\mathbb{E}} [\cdot]$ is taken w.r.t. all $b_i^{(k)}$ for $i=1,2$ and $k=1,2, \cdots, N$; and $\bar \ell_{2}^{(k)}$ can be defined similarly. Note that the contrastive loss is applied on the binary codes $b^{(k)}_i$ directly. This is because the similarity-based contrastive loss without projection head is more consistent with the objective of the hashing task.

The problem now reduces to how to minimize the loss function $\bar L_{cl}$ {\textit{w.r.t.}} the model parameters $\theta$, the key of which lies in how to efficiently compute the gradients $\frac{\partial \bar \ell_1^{(k)}}{\partial \theta}$ and $\frac{\partial \bar \ell_2^{(k)}}{\partial \theta}$. Recently, many efficient methods have been proposed to estimate the gradient of neural networks involving discrete stochastic variables \cite{BengioLC13,MaddisonMT17}. In this paper, we employ the simplest one, the straight-through (ST) gradient estimator \cite{BengioLC13}, and leave the use of more advanced estimators for future exploration. Specifically, the ST estimator first reparameterizes $b_i^{(k)}$ as
\begin{align}
\widetilde b_i^{(k)} = \frac{\mathrm{sign}(\sigma(f_\theta(v_i^{(k)})) - u) + 1}{2},
\end{align}
and use the reparameterized $\widetilde b_i^{(k)}$ to replace $b_i^{(k)}$ in \eqref{contrastive_loss_binary} to obtain an approximate loss expression, where $u$ denotes a sample from the uniform distribution $[0, 1]$. The gradient, as proposed in the ST estimator, can then be estimated by applying the backpropagation algorithm on the approximate loss. In this way, the entire model can be trained efficiently in an end-to-end manner. When the model is used to produce binary code for an image $x$ at the testing stage, since we want every image to be deterministically associated with a hashing code, we drop the effect of randomness in the training stage and produce the code by simply testing whether the probability $\sigma\left(f_\theta(x)\right)$ is larger than 0.5 or not.

\section{Improving under the IB Framework}
In this section, we first establish the connection between the probabilistic hashing model proposed above and the mutual information, and then reformulate the learning to hashing problem under the broader IB framework.

\subsection{Connections to Mutual Information}
For the convenience of presentation, we re-organize the minibatch of views $\{v_1^{(k)}, v_2^{(k)}\}_{k=1}^N$ into the form of $\{v_i\}_{i=1}^{2N}$. We randomly take one view  ({\textit{e.g.}}, $v_k$) as the target view for consideration. For the rest of $2N-1$ views $\{v_i\}_{i\ne k}$, we assign each of them a unique label from $\{1, 2, \cdots, 2N-1\}$ according to some rules. The target view $v_k$ is assigned the label same as the view derived from the same image. Without loss of generality, we denote the label as $y_k$. Now, we want to train a classifier to predict the label of the target view $v_k$. But instead of using the prevalent softmax-based classifiers, we require the classifier to be memory-based. Specifically, we extract a stochastic feature  $b_i \sim Bernoulli\left(\sigma(f_\theta(v_i))\right)$ for each view in the minibatch and predict the probability that $v_k$ belongs to label $y_k$ as
\begin{equation}
   p(y_k|v_k, {\mathcal{V}}) = \frac{e^{{sim(b_k, {\mathcal{B}}{\backslash b_k} (y_k))}/{\tau}}}
   {  \sum_{{{c}}\in {\mathcal{B}} \backslash b_k} e^{{sim(b_k, c)}/{\tau}}},
\end{equation}
where ${\mathcal{V}} \triangleq \{v_1, v_2, \cdots, v_{2N}\}$ is the set of views in the considered minibatch; and ${\mathcal{B}} \triangleq \{b_1, b_2, \cdots, b_{2N}\}$ is the set of stochastic features derived in the minibatch, while ${\mathcal{B}} \backslash b_k$ means the set that excludes the element $b_k$ from ${\mathcal{B}}$; and ${\mathcal{B}}{\backslash b_k} (y)$ denotes the feature that is associated with the label $y$ in the set ${\mathcal{B}} \backslash b_k$ .
Since the features $b_i\in {\mathcal{B}}$ are stochastic, we take expectation over the log-probability above and obtain the loss {\textit{w.r.t.}} the view-label pair $(v_k, y_k)$ under the considered minibatch ${\mathcal{V}}$ as
\begin{equation} \label{cross-entropy}
\ell_{ce}(v_k, y_k) \!=\! -{\mathbb{E}}_{p({\mathcal{B}}|{\mathcal{V}})} \! \left[\log \frac{e^{{sim(b_k, {\mathcal{B}}{\backslash b_k} (y_k))}/{\tau}}}
{  \sum_{{{c}}\in {\mathcal{B}} \backslash b_k} e^{{sim(b_k, c)}/{\tau}}} \right],
\end{equation}
where $p({\mathcal{B}}| {\mathcal{V}}) = \prod_{i=1}^{2N} {\mathbb{P}}(b_i|v_i)$ with
\begin{equation}
{\mathbb{P}} (b|v) \triangleq Bernoulli\left(\sigma(f_\theta(v))\right).
\end{equation}
By comparing \eqref{cross-entropy} to \eqref{contrastive_loss_binary}, we can see that the two losses are actually the same. Therefore, training the proposed probabilistic hashing model is equivalent to minimizing the cross-entropy loss of the proposed memory-based classifier.

The loss function in \eqref{cross-entropy} is only responsible for the $k$-th view under one minibatch. For the training, the loss should be optimized over a lot of view-label pairs from different minibatches. Without loss of generality, we denote the distribution followed by view-label pairs as ${{\mathbb{P}}(v, y)}$. Then, we can express the loss averaged over all view-label pairs as
\begin{equation}
L_{ce} = - {\mathbb{E}}_{{\mathbb{P}}(v, y)} {\mathbb{E}}_{p({\mathcal{B}}| {\mathcal{V}} )}\left[\log q(y|b)\right],
\end{equation}
where the distribution $ q(y|b)$ is defined as
\begin{equation}
  q(y|b) = \frac{e^{{sim(b, {\mathcal{B}}{\backslash b} (y))}/{\tau}}}
  {  \sum_{{{c}}\in {\mathcal{B}} \backslash b} e^{{sim(b, c)}/{\tau}}};
\end{equation}
and ${\mathcal{V}}$ represents the minibatch of views that $(v, y)$ is associated with. Without affecting the final conclusions, for the clarity of analysis, we only consider the randomness in $b \sim {\mathbb{P}}(b|v)$, while ignoring the randomness in ${\mathcal{B}}\backslash b$. Under this convention, the loss $L_{ce}$ can be written as
\begin{align} \label{Lce_lower_bound}
L_{ce} &= -\int {{\mathbb{P}}(b, y) \log q(y|b) dy db},
\end{align}
where ${\mathbb{P}}(b, y) = \int{{\mathbb{P}}(v, y){\mathbb{P}}(b|v)dv}$. From the inequality $\int {{\mathbb{P}}(b, y) \log {\mathbb{P}}(y|b) dy db} \ge \int {{\mathbb{P}}(b, y) \log q(y|b) dy db} $, which can be easily derived from the non-negativeness of KL-divergence, it can be seen that
\begin{align}
 L_{ce} \ge  -\int {{\mathbb{P}}(b, y) \log {\mathbb{P}}(y|b) dy db}  = H(Y|B),
\end{align}
where ${\mathbb{P}}(y|b)$ denotes the conditional distribution from the joint pdf ${\mathbb{P}}(b, y)$; $B$ and $Y$ denotes the random variables of binary representations and labels, whose marginal distributions are ${\mathbb{P}}(b)$ and ${\mathbb{P}}(y)$, respectively. Then, subtracting entropy $H(Y)$ on both sides gives
\begin{equation}
L_{ce} - H(Y) \ge - I(Y, B),
\end{equation}
in which the equality $I(Y, B)=H(Y) - H(Y|B)$ is used. Therefore, $L_{ce} - H(Y)$ is actually an upper bound of the negative mutual information $- I(Y, B)$. Because the entropy of labels $H(Y)$ is a constant, minimizing the loss function $L_{ce}$ is equivalent to minimizing the upper bound of negative mutual information. Thus, we can conclude that the proposed model essentially maximizes the mutual information, {\textit{i.e.}},
\begin{equation}
\max_\theta  I(Y, B),
\end{equation}
between binary representations $B$ and labels $Y$ under the joint probabilistic model ${\mathbb{P}}(v, y, b) = {\mathbb{P}}(v, y) {\mathbb{P}}(b|v)$. Here ${\mathbb{P}}(v, y)$ is the distribution of training data that is unchangeable, while ${\mathbb{P}}(b|v)$ is the distribution that could be optimized.

\subsection{Learning under the IB Framework}

\begin{figure}
	\begin{minipage}[]{0.48\textwidth}
		\includegraphics[width=1.0\textwidth]{process.pdf}
	\end{minipage}
	\caption{Illustration of the training procedures of CIBHash. An encoder $f(\cdot)$ is trained via maximizing distinctive semantic information, \textit{i.e.}, $Max:I(Y, B)$; and simultaneously minimizing superfluous information, \textit{i.e.}, $Min:I(B, V)$. }
	\vspace{-3mm}
	\label{fig:workflow}
\end{figure}

Information bottleneck (IB) is to maximize the mutual information between the representations and output labels, subjecting to some constraint \cite{tishby2000information}. That is, it seeks to maximize the objective function
\begin{equation} \label{IB_objective}
R_{IB} = I(Y, B) - \beta I(B, V),
\end{equation}
where $\beta$ is a Lagrange multiplier that controls the trade-off between the two types of mutual information; and $V$ denotes the random variable of inputs. Obviously, from the perspective of maximizing mutual information, the proposed probabilistic hashing method can be understood under the IB framework by setting the parameter $\beta$ to $0$. It is widely reported that the parameter $\beta$ can control the amount of information that is dropped from the raw inputs $v$, and if an appropriate value is selected, better semantic representations can be obtained \cite{AlemiFD017}. Therefore, we can train the proposed hashing model under the broader IB framework by taking the term $ I(B, V)$ into account.

Instead of directly maximizing $R_{IB}$, we seek to maximize its lower bound due to the computational intractability of $R_{IB}$. For the second term in \eqref{IB_objective}, by definition, we have
$
 I(B, V) = {\mathbb{E}}_{{\mathbb{P}}(v)}\left[KL\left({\mathbb{P}}(b|v) || {\mathbb{P}}(b)\right)\right].
$
From the non-negativeness of KL-divergence, it can be easily shown that
\begin{align}
I(B, V) \le {\mathbb{E}}_{{\mathbb{P}}(v)}\left[KL\left({\mathbb{P}}(b|v) || q(b)\right)\right],
\end{align}
where $q(b)$ could be any distribution of $b$. From \eqref{Lce_lower_bound}, we can also get the lower bound for $I(Y, B)$ as  $I(Y, B) \ge -L_{ce} + H(Y)$. Thus, we can obtain a lower bound for $R_{IB}$ as
\begin{equation}
 R_{IB} \ge -L_{ce} -\beta {\mathbb{E}}_{{\mathbb{P}}(v)}\left[KL\left({\mathbb{P}}(b|v) || q(b)\right)\right] + H(Y).
\end{equation}
Since $H(Y)$ is a constant and $L_{ce}$ is exactly the contrastive loss, to optimize the lower bound, we just need to derive the expression for the KL term. By assuming $q(b)$ follows a multivariate Bernoulli distribution $q(b)=Bernoulli(\gamma)$, the expression of KL-divergence $KL\left({\mathbb{P}}(b|v) || q(b)\right)$ can be easily derived to be
\begin{equation}
\resizebox{.89\linewidth}{!}{$
    \begin{aligned}
    KL({\mathbb{P}}(b|v)||q(b)) & = \sum^D_{d=1}  [\sigma(f_\theta(v))]_d  \log \frac{[\sigma(f_\theta(v))]_d}{[\gamma]_d} \\
    & + \sum^D_{i=1}  (1-[\sigma(f_\theta(v))]_d)  \log \frac{1-[\sigma(f_\theta(v))]_d}{1-[\gamma]_d}.
    \label{kl}
    \end{aligned}
    $}
\end{equation}
In our experiments, for a given view $v_1^{(k)}$, the value of $\gamma$ is set by letting $q(b) = p(b|v_2^{(k)})$; and $q(b)$ for the view $v_2^{(k)}$ can be defined similarly. Intuitively, by encouraging the encoding distributions from different views of the same image close to each other, the model can eliminate superfluous information from each view. In this way,  the lower bound of $R_{IB}$ can be maximized by SGD algorithms efficiently. We name the proposed model as CIBHash, standing for \textbf{Hash}ing with \textbf{C}ontrastive \textbf{I}nformation \textbf{B}ottleneck. The architecture is shown in Figure \ref{fig:workflow}.

\section{Experiments}
\subsection{Datasets, Evaluation and Baselines}
\paragraph{Datasets} Three datasets are used to evaluate the performance of the proposed hashing method. \emph{1) CIFAR-10}  is a dataset consisting of $60,000$ images from $10$ classes \cite{Krizhevsky2009Learning}. We randomly select $1,000$ images per class as the query set and $500$ images per class as the training set, and all the remaining images except queries are used as the database. \emph{2) NUS-WIDE}  is a multi-label dataset containing $269,648$ images from $81$ categories \cite{ChuaTHLLZ09}. Following the commonly used setting, the subset with images from the $21$ most frequent categories is used. We select 100 images per class as the query set and use the remaining as the database. Moreover, we uniformly select $500$ images per class from the database for training, as done in \cite{ShenQCY00S020}. \emph{3) MSCOCO} is a large-scale dataset for object detection, segmentation and captioning \cite{lin2014microsoft}. Same as the previous works, a subset of $122,218$ images from $80$ categories is considered. We randomly select $5,000$ images from the subset as the query set and use the remaining images as the database. $10,000$ images from the database are randomly selected for training.

\paragraph{Evaluation Metric} In our experiments, the mean average precision (MAP) at top $N$ is used to measure the quality of obtained hashing codes. Following the settings in \cite{CaoLWY17,ShenQCY00S020}, we adopt MAP@1000 for CIFAR-10, MAP@5000 for NUS-WIDE and MSCOCO.

\paragraph{Baselines} In this work, we consider the following unsupervised deep hashing methods for comparison: DeepBit \cite{LinLCZ16}, SGH \cite{DaiGKHS17}, BGAN \cite{SongHGXHS18}, BinGAN \cite{ZiebaSET18}, GreedyHash \cite{SuZHT18}, HashGAN \cite{DizajiZSYDH18}, DVB \cite{ShenLS19}, and TBH \cite{ShenQCY00S020}. For the reported performance of baselines, they are quoted from TBH \cite{ShenQCY00S020}.

\subsection{Training Details}
For images from the three datasets, they are all resized to $224 \times 224 \times 3$. Same as the original contrastive learning \cite{ChenK0H20}, during training the resized images are transformed into different views with transformations like random cropping, random color distortions and Gaussian blur, and then are input into the encoder network. In our experiments, the encoder network $f_\theta(\cdot)$ is constituted by a pre-trained VGG-16 network \cite{SimonyanZ14a} followed by an one-layer ReLU feedforward neural network with 1024 hidden units. During the training, following previous works \cite{SuZHT18,ShenLS19}, we fix the parameters of pre-trained VGG-16 network, while only training the newly added feedfoward neural network. We implement our model on PyTorch and employ the optimizer Adam for optimization, in which the default parameters are used and the learning rate is set to be $0.001$. The temperature $\tau$ is set to $0.3$, and $\beta$ is set to $0.001$.

\begin{table*}[!t]
\caption{MAP comparison with different state-of-the-art unsupervised hashing methods.}
\label{table:map}
\centering
\begin{tabular}{c|c|ccc|ccc|ccc}
\hline
\multirow{2}*{\textbf{Method}}  & \multirow{2}*{\textbf{Reference}}   & \multicolumn{3}{c}{\textbf{CIFAR-10}} & \multicolumn{3}{|c}{\textbf{NUS-WIDE}} & \multicolumn{3}{|c}{\textbf{MSCOCO}}\\

~          & ~         & 16bits & 32bits & 64bits & 16bits & 32bits & 64bits & 16bits & 32bits & 64bits  \\
\hline
\hline
DeepBit    & CVPR16    & 0.194  & 0.249  & 0.277  & 0.392  & 0.403  & 0.429  & 0.407  & 0.419  & 0.430 \\
SGH        & ICML17    & 0.435  & 0.437  & 0.433  & 0.593  & 0.590  & 0.607  & 0.594  & 0.610  & 0.618 \\
BGAN       & AAAI18    & 0.525  & 0.531  & 0.562  & 0.684  & 0.714  & 0.730  & 0.645  & 0.682  & 0.707 \\
BinGAN     & NIPS18    & 0.476  & 0.512  & 0.520  & 0.654  & 0.709  & 0.713  & 0.651  & 0.673  & 0.696 \\
GreedyHash & NIPS18    & 0.448  & 0.473  & 0.501  & 0.633  & 0.691  & 0.731  & 0.582  & 0.668  & 0.710 \\
HashGAN    & CVPR18    & 0.447  & 0.463  & 0.481  & -      & -      & -      & -      & -      & -     \\
DVB        & IJCV19    & 0.403  & 0.422  & 0.446  & 0.604  & 0.632  & 0.665  & 0.570  & 0.629  & 0.623 \\
TBH        & CVPR20    & 0.532  & 0.573  & 0.578  & 0.717  & 0.725  & 0.735  & 0.706  & 0.735  & 0.722 \\
\hline

\textbf{CIBHash}        & \textbf{Ours}      & \textbf{0.590}  & \textbf{0.622}  & \textbf{0.641} & \textbf{0.790}  & \textbf{0.807}  & \textbf{0.815}  & \textbf{0.737}  & \textbf{0.760}  & \textbf{0.775}  \\
\hline
\end{tabular}
\vspace{-2mm}
\end{table*}

\begin{table}[!t]
\caption{MAP comparison with variants of CIBHash.}
\centering
\begin{tabular}{c|c|cccc}
\hline
\multicolumn{2}{c|}{\textbf{Component Analysis}}&  16bits  &  32bits  &  64bits \\
\hline
\hline
\multirow{3}*{CIFAR-10}    &  Naive CL  & 0.493    & 0.574    & 0.606  \\
~                          &  CLHash  &  0.584   &  0.612   &  0.633  \\
~                          &  \textbf{CIBHash}    &  \textbf{0.590}   &  \textbf{0.622}   &  \textbf{0.641}  \\
\hline
\hline

\multirow{3}*{MSCOCO}      &  Naive CL  & 0.666    & 0.712    & 0.737  \\
~                          &  CLHash &  0.725   &  0.753   &  0.765  \\
~                          &  \textbf{CIBHash}    &  \textbf{0.737}   &  \textbf{0.760}   &  \textbf{0.775}  \\
\hline
\end{tabular}
\label{table:component}
\vspace{-2mm}
\end{table}

\subsection{Results and Analysis}

\paragraph{Overall Performance} Table \ref{table:map} presents the performances of our proposed model CIBHash on three public datasets with code lengths varying from $16$ to $64$. For comparison, the performance of baseline models is also included. From the table, it can be seen that the proposed CIBHash model outperforms the current SOTA method by a substantial margin on all three datasets considered. Specifically, an averaged improvement of $5.7\%$, $7.8\%$, $3.6\%$ (averaged over different code lengths) on CIFAR-10, NUS-WIDE, and MSCOCO datasets are observed, respectively, when it is compared to the currently best performed method TBH. The gains are even more obvious on the short code case. All of these reveal that the non-reconstruction-based hashing method is really better at extracting important semantic information than the reconstruction-based ones. This also reveals that when contrastive learning is placed under the information bottleneck framework, high-quality hashing codes can be learned. For more comparisons and results, please refer to Supplementary Materials.

\paragraph{Component Analysis}
To understand the influence of different components in CIBHash, we further evaluate the performance of two variants of our model. (\textit{i}) \textbf{Naive CL}: It produces hashing codes by directly binarizing the real-valued representations learned under the original contrastive learning framework using the median value as the threshold. (\textit{ii}) \textbf{CLHash}: CLHash denotes the end-to-end probabilistic hashing model derived from contrastive learning, as proposed in Section \ref{AdaptingCL2Hash}. As seen from Table \ref{table:component}, compared to Naive CL, CLHash improves the averaged performance by $5.2\%$ ,$4.3\%$ on CIFAR-10 and MSCOCO, respectively, which demonstrates the effectiveness of our proposed adaptations on the original contrastive learning, {\textit{i.e.}}, dropping the projection head and enabling end-to-end training. If we compare CIBHash to CLHash, improvements of $0.8\%$ and $1.0\%$ can be further observed on CIFAR-10 and MSCOCO, respectively, which fully corroborates the advantages of considering the CLHash under the broader IB framework.

\begin{table}[!t]
\centering
\caption{MAP comparison with different IB-based methods.}
\label{table:bvae}

\begin{tabular}{l|ccc}
\hline
\multirow{2}*{\textbf{Method}}& \multicolumn{3}{c}{\textbf{CIFAR-10}} \\

~      & 16bits & 32bits & 64bits \\
\hline
$\beta$-VAE    & 0.468  & 0.508  & 0.495  \\
Multi-View $\beta$-VAE &  0.465  & 0.492   &  0.522   \\
\textbf{CIBHash}          &  \textbf{0.590}   & \textbf{0.622}    &  \textbf{0.641}  \\
\hline
\end{tabular}
\vspace{-1mm}
\end{table}

\begin{figure}[!t]
    \centering
	\subfigure{
		\begin{minipage}[b]{0.47\columnwidth}
  	 	\includegraphics[width = 1\columnwidth]{diff_weight_cifar64.pdf}
		\end{minipage}
	}
	\subfigure{
		\begin{minipage}[b]{0.47\columnwidth}
  	 	\includegraphics[width = 1\columnwidth]{diff_batchsize_cifar64.pdf}
		\end{minipage}
	}
	\vspace{-1mm}
	\caption{Parameter analysis for the Lagrange multiplier $\beta$ and the batch size with 64-bit hashing codes on CIFAR-10.}
	\label{fig: hypersensitivity}
	\vspace{-0.5mm}
\end{figure}

\paragraph{Parameter Analysis} To see how the key hyperparameters $\beta$ and minibatch size influence the performance, we evaluate the model under different $\beta$ values and minibatch sizes. As shown in the left column of Figure \ref{fig: hypersensitivity}, the parameter $\beta$ plays an important role in obtaining good performance. If it is set too small or too large, the best performance cannot be obtained under either case. Then, due to the observed gains of using large batch sizes in the original contrastive learning, we also study the effect of batch sizes in our proposed CIBHash model. The results are presented in the right column of Figure \ref{fig: hypersensitivity}. We see that as the batch size increases, the performance rises steadily at first and then converges to some certain level when the batch size is larger than $64$.

\paragraph{Comparison with $\beta$-VAE}

$\beta$-VAE can be regarded as an unsupervised representation learning method under the IB framework \cite{AlemiFD017}, where $\beta$ controls the relative importance of reconstruction error and data compression. The main difference between $\beta$-VAE and our model CIBHash is that $\beta$-VAE relies on reconstruction to learn representations, while our model leverages contrastive learning that maximizes the agreement between different views of an image. We evaluate the $\beta$-VAE and multi-view $\beta$-VAE. Table \ref{table:bvae} shows that CIBHash dramatically outperforms both methods. This proves that the non-reconstruction-based method is better at extracting semantic information than the reconstruction-based methods again.

\section{Conclusion}
In this paper, we proposed a novel non-reconstruction-based unsupervised hashing method, namely CIBHash. In CIBHash, we attempted to adapt the contrastive learning to the task of hashing, by which its original structure was modified and a probabilistic Bernoulli representation layer was introduced to the model, thus enabling the end-to-end training.  By viewing the proposed model under the broader IB framework, a more general hashing method is obtained. Extensive experiments have shown that CIBHash significantly outperformed existing unsupervised hashing methods.

\section*{Acknowledgments}
This work is supported by the National Natural Science Foundation of China (No. 61806223, 61906217, U1811264), Key R\&D Program of Guangdong Province (No. 2018B010107005), National Natural Science Foundation of Guangdong Province (No. 2021A1515012299). This work is also supported by Huawei MindSpore.
\newpage
\bibliographystyle{named}


\section{Introduction}
\label{in}

Recognizing objects in the real world in the same way as human beings is an important goal for artificial intelligence research [1]. At present, the computer vision object recognition research in the field of artificial intelligence is inspired by human vision system research results. Of these, various convolutional neural networks (CNN) [2-6] are the most advanced and widely used. CNN is a feature extraction and object recognition algorithm [7-10] which includes hierarchical structure, nonlinear activation, maximum pooling, and other operations. In addition, CNN is inspired by how the visual input of mammals can be visually recognized through hierarchical filtering and pooling operations of simple and complex cells in the V1 region [11]. Although CNN is inspired by the human visual neural network, the current CNN artificial neural network is a black box model and lacks interpretability.

How does the CNN neural network identify cats? One widely accepted answer is that it does so by examining the cat’s shape, and evidence for this hypothesis comes from visualization techniques such as DeconvNet [12]. However, some of the most important and widely used visualization techniques, such as DeconvNet, have recently been proven to be misleading [13]. Geirhos et al. found that for the deep neural network, a cat with an elephant’s skin is an elephant, but for humans, it is still a cat. Current deep learning techniques for object recognition, such as AlexNet, VGG-16, Googlenet, ResNet-50, ResNet-152, Densenet-121, and Squeezenet1_1, etc., mainly rely on the texture of objects rather than their shapes [14]. They also found that deep neural networks follow a shortcut learning strategy, that is, instead of correctly identifying and classifying foreground objects with labels, they learn to classify objects using a background, texture, or other such shortcuts (which are not obvious to humans) of foreground objects. This shortcut learning strategy makes the generalization ability of deep neural networks worse, lacks domain generality, and makes the recognition mechanism of neural networks incomprehensible [15]. Therefore, CNNs (model-based object recognition methods that only use feature information of an object’s image) face two problems. Firstly, they are unable to distinguish between objects whose features are highly similar, such as a real object, a model of an object, and a photograph of an object; secondly, they are unable to accurately recognize a target object when it is disturbed by a background or blocked by other objects. For example, a car with a clock texture is recognized as a clock, and a bear with a water bottle surface texture is recognized as a water bottle [14].

To solve the problems faced by CNN, some researchers have begun to recognize objects by considering the context of the objects in addition to the objects’ features. The difficulty of object recognition based on an object’s context is how to build a relationship between an object and its context. For example, desks and chairs often appear at the same time, but elephants and beds almost never appear at the same time; a plate is most often on a table, but is also likely to appear in other places. The relationships between objects and their contexts have strong and weak points which are obtained statistically from low-level features in an object’s image [16]. Although the context of an object is helpful for interpreting what it is, it greatly increases the time taken to recognize the object compared with CNN, and fails to solve the two major problems faced by CNN for two reasons. Firstly, contextual-based inference doesn’t work when different categories of objects whose features are highly similar but real sizes are very different appear in the same scenes normally, or appear in infrequent scenes in unusual situations; secondly, for situations whereby a target object has interference such as a background or other objects, contextual inference is even more powerless in most cases [17-20]. In addition, as an object can be photographed at different distances and angles, the object’s size in an image comes out of proportion, which invalidates the context-based computer vision model. Therefore, object recognition based on its context has many limitations, and contextual-based inference cannot solve the two major problems that CNN faces.

How can humans solve them solve those two problems with CNN? Visual commonsense tells us that human beings can use the real sizes of objects (the size of the overall shape or key parts of the object in the real physical world). Firstly, this allows objects with highly similar features to be easily recognized, such as cars, photos of cars, and models of cars; secondly, this allows target objects that are interfered with by the background and other objects to be correctly recognized. For example, accurately identify zebra crossings (target object) that are interfered with by cars, pedestrians, and other objects. At present, the research of human visual neural networks also confirms that the real size of an object is the main basis for choosing the visual cortex regions to analyze the features of an object for human visual object recognition [21-23]. The recognition of objects’ real sizes in human beings is an automatic [24-25] recognition process that first appears in infants, and is mediated by the dorsal visual pathway, ventral visual pathway, parietal cortex region, lateral occipital complex, and other brain regions [26-28]. This process occurs prior to object feature recognition [29-31]. It can be seen that the object recognition of human vision is based on not only an object’s features, but also on the real size of an object. Inspired by this, this article proposes SizeNet which is an object recognition method based on an object’s real size and features. At present, object recognition methods that use the real sizes of objects as an important basis for object recognition have not been reported in the field of computer vision.

\section{SizeNet}

Figure~\ref{fig:fig1} illustrates the end-to-end object recognition network used in SizeNet. The entire network can be split into two parts. The first part recognizes the real size of an object, and filters out the categories that match the real size of an object in the categories label set to generate a new categories label for the downstream object recognition, which based on an object’s features. The other part operates on an object’s features in an object image to propose and classify an object. The category label in the output layer of the object recognition algorithm based on an object’s features, which belongs to the new label set and has the highest matching probability, is output as the recognition result. The label set indicating the real size range of each category is manually calculated and annotated in advance. Due to the fact that the real size of each object in the same category is different, and the real size of the same object under different shooting angles is also different, the real size of each category label is therefore not a number, but a range.

\begin{figure}[t]
	\begin{center}

		\includegraphics[width=0.8\linewidth]{fig1.eps}
		\setlength{\abovecaptionskip}{0pt}
		\setlength{\belowcaptionskip}{0pt}
	\end{center}
	\caption{Object recognition network of SizeNet.}
	\label{fig:fig1}
\end{figure}

\section{Experiments}

In order to verify whether SizeNet can effectively solve the two major problems faced by CNN, this paper created the dataset and designed the object recognition experiments.

\subsection{Dataset}
\label{dt}

The self-made Rsize dataset for images, which annotated the real sizes of the objects, is shown in Table~\ref{table1} and Figure~\ref{fig:fig2}. Table~\ref{table1} shows the label set of the Rsize dataset. Each object category in the label set has a shooting distance range which was manually calculated in advance. For the two problems faced by CNN, Rsize dataset 1 in the Rsize dataset was used for objects with highly similar object features in recognition experiments, and Rsize dataset 2 was used for target objects that were interfered by backgrounds and other objects in recognition experiments.

Figure~\ref{fig:fig2} show some object images in the Rsize dataset. Figure~\ref{fig:fig2a} shows the images in the training set of the Rsize dataset. The image numbers of each object category in the training sets of the Rsize dataset are 350. Figure~\ref{fig:fig2b} shows the images in the test sets of the Rsize dataset. The image numbers of each object category in the test sets of the Rsize dataset are 100. Since SizeNet recognizes objects based on the real size of an object, each image in the test sets were calibrated to the real size of the object, and the real size of the object to be recognized in each image in the test sets was manually calculated in advance and marked in the name of each image. The number between the underscore "_" and ".jpg" in each image name denotes the real size of the object in question. These test sets use the shooting distance of the object (in meters) to indicate the real size of the object, because the real size of the object in the image was indicated by the shooting distance (the distance between the camera and the object to be recognized), and the size of the object in the image. In addition, the size of the object to be recognized in the image was made as large as the image size so that only the converted shooting distance could represent the real size of the object.

\begin{table}[H]
	\setlength{\abovecaptionskip}{10pt}
	\setlength{\belowcaptionskip}{10pt}
	\normalsize
	\caption{The label set in the Rsize dataset, including Rsize dataset 1 and Rsize dataset 2. Rsize dataset 1 contains six categories of objects: Police cars (Pc), Police car models (Pcm), Fire trucks (Ft), Fire truck models (Ftm), Bullet trains (Bt), and Bullet train models (Btm); Rsize dataset 2 contains five categories of objects: Pedestrians (Pe), Cars (C), Crosswalks (Cw), Pillows (P), and Beds (B). Each object category in the label set has a shooting distance range (Sdr).}
	\label{table1}
	\centering
	\begin{tabular}{cccccccccccc}
		\toprule
		\multirow{2}{*}{} & \multicolumn{6}{c}{Rsize dataset 1} & \multicolumn{5}{c}{Rsize dataset 2} \\
		\cmidrule(r){2-7} \cmidrule(r){8-12}
		&  Pc   &  Pcm   &  Ft   &  Ftm   &  Bt   &  Btm
		&  Pe   &  C   &  Cw   &  P   &  B  \\
		\midrule
		Sdrs(m)    & 4-8 & 0.1-1 & 5-12 & 0.2-1 & 30-90 & 0.3-2 & 1-3.1 & 5-8 & 10-20 & 0.2-3 & 1.5-3.5  \\
		\bottomrule
	\end{tabular}
\end{table}

\begin{figure*}[htp]
	\normalsize
	\centering
	\subfigure[The object images in the training sets of the Rsize dataset.]
	{
		\label{fig:fig2a}
		\includegraphics[width=1.0\linewidth]{fig2a.eps}
	}
	\subfigure[The objects images marked with shooting distances in the test sets of the Rsize dataset.]
	{
		\label{fig:fig2b}
		\includegraphics[width=1.0\linewidth]{fig2b.eps}
	}
	\caption{The object images in the Rsize dataset. Figure~\ref{fig:fig2a} shows police car and a police car model in the training set of Rsize dataset 1, and some pedestrians, car, and a crosswalk in the training set of Rsize dataset 2. As the training images were used for training the object recognition algorithm based on object features, each image was not calibrated to the real sizes of the objects. Figure~\ref{fig:fig2b} shows a police car and a police car model marked with shooting distances in the test set of Rsize dataset 1, and a pedestrian, car, and a crosswalk marked with shooting distances in the test set of Rsize dataset 2.}
	\label{fig:fig2}
\end{figure*}

\subsection{Recognition architecture}
\label{ra}

Figure~\ref{fig:fig3} shows the object recognition architecture of SizeNet on the Rsize dataset. Step one, train the Imagenet pre-trained AlexNet which was used as the object recognition algorithm based on object features on training sets, and obtain the trained AlexNet. Step two, recognize the object image using the object’s real size in the test set one by one, and compare the object’s real size in the object image being recognized in the test set with the object real size ranges of all object categories in the object categories label set. This is done by choosing the object categories whose object real size ranges cover the real size of the object image being recognized to generate a new object categories label set. This process is downstream object recognition based on object features. Step three, recognize the object image being recognized using the trained AlexNet, and arrange the matching labels according to the matching probabilities from big to small in the Softmax layer when the input object image is passed to the Softmax layer. Step four, choose the matching labels in the Softmax layer from big to small, one by one, and determine whether the matching label belongs to the new label set or not. Step five, if the matching label does not belong to the new label set, i.e. “No”, then return to step four. Step six, if the matching label belongs to the new label set, i.e. “Yes”, then output the matching label as the object recognition result.

\begin{figure*}[t]
	\begin{center}

		\includegraphics[width=0.9\linewidth]{fig3.eps}
		\setlength{\abovecaptionskip}{0pt}
		\setlength{\belowcaptionskip}{0pt}
	\end{center}
	\caption{The object recognition architecture of SizeNet on the Rsize dataset.}
	\label{fig:fig3}
\end{figure*}

\subsection{Comparison with State-of-the-art Methods}
\label{cm}

In order to compare the object recognition performance of SizeNet, this paper used AlexNet, VGG-16, Inception V3, Resnet-18, and DenseNet-121 pre-trained on the ImageNet dataset (\url{https://download.pytorch.org/models}) to carry out the same object recognition experiments on the Rsize dataset. In order to study the influence of highly similar object features to carry out object recognition based on object features, this paper conducted object recognition experiments using SizeNet, AlexNet, VGG-16, Inception V3, Resnet-18, and DenseNet-121 on three datasets. One dataset includes objects such as police cars, fire trucks, and bullet trains; one includes object models such as police car models, fire truck models, and bullet train models; and the last one is the Rsize dataset 1, which includes both objects and object models such as police cars, fire trucks, bullet trains, police car models, fire truck models, and bullet train models. In order to study the influence of the background and other interfering objects to target object recognition, this paper conducted object recognition experiments on the Rsize dataset 2, whose training set includes the images which contain only the target objects, and a test set that includes the images which contain both the target objects and other interfering objects.

\section{Results}
\raggedbottom
Table~\ref{table2} shows the results of the object recognition experiments when using SizeNet, AlexNet, VGG-16, Inception V3, Resnet-18, and DenseNet-121 on the Rsize dataset 1. The results show that the average validation accuracies (Val acc. (\%)) and average test accuracies (Test acc. (\%)) for object recognition when using SizeNet, AlexNet, VGG-16, Inception V3, Resnet-18, and DenseNet-121 on the objects dataset or the objects models dataset in Rsize dataset 1 are all high, but on the Rsize dataset 1, which includes both objects and object models, the average test accuracies are all low (AlexNet 71.3\%, VGG-16 74.6\%, Inception V3 75.1\%, Resnet-18 74.8\%, and DenseNet-121 73.5\%). This was not the case for SizeNet however, which had a high test average accuracy of 98.0\%. The confusion matrix of real labels and prediction labels on the test set of Rsize dataset 1 using AlexNet is shown in Figure~\ref{fig:fig4}. From this, it can be concluded that AlexNet, VGG-16, Inception V3, Resnet-18, and DenseNet-121 cannot correctly recognize objects and object models whose features are highly similar, but SizeNet can correctly recognize the objects and object models based on their real sizes.

\begin{table}[t]
	\setlength{\abovecaptionskip}{10pt}
	\setlength{\belowcaptionskip}{10pt}
	\normalsize
	\caption{The results of object recognition experiments using SizeNet, AlexNet, VGG-16, Inception V3, Resnet-18 and DenseNet-121 on Rsize dataset 1.}
	\label{table2}
	\centering
	\begin{tabular}{cccccccccc}
		\toprule
		\multirow{2}{*}{} & \multicolumn{2}{c}{Objects} & \multicolumn{2}{c}{Object models} & \multicolumn{2}{c}{Rsize dataset 1} \\
		\cmidrule(r){2-3} \cmidrule(r){4-5} \cmidrule(r){6-7}
		&  Val acc.  &  Test acc.
		&  Val acc.  &  Test acc.
		&  Val acc.  &  Test acc.  \\
		\midrule
		SizeNet (ours)    & 98.0    & 100.0   & 95.1  & 98.0    & 86.5  & 98.0  \\
		AlexNet   & 97.5  & 99.0    & 93.1  & 95.6  & 84.6  & 71.3  \\
		VGG-16    & 96.8  & 99.1  & 94.2  & 98.6  & 85.8  & 74.6  \\
		Inception V3  & 93.1  & 99.0    & 89.9  & 96.3  & 80.1  & 75.1   \\
		Resnet-18   & 94.8  & 99.5  & 93.1  & 99.3  & 85.1  & 74.8 \\
		Densenet-121  & 96.2  & 99.6  & 92.8  & 98.1  & 85.0    & 73.5 \\
		\bottomrule
	\end{tabular}
\end{table}

\begin{figure}[htbp]
	\begin{center}

		\includegraphics[width=0.8\linewidth]{fig4.eps}
		\setlength{\abovecaptionskip}{0pt}
		\setlength{\belowcaptionskip}{0pt}
	\end{center}
	\caption{The confusion matrix of real and prediction labels in the test set using AlexNet on Rsize dataset 1. It can be seen that 31 of the 100 images with police car real labels were misrecognized as police car models, and 23 of the 100 images with police car model real labels were misrecognized as police cars, and so on.}
	\label{fig:fig4}
\end{figure}

\begin{table}[t]
	\setlength{\abovecaptionskip}{10pt}
	\setlength{\belowcaptionskip}{10pt}
	\normalsize
	\caption{The results of object recognition experiments using SizeNet, AlexNet, VGG-16, Inception V3, Resnet-18 and DenseNet-121 on Rsize dataset 2.}
	\label{table3}
	\centering
	\begin{tabular}{ccc}
		\toprule
		& Val acc. & Test acc. \\
		\midrule
		Size-Net (ours) & 93.3  & 99.0 \\
		AlexNet & 93.8  & 71.2 \\
		VGG-16 & 92.0    & 69.4 \\
		Inception V3 & 90.1  & 65.2 \\
		Resnet-18 & 91.2  & 67.6 \\
		Densenet-121 & 92.5  & 67.8 \\
		\bottomrule
	\end{tabular}
\end{table}

Table~\ref{table3} shows the results of object recognition experiments using SizeNet, AlexNet, VGG-16, Inception V3, Resnet-18, and DenseNet-121 on Rsize dataset 2. The results show that the average validation accuracies of object recognition using SizeNet, AlexNet, VGG-16, Inception V3, Resnet-18, and DenseNet-121 on the training set, which includes the images containing only target objects, are all high. However, on the test set which includes the images containing both target objects and other interfering objects, the average accuracies are all low (AlexNet 71.2\%, VGG-16 69.4\%, Inception V3 65.2\%, Resnet-18 67.6\% and DenseNet-121 67.8\%). But this was not the case for SizeNet, which obtained a high average test accuracy of 99.0\%. The confusion matrix of real and prediction labels on the test set of Rsize dataset 2 using AlexNet is shown in Figure~\ref{fig:fig5}. It can be concluded that AlexNet, VGG-16, Inception V3, Resnet-18, and DenseNet-121 cannot correctly recognize the target objects from other interfering objects whose real sizes are different, but SizeNet can correctly recognize and confirm the target objects from other interfering objects based on their real sizes.

This is because the SizeNet proposed in this paper recognizes the object is based on not only its features, but also on its real size. The real size of the object can avoid recognizing the object as a model or other objects, while their real sizes have a big difference.

\begin{figure}[t]
	\begin{center}

		\includegraphics[width=0.8\linewidth]{fig5.eps}
		\setlength{\abovecaptionskip}{0pt}
		\setlength{\belowcaptionskip}{0pt}
	\end{center}
	\caption{The confusion matrix of real and predict labels in the test set using AlexNet on the Rsize dataset 2. It can be seen that 18 of the 100 images with car real labels were misrecognized as pedestrians, and 37 of the 100 images with crosswalk real labels were misrecognized as pedestrians; 23 of the 100 images with pillow real labels were misrecognized as beds, and 39 of the 100 images with bed real labels were misrecognized as pillows, and so on.}
	\label{fig:fig5}
\end{figure}

\section{Discussion}
In order to solve the problems faced by CNN and context-based object recognition models, there are two issues that need to be addressed. Firstly, recognizing objects with highly similar object features; secondly, recognizing target objects that are interfered with by backgrounds and other objects. Inspired by the visual neural networks of humans, the SizeNet system proposed in this paper conducts object recognition based on the real sizes and features of objects. In addition, the Rsize dataset was used to address the two problems.

Object recognition experiments using SizeNet, AlexNet, VGG-16, Inception V3, Resnet-18, and DenseNet-121 were conducted on the Rsize dataset. The results show that the AlexNet, VGG-16, Inception V3, Resnet-18, and DenseNet-121 cannot correctly differentiate between police cars and the police car models, fire truck and fire truck models, and bullet train and bullet train models whose features are highly similar but real sizes are obviously different. This is because the AlexNet, VGG-16, Inception V3, Resnet-18, and DenseNet-121 only use the object features (shapes, textures, colors) to recognize the object, and do not consider the real sizes of the objects. The features of an object and an object model are highly similar, and therefore make it difficult to distinguish between them. The AlexNet, VGG-16, Inception V3, Resnet-18, and DenseNet-121 cannot correctly understand and distinguish target objects (such as pedestrians) from interference objects (such as crosswalks and cars), whose real sizes are different from the target objects. It is because the AlexNet, VGG-16, Inception V3, Resnet-18, and DenseNet-121 not only recognize the target object, but also the interference objects based on the object features, and cannot understand which object is the target object.

On the other hand, SizeNet can correctly recognize police cars and police car models, fire trucks and fire truck models, and bullet trains and bullet train models, and can also correctly understand and distinguish the target object from interference objects. This is because SizeNet not only uses the object’s features but also the real size of the object. Therefore, SizeNet does not correctly distinguish between the objects and object models from the highly similar features, but from the different real sizes, and correctly understands and distinguishes the target objects from the interference objects with the marked real sizes of the object images. SizeNet can solve the two problems faced by CNN and context-based object recognition models.

SizeNet can be integrated with any other object recognition algorithms based on object features, and can be used for image segmentation, object classification, and object detection. SizeNet has important theoretical and practical value for research into interpretable computer vision.

\section*{References}
\medskip
{
\small
[1] Yamins, Dlk\ \& Dicarlo, J. J.\ (2016) Using goal-driven deep learning models to understand sensory cortex. {\it Nature Neuroscience} {\bf 19}(3):356-365.

[2] Christian, S., Liu, W., Jia, Y., Pierre, S., Scott, R., Dragomir, A., Dumitru, E., Vincent, V.\ \& Andrew, R.\ (2015) Going deeper with convolutions. {\it 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)} {\bf 1}:1-9.

[3] Simonyan, K.\ \& Zisserman, A.\ (2014) Very Deep Convolutional Networks for Large-Scale Image Recognition. {\it Computer Science}.

[4] He, K.,  Zhang, X.,  Ren, S.\ \& Sun, J.\ (2016) Deep Residual Learning for Image Recognition. {\it 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)} {\bf 1}:770-778.

[5] Szegedy, C.,  Vanhoucke, V.,  Ioffe, S.,  Shlens, J.\ \& Wojna, Z.\ (2016) Rethinking the Inception Architecture for Computer Vision. {\it 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)} {\bf 1}:2818-2826.

[6] Huang, G.,  Liu, Z.,  Laurens, Vdm\ \& Weinberger, K. Q.\ (2017) Densely Connected Convolutional Networks. {\it 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)} {\bf 1}:2261-2269.

[7] Fukushima, K.\ (1980) Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. {\it Biological Cybernetics} {\bf 36}(4):193-202.

[8] LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W.\ \& Jackel, L. D.\ (1989) Backpropagation Applied to Handwritten Zip Code Recognition. {\it Neural Computation} {\bf 1}(4):541-551.

[9] Riesenhuber, M.\ \& Poggio, T.\ (1999) Hierarchical models of object recognition in cortex. {\it Nature Neuroscience} {\bf 2}(11):1019-1025.

[10] Krizhevsky, A., Sutskever, I.\ \& Hinton, G.\ (2012) ImageNet Classification with Deep Convolutional Neural Networks. {\it Advances in neural information processing systems} {\bf 25}(2):5249-5262.

[11] Hubel, D.H.\ \& Wiesel, T.N.\ (1959) Receptive fields of single neurones in the cat's striate cortex. {\it The Journal of Physiology} {\bf 148}(3):574-591.

[12] Zeiler, M.D.\ \& Fergus, R.\ (2014) Visualizing and Understanding Convolutional Neural Networks. {\it European Conference on Computer Vision} :818-833.

[13] Nie, W., Yang, Z.\ \& Patel, A.\ (2018) A Theoretical Explanation for Perplexing Behaviors of Backpropagation-based Visualizations. {\it Proceedings of the 35th International Conference on Machine Learning} {\bf 80}:3809-3818.

[14] Geirhos, R.,  Rubisch, P.,  Michaelis, C.,  Bethge, M.,  Wichmann, F. A.\ \& Brendel, W.\ (2019) ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. {\it International Conference on Learning Representations}.

[15] Geirhos, R.,  Jacobsen, J. H.,  Michaelis, C.,  Zemel, R.,  Brendel, W.,  Bethge, M.\ \& Wichmann, F. A.\ (2020) Shortcut Learning in Deep Neural Networks. {\it Nature Machine Intelligence} {\bf 2}:665-673.

[16] Torralba, A.\ \& Sinha, P.\ (2001) Statistical context priming for object detection. {\it Proceedings of the IEEE International Conference on Computer Vision} {\bf 1}:763–770.

[17] Wolf, L.\ \& Bileschi, S.\ (2006) A Critical View of Context. {\it International Journal of Computer Vision} {\bf 69}(2):251-261.

[18] Rabinovich, A.,  Vedaldi, A.,  Galleguillos, C.,  Wiewiora, E.\ \& Belongie, S.\ (2007) Objects in context. {\it 2007 11th IEEE International Conference on Computer Vision} :1-8.

[19] Galleguillos, C.\ \& Belongie, S.\ (2010) Context based object categorization: A critical survey. {\it Computer Vision and Image Understanding} {\bf 114}(6):712-722.

[20] Zhao, H.,  Shi, J.,  Qi, X.,  Wang, X.\ \& Jia, J.\ (2017) Pyramid Scene Parsing Network. {\it 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)} {\bf 1}:6230-6239.

[21] Konkle, T.\ \& Oliva, A.\ (2012) A Real-World Size Organization of Object Responses in Occipitotemporal Cortex. {\it Neuron} {\bf 74}(6):1114-1124.

[22] Holler, D. E., Behrmann, M.\ \& Snow, J. C.\ (2019) Real-world size coding of solid objects, but not 2-D or 3-D images, in visual agnosia patients with bilateral ventral lesions. {\it Cortex} {\bf 119}:555-568.

[23] Holler, D. E., Fabbri, S.\ \& Snow, J. C.\ (2020) Object responses are highly malleable, rather than invariant, with changes in object appearance. {\it Scientific Reports} {\bf 10}(4654).

[24] Murray, S. O., Boyaci, H.\ \& Kersten, D.\ (2006) The representation of perceived angular size in human primary visual cortex. {\it Nature Neuroscience} {\bf 9}(3):429-434.

[25] Konkle, T.\ \& Oliva, A.\ (2012) A familiar-size Stroop effect: real-world size is an automatic property of object representation. {\it Journal of Experimental Psychology Human Perception \& Performance} {\bf 38}(3):561-569.

[26] Murata, A., Gallese, V., Luppino, G., Kaseda, M.\ \& Sakata, H.\ (2000) Selectivity for the Shape, Size, and Orientation of Objects for Grasping in Neurons of Monkey Parietal Area AIP. {\it Journal of Neurophysiology} {\bf 83}(5):2580–2601.

[27] Konen, C. S.\ \& Kastner, S.\ (2008) Two hierarchically organized neural systems for object information in human visual cortex. {\it Nature Neuroscience} {\bf 11}(2):224-231.

[28] Nishimura, M.,  Scherf, K,  Zachariou, V.,  Tarr, M\ \& Behrmann, M.\ (2014) Size Precedes View: Developmental Emergence of Invariant Object Representations in Lateral Occipital Complex. {\it Journal of Cognitive Neuroscience} {\bf 27}(3):1-18.

[29] Wilcox, T.\ (1999) Object individuation: infants' use of shape, size, pattern, and color. {\it Cognition} {\bf 72}(2):125-166.

[30] Long, B.,  Konkle, T.,  Cohen, M. A.\ \& Alvarez, G. A.\ (2016) Mid-level perceptual features distinguish objects of different real-world sizes. {\it Journal of Experimental Psychology General} {\bf 145}(1):95-109.

[31] Long, B\ \& Konkle, T.\ (2017) A familiar-size Stroop effect in the absence of basic-level recognition. {\it Cognition} {\bf 168}:234-242.
}

\end{document}