\section*{Supplementary Materials for Paper ID 4759: MSI: Maximize Support-Set Information for Few-Shot Segmentation}}
\begin{center}
 {\section*{Supplementary Materials}}
\end{center}

MSI shows that background information overlooked by existing methods can help to segment a target. By utilizing both STF and SIF, MSI is able to obtain information on the target class from the entire support image while minimizing bias to the target class. Through STF, MSI is able to obtain more fine-grained target information which potentially gets removed by masking background features. By using SIF, it becomes possible to acquire more information about the target that might have been missed by the limited support mask. Furthermore, it utilizes other objects in the background of the support image to either avoid segmenting non-target objects or aid segmenting the target object in the query image. In this supplementary material, we include the following items:

\begin{enumerate}[label={A.}{{\arabic*}},noitemsep,itemindent=0.5cm]
\item Correlation map analysis.
\item Detailed architecture.
\item Training profiles.
\item Data augmentation.
\item Additional qualitative results.
\item Failure study.
\end{enumerate}

\subsection*{A.1~Correlation Map Analysis}
Since FM~(Feature Masking) eliminates background features, high correlation values exist only within the target object area~(Fig.~\ref{fig:correlation_compare}).
When the support mask does not accurately cover the entire target object, the target information received by a network is limited~(Fig.~\ref{fig:correlation_compare2}).

The support target features (STF) have high activation around the target object in the support image but not in the query image. Some background areas in the support image features (SIF) are activated because they match the background features of the query image. Although both SIF and STF weakly capture the target object of the query image in their correlation maps, once these correlation maps are fed to the encoder, in the encoded feature maps, we observe strong signals across the target class of the query image ~(see the feature map in Fig.~\ref{fig:correlation_compare}). On the contrary, the FM encoder produces a more scattered activation around the target.

\subsection*{A.2~Detailed Architecture}
In Fig.~\ref{fig:overall} of the main manuscript, the encoder part is shown briefly. In Fig.~\ref{fig:detail_architecture}, we reveal more details relevant to the encoder that are specific to each baseline architecture: HSNet~\cite{HSNet}, VAT~\cite{VAT2} and, ASNet~\cite{ASNet}. For HSNet, the SCM is divided into 3 convolutional blocks and used in 3 stages of the encoder step-by-step. VAT utilizes query features again in the decoding stage. ASNet uses pooling to reduce the input feature size.

\begin{figure}[h]
  \centering
    \includegraphics[height=6.0cm]{figure/feature_map.png}
    \caption{Correlation map and feature map visualization. Although STF and SIF weakly capture the target class of the query image in their correlation maps, once these correlation maps are fed to the encoder, we notice strong signals on the target object.}
    \label{fig:correlation_compare}
\end{figure}

\begin{figure}[h]
  \centering
    \includegraphics[height=7.8cm]{figure/detail_architecture.png}
    \caption{Detailed architectures showing where SCM is used for HSNet~\cite{HSNet}, VAT~\cite{VAT2}, and ASNet~\cite{ASNet}. (a)~HSNet~+~MSI (b)~VAT~+~MSI (c)~ASNet~+~MSI.}
    \label{fig:detail_architecture}
\end{figure}

\subsection*{A.3~Training Profiles} Figure~\ref{fig:train_curve} (main paper) shows that VAT~+~MSI provides 3.5x faster convergence compared to VAT~\cite{VAT2}. In Fig.~\ref{fig:train_curve_hsnet}, we reveal that our MSI with HSNet baseline, HSNet~+~MSI, enables 4.5x faster convergence in comparison to HSNet~\cite{HSNet}. This could be because STF provides a strong prior on the target boundary information to the encoder network.

\begin{figure}[h]
  \centering

    \includegraphics[height=8.3cm]{figure/train_curve_hsnet.png}

   \caption{Train.~and~Val.~profiles of HSNet~\cite{HSNet} and HSNet+MSI on PASCAL-5$^i$~\cite{pascal} with ResNet50~\cite{Resnet}. HSNet+MSI provides 4.5x faster convergence (to reach 60\% in mIoU) on average than HSNet. Red circles indicate when train accuracy reaches 60\% in mIoU.}
   \label{fig:train_curve_hsnet}
\end{figure}

\subsection*{A.4~Data Augmentation.}

\begin{table}[h]
\centering
\scalebox{0.7}{%
 \begin{tabular}{c | c c | c c}
 \hline
 Method \& Backbone & Data Aug.  & MSI & mIoU & FB-IoU \\
 \hline
 & - & -  &  65.5 & 77.8 \\
VAT~\cite{VAT2} & \checkmark & - & 65.3  &  77.4 \\
(ResNet50~\cite{Resnet}) & - & \checkmark  &  67.2 & 78.6 \\
 & \checkmark & \checkmark  &  \textbf{68.3} & \textbf{79.1} \\
\hline
 & - & -  &  67.9 & 79.6 \\
VAT~\cite{VAT2} & \checkmark & -  & 67.5 & 78.8 \\
(ResNet101~\cite{Resnet}) & - & \checkmark  &  68.9 & 79.4 \\
 & \checkmark & \checkmark  &  \textbf{70.0} & \textbf{82.3} \\
\hline
 \end{tabular}
 }
\caption{The impact of data augmentation on VAT~\cite{VAT2} and VAT~+~MSI on PASCAL-5$^i$~\cite{pascal}. Best results are shown in \textbf{bold}.}

\label{table:data_augmentation}
\end{table}

VAT~\cite{VAT2} does not recommend using data augmentation~\cite{data_aug_1, data_gug_2} as it causes performance drop. However, when using our proposed method with VAT (VAT~+~MSI), the performance improves using CATs data augmentation~\cite{data_gug_2} (see Tab.~\ref{table:data_augmentation}).

\subsection*{A.5 Additional Qualitative Results.}
Figs.~\ref{fig:pascal_msi}, \ref{fig:coco_msi}, and \ref{fig:fss_msi} show additional qualitative results of VAT~+~MSI on three benchmarks.

\subsection*{A.6~Failure Cases}
Although MSI shows significant gains over strong FSS baselines, we identify some challenging FSS cases where there is still room for improvement.
When a very small part of the target object in the support set is given, the target object might not be accurately segmented in the query image (see the fourth row in Fig.~\ref{fig:coco_fail}). Also, when the target object in the query image lies far in the background and appears blurry (see the fifth row in Fig.~\ref{fig:pascal_fail}), the segmentation accuracy could worsen.
Other challenging cases are when the color or shape of the target object is quite different ~(see the third row in Fig.~\ref{fig:pascal_fail} and the third row in Fig.~\ref{fig:coco_fail}) and when the object class in the support set and query image are not the same, but are similar in shape and semantics, such as a computer monitor vs. television (see the second row in Fig.~\ref{fig:pascal_fail}).

\begin{figure}[h]
  \centering
    \includegraphics[height=13.1cm]{figure/correlation_map.png}
    \caption{Correlation map visualization. Feature masking~(FM) with the inaccurate support mask causes inadvertent loss in the boundary information of the target. On the other hand, Support Image Features~(SIF) utilizes the entire image features and the Support Target Features~(STF) focuses on the target after removing the background at the input level. Note that in the Block 1, compared to FM, STF has distinct signals at the target class boundary. In the Block 3, compared to FM, in SIF, the correlation is evenly distributed across various areas in the target object.}
    \label{fig:correlation_compare2}
\end{figure}

\begin{figure}
  \centering
    \includegraphics[height=21.4cm]{figure/pascal_su.png}
    \caption{VAT~+~MSI Qualitative result using ResNet50~\cite{Resnet} on PASCAL-5$^i$~\cite{pascal}.}   \label{fig:pascal_msi}
\end{figure}

\begin{figure}
  \centering
    \includegraphics[height=21.4cm]{figure/coco_su.png}
    \caption{VAT~+~MSI Qualitative result using ResNet50~\cite{Resnet} on COCO-20$^i$~\cite{lin2015microsoft}.}    \label{fig:coco_msi}
\end{figure}

\begin{figure}
  \centering
    \includegraphics[height=21.4cm]{figure/fss_su.png}
    \caption{VAT~+~MSI Qualitative result using ResNet50~\cite{Resnet} on FSS-1000~\cite{FSS1000}.}
    \label{fig:fss_msi}
\end{figure}

\begin{figure}
  \centering
    \includegraphics[height=8.6cm]{figure/fail_pascal.png}
    \caption{Failure cases of VAT~+~MSI with ResNet50~\cite{Resnet} on PASCAL-5$^i$~\cite{pascal}.}
    \label{fig:pascal_fail}
\end{figure}

\begin{figure}
  \centering
    \includegraphics[height=8.6cm]{figure/fail_coco.png}
    \caption{Failure cases of VAT~+~MSI with ResNet50~\cite{Resnet} on COCO-20$^i$~\cite{lin2015microsoft}.}
    \label{fig:coco_fail}
\end{figure}

\section{Introduction}
\label{sec:intro}

\begin{figure}[t]
  \centering

    \includegraphics[height=6.0cm]{figure/teaser_1.png}

   \caption{Recent FSS baseline (VAT\cite{VAT2}) struggles to accurately segment the target object in several challenging scenarios in PASCAL~\cite{pascal} and COCO$^i$~\cite{lin2015microsoft}: (a) the same instance of the target class is not masked e.g., the sofa on the right, (b) the support mask is very small compared to the entire image e.g., flowers in the pot, (c) support mask is missing some target boundary information e.g., the front and chimney of the train, and (d) the background contains some important contextual information, unavailable in the support mask, e.g., shoes and grass. Our method (MSI) is capable of accurately segmenting target objects. It maximizes the support set information to compensate for the limited support mask information and can exploit relevant contextual information from the background.}

   \label{fig:4cases}
\end{figure}

\begin{figure}
  \centering

    \includegraphics[height=7.9cm]{figure/teaser3.png}
    \caption{(a) Many FSS methods use support mask to remove background features~\cite{Zhang2020SGOneSG}, denoted as feature masking (FM), and so rely only on target object features. (b) Recent work, HM~\cite{hmmasking}, merges both image masking and feature masking to achieve hybrid masking for improving target information. (c) We propose to maximize the support set information (MSI) to compensate for the limited support mask information and exploit relevant contextual information from the background.
    }

    \label{fig:teaser}
\end{figure}

Deep convolutional neural networks (DCNNs) have achieved state-of-the-art results across several mainstream computer vision (CV) problems, including object detection \cite{redmon2016you,ren2015faster} and semantic segmentation \cite{FCN, Deeplab, pyramidscene}. An important factor underlying the success of DCNNs is large-scale annotated datasets, which are costly and cumbersome to acquire in many dense prediction tasks, such as semantic segmentation. Moreover, these models struggle to segment novel objects when there only a few annotated examples are available. To address these limitations, we have seen a plethora of few-shot segmentation (FSS) approaches \cite{Co-FCN,AMP-2,PMM,FWB,PFENet,PANet,CANet,ASGNet,DAN,FSOT,Zhang2020SGOneSG,CWT,HSNet,ASNet,BAM,VAT2,hmmasking}. The problem settings in FSS require accurate segmentation of a target object in a query image, given a few annotated images, termed as the support set, from the target class.

Shaban et al.'s work~\cite{OSLSM} is believed to have introduced the FSS problem. It proposes the first FSS model, in which the masked support image is used to extract only the target features. Using the target features, both segmentation and conditioning branches are trained to segment an object of the target class. Later, Zhang et al.~\cite{Zhang2020SGOneSG} show that extracting target features using masked average pooling~(MAP) is more beneficial for network learning than obtaining features by masking images. Since then, many FSS models~\cite{PMM,PANet,PFENet,FWB,CANet,ASGNet,zhang2021fewshot,HSNet, VAT2,ASNet,BAM} consider MAP as the de-facto technique for obtaining target features and focus on improving the encoder/decoder network. Recently, HSNet~\cite{HSNet} propose to utilize more meaningful target features by extracting deep features from multiple layers of a backbone network. Likewise, ASNet~\cite{ASNet} and VAT~\cite{VAT2} propose different network architectures to harness deep features.

Despite showing promising results, many recent methods, including HSNet~\cite{HSNet}, ASNet~\cite{ASNet}, and VAT~\cite{VAT2}, struggle with the following types of challenging FSS cases (Fig.~\ref{fig:4cases}): (a) When the support mask does not mask all instances of the same target class. (b) The support mask is unable to faithfully capture object boundaries. (c) The support mask is too small, carrying limited object information. (d) The background contains some important contextual information, unavailable in support mask, for accurately segmenting the target object. We conjecture that this happens because many current SOTA methods rely on support masks to completely remove the background (Fig.~\ref{fig:teaser}), which limits information in several challenging FSS cases.

In this paper, we propose a new method to overcome the limited information bottleneck from the support mask. It is based on the intuition that upon maximizing the information from support set images (MSI), it is possible to compensate for the limited support mask information and also utilize the important contextual information available in the background (Fig.~\ref{fig:teaser}).
MSI jointly exploits two complementary sources of features. The first set of features is obtained by using masked support images, which only activate the target-related features in the query image. The second set of features is generated from the full support images, which activate the features of all similar objects shared between the support and query images. The former features act an anchor for the latter in localizing a certain target class and also supplements it with the target boundary information. We summarize our key contributions as follows:

\begin{itemize}[topsep=-3pt, noitemsep]

    \item We propose a new approach (MSI) to fully harness the entire support image in the end-to-end learning pipeline of FSS.

    \item We perform extensive experiments on three challenging FSS benchmarks: PASCAL-$5^i$~\cite{pascal}, COCO-$20^i$~\cite{lin2015microsoft}, and FSS-1000~\cite{FSS1000}. Results show that MSI consistently achieves notable mIoU improvements in one-shot over all three strong baselines, including HSNet~\cite{HSNet}, ASNet~\cite{ASNet}, and VAT~\cite{VAT2}.
    \item MSI improves the training speed of recent baseline models on PASCAL-5$^i$~\cite{pascal}, with VAT~\cite{VAT} and HSNet~\cite{HSNet}, 3.3x and 4.5x on average, respectively.
\end{itemize}

\section{Related Works}
\label{sec:related_works}

\noindent\textbf{Few-shot segmentation:} OSLSM~\cite{OSLSM} is considered to be the first work introducing FSS problem. OSLSM proposed a model consisting of a condition branch and a segmentation branch. Since OSLSM, various methods have been proposed to solve the FSS problem~\cite{Co-FCN,AMP-2,PMM,FWB,PFENet,PANet,CANet,ASGNet,DAN,FSOT,Zhang2020SGOneSG,CWT,HSNet,ASNet,BAM,VAT2,hmmasking}. co-FCN~\cite{Co-FCN} proposed a conditional network and achieved competitive accuracy. AMP~\cite{AMP-2} proposed adaptive masked proxies which builds the weights of the final segmentation layer utilizing multi-resolution imprinting. PMM~\cite{PMM} proposed to activate target objects and deactivate other objects. Khoi and Sinisa~\cite{FWB} proposed a feature weighting and boosting method to improve the discriminativeness of features.
PFENet~\cite{PFENet} utilized high-level features with training-free prior. PANet~\cite{PANet} used prototype representations from an embedding space and segmented each pixel by comparing to the prototypes. CANet~\cite{CANet} proposed the dense comparison module~(DCM) for multi-level feature comparison and the iterative optimization module~(IDM) for improving quality of segmentation. Similarly, ASGNet~\cite{ASGNet} proposed the two modules superpixel-guided clustering~(SGC) and guided prototype allocation~(GPA). DAN~\cite{DAN} designed democratic graph attention to perform finding matching features.  FSOT~\cite{FSOT} proposed correspondence matching network(CMNet) to find correlation between query and support features. SG-One~\cite{Zhang2020SGOneSG} proposed masked average pooling~(MAP) to generate guidance features. CWT~\cite{CWT} suggested training only classifier while freezing encoder and decoder. Recently, CyCTR~\cite{zhang2021fewshot} proposed cycle-Consistent Transformer to perform cross-attention between support and query features.
HSNet~\cite{HSNet} designed efficient 4D convolution to utilize spatial information in deep features. To address both few-shot classification and few-shot segmentation, ASNet~\cite{ASNet} proposed an integrated few-shot learning technique. VAT~\cite{VAT2} designed 4D Convolutional Swin Transformer in an efficient way.
BAM~\cite{BAM} proposed to use base and the meta learners, where meta learner's segmentation is refined by the base learner.

\noindent\textbf{MAP~(Masked Average Pooling):} Zhang et al.~\cite{Zhang2020SGOneSG} proposed the MAP to collect target information from the support set. MAP masks features instead of masking support image and uses average pooling to extract target information. They argued that removing background image from support image increases the variance of the input data for a unified network. Also, masking image will make the network biased toward target image. For these reasons, the MAP is recommended to get target features. Since Zhang et al.~\cite{Zhang2020SGOneSG}, many few shot works~\cite{PMM,PANet,PFENet,FWB,CANet,ASGNet,zhang2021fewshot,HSNet, VAT2,ASNet,BAM} are following the MAP to extract target features. However, by performing average pooling, spatial information is inevitably lost \cite{gap}. In this work, we utilized the masked support image to generate target features without using MAP. Therefore, we can retrieve fine-detail texture information and preserve spatial information.

\noindent\textbf{Multi-layer features:}~Instead of using MAP, HSNet~\cite{HSNet} proposed to use deep features extracted from multiple layers and designed effective convolution to process the features. To process the deep features effectively and efficiently, VAT~\cite{VAT2} proposed a model based on swin transformer~\cite{swin_transformer} and ASNet~\cite{ASNet} proposed attentive squeeze network. HM~\cite{hmmasking} proposes hybrid masking to compensate for the lost details in feature masking. Although the spatial information that MAP lost was preserved, these works extracted target features counting on support masks. Therefore, when the support masks give limited information, limited target information is extracted. In this paper, not only the information loss when the target features are extracted by relying on the support mask is minimized but also more meaningful target features are extracted by utilizing both the entire support image and the masked support image.

\noindent\textbf{Utilizing background information:} There have been several attempts to improve FSS accuracy by using background information~\cite{PMM,BAM}. PMM~\cite{PMM} proposed a method of deactivating the background and activating the target object using the duplex manner. Negative learning was carried out with objects in the background, and on the other hand, positive learning was conducted using the foreground object. Afterwards, the two learning results were used to segment target object accurately. Similarly, BAM~\cite{BAM} also proposed to use two learners. Base learner is trained using the objects in the background and meta learner is trained with foreground object. These two learners are integrated to predict segmentation accurately. However, they assume that the background lacks meaningful target information. Therefore they missed target information that might exist in the background. We obtain more target information even from the background and simultaneously utilize background information to avoid segmenting wrong objects.

\section{Overall Method}

\label{sec:method}

\begin{figure*}
  \centering
    \includegraphics[height=5.8cm]{figure/main3.png}
    \caption{Overall architecture. To maximize the usability of the support set, first we extract support image features~(SIF) and support target features~(STF) through backbone. SIF branch creates features using the entire support image, and STF branch extracts the features after removing background pixels. The query features~(QF) are obtained from the query. We obtain correlation maps by calculating the cosine similarity of the STF and SIF with the QF, respectively, and these correlation maps are concatenated to make super correlation maps~(SCM). $CS_2$ is finding cosine similarity between target object and the query image. Therefore, only target-related information in the query image will be activated. Whereas $CS_1$ is finding cosine similarity between the entire support image and the query image. So, all the similar objects between the support image and the query image are activated.}

    \label{fig:overall}
\end{figure*}

Fig.~\ref{fig:overall} displays the overall architecture of our method. Following recent works~\cite{HSNet,ASNet,VAT2}, it consists of a backbone network (ResNet-50~\cite{Resnet} or ResNet-101~\cite{Resnet}) pre-trained on ImageNet~\cite{image_net} to obtain multi-layer features, and an encoder/decoder network to predict the segmentation mask.
We propose a new plug-and-play module to maximize the support set information (MSI), which compensates for limited support mask information by exploiting relevant contextual information from the background.
MSI extracts support target features~(STF), containing target class information from the support images, and support image features~(SIF), accounting for the information in the entire support image.
Next, STF and SIF are leveraged to obtain their respective correlation maps by computing the cosine similarity with respect to the query features~(QF) obtained from the query image.
Then, the correlation maps corresponding to STF and SIF are utilized to get super correlation maps (SCM).
Finally, this SCM is used as the input to the encoder and decoder which can be from recent FSS methods: \cite{HSNet}, \cite{VAT2}, and \cite{ASNet}.

\subsection{Preliminaries}

The goal of few-shot segmentation is to train a model that can segment the target object in a query image when provided a few annotated example images from the target class.
Following prior work in FSS ~\cite{HSNet,ASNet,VAT2}, we utilize the episodic scheme for training our model.

Specifically, we assume the availability of two disjoint sets, $C_{train}$ and $C_{test}$ as training and testing classes, respectively.
The training data $D_{train}$ is sampled from $C_{train}$ and the testing data $D_{test}$ is from $C_{test}$.
We then construct multiple episodes from $D_{train}$ and $D_{test}$.
An episode is comprised of a support set, $S = (I^s, M^s)$, and a query set, $Q = (I^q, M^q)$, where $I^*$ and $M^*$ denote an image and its corresponding mask.
Furthermore, $D_{train} = \{(S_i, Q_i)\}^{N_{train}}_{i=1} $ and $D_{test} = \{(S_i, Q_i)\}^{N_{test}}_{i=1}$, where $N_{train}$ represents the number of episodes for training and $N_{test}$ is the number of episodes for testing.
During training, we iteratively sample episodes from $D_{train}$ to train a model that learns a mapping from $(I^s, M^s, I^q)$ to query mask $M^q$.
The learned model is used without further optimization by randomly sampling episodes from the testing data $D_{test}$ in the same manner and comparing the predicted query masks to the ground truth.

\subsection{Maximizing Support-Set Information: MSI}

An important question for the FSS pipeline is where in a network it is most suitable to use masking to provide target information to the network. Earlier attempts, such as the work of Shaban et al.~\cite{OSLSM} suggested masking the background in the input image so that only target features are obtained. The rationale was that the network would supposedly be biased toward large objects, so removing the background would decrease the variance of the output parameters.
Later, Zhang et al.~\cite{Zhang2020SGOneSG} proposed to mask the background after extracting features because masking the input image makes a network biased towards the target image and changes the statistical distribution of the support image set. Many recent state-of-the-art methods have conformed to this approach~\cite{PMM,PANet,PFENet,FWB,CANet,FSOT,ASGNet,zhang2021fewshot,HSNet,VAT2,ASNet,BAM}.

\noindent\textbf{Motivation:} Although these methods report promising performance, we note that many recent methods, including HSNet~\cite{HSNet}, ASNet~\cite{ASNet}, and VAT~\cite{VAT2}, struggle to accurately predict the segmentation mask in several challenging scenarios (Fig.~\ref{fig:4cases}). We believe that this occurs because many recent FSS methods fully rely on the support mask to remove the background features (Fig.~\ref{fig:teaser}). These support masks likely present an information bottleneck in such challenging cases. For instance, when they are very small, fail to properly encapsulate target object boundaries, or do not capture all instances of the target object.

We put forth a new approach (MSI) to alleviate the aforementioned limitations of support masks. We believe that by maximizing the information from support images, it becomes possible to compensate for the limited information in the support mask and also leverage important contextual information available in the background.
Specifically, we extract support target features~(STF), which contain target class information in the support image, and support image features~(SIF), which contain information from the entire support image. This allows us to obtain target class information from the entire support image while also reducing bias toward the target class. Additionally, the STF enables the acquisition of more fine-grained target information which potentially gets removed when relying only on support mask. Target information may exist even in the background features because the receptive field can cover even the target object area at deep features~\cite{receptive_analysis,hmmasking}.
We leverage STF and SIF to obtain their respective correlation maps by computing their cosine similarity with the query features~(QF), which are obtained from the query image.
The correlation maps corresponding to STF and SIF are then combined into Super Correlation Maps (SCM), which are inputted to the encoder and decoder.

\noindent\textbf{Formalizing STF and SIF:} Support image features~(SIF),~$\alpha^N_{i=1}$, where $N$ is the number of features, refer to the features of the entire support image, $I^S\in\mathbb{R}^{3 \times w \times h }$, where $w$ is the width and $h$ is the height of the image. Support target features~(STF),~$\beta^N_{i=1}$, are the features from the target image, $I^T\in\mathbb{R}^{3 \times w \times h }$, which is a masked support image using support mask $\mathbf{M}^S \in \{0,1\}^{w \times h}$. Formally,
\begin{align}  \label{eq:2}
  I^{T} & =  I^S \odot \zeta (M^S),
\end{align}
where $\zeta ( \cdot ) $ resizes the mask $M^S$ to fit the dimension of the image $I^S$ and $\odot$ denotes the Hadamard product. Query features~(QF),~$\kappa^N_{i=1}$, are obtained from the query image, $I^Q\in\mathbb{R}^{3 \times w \times h }$.

For each image ($I^S$, $I^T$, and $I^Q$), the backbone network, $\lhd$, produces $N$ number of features ($\alpha^N_{i=1}$, $\beta^N_{i=1}$, and $\kappa^N_{i=1}$ respectively) from its intermediate layers.
\begin{align} \label{eq:3}
   {\alpha}^N_{i=1} =  \lhd(I^S),~     {\beta}^N_{i=1} =  \lhd(I^T),~
{\kappa}^N_{i=1} =  \lhd(I^Q),
\end{align}
where features, $\alpha_i,\beta_i,\kappa_i  \in\mathbb{R}^{c_i \times w_i \times h_i }$, have different sizes of channel and spatial dimensions. Features extracted from deeper layers have a large number of channels with smaller width and height dimensions.

\noindent\textbf{Super Correlation Maps (SCM):} We calculate the cosine similarities of ${\alpha}^N_{i=1}$ and ${\beta}^N_{i=1}$ with respect to ${\kappa}^N_{i=1}$ using the Eq.~\ref{eq:4} to get correlation maps $C_i \in \mathbb{R}^{ 1 \times w_i \times  h_i \times w_i \times h_i }$ and $S_i \in \mathbb{R}^{ 1 \times w_i \times  h_i \times w_i \times h_i }$. To perform multiplication between features, we reshape and transpose the features, ${\alpha}^N_{i=1}$, ${\beta}^N_{i=1}$, and ${\kappa}^N_{i=1}$ into ${\alpha'}^N_{i=1} \in\mathbb{R}^{ (w_i \cdot h_i) \times c_i }$, ${\beta'}^N_{i=1} \in\mathbb{R}^{ (w_i \cdot h_i) \times c_i }$, and ${\kappa'}^N_{i=1} \in\mathbb{R}^{ c_i \times (w_i \cdot h_i) }$.
\begin{equation}
  \begin{aligned}  \label{eq:4}
    CS_{1}={C}^N_{i=1} & = \mathrm{ReLU} \bigg(\frac{{\alpha'}^N_{i=1}\cdot {\kappa'}^N_{i=1}}{ \Vert {\alpha'}^N_{i=1} \Vert \Vert {\kappa'}^N_{i=1} \Vert  } \bigg),\\
    CS_{2}={S}^N_{i=1}  & = \mathrm{ReLU} \bigg(\frac{{\beta'}^N_{i=1}\cdot {\kappa'}^N_{i=1}}{ \Vert {\beta'}^N_{i=1} \Vert \Vert {\kappa'}^N_{i=1} \Vert  } \bigg),
  \end{aligned}
\end{equation}
where $N$ the is number of features and ReLU~\cite{relu} removes inactivated and noisy values in the correlation map $C_i$ and $S_i$.
Correlation maps ${C}^N_{i=1}$ and ${S}^N_{i=1}$ are concatenated along the first dimension to obtain the Super Correlation Maps (SCM), ${P}^N_{i=1}  \in \mathbb{R}^{2 \times w_i \times  h_i \times w_i \times h_i }$ ~(Eq.~\ref{eq:5}).
\begin{align} \label{eq:5}
 SCM = {P}^N_{i=1} = [{C}^N_{i=1} \oplus {S}^N_{i=1}],
\end{align}
where $\oplus$ denotes the concatenation.

\subsection{Encoder and Decoder Architecture}
We adopted the encoder and decoder from recent FSS models: HSNet~\cite{HSNet}, ASNet~\cite{ASNet}, and VAT~\cite{VAT2}. The input channel size of encoder in the models was doubled to fit the size of the SCM. Otherwise, the architectures of the models were unchanged.

\section{Experiments}
\label{sec:result}

\noindent\textbf{Datasets:}
We use three widely used and publicly available datasets for evaluating our method (MSI): PASCAL-$5^i$~\cite{pascal}, COCO-$20^i$~\cite{lin2015microsoft} and FSS-1000~\cite{FSS1000}.
PASCAL-$5^i$ consists of 20 classes, whereas COCO-$20^i$ has 100 classes and FSS-1000 contains 1000 classes. Following previous works~\cite{HSNet,ASNet,VAT2}, we cross-validate using 4 folds for PASCAL-$5^i$ and COCO-$20^i$ and we divide the classes into 4 groups for training and testing.
Therefore, 5 and 20 classes are used for each fold testing on PASCAL-$5^i$ and COCO-$20^i$, respectively. Other remaining classes are used for training. For FSS-1000, 1000 classes are divided into 520, 240 and 240 for the training, validation and test. Lastly, in order to evaluate the generalizability of our method, COCO-$20^i$ is used for training, and PASCAL-$5^i$ for testing. To avoid class overlapping between training and testing, following previous works~\cite{HSNet,PFENet}, we change the PASCAL-$5^i$ class order for each fold .

\noindent\textbf{Implementation details:}
We incorporate MSI into three baseline models, HSNet~\cite{HSNet}, VAT~\cite{VAT2}, and ASNet~\cite{ASNet}, and refer to them as HSNet~+~MSI, ASNet~+~MS and VAT~+~MSI. The backbones networks, ResNet50~\cite{Resnet} and ResNet101~\cite{Resnet}, are pre-trained on ImageNet~\cite{image_net}, and are used to extract deep features following HSNet, ASNet, and VAT~(features from conv3\_x to conv5\_x before the ReLU~\cite{relu} activation of each layer stacked to form the deep features). No fine-tuning of backbones was performed.
For a fair comparison with the existing models, we keep their default hyperparameters, as listed in their codebases. For VAT training, CATs data augmentation~\cite{data_gug_2} was used and the batch size was reduced due to GPU memory limitations. Specifically, the batch sizes 4, 8, and 4 were used for PASCAL-5$^i$~\cite{pascal}, COCO-20$^i$~\cite{lin2015microsoft}, and FSS-1000~\cite{FSS1000} respectively. For VAT~+~MSI training, we used the data augmentation used in \cite{data_aug_1, data_gug_2, VAT}. No data augmentation was employed for training HSNet~+~MSI and ASNet~+~MSI.

\noindent\textbf{Evaluation metrics:}
We report FSS performance using mean Intersection-over-Union~(mIoU) and Foreground and Background IoU~(FB-IoU), which are widely used by existing methods~\cite{FSOT,PFENet,ASGNet,HSNet,ASNet,VAT2,hmmasking}. We calculate the IoU of all classes' cases using Eq.~\ref{iou}, and use mIoU $=  \frac{1}{n}\sum_{1}^{n}IoU$ where $n$ is the number of test cases and FB-IoU$= \frac{1}{2}{(IoU_F + IoU_B)}$ where $F$ is foreground and $B$ is background without considering classes.
\begin{align} \label{iou}
    IoU = \frac{Intersection}{Union} = \frac{TP}{TP+FP+FN}
\end{align}
where FP stands for false positives, FN for false negatives, and TP for true positives. Note that mIoU is a better metric than FB-IoU for finding the target class because mIoU measures only the target class accuracy.

\begin{table*}

\centering
\scalebox{0.8}{
\resizebox{\textwidth}{!}{%
\begin{tabular}{@{}cc|cccccc|cccccc@{}}
\toprule
\multirow{2}{*}{Backbone} & \multirow{2}{*}{Methods} & \multicolumn{6}{c|}{1-shot} & \multicolumn{6}{c}{5-shot}
\\
 &  & $5^0$ & $5^1$ & $5^2$ & $5^3$ & mIoU & FB-IoU & $5^0$ & $5^1$ & $5^2$ & $5^3$ & mIoU & FB-IoU  \\ \midrule
\multirow{12}{*}{ResNet50~\cite{Resnet}}

 & CWT~\cite{CWT} & 56.3 & 62.0 & 59.9 & 47.2 & 56.4 & - & 61.3 & {68.5} & 68.5 & 56.6 & 63.7 & - \\
 & RePRI~\cite{RePRI} & 59.8 & 68.3 & {62.1} & 48.5 & 59.7 & - & 64.6 & 71.4 & {71.1} & 59.3 & 66.6 & - \\
 & CyCTR~\cite{zhang2021fewshot} & {67.8} & {72.8} & 58.0 & 58.0 & 64.2 & - & {71.1} & {73.2} & 60.5 & 57.5 & 65.6 & - \\
  & BAM~\cite{BAM} & {69.0} & {73.6} & \textbf{67.6} & 61.1 & 67.8 & - & {70.6} & 75.1 & \textbf{70.8} & 67.2 & 70.9 & - \\

\cmidrule(l){2-14}

 & HSNet~\cite{HSNet} & 64.3 & 70.7 & 60.3 & 60.5 & 64.0 & {76.7} & 70.3 & {73.2} & 67.4 & {67.1} & {69.5} & {80.6} \\
& HSNet~+~MSI & 68.1 & {71.5} & 58.2 & {62.9} & {65.2}~\textcolor{red}{(+1.2)} & 76.5  & 70.7 & 72.8 & 61.5 & 66.6 & 67.9~\textcolor{blue}{(-1.6)}  & 78.2 \\  \cmidrule(l){2-14}

 & ASNet~\cite{ASNet} & {68.9} & {71.7} & {61.1} & {62.7} & {66.1}  & 77.7 & 72.6 & 74.3 & 65.3 & 67.1 & 70.8 & 80.4 \\

& ASNet~+~MSI & {69.2} & {71.7} & {59.7} & {64.4} & {66.3}~\textcolor{red}{(+0.2)}  & 77.9 & 72.0 & 73.2 & 64.0 & 68.0 & 69.3~\textcolor{blue}{(-1.5)} & 80.2 \\

  \cmidrule(l){2-14}

 & VAT$^\ast$~\cite{VAT} & 67.6 & 71.2 & 62.3 & 60.1 & {65.3} & {77.4} & {72.4} & {73.6} & {68.6} & {65.7} & {70.0} & {80.9} \\
& VAT~\cite{VAT2} & 67.6 & 72.0 & 62.3 & 60.1 & {65.5} & {77.8} & {72.4} & {73.6} & {68.6} & {65.7} & {70.1} & {80.9} \\

& VAT$^\ast$~+~MSI & {71.0} & {72.5} & {63.8} & {65.9} & {68.3}~\textcolor{red}{(+2.8)} & {79.1}  & {73.0} & {74.2} & 66.6 & {70.5} & {71.1} ~\textcolor{red}{(+1.0)}  & {81.2}  \\

  \midrule

 & CWT~\cite{CWT} & 56.9 & 65.2 & 61.2 & 48.8 & 58.0 & - & 62.6 & 70.2 & {68.8} & 57.2 & 64.7 & - \\
 & RePRI~\cite{RePRI} & 59.6 & 68.6 & {62.2} & 47.2 & 59.4 & - & 66.2 & 71.4 & 67.0 & 57.7 & 65.6 & -  \\
 & CyCTR~\cite{zhang2021fewshot} & {69.3} & {72.7} & 56.5 & 58.6 & 64.3 & 72.9 & {73.5} & 74.0 & 58.6 & 60.2 & 66.6 & 75.0 \\
\cmidrule(l){2-14}

\multirow{5}{*}{ResNet101~\cite{Resnet}}
 & HSNet~\cite{HSNet} & 67.3 & {72.3} & 62.0 & 63.1 & {66.2} & 77.6 & 71.8 & {74.4} & 67.0 & {68.3} & {70.4} & {80.6} \\

  & HSNet~+~MSI  & {70.5} & 72.9 & 60.6 & {64.3} & {67.1}~\textcolor{red}{(+0.9)}  & {77.8} & {71.9} & 74.9 & 64.1 & 67.7 & 69.7~\textcolor{blue}{(-0.7)} & 79.5 \\

   \cmidrule(l){2-14}

 & ASNet~\cite{ASNet} & {69.0} & {73.1} & {62.0} & {63.6} & {66.9}  & 78.0 & 73.1 & 75.6 & 65.7 & 69.9 & 71.1 & 81.0 \\

  & ASNet~+~MSI & {70.5} & \textbf{73.8} & {61.3} & {65.5} & {67.8}~\textcolor{red}{(+0.9)}  & {78.8} & {73.4} & {75.5} & 66.2 & 71.0 & 71.5~\textcolor{red}{(+0.4)} & 81.3 \\

  \cmidrule(l){2-14}

& VAT$^\ast$~\cite{VAT} & {68.4} & 72.5 & {64.8} & {64.2} & {67.5}  & {78.8} & {73.3} & {75.2} & {68.4} & {69.5} & {71.6} & {82.0} \\
& VAT~\cite{VAT2} & {70.0} & 72.5 & {64.8} & {64.2} & {67.9}  & {79.6} & \textbf{75.0} & {75.2} & {68.4} & {69.5} & {72.0} & \textbf{83.2} \\

& VAT$^\ast$~+~MSI  & \textbf{73.1} & {73.3} & {64.7} & \textbf{68.8} & \textbf{70.0}~\textcolor{red}{(+2.1)}  & \textbf{82.3} & {73.6} & \textbf{75.7} & 68.0 & \textbf{71.3} & \textbf{72.2} ~\textcolor{red}{(+0.2)} & {82.3} \\

 \bottomrule

\end{tabular}
}
}
\caption{Performance evaluation on Pascal-5$^i$~\cite{pascal}. $^\ast$~indicates that data augmentation used for training. Best results are shown in \textbf{bold}.}
\label{table:performance_pascal}

\end{table*}

\begin{table*}
  \centering
    \begin{minipage}{.60\textwidth}
\centering
\resizebox{\textwidth}{!}{%
\begin{tabular}{@{}cc|ccccc|ccccc@{}}
\toprule
\multirow{2}{*}{Backbone} & \multirow{2}{*}{Methods} & \multicolumn{5}{c|}{1-shot} & \multicolumn{5}{c}{5-shot}
\\
 &  & $20^0$ & $20^1$ & $20^2$ & $20^3$ & mIoU & $20^0$ & $20^1$ & $20^2$ & $20^3$ & mIoU \\ \midrule
\multirow{10}{*}{ResNet50~\cite{Resnet}}

 & RePRI~\cite{RePRI} & 32.0 & 38.7 & 32.7 & 33.1 & 34.1  & 39.3 & 45.4 & 39.7 & 41.8 & 41.6  \\
& CyCTR~\cite{zhang2021fewshot} & {38.9} & 43.0 & {39.6} & {39.8} & {40.3}  & 41.1 & 48.9 & 45.2 & {47.0} & 45.6  \\

& BAM~\cite{BAM} & {43.4} & {50.6} & {47.5} & {43.4} & {46.2}  & \textbf{49.3} & 54.2 & 51.6 & {49.6} & 51.2  \\

 \cmidrule(l){2-12}

 & HSNet~\cite{HSNet} & 36.3 & {43.1} & 38.7 & 38.7 & 39.2  & {43.3} & {51.3} & {48.2} & {45.0} & {46.9}  \\

& HSNet~+~MSI & {40.9} & {46.9} & {48.8} & {45.3} & {45.5}~\textcolor{red}{(+6.3)}   & {45.3}  & {53.5} & {53.1}  & {49.3}  & {50.3}~\textcolor{red}{(+3.4)}  \\

 \cmidrule(l){2-12}

& ASNet~\cite{ASNet} & {41.5} & 44.1 & 42.8 & {40.6} & {42.2}   & {47.6} & {50.1} & 47.7 & {46.4} & {47.9}  \\

 & ASNet~+~MSI & {41.5} & {46.3} & {43.5} & {42.1} & {43.4}~\textcolor{red}{(+1.2)}  & 46.0 & 50.7 & 47.5 & {46.9} & {47.8}~\textcolor{blue}{(-0.1)} \\

 \cmidrule(l){2-12}

& VAT$^\ast$~\cite{VAT2} & {39.0} & 43.8 & 42.6 & {39.7} & {41.3} & {44.1} & 51.1 & 50.2 & 46.1 & 47.9  \\

& VAT$^\ast$~+~MSI & {42.4} & {49.2} & {49.4} & {46.1} & {46.8}~\textcolor{red}{(+5.5)}  & 47.1 & {54.9} & {54.1} & {51.9} & {52.0}~\textcolor{red}{(+4.1)}  \\

 \midrule

\multirow{7}{*}{ResNet101~\cite{Resnet}}

& HSNet~\cite{HSNet} & 37.2 & {44.1} & {42.4} & {41.3} & {41.2} & {45.9} & {53.0} & {51.8} & {47.1} & {49.5} \\

& HSNet~+~MSI & {42.4} & {50.1} & {49.5} & \textbf{48.3} & {47.6}~\textcolor{red}{(+6.5)} & {48.0}  & {57.3} & {52.6}  & {52.6}  & {52.6}~\textcolor{red}{(+3.1)} \\

\cmidrule(l){2-12}

& ASNet~\cite{ASNet} & {41.8} & 45.4 & 43.2 & {41.9} & {43.1} & {48.0} & 52.1 & 49.7 & 48.2 & {49.5} \\

 & ASNet~+~MSI & {42.9} & 45.2 & {44.3} & {43.4} & {44.0}~\textcolor{red}{(+0.9)}& {47.7} & {49.9} & 49.0 & {48.8} & {48.9}~\textcolor{blue}{(-0.6)} \\

\cmidrule(l){2-12}

& VAT$^\ast$ & 39.5 & 44.4 & 46.1 & 40.4 & 42.6 & 45.2 & 54.1 & 51.1 & 47.1 & 49.4 \\

& VAT$^\ast$~+~MSI & \textbf{44.8} & \textbf{54.2} & \textbf{52.3} & 46.9 & \textbf{49.5}~\textcolor{red}{(+6.9)} & \textbf{49.3} & \textbf{58.0} & \textbf{56.1} & \textbf{52.8} & \textbf{54.1}~\textcolor{red}{(+4.7)}\\

  \bottomrule

\end{tabular}
}
\caption{Performance evaluation on COCO-20$^i$~\cite{lin2015microsoft}. $^\ast$~indicates that data augmentation used for training. Best results are shown in \textbf{bold}.}
\label{table:performance_coco}
    \end{minipage}
    \begin{minipage}{.32\textwidth}
\centering

\resizebox{\textwidth}{!}{

\begin{tabular}{@{}cc|cc@{}}
\toprule
\multirow{2}{*}{Backbone} & \multirow{2}{*}{Methods} & \multicolumn{2}{c}{mIoU} \\
 &  & 1-shot & 5-shot \\ \midrule
\multirow{5}{*}{ResNet50~\cite{Resnet}} & HSNet~\cite{HSNet} & {85.5} & 87.8 \\
 & HSNet~+~MSI & 87.5~\textcolor{red}{(+2.0)} &
 88.4~\textcolor{red}{(+0.6)} \\

  \cmidrule(lr){2-4}

  & VAT$^\ast$~\cite{VAT} & {{89.5}} & {90.3} \\
 & VAT~\cite{VAT2} & { {90.1}} & {90.7} \\

& VAT$^\ast$~+~MSI & 90.0~\textcolor{blue}{(-0.1)} & 90.6~\textcolor{blue}{(-0.1)} \\

\midrule

\multirow{5}{*}{ResNet101~\cite{Resnet}} & HSNet~\cite{HSNet} & 86.5 & 88.5 \\
& HSNet~+~MSI & 88.1~\textcolor{red}{(+1.6)} & 89.2~\textcolor{red}{(+0.7)} \\

\cmidrule(lr){2-4}

& VAT$^\ast$~\cite{VAT} & {90.0}  & {90.6} \\
& VAT~\cite{VAT2} & {90.3}  & {90.8} \\

& VAT$^\ast$~+~MSI & \textbf{90.6}~\textcolor{red}{(+0.3)} & \textbf{91.0}~\textcolor{red}{(+0.2)} \\

 \bottomrule
\end{tabular}
}
\caption{Performance evaluation on FSS-1000~\cite{FSS1000}.}
\label{table:performance_fss}
    \end{minipage}
  \end{table*}

\begin{table}[!htp]
\centering
\scalebox{0.6}{%
\begin{tabular}{@{}cc|c|c@{}}
\toprule
\multirow{2}{*}{Backbone} & \multirow{2}{*}{Methods} & \multicolumn{1}{c|}{1-shot} & \multicolumn{1}{c}{5-shot}  \\
&  &  mIoU &  mIoU    \\ \midrule
\multirow{4}{*}{ResNet50~\cite{Resnet}}
 & HSNet~\cite{HSNet} & 61.6  & {68.7} \\

   & HSNet~+~MSI & {66.0}~\textcolor{red}{(+4.4)} & {70.8}~\textcolor{red}{(+2.1)} \\

 \cmidrule(l){2-4}

   & VAT$^\ast$& 64.5 & 69.7 \\

  & VAT$^\ast$~+~MSI&  {67.8}~\textcolor{red}{(+3.3)} & {72.6}~\textcolor{red}{(+2.9)} \\

   \midrule

\multirow{4}{*}{ResNet101~\cite{Resnet}}  & HSNet~\cite{HSNet} & {64.1} & {70.3}\\

  & HSNet~+~MSI&  {67.3}~\textcolor{red}{(+3.2)} &  {72.3}~\textcolor{red}{(+2.0)} \\

 \cmidrule(l){2-4}

   & VAT$^\ast$& 66.8 & 71.4 \\

  & VAT$^\ast$~+~MSI&  \textbf{69.1}~\textcolor{red}{(+2.3)} & \textbf{74.2}~\textcolor{red}{(+2.8)} \\

 \bottomrule

\end{tabular}
}
\caption{Generalizability performance evaluation on PASCAL-5$^i$~\cite{pascal} after training with COCO-20$^i$~\cite{lin2015microsoft}. $^\ast$~indicates that data augmentation used for training. Best results are shown in \textbf{bold}.}
\label{table:performance_pascal_shift}

\end{table}

\subsection{Results}
\label{subsection:Results}
\noindent\textbf{PASCAL-$5^i$~\cite{pascal}:} Tab.~\ref{table:performance_pascal} compares the results of the proposed method (MSI) with baseline models on PASCAL-$5^i$. Our method provides improvement in almost all experiments. We observe that VAT~+~MSI provides noticeable gains in both 1-shot and 5-shot tests, particularly in 1-shot. In the 1-shot test, VAT~+~MSI provides a 2.8\% gain with ResNet50~\cite{Resnet} and a 2.1\% gain with ResNet101~\cite{Resnet} in mIoU.

\noindent\textbf{COCO-$20^i$~\cite{lin2015microsoft}:}
Tab.~\ref{table:performance_coco} reports results on the COCO$20^i$ dataset. HSNet~\cite{HSNet}, ASNet~\cite{ASNet}, and VAT~\cite{VAT} are selected and compared with the proposed method~(see Tab.~\ref{table:performance_coco}).
Our method delivers consistent improvement in almost all experiments. VAT~+~MSI provided a gain of 5.5\% with ResNet50~\cite{Resnet} and 6.9\% with ResNet101~\cite{Resnet} in mIoU in the 1-shot test, and delivers a gain of 4.1\% and 4.7\% in the 5-shot test.
HSNet~+~MSI improved by 6.3\% with ResNet50~\cite{Resnet} and 6.5\% with ResNet101~\cite{Resnet} in the 1-shot test. Moreover in the 5-shot, HSNet~+~MSI improved by 3.4\% and 3.1\% in mIoU with ResNet50~\cite{Resnet} and Resnet101~\cite{Resnet}, respectively.

\noindent\textbf{FSS-1000~\cite{FSS1000}:}
Tab.~\ref{table:performance_fss} compares the performance of our MSI on FSS-1000 using two baselines: HSNet~\cite{HSNet} and VAT~\cite{VAT2}. HSNet~+~MSI delivered 2.0\% and 0.6\% mIoU improvements on 1-shot and 5-shot tests, respectively, with ResNet50~\cite{Resnet}. HSNet~+~MSI  with ResNet101~\cite{Resnet} showed 1.6\% and 0.7\% in mIoU. VAT~+~MSI showed slightly inferior performance to VAT with ResNet50~\cite{Resnet}; however gains of 0.3\% and 0.2\% in mIoU can be seen on 1-shot and 5-shot tests with ResNet101~\cite{Resnet}.

\noindent\textbf{Generalization Test:} We tested the generalizability of trained models in Tab.~\ref{table:performance_pascal_shift}.
HSNet~\cite{HSNet} and VAT~\cite{VAT} were compared with HSNet~+~MSI and VAT~+~MSI, respectively.~All models were trained on COCO-20$^i$~\cite{lin2015microsoft} and then tested on PASCAL-5$^i$~\cite{pascal}. Both HSNet~+~MSI and VAT~+~MSI delivered significant mIoU gains on 1-shot and 5-shot tests. HSNet~+~MSI with ResNet50~\cite{Resnet} provided 4.4\% and 2.1\% improvements and VAT~+~MSI showed 3.3\% and 2.0\% gains.

\begin{table}
\centering
\scalebox{0.7}{%
 \begin{tabular}{c | c c c | c c}
 \hline
 Method \& Backbone & FM & STF & SIF & mIoU & FB-IoU \\
 \hline
  & &  & \checkmark & 60.7 & 72.0 \\
 & \checkmark &   &  & 65.3 (Baseline~\cite{VAT}) & 77.4 \\
 &  & \checkmark &  & 65.2 &  77.4\\
VAT~(ResNet50~\cite{Resnet}) & \checkmark & \checkmark &  & 66.3~(HM~\cite{hmmasking}) & 78.0 \\
 &  & \checkmark & \checkmark & \textbf{68.3} (Ours) & \textbf{79.1} \\
 & \checkmark &  & \checkmark & 65.7 & 77.5 \\
 & \checkmark & \checkmark & \checkmark  & 67.1 & 78.8 \\
 \hline

 \end{tabular}
 }
\caption{\small Ablation study of the different combination of features with VAT~\cite{VAT}~(ResNet50~\cite{Resnet}) on PASCAL-5$^i$~\cite{pascal}). \checkmark denotes that these features are utilized to generate super correlation maps~(SCM). Best results are shown in \textbf{bold}.}

\label{table:Ablation_study1}
\end{table}

\begin{table}
  \centering
    \begin{minipage}{.47\textwidth}
\centering
\resizebox{\textwidth}{!}{
\begin{tabular}{@{}cc|ccccc}
\toprule
\multirow{2}{*}{Backbone} & \multirow{2}{*}{Methods} & \multicolumn{5}{c}{1-shot}
\\
 &  & $20^0$ & $20^1$ & $20^2$ & $20^3$ & mIoU   \\ \midrule
\multirow{5}{*}{ResNet50~\cite{Resnet}}
& VAT$^\ast$~\cite{VAT}~(Baseline) & 67.6 & 71.2 & 62.3 & 60.1 & {65.3} \\
& VAT$^\ast$+MSI~(Feature Add.)& 68.5 & 71.0 & 60.8 & 62.0 & 65.6 \\
 & VAT$^\ast$+MSI~(Correlation Add.) & 67.0 & 68.2 & 53.5 & 56.0 & 61.2 \\

  & VAT$^\ast$+MSI~(Attention) & 69.9 & \textbf{72.5} & 63.4 & 63.8  & 67.4 \\

 & VAT$^\ast$+MSI~(Ours) & \textbf{71.0} & \textbf{72.5} & \textbf{63.8} & \textbf{65.9} & \textbf{68.3} \\

\bottomrule
\end{tabular}

}

    \end{minipage}
    \caption{\small Different methods of generating super correlation maps~(SCM) on PASCAL-5$^i$~\cite{pascal}. Best results are shown in \textbf{bold}. }
\label{table:Ablation_study2}

  \end{table}

\begin{table}
  \centering
    \begin{minipage}{.47\textwidth}
\centering
\resizebox{\textwidth}{!}{
\begin{tabular}{@{}cc|ccccc}
\toprule
\multirow{2}{*}{Backbone} & \multirow{2}{*}{Methods} & \multicolumn{5}{c}{1-shot}
\\
 &  & $20^0$ & $20^1$ & $20^2$ & $20^3$ & mIoU   \\ \midrule
\multirow{2}{*}{ResNet50 (PASCAL-5$^i$)}
 & VAT$^\ast$~\cite{VAT} & 59.9 & 42.4 & 42.0 & 45.5 & 47.5 \\
& VAT$^\ast$~+~MSI & \textbf{68.1} & \textbf{46.6} & \textbf{43.6} & \textbf{47.1} & \textbf{51.4}~\textcolor{red}{(+3.9)}  \\

 \midrule
\multirow{2}{*}{ResNet50 (COCO-20$^i$)}
& VAT$^\ast$~\cite{VAT} & 29.7 & 32.4 & 36.0 & 29.0 & 31.8 \\
 & VAT$^\ast$~+~MSI & \textbf{30.7} & \textbf{37.9} & \textbf{40.9} & \textbf{33.9} & \textbf{35.9}~\textcolor{red}{(+4.1)} \\

\bottomrule
\end{tabular}

}

    \end{minipage}
    \caption{\small Performance comparison on PASCAL-5$^i$~\cite{pascal} and COCO-20$^i$~\cite{lin2015microsoft} for cases where support masks occupy below 5\% of the support images. Best results are shown in \textbf{bold}. }
\label{table:Ablation_study3}

  \end{table}

\begin{table}
  \centering
    \begin{minipage}{.47\textwidth}
\centering
\resizebox{\textwidth}{!}{
\begin{tabular}{@{}cc|ccccc}
\toprule
\multirow{2}{*}{Backbone} & \multirow{2}{*}{Methods} & \multicolumn{5}{c}{1-shot}
\\
 &  & $20^0$ & $20^1$ & $20^2$ & $20^3$ & mIoU   \\ \midrule
\multirow{4}{*}{ResNet50~\cite{Resnet}}
&VAT$^\ast$~(w/o person in BG) & 67.7 & 70.9 & 61.3 & 61.6 & 65.2  \\
& VAT$^\ast$~\cite{VAT}~(w/ person in BG) & {67.6} & {71.2} & {62.3} & {60.1} & {65.3} \\

&VAT$^\ast$~+~MSI~(w/o person in BG) & 69.4 & 71.4 & 60.5 & 64.1 & 66.3  \\
& VAT$^\ast$~+~MSI~(w/ person in BG) & \textbf{71.0} & \textbf{72.5} & \textbf{63.8} & \textbf{65.9} & \textbf{68.3} \\

\bottomrule
\end{tabular}

}

    \end{minipage}
    \caption{\small Training VAT~+~MSI on PASCAL-5$i$\cite{pascal} without the person class existing as background of the support image.}

\label{table:Ablation_study4}

  \end{table}

\begin{figure*}[!htp]
  \centering
    \includegraphics[height=6.0cm]{figure/qualitative_comparison.png}
    \caption{Visual comparison of VAT~\cite{VAT} and VAT~+~MSI with ResNet50~\cite{Resnet} on PASCAL-5$^i$~\cite{pascal} and COCO-20$^i$~\cite{lin2015microsoft}.

    \label{fig:visual_comparison}
\end{figure*}

\subsection{Why MSI is useful?}
Fig.~\ref{fig:visual_comparison} draws visual comparisons between VAT~+~MSI and the baseline VAT~\cite{VAT}, which depends only on target features. Tab.~\ref{table:Ablation_study1} highlights the superiority of the VAT~+~MSI, which utilizes the entire features from the support image. These results indicate that using the entire support image improves results compared to using only the target information. Now, we describe four challenging FSS cases where MSI is effective (Fig.~\ref{fig:4cases}).

\begin{figure}[!htp]
  \centering

    \includegraphics[height=1.6cm]{figure/features.png}

   \caption{Comparison of VAT~+~MSI on PASCAL-5$^i$~\cite{pascal} with ResNet50~\cite{Resnet}~using different kinds of features in SIF: SIF with only target features, SIF with only background features, and SIF with the entire image's features.

   \label{fig:features}
\end{figure}

\noindent\textbf{When the same instance of the target class is not masked:}
Fig.~\ref{fig:features} visualizes different kinds of features in SIF using VAT~+~MSI on PASCAL-5$^i$~\cite{pascal}~(ResNet50~\cite{Resnet}): SIF with only target features, SIF with only background features, SIF with the entire image's features.
We observe that the support mask fails to mask an instance of the same class, i.e., the sofa on the right side. Therefore, SIF with target features are generated only from the upper sofa.
Next, SIF with background features are not much effective overall, even though they better segment the magnified part in the cyan box.
Finally, SIF with the entire image's features, which incorporate both the former and the latter features, allows accurate segmentation of the target object, both inside and outside the cyan box. This indicates that background features have target information and utilizing the features improves the performance of FSS.

\noindent\textbf{Support mask is very small:}
In PASCAL-5$^i$~\cite{pascal} and COCO-20$^i$~\cite{lin2015microsoft}, there are several cases where the support masks are very small because the target object only occupies a small part of the support image. To show that MSI is more effective than the existing methods on very small target objects, in both PASCAL-5$^i$~\cite{pascal} and COCO-20$^i$~\cite{lin2015microsoft}, we compare the mIoU for cases when the support mask occupies less than 5\% of the support image. Compared to VAT~\cite{VAT}, VAT~+~MSI displays noticeable gains of 3.9\% and 4.1\% (in mIOU) for PASCAL-5$^i$ and COCO-20$^i$, respectively.

\noindent\textbf{Support mask is missing some target boundary:}  We observe that in PASCAL-5$^i$~\cite{pascal}, COCO-20$^i$~\cite{lin2015microsoft}, and even FSS-1000~\cite{FSS1000}, the masks around boundaries can be imperfect (Fig.~\ref{fig:boundary_error}). By masking the features with an inaccurate mask, the boundary information of the target disappears inadvertently (Fig.~\ref{fig:boundary_error}). Therefore, compared to ours, previous methods face difficulties in finding the target in the query image with only limited target information.

\begin{figure}[t]
  \centering

    \includegraphics[height=2.05cm]{figure/boundary_error.png}

   \caption{Missing information around boundary on PASCAL-5$^i$~\cite{pascal}. The mask is not perfect to cover entire target object.}
   \label{fig:boundary_error}
\end{figure}

\noindent\textbf{When some background is helpful:} In some FSS cases, the background offers relevant context that is unavailable in the support mask, for accurate segmentation. However, many recent methods depend on support masks
to completely remove the background (Fig.~\ref{fig:background_info}). To empirically validate the relevance of background in some challenging cases, we train VAT+MSI and VAT~\cite{VAT} without the person class existing in the background of the support image ~(Tab.~\ref{table:Ablation_study4}). VAT+MSI shows a performance degradation when background information in the support set is unavailable. Compared to VAT+MSI, the degradation from VAT is very small because the background features are not utilized.

\begin{figure}[t]
  \centering

    \includegraphics[height=2cm]{figure/background_info.png}

   \caption{Comparison of performance when background information of support set is relevant to the background in query image.}

   \label{fig:background_info}
\end{figure}

\subsection{Ablation Study}

\noindent\textbf{Best features for SCM:} In Tab.~\ref{table:Ablation_study1}, we show results when using various combinations of features to obtain SCM. For this purpose, VAT~\cite{VAT} with ResNet50~\cite{Resnet} backbone is used as a baseline and PASCAL-5$^i$~\cite{pascal} is chosen for training and testing the model. FM refers to the feature masking utilized by several previous FSS models, obtained by masking background features. The ablation study reveals that harnessing both SIF and STF via concatenation achieves the best mIoU of 68.3\% among all others.
Please see the supplementary text for visualizations of the correlation value distributions for SIF and STF.

\noindent\textbf{On different ways of fusing SIF and STF:}
The proposed method, MSI, calculates cosine similarity between $\langle$STF, QF$\rangle$ and $\langle$SIF, QF$\rangle$, and concatenates the two generated correlation maps.
To see the effectiveness of this method, we experimented with different approaches to fuse SIF and STF to obtain SCM~(see Tab.~\ref{table:Ablation_study2}).
Feature addition refers to the element-wise addition of features before calculating cosine similarity to get correlation map.
Correlation addition means simply adding two correlation maps element-wise. The attention method merges two correlation maps through 1x1 depth-wise attention based on Attention~U-Net~\cite{attention_unet}. Among all design choices for fusion, concatenating the two correlation maps showed the best performance.

\noindent\textbf{Fast training convergence:} We have noticed that the proposed method (MSI) with VAT~\cite{VAT2} and ResNet50~\cite{Resnet} on PASCAL-5$^i$~\cite{pascal}, VAT~+~MSI, provides 3.3x faster convergence (i.e. to reach mIoU of 60\%) on average than VAT~\cite{VAT} ~(Fig.~\ref{fig:train_curve}), along with a remarkable improvement in performance. Similarly with HSNet, HSNet+MSI allows faster convergence  by 4.5x on average. See supplementary for plots.

\begin{figure}[t]
  \centering

    \includegraphics[height=3.3cm]{figure/train_curve3.png}

   \caption{Train. and val. profiles of VAT~\cite{VAT} and VAT~+~MSI on PASCAL-5$^i$~\cite{pascal} with ResNet50~\cite{Resnet}. VAT~+~MSI provides 3.3x faster convergence (to reach 60\% in mIoU) on average than VAT. Red circles indicate when train accuracy reaches 60\% in mIoU.}
   \label{fig:train_curve}
\end{figure}

\section{Conclusion}

We have proposed a new method, MSI, maximizing the information of the support set, deviating from the masking method~\cite{Zhang2020SGOneSG} widely used in many FSS models~\cite{PMM,PANet,PFENet,FWB,CANet,ASGNet,zhang2021fewshot,HSNet, VAT2,ASNet,BAM}. The joint exploitation of SIF and STF develops a synergy between them, which allows handling several challenging FSS cases. Results show significant performance improvements across all benchmarks upon recent, strong FSS baselines~(HSNet~\cite{HSNet}, ASNet~\cite{ASNet}, and VAT~\cite{VAT2}).

{\small
\bibliographystyle{ieee_fullname}