\section*{Appendices}

\section{Conclusion}
We presented PAN, a method of incorporating additional attribute annotations in image datasets to learn a better similarity predictor. We saw that PAN's method of decomposing similarity prediction into multiple conditions is general, functions with a range of different image encoders and is flexible in using attribute annotation, possibly sparse, when available. PAN outperformed state of the art on a diverse set of three tasks---by 4-9\% on fashion item compatibility prediction on Polyvore Outfits, 5\% on few shot classification on CUB and over 1\% Recall@1 on In-Shop Clothing Retrieval---contrary to prior approaches of using attribute supervision, which were unable to outperform methods that automatically learned concepts in different similarity conditions. In showing these benefits of PAN we factored out contributions from training parameters like batch-sizes, hopefully informing future work with our analysis.
\smallskip

\noindent\textbf{Acknowledgements:} This work was funded in part by DARPA and by the Hariri Data Science Faculty Fellowship.
\section{Datasets and Tasks}

\noindent\textbf{Polyvore Outfits~ \cite{vasileva2018learning}} contains 53K outfits (sets of fashion items) for training, 5K for validation and 10K for testing. It also provides fine-grained category information and text descriptions of items.  We use the 205 sparsely annotated attributes from \cite{plummerSimilarity} as labels for supervising similarity conditions. %
Evaluation involves two tasks.  First, in the fill-in-the-blank (FITB) outfit completion task a model is given a partial outfit and has to select from four possible answers what item would best complete it.  Performance is measured by how often the answer was correct. Second, in outfit compatibility a model is asked to discriminate between good and bad outfits. Performance is measured using area under a receiver operating characteristic curve (AUC). Following~\cite{vasileva2018learning,cucurull2019context}, outfit compatibility scores are computed by averaging the similarity prediction over all pairs of items in the outfit. There are 10K FITB questions and 10K each positive and negative samples for outfit compatibility (20K total) in the test split.

Since current methods get almost perfect performance on the original outfit compatibility task, we created a more challenging testing set of the same size by modifying the procedure outlined in \cite{vasileva2018learning}, that we refer to as the \emph{resampled} set. For outfit compatibility we collected new negative outfits by replacing only part of a ground truth outfit, unlike the original split which replaced all items.  We randomly selected the number of items to replace, and each item is replaced with another of the same type in the same split (\ie, a top could only be replaced with another top).   Similarly, we made a more challenging FITB task, where a model must select between 10 candidate answers (the original test had 4).  As with the original sampling, we ensured any replaced items and candidate answers were of the same type.
\smallskip

\noindent\textbf{CUB-200-2011 \cite{wah2011caltech}} consists of 200 classes and a total of 11,768 images of birds. We use the split provided by \cite{chen2019closer} for our experiments, which contains 100 base classes, 50 validation and 50 novel classes. The CUB dataset also has 312 fine-grained binary attributes labeled for each image, with an accompanying score on a 4 point scale indicating the confidence of the assigned label. We drop all attribute labels that have a confidence score less than or equal to 2.

We use the 5-way 5-shot classification task for evaluation. Reported accuracies are averaged over 3 training runs from different random initializations accompanied by 95\% confidence intervals. A test \emph{episode} consists of a random sample of 5 classes and 5 support images from the 50 classes in the novel split of the dataset. 16 query images, distinct from support images, are also sampled for each of these 5 classes. The accuracy for an episode is the 5-way accuracy of a classifier over the 16 x 5 = 80 query images. A few shot learning model is evaluated using its average classification accuracy over 600 randomly generated test episodes.
\smallskip

\noindent\textbf{In-Shop Retrieval \cite{liu2016deepfashion}} contains 52,712 images of clothes from 11,967 classes. There are 14,218 query images and 12,612 gallery images for testing. Given a query image, the task is to retrieve an image of the same item from the gallery set. Note that the query and gallery sets do not overlap with the training set. There are 463 attributes of clothes in total, we use these attribute labels for our PAN-Supervised model. Methods are ranked based on Recall@1.

\section{Results} \label{subsec:results}

\subsection{Comparison with prior work}
Table \ref{tab:compare_prior_work_po}, Table \ref{tab:compare_prior_work_cub}, and Table \ref{tab: inshop_results} compare the best settings (encoder, number of unsupervised similarity conditions, \etc) used by our model %
to representative state-of-the-art results reported in prior work on Polyvore Outfits, CUB, and In-Shop Retrieval, respectively. As shown in Table~\ref{tab:compare_prior_work_po} we obtain a 4\% better FITB accuracy and 9\% AUC boost over the state-of-the-art on the fashion compatibility task using our more challenging resampled test set for both tasks, while also increasing FITB accuracy by 8\% on the original split.  Similarly, in Table~\ref{tab:compare_prior_work_cub} and Table \ref{tab: inshop_results} we observe a 5\% and 1\% performance improvement over the state-of-the-art on fine-grained few shot classification and In-Shop Retrieval, respectively. Improvement over the diverse set of tasks demonstrates PAN's ability to generalize.  Our PAN model can also be useful when no supervision is provided, as our PAN-Unsupervised model obtains a 3-4\% gain over prior work on Polyvore Outfits and CUB, while also boosting performance on In-Shop Retrieval.  Note that fashion compatibility benefited from using a graph image encoder (GE), while few-shot classification reported best performance with a CNN encoder, which we shall discuss further in Section~\ref{subsec:context}.

In addition to comparison with prior work, Table \ref{tab:compare_prior_work_po}, Table \ref{tab:compare_prior_work_cub}, and Table \ref{tab: inshop_results} also provide two alternative methods of using attributes in an image similarity model.  In ``X + Attr.\ Multitask'' we use a hard parameter sharing multitask approach~\cite{caruanaMULTI-task}, where we share an image encoder, but have separate output heads for each of the two tasks (one of them being attribute classification, the other similarity link prediction). In ``Attr.\ Similarity'' we use a standard framework where we predict attributes for each image and then learn a classifier implemented as a fully connected layer that takes both attributes as input and predicts similarity (the general framework used by~\cite{farhadi2009describing}).  Notably, both baseline methods that use attributes only improve performance on few-shot classification, but either make no difference, or are even harmful to performance on the other two datasets (\eg, Attr. Similarity \emph{reduces} FITB performance by 5-8\% compared with the CGAE baseline).  In contrast, our PAN-supervised model outperforms all other methods, including on the fashion compatibility task where we report a staggering 6-17\% boost over the attribute baselines on the resampled test set.
\smallskip

\input{tables/po_results}

\input{tables/cub_results}

\input{tables/inshop_results}

\begin{figure}[t]
    \centering
    \begin{subfigure}[t]{0.5\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/plot_lam_polyvore_testval.pdf}
    \caption{FITB accuracy of PAN-Supervised-GE on Polyvore Outfits.}
    \end{subfigure}
    \begin{subfigure}[t]{0.5\textwidth}
    \centering
    \includegraphics[width=\linewidth]{figures/plot_lam_testval.pdf}
    \caption{5-way 5-shot classification accuracy of PAN-Supervised on CUB}
    \end{subfigure}
    \caption{Sensitivity to $\lambda$ from Eq.~(\ref{eq:total_loss}). }
    \label{fig:plot-lam}
    \vspace{-5mm}
\end{figure}

\input{tables/analysis}

\input{tables/att_combn_type}

\subsection{Choice of image encoder and batch size}
\label{subsec:context}

Prior work has been inconsistent in its training methods and controlling for hyperparameters like batch size, which can significantly affect performance.  Table~\ref{tab:analysis} remedies this by comparing training with the whole training split vs. using minibatches. It also compares the effect of using a graph encoder (GE) instead of a simple CNN. We see comparing the numbers of row 3 of Table \ref{tab:analysis} to those of prior methods in Tables \ref{tab:compare_prior_work_po} and \ref{tab:compare_prior_work_cub} that training with the whole training set can significantly improve performance, making even a simple Siamese network trained using a triplet loss outperform most recent methods on both tasks. We note here that in our experiments with single batch training, we use a pre-trained CNN to extract image features, and do not finetune it.

It is also notable that using GE performs worse than the simpler Siamese Network baseline on CUB, which we believe is due to differences between tasks. Specifically, in fashion compatibility links exist between items that may be very different from each other. Thus, the additional context provided through the GE may be more important than in CUB, which has less variation between linked items since they all contain the same bird.

\subsection{Choice of attribute combination $f_a$}
\label{subsec:logical_function}

As discussed in Section \ref{subsec:def-similarity-conds}, choosing $f_a$ as one of common logical functions can lead to different interpretations of the attribute supervision provided.
Table~\ref{tab:att_comb_type} compares these functions that convert image attribute labels to pairwise labels for use in our PAN model.

At first glance, predicting $1$ when either both images have an attribute or neither of them does (\ie using $XNOR$) seems like an intuitive choice. This would indicate how many attributes match between two images. %
However, some functions like fashion compatibility, where images may match \emph{because} they contain different attributes, $XNOR$ would not be appropriate since any non-shared attributes would be ignored (\ie have a supervision label $0$). $AND$, which encourages models to predict $1$ only when both images have the attribute has same issue. However $OR$ would not face the issue since it can be $1$ when either of the images has an attribute, allowing the model to use its relevance weights to decide whether a combination of attributes is relevant for similarity. In Table \ref{tab:att_comb_type}, we see $OR$ perform the best on fashion compatibility on Polyvore Outfits.

Notably, we also see that $OR$ is still competitive with $XNOR$ on the CUB dataset, where the goal is to determine similarity in the more conventional sense, \ie, similar images should have matching attributes. This task seems like a good fit for $XNOR$.
However, on CUB, many of the attributes are mutually exclusive (\eg, a bird either has a ``red beak'' or a ``black beak'', but not both).
If a model can reason jointly about dissimilar attributes inferring that they should not coexist, it can correctly function on this task. Relevance weights in our model allow for this joint reasoning over different attributes.
Thus, training the model to predict $1$ when either image has an attribute ($OR$ supervision), can also perform well on this task, and from the empirical results in Table \ref{tab:att_comb_type}, we see that it does.

Providing both the $AND$ and $XOR$ outputs (as a concatenation) seems lucrative since it seems more informative than $OR$, but we found that the model uses its additional capacity to overfit to training data. This is also challenging to use since attribute predictions are noisy (see Appendix \ref{sec:attr_recog} for attribute recognition performance).

The In-Shop retrieval task involves fetching matches from a gallery of different views of an object. $XNOR$ is ideal for matching different views in this case, since matching attributes can be directly translated to the inference that two views belong to the same object, therefore, are similar.

\subsection{PAN sensitivity to $\lambda$}
\label{subsec:lambda}
Figure \ref{fig:plot-lam} shows PAN's sensitivity  to different weights ($\lambda$) of the attribute supervision loss. Performance is plotted for both the testing and validation sets. On Polyvore Outfits, we see that the PAN-Supervised-GE model performs well when attribute supervision weight is relatively high (best accuracy at $\lambda=10$). Model performance decreases on either side of this, with a heavier decrease when $\lambda$ is increased significantly (to $100$). On the CUB dataset, we see a somewhat different behavior where the best model performance is achieved at $\lambda=10^{-5}$, which is much lower, indicating relatively lower attribute supervision is optimal for the task.  Note that on the In-Shop task we set $\lambda=1$ and did not tune it on that task, demonstrating that PAN can be readily adapted to improve performance on other tasks/models.

\section{Introduction}

\begin{figure}[t!]
    \centering
    \includegraphics[width=0.95\linewidth,trim=0cm 0cm 0cm 0cm,clip]{figures/intro_fig_v2.pdf}
    \caption{In prior work (\eg~\cite{koh2020concept,plummerPLCLC2017,vasileva2018learning,yang2019interpretable}), shown in (a), attributes used for image similarity are predicted for each image and then are used as input to the image similarity model.  However, this can result in loss of important information about how attributes are expressed (\eg, different shades of the attribute \emph{yellow breast}).  Thus, in our work, shown in (b), we avoid this loss of information by using a joint representation of the two images to compute multiple disentangled similarity scores, each corresponding to an attribute, and relevances of each similarity score in the final similarity prediction. This allows for more fine-grained reasoning about different attribute manifestations, boosting performance.}
    \label{fig:comparison}
\end{figure}

\begin{figure*}[t]
    \centering
    \includegraphics[width=0.8\linewidth]{figures/pipeline_v2.pdf}
    \caption{\textbf{PAN overview.}  Given a pair of images, the goal of PAN is produce its similarity score.  We begin by using the image encoder to generate feature vectors for input images. %
    The image features are then fed into the Concept-conditioned Similarity Module (CSM) that uses these features to generate a set of similarity scores with corresponding relevance weights. This enables PAN to identify that two images do contain the same attribute, but that they are not relevant to the similarity score since they are different manifestations of the attribute (see Fig~\ref{fig:comparison} for an example).  The final similarity score $p \in [0, 1]$ is produced using a weighted combination of the similarity conditions and their relevance. Note that the different colored lines (blue, pink) represent information flow pertaining to individual images.  %
    }
    \label{fig:pipeline}
\end{figure*}

Learning similarity metrics between images is a central problem in computer vision with wide-ranging applications such as face recognition \cite{liu2017sphereface,schroff2015facenet}, image retrieval \cite{gordo2016deep,noh2017large,yang2019efficient}, prototype based few shot image classification \cite{koch2015siamese,snell2017prototypical,sung2018learning,vinyals2016matching}, continual learning of image classification \cite{castro2018end,rebuffi2017icarl,stojanov2019incremental}, and fashion compatibility or recommendation  \cite{cucurull2019context,tan2019learning,vasileva2018learning,veit2017conditional,veit2015learning,zhang2018become}. There has been a recent trend of learning these metrics by decomposing the problem into multiple axes of similarity or \emph{similarity conditions}, which has improved performance on a variety of tasks~\cite{HeICDM2016,LIN_2020_CVPR,plummerHint2019,plummer2018conditional,plummerPLCLC2017,tan2019learning,vasileva2018learning,veit2017conditional}. Generally speaking, methods that automatically learn what these conditions represent~\cite{plummer2018conditional,tan2019learning} have reported better performance than those that predefine this knowledge using information like labeled image attributes and item categories~\cite{LIN_2020_CVPR,veit2017conditional,vasileva2018learning,plummerPLCLC2017}.
We argue this is primarily due to prior work using attributes to predict their presence on single images (\eg~\cite{koh2020concept,yang2019interpretable,vasileva2018learning,plummerPLCLC2017}), and subsequently using these predictions for predicting similarity (Fig \ref{fig:comparison} (a)). This incurs a loss in information about the different manifestations of an attribute, differences that could affect similarity prediction, but may not be distinguishable in attribute annotations. While this could be addressed by collecting a complete set of annotations of every possible attribute and their different manifestations that could affect similarity, such a collection would be expensive.  In addition, it is often impossible to articulate every fine-grained attribute that may affect similarity.

In this paper, we introduce a Pairwise Attribute-informed similarity Network (PAN) that effectively learns to use supervisory information in the form of attribute labels, avoiding information loss, to create a powerful image similarity model that performs well on a range of diverse tasks.
To illustrate how we do this we refer to the example in Fig \ref{fig:comparison}(b). The figure shows two birds (of different categories) from the Caltech-UCSD Birds (CUB) dataset \cite{wah2011caltech}, where they are both labeled positively for the binary attribute \texttt{has\_breast\_color::yellow}, indicating that they have yellow breasts. Prior work (\eg~\cite{koh2020concept,plummerPLCLC2017,vasileva2018learning,yang2019interpretable}) directly predicts attributes for each image,  which tends to lose information about subtle differences in the manifestations of the attributes, like the shades of the color yellow.
Our PAN model avoids this issue by first comparing images in a feature space rather than attribute space, as illustrated in Fig~\ref{fig:comparison}(b). Using the joint image features it then predicts both a similarity score and a relevance for different similarity conditions defined by the attributes. Even when the similarity score may coarsely indicate that the two images are similar since they have the same attribute, the model can pick up on finer attribute differences and decide that the mere presence of the same attribute is of low relevance to a positive similarity prediction.
As our experiments will show, this difference can make a dramatic impact on the performance of the learned image similarity model.

One major challenge we face is the considerable difference in how attributes relate to the similarity functions that arise in different tasks.  For example, in few-shot classification, where we use the labeled support images in a nearest-neighbors classifier, the goal of a visual similarity classifier would be to simply measure similarity by matching attributes between the test and support images.  In contrast, for tasks like fashion compatibility, where two images are deemed similar if they complement each other when worn together in an outfit, image pairs with different attributes (\eg, black and orange) can indicate they are highly compatible.  Furthermore, simply modeling which attribute pairs indicate compatibility is insufficient, since two attributes which often result in compatible pairs could be deemed incompatible depending on the other attributes that are present.  For example, \emph{black} and \emph{orange} items are often compatible, except when some other attributes like \emph{red} are also present.  Thus, visual similarity models must learn a far more complex set of relationships between attributes when learning fashion compatibility.  These differences mean that methods that do well on few-shot classification often perform poorly on fashion compatibly and vice-versa. PAN, however, can take this into account via the method we use to convert the incomplete attribute labels for single images into supervisory signals for image pairs and improve performance across diverse tasks.

As we will discuss in Sec \ref{sec:model} (and illustrated in Fig \ref{fig:pipeline}), PAN naturally allows for training and automatically learning similarity conditions in the absence of any additional data like attributes. We also find that PAN can improve performance even in cases where only sparse attribute labels are available.

Summarizing our contributions:
\begin{itemize}
    \item We propose a Pairwise Attribute-informed similarity Network (PAN), which incorporates fine-grained attribute information during training based on a joint representation of two images enabling us to avoid the loss of information suffered by prior work.
    \item While prior methods of incorporating attribute information under-perform prior art, PAN outperforms them on three diverse tasks---by 4-9\% on fashion item compatibility prediction on Polyvore Outfits~\cite{vasileva2018learning}, 5\% on few shot classification on CUB \cite{wah2011caltech} and over 1\% Recall@1 on In-Shop Clothing Retrieval~\cite{liu2016deepfashion}, demonstrating PAN's generality. In comparison to prior approaches of incorporating attribute supervision, PAN is better by a wider margin, \eg, it outperforms them by a sizeable 6-17 \% on Polyvore Outfits.
    \item We propose different methods of using attributes for supervising predictions along similarity conditions, delving into the interpretations of each, providing insights for their applicability in different tasks.
    \item Our analysis also outlines the contributions of the training procedures, specifically training batch-size. This has commonly been overlooked in prior work, but could have significant impact on final model performance. In doing so, we factor out contribution of the training procedure in demonstrating PAN's benefits.
\end{itemize}

\section{Pairwise Attribute-informed similarity Network (PAN)} \label{sec:model}

 Given two images $\boldsymbol x_1$ and $\boldsymbol x_2$, the goal of our PAN model is to output a score in $[0, 1]$ representing the probability that the two images are similar.  The primary contribution of our approach lies in its Concept-conditioned Similarity Module (Sec~\ref{subsec:dec-desc}), which takes features  $\boldsymbol h_1, \boldsymbol h_2 \in \mathbb{R}^d$ representing the input images (computed using an image encoder from Sec~\ref{subsec:enc-desc}), and predicts their similarity $p$ using a weighted combination of predictions along multiple axes of similarity $\boldsymbol\rho$ and their relevance $\boldsymbol\omega$.  As discussed in the introduction, learning relevance weights over the attribute defined similarity conditions can help us selectively ignore them when said attributes are present, but are not quite relevant in determining similarity between the images.

As our experiments demonstrate, the similarity conditions can be unsupervised as in prior work~\cite{plummer2018conditional,tan2019learning}, but the aim of PAN is to learn to supervise these conditions so they represent a particular concept (\ie, a particular attribute).  These attributes can represent any concept that has been annotated in a dataset.  For example, for CUB~\cite{wah2011caltech} these represent parts of birds (\ie, the type of beak or tail features).  In Polyvore Outfits~\cite{vasileva2018learning} and InShop Retrieval~\cite{liu2016deepfashion} these attributes can contain low-level concepts like colors and textures as well as high-level concepts like ``formal'' and ``fashionable''.  We discuss how we convert the attributes/concepts, which are annotated per image, into labels for image pairs in Sec~\ref{subsec:def-similarity-conds}. Fig \ref{fig:pipeline} provides an overview of our approach.

\subsection{Concept-conditioned Similarity Module (CSM)} \label{subsec:dec-desc}

Given features $\boldsymbol{h}_i, \boldsymbol{h}_j$ $\in \mathbb{R}^{d}$ of two images, our Concept-conditioned Similarity Module (CSM) generates a set of $M$ similarity scores  $\boldsymbol{\rho} = [\rho_1, \ldots, \rho_M] \in \mathbb{R}^M$ and corresponding relevance weights $\boldsymbol{\omega} = [\omega_1, \ldots, \omega_M] \in \mathbb{R}^M$ which represent the importance of each similarity condition to the final similarity score:

\begin{align}
    \boldsymbol{\rho} &= \sigma \left( \boldsymbol{W}_1^\top |\boldsymbol{h}_i - \boldsymbol{h}_j| + \boldsymbol{b}_1 \right) \\
    \boldsymbol{\omega} &= \text{softmax} \left( \boldsymbol{W}_2^\top |\boldsymbol{h}_i - \boldsymbol{h}_j| + \boldsymbol{b}_2 \right)
\end{align}
where $M$ is the number of distinct similarity conditions, $|\cdot|$ represents an element-wise absolute value, and $\sigma(\cdot)$ an element-wise sigmoid function. $\boldsymbol{W}_1, \boldsymbol{W}_2 \in \mathbb{R}^{d \times M}$, and $\boldsymbol{b}_1, \boldsymbol{b}_2 \in \mathbb{R}^{M}$ are learnable parameters.  Note that $\boldsymbol\rho$ is supervised using attribute labels, but the relevance scores $\boldsymbol\omega$ are treated as a latent variable and automatically learned.
The final similarity score $p \in [0, 1]$ is calculated as the sum of similarity conditions weighed by their relevance, \ie,
\begin{align}
    p = \sum_{m=1}^{M} \rho_m \omega_m = \boldsymbol{\rho}^\top \boldsymbol{\omega}.
\end{align}

Note that prior work that predicted multiple similarity conditions in both the supervised and unsupervised setting did so based off the features of a single image (\eg,~\cite{LIN_2020_CVPR,plummerHint2019,vasileva2018learning,veit2017conditional}). In contrast, CSM predicts these conditions off a joint representation of both images.  As we show in Appendix \ref{sec:more_questions}, this results in a significant boost to performance when combined with relevance scores.  We believe this is due, in part, to the fact that this joint representation makes it easier to identify differences in attribute manifestations (due to taking the difference of features for the two images). Thus, our approach can more accurately identify when to ignore attribute predictions.

\subsubsection{Defining Similarity Conditions} \label{subsec:def-similarity-conds}

Depending on the availability of labelled attributes for the images, we can choose to supervise similarity conditions to give them semantic meaning. This choice results in two kinds of similarity conditions as described below:
\smallskip

\noindent\textbf{Unsupervised similarity conditions.}  Similarity conditions are treated as latent variables as done in \cite{tan2019learning}.  The benefit of this approach is that it requires no additional annotations.  Note that all the conditions we predict are based on a joint representation of the two images, unlike in \cite{tan2019learning} where they were computed per-image.
\smallskip

\noindent\textbf{Supervised similarity conditions.} Unsupervised similarity conditions need no attribute annotations. However, we would expect that with some expert knowledge of what might be important in the image, as provided with attributes, we can improve performance using these attributes effectively. Hence, rather than treating each similarity condition as a latent variable, supervised similarity conditions are trained to reflect a specific concept. Since attribute annotations are defined per image, and we predict attributes based off a joint representation, we convert these labels to represent both images, as described next.

Suppose the images have $M$ labeled binary attributes. Each image $i$ is then accompanied with an $M$ dimensional vector $\boldsymbol{a}_i \in \{0, 1\}^{M}$. For a pair of images $i$ and $j$, we can use a function $f_{a} : \{0, 1\} \times \{0, 1\} \rightarrow [0, 1]$ to get an $M$ dimensional vector $\boldsymbol{a}_{i, j} = f_{a}\left( \boldsymbol{a}_i, \boldsymbol{a}_j \right)$. Elements of $\boldsymbol{a}_{i, j}$ can then be used as labels for supervising the similarity conditions in the model output scores $\boldsymbol{\rho}$. Note that if there are missing entries in $\boldsymbol{a}_{i, j}$ because of missing attribute labels, these can be handled by zeroing out the loss resulting from them using a binary mask over the indices of $\boldsymbol{a}_{i, j}$. %

In Section~\ref{subsec:logical_function}, we experiment with common logical functions as $f_a$, which map to clear semantic meanings. For instance, using $f_a = logical~AND$, with the similarity score $\boldsymbol \rho$, the model is asked to predict whether a given attribute appears in both images. Similarly with $OR$, model predicts whether the attribute is in either image, with $XOR$ it predicts whether it is exclusively in one image, and with $XNOR$ it predicts whether the attribute is in both images or in neither. A more detailed discussion of opting for these 4 choices for $f_a$ is in Appendix \ref{sec:more_questions}.  In practice for a given dataset, a logical function can be selected using some prior knowledge about how the attributes relate to the similarity score, or can be selected empirically using held-out data.

\subsection{Types of Image Encoder} \label{subsec:enc-desc}

As mentioned previously, the image encoder generates a lower dimensional feature representation $\boldsymbol h$ for an image $\boldsymbol x$. We experiment with three different image encoders.

\smallskip

\noindent\textbf{Convolutional Network.}  Unless specified otherwise, we use a simple convolutional neural network (CNN), specifically a ResNet~\cite{he2016deep} to obtain our image feature representation (details in the Appendix \ref{sec:implementation_details}).
\smallskip

\noindent\textbf{Graph Encoder (GE)~\cite{cucurull2019context}.} For some image similarity tasks like fashion compatibility, context can be an important cue in determining how similar two items are. Thus, the second encoder we explore is a graph convolutional network (GCN) that operates on features extracted by a CNN. The GE (composed of the CNN and GCN) takes in as input both images from a dataset and an adjacency matrix over them and simultaneously generates features for all images. This encoder was also used by Cucurull~\etal~\cite{cucurull2019context} and we refer readers to Section 3.1 of their paper for complete details.
\smallskip

\noindent\textbf{ProxyNCA++~\cite{wern2020proxynca++}.} Many tasks may also find context unhelpful or that GCNs may be too computationally expensive to use~\cite{LIN_2020_CVPR}.  For example, in retrieval tasks a particular emphasis is placed on speed as methods may have to search through millions of images in order to locate the desired item.  As such, for our last encoder we use a state-of-the-art retrieval method when evaluating on the In-Shop Retrieval task~\cite{liu2016deepfashion}. ProxyNCA++ at its core learns a distance metric between images based on learning proxy feature representations for each class. Consequently, it relies on annotated categories for images, and cannot directly be applied for similarity metric learning where no such annotation is available. We refer readers to \cite{wern2020proxynca++} for complete details of this encoder.

\subsection{Model Objective and Training} \label{subsec:model-objective}

The final objective function on a pair of images $\boldsymbol x_i$ and $\boldsymbol x_j$ is then defined as:
\begin{align}
    \mathcal{L}(\boldsymbol{x}_i, \boldsymbol{x}_j, e_{i, j}, \boldsymbol{a}_{i, j}) = \mathcal{L}^{BCE} (e_{i, j}, p) + \lambda \mathcal{L}_{el}^{BCE} (\boldsymbol{a}_{i, j}, \boldsymbol{\rho}),
    \label{eq:total_loss}
\end{align}
where $\lambda$ is a tunable hyperparameter, $e_{i, j} \in \{ 0, 1 \}$ is the ground truth similarity label between images $\boldsymbol x_i$ and $\boldsymbol x_j$, $\mathcal{L}^{BCE}$ is the binary cross-entropy loss and $\mathcal{L}_{el}^{BCE}$ is the mean element-wise binary cross-entropy. Note that when there are no supervision attributes, the second term in Equation \ref{eq:total_loss} is $0$. For training, an equal number of positive and negative pairs are sampled randomly from the training split and the model is trained to predict similarity between them. Details regarding the exact process for each encoder are in Appendic \ref{sec:implementation_details}.

\section{Related Work}

\noindent\textbf{Visual Similarity Learning.} Learning visual similarity can be used for a wide range of visual tasks. A couple widely used evaluation tasks for similarity learning are face recognition \cite{chopra2005learning,liu2017sphereface,nguyen2010cosine,schroff2015facenet} and image retrieval \cite{noh2017large,gordo2016deep,yang2019efficient}, the latter itself subcategorized multiple ways depending on type of images involved, \eg fashion images, natural images. Image retrieval has a fairly direct application in e-commerce.
A major portion of this industry consists of websites selling clothing and accessories, giving rise to the challenging task of predicting fashion compatibility \cite{veit2015learning, kang2017visually, vasileva2018learning}, which, as discussed in the introduction, is a form of visual similarity different from the conventional notion. However, we use similarity and fashion compatibility interchangeably as they are addressed in the same manner in our paper.

Some prior visual similarity learning approaches focused on learning a single similarity space~\cite{veit2015learning,kang2017visually,han2017learning}. More recent work \cite{veit2017conditional,plummerHint2019,vasileva2018learning,tan2019learning,LIN_2020_CVPR} found that learning multiple \emph{similarity conditions}, each capturing a different concept, performs better. Concurrent to some of these, the role of using contextual information in image encodings was discussed by Cucurull~\etal~\cite{cucurull2019context}. They used a graph convolutional network (GCN) on images, with similarity links defining the graph structure, to achieve state of the art performance in fashion compatibility prediction.

Up until now, methods using predefined similarity conditions~\cite{plummerHint2019,veit2017conditional,vasileva2018learning} have underperformed methods that automatically learned these conditions~\cite{tan2019learning,LIN_2020_CVPR}.  Via PAN, we show a method of incorporating additional attribute annotations to supervised similarity conditions while improving final task performance, breaking this trend.
\smallskip

\noindent\textbf{Few Shot Learning.}
Given the cost associated with acquiring human-annotated labels, learning with few labelled examples is well sought after in computed vision, with a range of prior work exploring possible solutions~\cite{miller2000learning,li2006one,lake2011one,koch2015siamese}. Given a few examples with labeled classes, \cite{vinyals2016matching} classified novel examples using attention weights to compute a probability distribution over known classes. They introduced an episodic training paradigm,
later also adopted by \cite{ravi2016optimization,snell2017prototypical}. Each episode in an N-way K-shot classification task is a sample of N classes, with K images from each class available to the learning algorithm as ``labeled'' or ``support'' examples for learning few shot classification. Each episode is also accompanied by some query examples which the learner is supposed to predict class labels for. For training, the learning algorithm is provided episodes sampled from a base training dataset, and it is typically evaluated on test episodes sampled from a novel split of the dataset, containing classes different from those appearing in the base split. Both \cite{vinyals2016matching} and \cite{snell2017prototypical} adopted a strategy of minimizing a distance metric in feature space between query images and a prototypical support vector for training. Subsequent work \cite{sung2018learning} improved few shot classification performance by learning a parametric distance measure rather than using a closed-form metric without learnable parameters.

Another body of work uses meta-learning \cite{finn2017model,Qiao_2019_ICCV,Sun_2019_CVPR} to initialize a classification model's parameters allowing for fast adaptation to a new few-shot task using just a few parameter updates. \cite{chen2019closer} performed a thorough study of several recent few-shot learning methods and proposed a strong baseline classifier using cosine similarity and data augmentation. %
Some recent methods have also employed graph neural networks for few shot classification~\cite{garcia2018fewshot,Kim_2019_CVPR,Schonfeld_2019_CVPR}.

Like \cite{veit2017conditional, vasileva2018learning, tan2019learning}, PAN lies in the space of decomposing similarity prediction into multiple similarity conditions, but unlike them, PAN can use attribute supervision when available to supervise these conditions and improve performance. PAN is a general similarity learning approach and its benefits with multiple similarity condition spaces and attribute supervision can be seen with different image encoders, be it a simple CNN or a contextual GCN encoder similar to that used by \cite{cucurull2019context}, and on different tasks, like fashion compatibility and few shot classification.


\section{Introduction}
Urban design solutions determine many important aspects of urban life: economical, ecological, social aspects, transportation, pedestrian flows, consequently, multiple parameters are involved. The necessity to consider many different factors interacting with each other made this problem a challenge in the field of generative design. The result of an urban project has a long-lasting effect on various spheres of life, which makes finding an optimal solution even more important. The issue arises when it is necessary to define a set of parameters determining an optimal solution for a given area. Some of them are dictated by the local regulations, some by common knowledge, some by the sense of aesthetics; sometimes it is difficult to formulate them using words while algorithms would require formulae or numeric representation.

One of the problems involving a manifold of parameters is a design of an urban block. This problem takes into consideration the shapes of the buildings, their proportions, relationships with the adjacent buildings, the composition of the open space, relationships of buildings with the open space, with the urban block itself, relationships with the buildings of the adjacent blocks. Block design involves the knowledge of the typologies of the blocks in a city, surroundings (streets, squares, railways nearby), etc. Moreover, it should smoothly integrate into the existing urban context, which means taking into consideration the disposition and typology of the adjacent blocks. All of the parameters mentioned above would be difficult to define for each block a design should be produced to.

With the recent introduction of Generative Adversarial Networks (GANs)\cite{goodfellow2014generative}, it became possible to use the neural networks' ability to autonomously learn visual features in order to generate new content. The main advantage of this technique applied to generative design is the elimination of the necessity to determine any parameters of the desired design solution and learning the visual features determining the existing city structure and consequently generating blocks imitating this structure.

\section{Related Work}
As the concept of GANs has been introduced quite recently\cite{goodfellow2014generative} there has not been much research on the topic concerning architecture and urban design, although it is well-known in the field of computer science.

Prior work in the generative urban design field is mostly based on a set of defined rules and constraints according to which several design options are generated\cite{rule_based_design}. Nagy, Villaggi, and Benjamin\cite{inproceedings} explore optimization of the problem complexity with the set of two principal goals and the use of genetic algorithms. However, their approach, too, needs a set of constraints and requirements; their model outputs a design space out of which, after optimization, only a few designs are chosen as final.

The use of GANs has been explored in interior design as well as in visualization\cite{nvidia} and simulation. Stanislas Chaillou\cite{archigan} explored the functionality of the method applied to the design of the apartments generating their shape, internal structure, functional zoning, and furniture placement. The same approach at the same architectural scale\cite{daylightgan} was used for the simulation of natural illumination in an apartment provided its plan. Another work published in 2020 used a GAN with a different architecture with respect to ours for landuse planning\cite{landuseai} with the use of predefined parameters that a planner considers important for his solution and the zone design. However, this approach produces only a large-scale distribution of the functional zoning in the city and is not applicable to the urban design.
\newline The present work tackles a different scale problem: urban block level design. Instead of using a set of rules exploited by the generative urban design models previously, the GAN learns the set of parameters present in the city block design from the visual features of the images. A work that uses GANs in architecture on a scale that is similar to ours is the project This Map Does Not Exist\cite{thismapdoesnotexist}. However, this work uses a different architecture that does not allow the production of the designs based on the existing input: the maps are not attached to any existing point in the real world and the existing conditions of an urban environment are not taken into account when producing an image. One more work\cite{ntg} uses generative networks for the generation of an urban street layout. This approach is the closest to our work both in terms of urban scale and the possibility of conditional generation. The problem is explored from a different point of view: the authors use graph generation to produce a road design while our research tackles the generation of the urban blocks.
\newline Importantly, our research combines both the challenges faced in the previously mentioned works using GANs: production of a design solution based on the chosen parameters of the surrounding environment and the integrity of the solution within its urban block.

\section{Data sampling and acquisition}
Firstly, the data of the city of Milan was used to make a dataset. This city has been chosen due to its characteristic morphology and relatively consistent urban fabric inside the external ring road and within a 1 km distance from it.

The dataset consists of images that are diagrammatic representations of the city zones. It has been decided to start with the basic parameters determining the composition of a building block: building footprints, building heights' categories, road structure, and railways. The heights of the buildings have been divided into 3 ranges: low-rise structures, medium-rise structures, and high-rise structures depending on the distribution of the heights in the city. Heights in the diagrammatic representation are defined as three gray values. In case of height not having been mentioned for a building, a mean value for the city was used to denote its height.

The dataset has been generated with the use of QGIS software \cite{QGIS_software} with the condition to have more than 12 percent of a block of interest occupied. Urban blocks containing exclusively industrial buildings have been excluded from the datasets as their presence introduces additional variance. Since the information about the landuse and the building type is not provided to the algorithm the experiment is concentrated on the buildings with residential and mixed functions. Moreover, the proportions of the industrial buildings are similar across different geographical regions and do not introduce any characteristics typical to a specific city.

The images were produced in scale 1:3000 in RGB colorspace; these images were scaled down from 2100x2100 to 256x256 pixels to be fed into the model.

The dataset consists of two sets: set B includes the diagrammatic representations of the city and set A holds the same images with an empty polygon in the center of the image, destined to be filled with the buildings as in Figure~\ref{fig:figure1}. Code for dataset generation is available in the repository\cite{github}.

The data was acquired from open sources: Open Data Portal of Comune di Milano \cite{Milano_geoportale} available under CC License \cite{cc_license} and Openstreetmap.org \cite{OpenStreetMap} available under the Open Database Licence. The absent parameters have been considered as a mean value for the city. No modifications have been made to the datasets provided.
\begin{figure}[!h]
    \centering
    \includegraphics[width=1.0\columnwidth]{sample.png}
    \caption{Sample from Milan Dataset of urban blocks: on the left an image from set A, on the right a corresponding image from set B. The white block in the center of the left image is the block to be designed.}
    \label{fig:figure1}
    \Description{Two images showing a diagrammatic representation of the top view on a part of the city of Milan (Piazzale Segesta). The image on the left is the input condition, the image on the right is the ground truth.}
\end{figure}

\begin{figure*}[th!]
    \centering
    \includegraphics[width=1.0\textwidth]{milan_test_results.png}
    \centering
    \caption{The results were obtained on the test set of Milan images with the model trained on Milan urban blocks. The images are presented in sets of 3: input, output, output with the zone of interest highlighted (for the clarity of reading the results). The zones have been taken from the Riformare Milano project\cite{riformaremilano}. 1 - Dergano zone; 2 - Oriani zone; 3 - Caserma Perucchetti; 4 - Dergano zone; 5 - Mecenate zone; 6 - San Cristoforo zone}
    \label{fig:milan_test}
    \Description{A 3x6 matrix showing 6 samples from the Milan test set results: the first column is the input image, the second is the output from the model and the third is the ground truth. The images are diagrammatic representations of the city plan.}
    \centering
\end{figure*}

\section{Architecture}
The architecture used for this research is a GAN for Image-to-Image translation \cite{isola2018imagetoimage}. The algorithm consists of two models: Generator Network based on Unet architecture and Discriminator Network based on PatchGAN Convolutional Network. The principle is in mutual learning: the generator generates an output image based on the condition (input image). The discriminator model receives generated images and target images and learns to distinguish between them, classifying each of them into a 'real' or a 'fake' class. In the case of PatchGAN, the model learns to classify patches of a given image as real or fake instead of giving a result for the whole image sample, so it outputs the result over which areas of an image should be improved by the generator.

Three different implementations of the algorithm were used, which are: Tensorflow-based \cite{pixtf}, PyTorch-based without Unet \cite{pixtorch}, Pytorch-based with Unet \cite{github} \cite{ganspecialization}. The use of adaptive learning rate tends to give better results. An algorithm compares the mean loss from 250 iterations with the previous accumulated losses. In case the latest loss is larger than the largest of the previous three losses, the learning rate reduces. This technique results in more distinct shapes of the building contours and more efficient training in terms of time. The algorithm has been trained using Google Colab environment with a single NVIDIA Tesla K80 GPU.

\section{Generation of Milan Urban Style}
The specifics of Milan morphology and consistency in the urban fabric of the center of the city make it a good location for the initial experiments. The relative homogeneity of the urban morphology reduces the variance in the data. The model has been trained for 20 epochs with a dataset of 1200 images.

After the training, the model has been tested on a new set of images that were not included in the training set. The areas chosen for the test have been previously defined as 'problematic'\cite{riformaremilano} and requiring a transformation. However, the main point is not a proposal of a solution to these areas, but a demonstration of the possibilities of GAN-based generative urban design, even in the complex urban contexts that are different from the residential ones. The results in Figure~\ref{fig:milan_test} show designs of the urban blocks coherent with the traditional Milanese urban blocks\cite{Biraghi:2018:ISBN:978-5-7638-4127-5} that tend to have a closed structure and an internal courtyard system in their majority. The model successfully recognizes the zone of interest to construct the block within and manages to adapt the learned visual characteristics of the city into a new context. Samples 1, 2, 4 are the samples with the irregular urban fabric and sample 5 is an example of an urban context with a high presence of industrial buildings differing in proportions from the forms in the training set.

At the same time model fails to produce a clear solution in a few samples (3, 6 in Figure~\ref{fig:milan_test}). This behavior occurs in the samples having a low urban density of the surroundings or their absence. Another reason is connected to the absence of the surroundings: their small proportion in the image to the zone of interest (e.g. when the zone occupies more than 30 percent of an image). This factor is an indication of the fact that the model produces solutions based on the visible surroundings; at the same time, it introduces a limitation of its use as well as a constraint for the input images: the model is not capable of producing design without any surroundings. This behavior is consistent on different urban scales and is related mostly to the proportion of the block to the image.
\begin{figure}[h!]
    \centering
    \includegraphics[width=0.95\columnwidth]{scale_comparison1.png}
    \caption{Illustration of the model's performance on the urban contexts in different scales. The images are on scales 1:2500, 1:5600, and 1:8500 respectively; the upper row represents the city of Milan, Mecenate zone; the second row is the city of Bengaluru.}
    \label{fig:scales}
    \Description{A 2x3 matrix with a diagrammatic representation of Milan and Bengaluru top views in different scales (1:2500, 1:5600, 1:8500) and with a different area of the zone of interest within one scale with the results from the generative models.}
    \centering
\end{figure}
\begin{figure}[h!]
    \centering
    \includegraphics[width=0.95\columnwidth]{scale_comparison2.png}
    \caption{Illustration of the model's performance on the zone of interest with different proportions with respect to the image within the same Bengaluru urban context.}
    \label{fig:proportions}
    \Description{4 images with a diagrammatic representation of the city top view in scale 1:2500 with different proportion of the area defined as a block of interest. This shows that the model fails when the proportion of the block with respect to the image is large.}
    \centering
\end{figure}

Furthermore, the model was tested on a set of images from the same locations but with different scales to see whether the design capabilities depend on the scale and the proportions of the zone's size with respect to the surroundings. As it can be observed in Figure~\ref{fig:scales} performance of the model differs based on the scale. At the same time, it should be noticed that difference in performance is not related only to the scale but also to the proportion of the zone of interest: the block of Bengaluru (see Figure~\ref{fig:scales} second row, first column) at the scale of 1:2500 causes model's failure as the area of the block is too large with respect to the surroundings' area. On the scale of 1:5600, however, the performance is close to the one of Milan's sample at the scale of 1:2500. The areas of the respective blocks are similar. The same observation can be made for the last sample of Bengaluru and the second sample of Milan. Besides, the model was checked on an image on a large scale cropping the block to different proportions with respect to surroundings (Figure~\ref{fig:proportions}). The experiment demonstrates that the performance of the model depends on the proportion of the surroundings in an input image not on the urban scale and the fact that the design is conditioned by the context. This is the reason the importance of the presence of the context should be noted: firstly, the empty area is likely to be treated as the construction site and, secondly, there will be no surrounding blocks to condition the design of the current block of interest (see model's behavior in Figure~\ref{fig:milan_test} in samples 3, 6).
\newline However, there is another limitation related to the chosen training process and the generated images. The dimensions of the generated buildings remain the same independently from the scale of the input image (for instance, in Figure~\ref{fig:scales}, row 2): the width of the building volumes does not change with the modification of the scale of the surroundings. This behavior was expected, as all the images in the training set were produced on one scale, thus the distances and the units in the images are perceived by the model within this scale only.

\section{Style Translation}
Having obtained the results from the model trained on the city of Milan, it was decided to experiment further and see which features the model will learn from a completely different morphology. For this reason, several cities with a morphology that is distant from the morphology of Milan have been chosen:
\begin{itemize}
    \item Amsterdam
    \item Bengaluru
    \item Tallinn
    \item Turin
\end{itemize}
\begin{figure}[!h]
    \centering
    \includegraphics[width=0.9\columnwidth]{cities_morphology.png}
    \caption{Illustration of the difference in urban morphology of the chosen cities. 1 - Milan, 2 - Amsterdam, 3 - Tallinn, 4 - Turin, 5 - Bengaluru}
    \Description{Five images showing the morphology of the following cities: Amsterdam, Bengaluru, Tallinn, Turin, Milan. The morphology is shown by drawing white building footprints from a part of each city on the black background.}
    \label{fig:morphology}
\end{figure}
The datasets for these cities are built with the use of scripting and QGIS software in the same way as the dataset of Milan urban blocks. The cities of Tallinn and Bengaluru had a condition of having more than 8 buildings in the block of interest due to the composition of their urban fabric.

As can be seen in Figure~\ref{fig:morphology} the cities vary greatly in urban density, buildings' types (isolated structures in Bengaluru and Tallinn; in connected structures and internal courtyard forming in Milan, Turin, and Amsterdam), relationships between the buildings and the block they belong to, the composition of the blocks. For instance, Milan, Turin, and Amsterdam have closed block structure often with an internal courtyard; at the same time the density of the blocks vary from city to city as well as the composition of the blocks: in Amsterdam typically a closed block is formed by many buildings while in Turin just by a few and the shape of a footprint of a block is more clear and geometrical. The differences in the urban fabric of the cities are helpful in the estimation of the learning results of the model and comparison of the results produced.

After the training and testing of these models on the corresponding cities each model was tested on the urban contexts from the other cities. This test gives an understanding of whether it is possible to translate an urban style from one city to another with the use of GAN, whether the model is able to adapt to completely different surroundings, and the coherence of the results with the urban blocks of training. The results were presented as a confusion matrix in Figure~\ref{fig:confusion_matrix}.
\newline The resulting models are published\cite{urbangen} and are available in open access under MIT License. The code for model testing is provided as well.

\section{Estimation}
The result of training for the case of urban block design should be evaluated both quantitatively and qualitatively. The qualitative approach is based on the use of a neural network predicting the class for the given images. Unfortunately, there was no possibility to have a survey based on the experts' opinion to obtain the qualitative estimation. This method serves as an alternative to experts' judgment on the blocks' design as it exploits the visual comprehension and appearance of an urban block in general that allows to attribute it to a particular city. This neural network classifier is trained independently of GAN and it represents a Convolutional Neural Network with a resnet50 backbone. This classifier is trained to distinguish among the urban blocks of all the cities the models were trained on.

The quantitative approach is based on the analysis of the block design from the urban and morphological perspective relative to the city it is situated in. The numerical parameters determining the morphology of an urban block are taken into account: relationships between the buildings within a block, relationships between the buildings and the block, average individual parameters of the buildings.

\subsection{Qualitative Estimation}

A separate convolutional neural network with a resnet50 backbone \cite{DBLP:journals/corr/HeZRS15} was trained to classify between the images of the urban blocks of all the five cities. The training and validation sets were composed of the images from the real cities' structures \cite{github} which can be seen in Figure~\ref{fig:classification_sample} while the test images were the images generated by the GAN model. The results of the classification are reported in the Table ~\ref{tab:class_results}. The accuracy achieved on real samples is compared with the accuracy achieved on the generated samples.
\begin{figure}[!h]
    \centering
    \includegraphics[width=0.9\columnwidth]{classification_sample.png}
    \caption{Samples from the dataset used for urban blocks classification.}
    \label{fig:classification_sample}
    \Description{Four samples from the dataset used for classification purposes. Each sample contains a white mask of an urban block on the black background.}
\end{figure}

\begin{figure*}[th!]
    \centering
    \includegraphics[width=1.0\textwidth]{confusion_matrix_cities}
    \centering
    \caption{Results obtained on the test set. Rows represent the existing urban context of the city, columns represent the cities the models were trained on (the expected morphology of the urban block). E.g. cell [1, 3], row 1 column 3 represents the model trained on the city of Amsterdam tested within Milan urban context: Amsterdam urban block design in Milan. The diagonal cells represent the models trained and tested on the same city.}
    \Description{A 5x5 matrix where every cell has an image of a city plan as a diagrammatic representation with different urban block designs produced by the generative models.}
    \label{fig:confusion_matrix}
    \centering
\end{figure*}

As in the case with the datasets for GAN, the classification dataset is divided into training and validation sets. Training images are the images the CNN learns from while validation images are used to see the results of the training process, these are the data the algorithm has not seen before. From the confusion matrix of results produced from the training set as seen in Table~\ref{tab:train_matrix} and validation set (Table~\ref{tab:valid_matrix}), it can be seen that urban blocks of some cities are confused more often with some specific cities. For example, Bengaluru and Turin do not have a large amount of misclassified images while the samples from Milan are often predicted to be from Tallin and vice versa.
\begin{table}[!h]
  \centering
  \caption{Classification Confusion Matrix from Training set results. Columns represent the predicted classes, rows represent the actual classes. E.g. cell [1, 3] row Amsterdam column Milan shows the number of Amsterdam sample images classified as Milan images.}

  \begin{tabular}{cccccc}
	\toprule
	 &
	\multicolumn{1}{p{0.1\columnwidth}}{\centering{Amster.}} &
    \multicolumn{1}{p{0.15\columnwidth}}{\centering{Bengaluru}} &
    \multicolumn{1}{p{0.1\columnwidth}}{\centering{Milan}} &
    \multicolumn{1}{p{0.1\columnwidth}}{\centering{Turin}} &
    \multicolumn{1}{p{0.1\columnwidth}}{\centering{Tallinn}} \\
    \midrule
    Amsterdam & 926 & 21 & 47 & 20 & 6 \\
    Bengaluru & 16 & 951 & 24 & 14 & 7 \\
    Milan & 55 & 22 & 866 & 14 & 38\\
    Tallinn & 54 & 26 & 32 & 484 & 10 \\
    Turin & 19 & 11 & 47 & 8 & 570\\
    \bottomrule
  \end{tabular}
  \label{tab:train_matrix}
 \end{table}

  \begin{table}[h]
  \centering
  \caption{Classification Confusion Matrix from Validation set results. Columns represent the predicted classes, rows represent the actual classes.}
  \begin{tabular}{cccccc}
	\toprule
	  &
	\multicolumn{1}{p{0.1\columnwidth}}{\centering{Amster.}} &
    \multicolumn{1}{p{0.15\columnwidth}}{\centering{Bengaluru}} &
    \multicolumn{1}{p{0.1\columnwidth}}{\centering{Milan}} &
    \multicolumn{1}{p{0.1\columnwidth}}{\centering{Turin}} &
    \multicolumn{1}{p{0.1\columnwidth}}{\centering{Tallinn}} \\
    \midrule
    Amsterdam & 146 & 3 & 0 & 17 & 13 \\
    Bengaluru & 1 & 167 & 0 & 12 & 3 \\
    Milan & 8 & 13 & 102 & 31 & 44 \\
    Tallinn & 19 & 2 & 4 & 82 & 1 \\
    Turin & 4 & 0 & 8 & 0 & 77 \\
    \bottomrule
  \end{tabular}
  \label{tab:valid_matrix}
\end{table}
 \begin{table}[h!]
  \centering
  \caption{Statistics from the classification model trained on the images of the urban blocks of Milan, Amsterdam, Bengaluru, Tallinn, Turin. The test accuracy is the confidence of prediction over the generated images from the Generator's model training set.}
  \begin{tabular}{ccccc}
    \toprule
    Classification &
    \multicolumn{1}{p{0.15\columnwidth}}{\centering{Training Accuracy}} &
    \multicolumn{1}{p{0.15\columnwidth}}{\centering{Validation Accuracy}} &
    \multicolumn{1}{p{0.15\columnwidth}}{\centering{Accuracy on Test Gen samples}} &
    \multicolumn{1}{p{0.15\columnwidth}}{\centering{Confidence On Test Gen Samples}} \\
    \midrule

    Amsterdam & 0.815 & 0.907 & 0.952 & 0.791 \\
    Bengaluru & 0.913 & 0.94 & 0.352 & 0.799 \\
    Milan & 0.515 & 0.87 & 0.619 & 0.877 \\
    Tallinn & 0.759 & 0.799 & 0.523 & 0.85 \\
    Turin & 0.865 & 0.87 & 0.5 & 0.795 \\
    Overall & 0.758 & 0.885 & 0.566 & 0.822 \\
    \bottomrule
  \end{tabular}
  \label{tab:class_results}
\end{table}

This classification model has been tested on the images generated by GAN. The test set consists of 50 images, 10 images per city. The results produced from Classifier over this set are reported in Table~\ref{tab:class_results}. As can be seen, the generated images are classified with more than 0.5 accuracy to the correct class apart from Bengaluru samples with a confidence of more than 0.75. Bengaluru model sometimes fails to produce a solution in Tallinn blocks due to the reason of low density.

\subsection{Quantitative Estimation}
Taking into consideration the city of Milan it can be noticed that from the morphological point of view its blocks could be divided into a few groups sharing some common features across the city based on the following characteristics: shape, dimension, number of parcels, and topography\cite{Biraghi:2018:ISBN:978-5-7638-4127-5}. While the first two are the conditions determined by the input image, the focus is on the latter two. The number of parcels and the topography of the blocks could be described numerically as parameters describing the relationships between the urban forms. These relationships have the same importance for the other cities as well. For this evaluation, the factors considered are the relationships between the buildings within a block (distance between the neighboring buildings), the relationships between a block and its buildings (building density, the proportion of built space in a block; the distance between the street and the closest building in a block) and the individual parameters of the buildings (average area of a building footprint in a block). We acknowledge that these parameters vary across one city, however, it is possible to deduce the most common characteristics shared by the majority of the blocks by taking the median value across the city. These most common characteristics represent the urban fabric, the feature that distinguishes one city from another~\ref{fig:morphology}. Moreover, in our case, the "inter-city" similarity is higher than the "intra-city" similarity as the cities have been chosen to have a 'contrasting' morphology. However, this research is limited by the number of comparison parameters for urban blocks; it would be a valuable continuation of the work to develop a complete set of comparison criteria.

\begin{table}
  \centering
  \caption{Quantitative estimation of the solutions produced by 5 trained GAN models: comparison of the average parameters by city and the parameters in the samples produced by the models.}
  \begin{tabular}{ccccc}
    \toprule
    City &
    \multicolumn{1}{p{0.11\columnwidth}}{\centering{City Dataset: Median Density}} &
    \multicolumn{1}{p{0.13\columnwidth}}{\centering{Generated Blocks: Median Density}} &
    \multicolumn{1}{p{0.15\columnwidth}}{\centering{City Dataset: Median Building Area, px}} &
    \multicolumn{1}{p{0.14\columnwidth}}{\centering{Generated Blocks: Median Building Area, px}} \\
    \midrule
    Amsterdam & 0.537 & 0.331 & 104 & 124 \\
    Bengaluru & 0.484 & 0.307 & 112 & 69 \\
    Milan & 0.489 & 0.521 & 258 & 215 \\
    Tallinn & 0.236 & 0.307 & 13 & 31 \\
    Turin & 0.553 & 0.594 & 283 & 389 \\
    \bottomrule
  \end{tabular}
  \label{tab:block_metrics}
\end{table}
\begin{table}
  \caption{Quantitative estimation of the solutions produced by 5 trained GAN models: comparison between the offset of the buildings from the street by city and the parameters in the samples produced by the models. Distance from the street is a minimum distance between the buildings in a block and the surrounding streets; the resulting value is a median value for the dataset samples and the samples produced by the models.}
  \centering
  \begin{tabular}{ccc}
    \toprule
    City &
    \multicolumn{1}{p{0.30\columnwidth}}{\centering{City Dataset: Median Distance Between the Buildings and the Street, px}}&
    \multicolumn{1}{p{0.35\columnwidth}}{\centering{Generated Blocks: Median Distance Between the Buildings and the Street, px}} \\
    \midrule
    Amsterdam & 1.14 & 1.414 \\
    Bengaluru & 1.01 & 3.4 \\
    Milan & 0.346 & 2.0 \\
    Tallinn & 1.85 & 2.4 \\
    Turin & 1.23 & 2.8 \\
    \bottomrule
  \end{tabular}
  \label{tab:street_distance}
\end{table}
\begin{table}
  \centering
  \caption{Quantitative estimation of the solutions produced by 5 trained GAN models: comparison between the median distance between the adjacent buildings by city and the parameters in the samples produced by the models. The distance between the adjacent buildings is an average of minimum distances between the buildings in a block; the resulting value is a median value.}
  \begin{tabular}{ccc}
    \toprule
    City &
    \multicolumn{1}{p{0.33\columnwidth}}{\centering{City Dataset: Median Distance Between the Adjacent Building}}&
    \multicolumn{1}{p{0.33\columnwidth}}{\centering{Generated Blocks: Median Distance Between the Adjacent Buildings}} \\
    \midrule
    Amsterdam & 2.6 & 2.5 \\
    Bengaluru  & 2.8 & 3.0 \\
    Milan & 1.5 & 2.2 \\
    Tallinn & 3.1 & 3.4 \\
    Turin & 0.5 & 0.0 \\
    \bottomrule
  \end{tabular}
  \label{tab:building_distance}
\end{table}
These parameters have been measured for all the blocks in each city dataset, then the median value has been calculated for each city. The median has been chosen over the mean value as it takes into consideration the presence of the outliers which are likely to be present in the datasets due to the urban variety in each city. The metrics have been calculated from the images with the use of Computer Vision\cite{github} and QGIS; the units of absolute values such as the median building area or the distance between adjacent buildings are pixels.

From Tables~\ref{tab:block_metrics},~\ref{tab:building_distance} it can be seen that urban blocks produced by the GAN in different urban contexts are coherent with the city of training: the parameters of building density, building area as in Table~\ref{tab:block_metrics} are close to the actual values, which confirms the observations made in the confusion matrix (Figure~\ref{fig:confusion_matrix}). At the same time, the distance from the street to the closest building in a block differs as in Table~\ref{tab:street_distance}: it is a constraint taken from the surroundings present in the input image (Figure~\ref{fig:street}).

\section{Contribution}
This paper aims to contribute to the field of generative design by:
\begin{itemize}
    \item introducing a new approach towards the context-based generative design of an urban block without a predefined set of constraints, by learning the visual parameters from the existing environment instead;
    \item introducing an application of an Image-to-Image translation GAN to the field of urban design and evaluating the produced design solutions from quantitative and qualitative points of view.
\end{itemize}
\begin{figure}[h!]
    \centering
    \includegraphics[width=0.95\columnwidth]{street_distance.png}
    \caption{Samples from the test set. The block design is aligned with the surrounding blocks' street lines.}
    \Description{Six results from the test set with the annotations showing an alignment of the generated buildings to the existing buildings.}
    \label{fig:street}
\end{figure}
\section{Results}

The study shows that it is possible to achieve a generation of an urban block without the explicit definition of its parameters with the use of GAN. Not only does this method allow for learning the parameters from the existing urban context without their explicit definition, but it also produces results coherent with the urban surroundings of a block as in Figure~\ref{fig:confusion_matrix}. Moreover, this method produces a design solution with the characteristics of a generated urban block very close to the actual parameters of the blocks of a city of interest.

The experiments have shown that the model is applicable to diverse urban contexts with different urban morphology. Besides, urban parameters and morphology learned by a model can be translated to another city with a completely different context. The algorithm adapts the parameters learned previously to a new urban surrounding.

Some limitations of the algorithm include the size of an urban block within the image, the presence of the urban surroundings in the image, the scale of the buildings in the image. The size of a block is directly connected to the presence of the urban surroundings: the block design is shaped by the context, furthermore, the algorithm learns from the visible context. Thus it is important to have it present in a considerable portion of the image (determined experimentally). The scale of an urban block is limited due to the specifics of the current training preset: all the images from the training set were of the same scale, which means it is reasonable to test the algorithm with the images of the same scale. The preset is limited to a certain scale (which may vary from city to city based on the average block size) due to the necessity of having a sufficient part of the image filled with urban surroundings of a block of interest.

Besides, it should be noted that the design solutions produced do not have the same clarity as the vector images produced by the architects. However, urban shapes in the generated images are clearly visible and give a good understanding of the blocks' design.

Another obvious limitation is the production of an urban block design that is similar to the existing blocks in the city. It is an approach to design that might be considered non-creative or lacking other important factors of urban design theory. We do not insist on the application of the suggested approach 'as-is'. This work highlights the ability of the algorithm to autonomously learn from the visual features, which could be further explored to take into account the other planning criteria and optimize the block's design from different points of view.

\section{Conclusion}
The experiment and evaluation of the GAN-based approach to the generative design of an urban block confirmed the possibility of the generation of a piece of urban fabric consistent with the surroundings and the desired urban style with the use of neural networks. Nonetheless, the images lack clarity of the vector images, the urban forms are compliant with the actual forms present in the city. What is more, these results have been achieved without image annotations, explicit definition of the parameters needed for each typology and shape of an urban block, typical characteristics of a city: all of these attributes are learnt by the neural network during the training process, from the experience having seen multiple urban surroundings.

In addition, the time and effort that are consumed for the block's design generation are much less compared to the human-generated design. It takes 2-4 hours to train one model after which it is able to generate the design solutions within 10 seconds.

This approach can not, obviously, substitute human participation in a city design process: the algorithm imitates the existing urban environment after having learned from it. It does not invent new space organization principles, does not rethink the comprehension of space by its users, and does not plan important landmarks of a city. However, considering that cities majorly consist of residential buildings and not landmarks, this approach could be used as a support tool for a planner or an architect by giving insights about the possible solutions and existing building composition of a city. Alternatively, it could be a design solution meant for typical blocks' design which would have a consistent integration into the existing surroundings. Moreover, as already mentioned, this approach could become a base for the consequent integration of the other urban design parameters into the model training which would result in a design that accustoms to multiple factors.

\begin{acks}
I would like to express my very great appreciation to Professor Luigi Mussio and Professor Federico Deambrosis who dedicated their time and effort to guiding and directing the work and encouraging me in the doubtful moments. My thanks also go to the professors of Architecture and Urban Design of Politecnico di Milano: Professor Lemes, Professor Caja, Professor Montedoro who had found time to give me their valuable suggestions from the urbanistic and architectural point of view as well as insights on the morphology of the cities. I would also like to thank Dr. Carlo Biraghi for the constructive advice.
\newline My gratitude is extended to Andrey Baran, Alberto Floris, Ram Kummamuru and Viktor Petukhov for the productive discussions and inspiration.
\end{acks}

\bibliographystyle{ACM-Reference-Format}


\section{Introduction}
Among the candidate theories beyond the standard model (SM),  supersymmetry (SUSY)~\cite{Fayet:1976cr,Nilles:1983ge,Martin:1997ns} is the most celebrated one thanks to its ability in explaining a wide range of phenomena that are not addressed by the SM. It postulates a symmetry between the fermions and the bosons introducing a set of new particles which differ by spin half-unit from the SM particles. It also posits the existence of the lightest superpartner (LSP) if a new symmetry, namely the R-parity~\cite{Fayet:1974pd}, is invoked. The LSP ($\tilde{\chi}_{1}^{0}$) is considered to be a suitable dark matter candidate~\cite{Farrar:1978xj}. Therefore, energy-frontier experiments such as CMS~\cite{Chatrchyan:2008aa} are focused to venture on experimental signatures of such SUSY theories. Recent results from CMS are exciting with innovative analysis strategies that target a broad spectrum of SUSY production scenarios. In this report, we present a few selected SUSY search results based on the Run-2 data corresponding to a total integrated luminosity of $\sim$ 140 $\mathrm{fb}^{-1}$ collected by the CMS detector.

\section{Searches for Gluino Production}
Strong production of SUSY particles can be searched for at higher masses due to their larger cross sections as compared to the other production modes. In particular, gluinos ($\tilde{g}$) -- superpartners of gluons -- are produced with the highest cross section at the LHC. Their searches are typically characterized by large jet multiplicity $\mathrm{N_{jets}}$ and the presence of large missing transverse momentum ($\mathrm{p_T^{miss}}$).  Additional discriminating variables are the number of leptons or photons and the b-tagged jets $\mathrm{N_{b-jets}}$ in the final state. The other kinematic variables that are also exploited are the scalar sum of jet transverse momenta ($\mathrm{H_T}$), jet masses, variables related to large-radius jets, etc. A few example event diagrams of the gluino pair production at the LHC are shown in Fig.~\ref{fig:001}.

\begin{figure}
    \centering
    \includegraphics[width=0.9\linewidth]{figures/CMS-SUS-19-006_Figure_001.pdf}
    \caption{Example event diagrams of the gluino pair production at the LHC, where the gluinos decay into heavy quarks and $\tilde{\chi}^0_1$ (top), light quarks (bottom left) and $\tilde{\chi}^0_1$, and light quarks and SM vector bosons (bottom right).}
    \label{fig:001}
\end{figure}

Searches pursued using complementary analysis strategies provide independent exclusion limits on the gluino production cross section. A classic example~\cite{Sirunyan:2019ctn} involves pair-produced gluinos decaying into purely hadronic final states. The aforementioned variables, namely $\mathrm{N_{jets}}$, $\mathrm{N_{b-jets}}$, $\mathrm{H_T}$ and $\mathrm{p_T^{miss}}$ are used to divide the available search region into 174 independent bins. The search excludes gluino masses up to 2310 GeV at 95\% confidence level (CL) as shown in Fig.~\ref{fig:002}.

\begin{figure}
    \centering
    \includegraphics[width=0.4\linewidth]{figures/CMS-SUS-19-006_Figure_013-a.pdf}
    \includegraphics[width=0.4\linewidth]{figures/CMS-SUS-19-006_Figure_013-b.pdf}
    \caption{Exclusion limits at 95\% CL on gluino masses at different neutralino ($\tilde{\chi}^0_1$) mass points for gluino to top (left) and bottom (right) quark decay modes from inclusive hadronic searches based on the jet multiplicity and missing transverse momentum~\cite{Sirunyan:2019ctn}. The solid black curve represents the observed exclusion and the dashed red curve represents the expected exclusion limits in the plane of gluino and $\tilde{\chi}^0_1$ masses.}
    \label{fig:002}
\end{figure}

Another interesting search~\cite{Sirunyan:2020zzv} exploits the topology of large-radius jets in the final state that can provide a common decay scenario for a TeV scale ($\simeq$ 3 TeV) initial SUSY particle. In this search, a pair of gluinos is produced, each decaying into a high momentum ($\mathrm{p_T}>200$ GeV) Z boson and $\mathrm{p_T^{miss}}$. A small mass splitting between the gluino and the intermediate neutralino ensures a large momentum for the final state Z bosons, provided that the LSP mass is fixed at 1 GeV. The Z boson gives rise to a large radius jet providing a narrow search window in the jet mass. The latter variable is fitted with a linear function to predict the number of background events in the search region. An exclusion limit on the gluino mass of 1920 GeV (Fig.~\ref{fig:003}) is achieved with this novel search strategy.

\begin{figure}
    \centering
    \includegraphics[width=0.5\linewidth]{figures/CMS-SUS-19-013_Figure_009.pdf}
    \caption{Exclusion limits at 95\% CL on gluino mass from the events with high-momentum Z bosons and missing transverse momentum~\cite{Sirunyan:2020zzv}. The solid black curve shows the observed exclusion limit.  The dashed black curve shows the expected limit and the green and yellow bands represent the $\pm$1 and $\pm$2 standard deviation uncertainty ranges. }
    \label{fig:003}
\end{figure}

Unlike the previous ones described, the analysis that deals with one lepton and the sum of masses of large-radius jets final states~\cite{CMS:2019tlp}, makes use of the jets emanating from initial-state radiation. The sum of the masses of large-radius jets ($\mathrm{M_{jet}}$) and the transverse mass ($\mathrm{m_T}$) variables are used to categorize the search and control regions. An ABCD method is deployed in these two variables to estimate the background events in the search regions. Scenarios with gluino masses up to about 2150 GeV are excluded at 95\% CL as shown in Fig.~\ref{fig:004}. A similar level of exclusion on the gluino mass $\sim$2 TeV is achieved by other search strategies~\cite{Sirunyan:2020ztc,Sirunyan:2019xwh}.

\begin{figure}
    \centering
    \includegraphics[width=0.4\linewidth]{figures/CMS-SUS-19-007_Figure_010.pdf}
    \caption{Exclusion limits at 95\% CL on gluino masses from the SUSY search in the final states with a single lepton using the sum of masses of large-radius jets~\cite{CMS:2019tlp}.}
    \label{fig:004}
\end{figure}

\section{Searches for Stop Production}
It is important for a SUSY theory to be ``natural", which means that there are minimal fine-tunings of the parameters of the theory. A ``natural SUSY"~\cite{Moreno:2015zeg} scenario introduces mildest possible fine-tuning. The searches for the third generation stop ($\tilde{t}$) production are of special importance from such naturalness arguments~\cite{Barger:2012hr,KING2012207}.  A few example event diagrams for different decay modes are shown in Fig.~\ref{fig:005}.

\begin{figure}
    \centering
    \includegraphics[width=0.8\linewidth]{figures/CMS-SUS-19-011_Figure_001.pdf}
    \caption{Example event diagrams of the stop pair production at the LHC, where the stop decays into a SM top and $\tilde{\chi}^0_1$ (left), SM bottom, W boson and $\tilde{\chi}^0_1$ (middle), and SM bottom, neutrinos, leptons and $\tilde{\chi}^0_1$ (right).}
    \label{fig:005}
\end{figure}

The search for top squark pair production using dilepton final states~\cite{Sirunyan:2020tyy} focuses on the requirement of the two opposite charged leptons ($\mu$,e) in the final state. Further, the search region is binned in transverse mass ($\mathrm{M_{T_2}}$) and $\mathrm{p_{T}^{miss}}$ significance (S) variables. A lower limit at 95\% CL on the top squark mass is placed up to 925 GeV as shown in Fig.~\ref{fig:006}.

\begin{figure}
    \centering
    \includegraphics[width=0.8\linewidth]{figures/CMS-SUS-19-011_Figure_006.pdf}
    \caption{Exclusion limits at 95\% CL on top squark masses from the SUSY search in the final states with two opposite charge leptons~\cite{Sirunyan:2020tyy}. The left plot shows the limit for top squark decaying into top and $\tilde{\chi}^0_1$, whereas the right plot shows the top squark decaying into a W boson and $\tilde{\chi}^0_1$ via an intermediate chargino production. The solid black curve shows the observed exclusion and the red dashed curve represents the expected exclusion on the plane of $\tilde{\chi}^0_1$ and top squark masses. }
    \label{fig:006}
\end{figure}
Another search for direct top squark pair production involves events with one lepton, multiple jets, and large transverse momentum imbalance~\cite{Sirunyan:2019glc}. An exclusion limit at 95\% CL is set on cross sections for the top squark masses at 1.2 TeV as shown in Fig.~\ref{fig:007}. A similar level of exclusion on the stop mass $\sim$1 TeV is achieved by other search strategies~\cite{Sirunyan:2019ctn,Sirunyan:2020ztc,Sirunyan:2019xwh}.

\begin{figure}
    \centering
    \includegraphics[width=0.4\linewidth]{figures/CMS-SUS-19-009_Figure_008.pdf}
    \caption{Exclusion limits at 95\% CL on top squark masses from the SUSY search with one lepton, jets, and missing transverse momentum~\cite{Sirunyan:2019glc}, where the top squark decays into either top and $\tilde{\chi}^0_1$ or a bottom quark,  boson and $\tilde{\chi}^0_1$ via an intermediate chargino. The mass splitting between the chargino and the $\tilde{\chi}^0_1$ is set at 5 GeV and the branching ratios of each of the decay scenarios is 50\%. }
    \label{fig:007}
\end{figure}

\section{Searches for Electroweak Production}
The present LHC luminosity limits probing electroweak decay scenarios of SUSY particles due to their low production cross sections. Despite this, several interesting channels exist that provide very clean signatures. Many scenarios have been looked at so far, including the direct production of chargino/neutralino, gauge-mediated, or direct production of weakly interacting massive particles. The interesting models are R-parity conserving ones where the LSP could undergo annihilation-production interactions with SM particles and providing the observed dark matter relic density, assuming coannihilation models of the stau ($\tilde{\tau}$) and the $\tilde{\chi}^0_1$~\cite{Spergel:2003cb,planck}. One such search is based on a compressed mass spectrum in events with a soft $\tau$ lepton, a highly energetic jet, and large missing transverse momentum~\cite{Sirunyan:2019mlu}. The results are interpreted with a small mass difference (50 GeV) between the chargino and the neutralino. The mass of the top squark is assumed to be the average of the chargino and neutralino masses. The data show no deviation from the SM and upper limits at 95\% CL are set on the sum of the chargino, neutralino, and the stop production cross sections, resulting in a lower limit of 290 GeV on the mass of the chargino. This is the most stringent limit available to the date surpassing the bounds from LEP experiments. This exclusion limit is shown in Fig.~\ref{fig:008}.
\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.5\linewidth]{figures/CMS-SUS-19-002_Figure_002-a.pdf}
    \caption{Exclusion limits at 95\% CL on the chargino production cross section in events with a soft $\tau$ lepton, a highly energetic jet, and large missing transverse momentum~\cite{Sirunyan:2019mlu}. The blue curve represents the theoretical cross sections, whereas the solid black curve shows the observed exclusion limit. The dashed black curve shows the expected exclusion limit and the green and yellow bands represent the $\pm$1 and $\pm$2 standard deviation on the prediction, respectively.}
    \label{fig:008}
\end{figure}

\section{Searches for Long Lived Particles}
A whole enterprise of long-lived particle (LLP) searches is becoming more popular across the collider community since LLPs are a common signature for a wide range of beyond-the-SM theories. Such searches are usually signature-driven that give rise to disappearing tracks or displaced vertices. One such exciting result for the Run-2 data is from the search for disappearing tracks~\cite{Sirunyan:2020pjd}.
The upper limits are set on chargino production in the context of an anomaly-mediated SUSY breaking model for purely wino and higgsino neutralino scenarios. The chargino mass is excluded at 95\% CL up to  884 (474) GeV for a lifetime of 3 (0.2) ns for a wino case and below 750 (175) GeV for a lifetime of 3 (0.05) ns for a higgsino case as shown in Fig.~\ref{fig:009}.
\begin{figure}[htpb]
    \centering
    \includegraphics[width=0.4\linewidth]{figures/CMS-EXO-19-010_Figure_001-b.pdf}
    \includegraphics[width=0.4\linewidth]{figures/CMS-EXO-19-010_Figure_003.pdf}
    \caption{Exclusion limits at 95\% CL on chargino masses from the wino-like production scenario for a chargino lifetime of 3.34 ns (left) and higgsino-like  production scenario (right) in searches with disappearing tracks ~\cite{Sirunyan:2020pjd}. }
    \label{fig:009}
\end{figure}
\section{Summary and Future Prospects}
With a delivered luminosity of $\sim$140 $\mathrm{fb}^{-1}$ in Run-2, CMS has provided very stringent limits on several SUSY production scenarios. So far, no significant deviation from the SM is observed. The exclusion limits on the gluino masses are set at almost 2 TeV while that on the top squark masses are as large as 1.2 TeV. Stronger limits on the electroweak searches for slepton mediated decays exclude charginos between 0.6-1.1 TeV, followed by the boson mediated ones that exclude charginos of 0.4-0.7 TeV, whereas weaker limits for compressed regions and direct stau pair production are set at 0.3 TeV. A summary of the CMS results is presented in Fig.~\ref{fig:011}.

With the upgraded High-Luminosity LHC scenario (HL-LHC) at Run-4 we expect more exciting results from CMS. An expected exclusion reach from HL-LHC on gluino masses is about 3 TeV and on stop masses at 2 TeV~\cite{yellowreport}. With the expected increment in luminosity, the electroweak and LLP searches would also become even more promising.
\begin{figure}[h]
    \centering
    \includegraphics[width=0.32\linewidth]{figures/T1qqqq_limits_summary_cms.pdf}
    \includegraphics[width=0.32\linewidth]{figures/T2tt_limits_summary_cms.pdf}
    \includegraphics[width=0.32\linewidth]{figures/EWK-best_limits_summary_cms.pdf}
    \caption{Summary of limits obtained by CMS for pair produced gluinos (left), pair produced top squark (middle) and electroweak searches (right)~\cite{Suspubliccms}.}
    \label{fig:011}
\end{figure}

\bibliographystyle{unsrtnat}