\section{Introduction}
\label{submission}

This work explores how to adapt a single large base model to work with multiple tasks. In particular we focus on using deep neural networks, pre-trained on large amounts of English text, for multi-task learning on several natural language understanding (NLU) tasks.

Some multi-task learning approaches consider learning a general-purpose model that shares all parameters across tasks \citep[e.g., the NLP decathlon introduced by][]{deca}. This setting requires all tasks to have the same input and output space, and the input indicates the task. Instead, we consider the setting where we share most parameters across all tasks, but have a small number of task-specific parameters which adapt the shared model.

Sharing parameters, and thus a common representation, between tasks can sometimes lead to better generalization. However, fine-tuning separate models for each task often works better in practice. Although we are interested in multi-task methods that give results close to (or better than) state-of-the-art, there are separate motivations for maintaining shared parameters between tasks:
\begin{itemize}

 \item On applications like mobile devices we may have constraints on battery life. Applying several different neural networks to the same input costs energy. If only the `tops' of our models are task-specific, we can apply a shared transformation only once to the input, and use this transformed representation multiple times, as input to each task-specific function.
 \item  Again on mobile devices, running several different neural networks for various tasks
 can incur a computational and energy overhead due to swapping parameters on a dedicated integrated circuit \citep{adapt}.
    \item An application with a large number of tasks may have constraints on the number of parameters that can be stored. For example, web-scale applications may need to avoid storing a separate large model for every user.
\end{itemize}

Given a large number of shared parameters in a base model, and a small number of task-specific parameters, our key questions are: where should we be transforming the base model? What form should these transformations take? We assume the task is always known, so the model can always choose the correct adaptation parameters and output space.

We experiment on a set of eight NLU tasks from the GLUE benchmark \citep{glue},
which include question answering, sentiment analysis, and textual entailment.
The number of training examples varies widely across the tasks, so we explore how
to schedule training to not unduly favor the well-resourced tasks, or overfit
the low-resource tasks.

We use the BERT model \citep[Bidirectional Encoder Representations from Transformers,][]{bert} as our base pre-trained model. Pre-trained BERT representations can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, including the GLUE benchmark. However, the entire model is fine-tuned, meaning we need a separate model for each task. The transformer architecture that BERT is based on is powerful and popular, so finding the best way to adapt the parameters of this architecture for multi-task learning may be useful in other contexts, such as multilingual machine translation.

Our main contributions are:
1)~We introduce the `Projected Attention Layer' (PAL), a low-dimensional multi-head attention layer that is added in parallel to normal BERT layers.
2)~We introduce a novel method for scheduling training, where we sample tasks proportional to their training set size at first, and de-emphasize training set size as training proceeds.
3)~We perform an empirical comparison of alternative adaptation modules for self-attention-based architectures.

Making links to the vision literature, we identify shared lessons for where to add task-adaptation parameters depending on resource constraints.
On the GLUE benchmark, we show that PALs enable comparable performance to fine-tuned BERT-base (the smaller of the two models considered by \citealt{bert}) on many tasks with $\approx$7 times fewer parameters. We improve the performance of BERT-base on the recognising textual entailment (RTE) task, achieving 76.6\% accuracy, surpassing the performance of fine-tuned BERT-large (70.1\%) and the MT-DNN model \cite{mtdnn} (75.5\%) which also uses BERT and multi-task learning. We also find that the more parameter sharing we have, the better we do on the RTE task.

\section{Background}
Multi-task learning aims to provide an inductive bias that means models have
to learn features that are general enough to perform well on many tasks \citep{Caruana}. In NLP, examples of previous work include using a single model for chunking, tagging, named entity recognition, and semantic role labeling by applying a shared neural network to text, with different output layers \citep{Collobert}. Another approach outputs predictions at different layers using the idea of a linguistic hierarchy \citep{hash, hmtl}. \citet{sub} train a sequence-to-sequence RNN model on tasks including machine translation and natural language inference, and learn sentence representations useful for downstream tasks. Outside NLP, multi-task learning has been applied to diverse domains such as speech recognition \citep{deng} and reinforcement learning \citep{rein}. \citet{ruder} provides a more general overview.

Many multi-task learning approaches can be categorized as either `hard parameter sharing' or `soft parameter sharing'. Hard parameter sharing uses the same hidden layers for all tasks, with task-specific output layers.
Soft parameter sharing gives each task its own model, but the distances between the parameters of the models are regularized to encourage the parameters to be similar. For example \citet{duong} use the L2 distance, and \citet{yang} use the trace norm. In this work we assume that soft-parameter sharing with the whole of BERT requires too many parameters. We instead explore how to do hard-parameter sharing, by adding adapters to shared layers, as well as the usual separate output layers.

\subsection{Adaptation Parameters}
\label{adapt}

Various strategies for adding adaptation parameters have been explored. \emph{Learning hidden unit contributions} \citep[LHUC,][]{7078569} modifies a neural network by multiplying each hidden unit by a learnable scalar. Since the number of units is much smaller than the number of parameters in the network, this approach adds a small number of parameters compared to other methods we consider.

\emph{Residual adapter modules} \citep{adapt} adapt large pre-trained residual networks \citep{resnet} for multi-task learning in computer vision.
Each adapter module contains a 1$\times$1 filter bank with a skip connection, which can be inserted \emph{in series}, between the original network layers, or \emph{in parallel}, as additional inputs to a layer. For a layer with $C$ channels, the module contains an additional $C \!\times\! C$ matrix per layer for each task, containing $C$ 1$\!\times\!$1 convolutional filters.
This $C \!\times\! C$ matrix can be compressed by replacing it with a low-rank approximation, so that the
adapters contain a small fraction of the model parameters (e.g., less than 10\% for each task).
Several of our methods were inspired by the idea of using a low-rank approximation to the key operation of a model: the convolutional layer when dealing with images, or multi-head attention in the transformer.

\subsection{Fine-tuning Approaches}

A recent trend in transfer learning is to pre-train some model architecture
on a language
modeling objective before fine-tuning
that same model for a supervised downstream
task \citep{dai,ulmft,gpt}. BERT uses a similar approach, but was pre-trained with two objectives: 1)~filling in words `masked' out of an input sentence, and 2)~classifying whether two input sentences are adjacent in a corpus of text. Unlike a normal language modeling objective, BERT conditions on both left and right context when predicting the masked words.

The neural network layers in BERT are taken from the Transformer model \citep{NIPS2017_7181}, a sequence to sequence model that achieved state-of-the-art results in machine translation. Transformer layers have subsequently been used more broadly, e.g.\ for language modeling \citep{xl}, image generation \citep{sagan}, and generalized to video classification, object detection/segmentation and human pose estimation \citep{NonLocal2018}.

A concurrent approach by \citet{bertadapt}, introduces adapters similar to our `low-rank' layers (section~\ref{sec:within}), but added within each layer before each application of layer-norm. This work also keeps the BERT model fixed while training adapter modules. We concentrated on jointly fine-tuning the entire BERT model on all tasks, which has downsides: 1)~interference and `forgetting' of stored knowledge is possible; 2)~we require access to all tasks at training time. However the multi-task setup requires less adaptation parameters for good performance (we use 1.13$\times$ parameters compared to their 1.3$\times$ parameters\footnote{Although the results are not directly comparable since \citet{bertadapt} use BERT-large and we use BERT-base.} to match having separate models for each GLUE task.), and is crucial for the transfer effects that gave us good performance on RTE.

\section{Adapting Self Attention}
The BERT model we are adapting is a multi-layer bidirectional Transformer encoder based on the original model of \citet{NIPS2017_7181}. We only consider the smaller BERT-base model, which contains 110 million parameters.
We somewhat arbitrarily limit ourselves to a 1.13$\times$ increase in total parameters, which is equivalent to 15 million, or 1.9 million parameters per task. This choice avoids the extremes of having nearly no extra task-specific parameters, or giving each task its own whole model.

In the following sections we first introduce various components of the full BERT model, and discuss how many parameters they require (section~\ref{sec:param}). We then show the exact form our parameter additions took, distinguishing between adding to the `top' of the model, just before the output space (section~\ref{sec:top}),  or within each layer of the BERT-base architecture (section~\ref{sec:within}).
\subsection{Model Architecture and Multi-head Attention}
\label{sec:param}

BERT takes in a sequence (one or two English sentences in our case) and outputs a vector representation of that sequence. Each token in the sequence has its own hidden vector, and the first token of every sequence is always a special classification embedding (\texttt{[CLS]}). At each layer of BERT the hidden states of every sequence element are transformed, but only the final hidden state of \texttt{[CLS]} is used for classification/regression tasks. We now describe how the vector for one element of the sequence is transformed.

The multi-head attention layer \citep{NIPS2017_7181} is the core of the transformer architecture that transforms hidden states for each element of a sequence based on the other elements (the fully-connected layers act on each element separately). The multi-head layer, which we write as $\mathrm{MH}(\cdot)$, consists of $n$ different dot-product attention mechanisms. At a high level, attention represents a sequence element with a weighted sum of the hidden states of all the sequence elements. In multi-head attention the weights in the sum use dot product similarity between transformed hidden states.

Concretely, the $i$th attention mechanism `head' is:
\begin{equation}
    \mathrm{Attention}_i(\mathbf{h}_j) = \sum_t \mathrm{softmax}\bigg(\frac{W^q_i \mathbf{h}_j\cdot W^k_i \mathbf{h}_t}{\sqrt{d/n}}\bigg) W^v_i \mathbf{h}_t
\end{equation}
where $\mathbf{h}_j$ (we drop the $j$ index in the following discussion) is a $d$ dimensional hidden vector for a particular sequence element, and $t$ runs over every sequence element. In BERT the $W^q_i$, $W^k_i$ and $W^v_i$ are matrices of size $d/n \times d$, and so each `head' projects down to a different subspace of size $d/n$, attending to different information.
Finally the outputs of the $n$ attention heads (each of size $d/n$) are concatenated together (which we show as $[\cdot,...,\cdot]$) and linearly transformed:
\begin{equation}
    \mathrm{MH}(\mathbf{h}) = W^o \,[\mathrm{Attention}_1(\mathbf{h}),...,\mathrm{Attention}_n(\mathbf{h})
\end{equation}
with $W^o$ a $d \times d$ matrix\footnote{\citet{NIPS2017_7181} provide a more detailed motivation and discussion.}.
Throughout this section, we ignore terms linear in $d$ (like bias terms) to avoid clutter, as they don't add significantly to the parameter count. The matrices in a multi-head layer have $3 n d^2/n + d^2 = 4d^2$ parameters.

We further define another component of a BERT layer, the self-attention layer, which we write as $\mathrm{SA}(\cdot)$:
\begin{equation}
    \mathrm{SA}(\mathbf{h}) = \mathrm{FFN}(\mathrm{LN}(\mathbf{h} + \mathrm{MH}(\mathbf{h}))),
\end{equation}
LN($\cdot$) is \emph{layer normalisation} \citep{Ba2016LayerN}, requiring $2d$ parameters.
FFN is a standard \emph{feed-forward network},
\begin{equation}
    \mathrm{FFN}(\mathbf{h}) = W_2f(W_1\mathbf{h} + b_1) + b_2,
\end{equation}
with $f(\cdot)$ a non-linearity, GeLU \citep{gelu} in BERT\@. Matrix $W_1$ has size $d_{ff} \times d$ and $W_2$ has size $d \times d_{ff}$, so overall we require $2d d_{ff}$ parameters from the FFN component.

Putting this together, a BERT layer, which we write BL$(\cdot)$, is layer-norm applied to the output of a self-attention layer, with a residual connection.
\begin{equation}
    \mathrm{BL}(\mathbf{h}) = \mathrm{LN}(\mathbf{h} + \mathrm{SA}(\mathbf{h}) )
\end{equation}
We have $4d^2 + 2dd_{ff}$ total parameters from a BERT layer

The entire BERT model is simply a stack of 12 BERT layers, followed by (in our case) a transformation to take us to the output space for a NLU task. We write the dimensions of the hidden states in BERT-base as $d_m\!=\!768$. The final hidden state of the first token of every sequence is all that is used for the transformation to the output.

The exact form of the transformation applied to the final hidden state of the \texttt{[CLS]} token is a simple $d \times d$ linear transformation, known as a \textbf{`pooling layer'}, followed by a nonlinearity then another matrix multiply that projects to the output space. The output space is always three dimensional or less in our case, and so this projection does not require many parameters. However separate pooling layers add $d^2$ parameters for each task. When sharing this layer we needed to use a non-standard training schedule; see section~\ref{samp}.

\subsection{Adding Parameters to the Top}
\label{sec:top}
The simplest way to add parameters to a model is to add them at the `top' of the model, i.e.\ just before the classification layer.

We get our final hidden state for \texttt{[CLS]}, $\mathbf{h}^{f}$, from the original vector embeddings of the tokens in the sequence (of length $l$), $\{\mathbf{h}_t\}_{t=0}^{l}$, by
\begin{equation}
    \mathbf{h}^{f} = \mathrm{TS}( \mathrm{BERT}(\{\mathbf{h}_t\}_{t=0}^{l} )),
\end{equation}
where $\mathrm{TS}(\cdot)$ is a task-specific function that can potentially operate on a single vector, but depends on the entire sequence when it contains attention layers. $\mathrm{BERT}(\cdot)$ always depends on the entire sequence, and is shared across tasks.

The benefits of this form are that at inference time we only apply $\mathrm{BERT}(\{\mathbf{h}_t\}_{t=0}^{l})$ once (assuming the setting where we perform multiple tasks on the same piece of text), which saves significantly on total operations because each $\mathrm{TS}(\cdot)$ requires much fewer operations than the main BERT model.

The simplest form for the task-specific transformation of the hidden state $\mathrm{TS}(\cdot)$ would be a linear transform followed by a nonlinearity. However this requires $d_m^2$ parameters, and $d_m$ is fairly large even for BERT-base. The linear transform does not violate our 15 million parameter constraint, but we expect there are more efficient ways to add parameters.

Another obvious transformation, adding an extra BERT layer for each task, results in approximately a 1.67$\times$ increase in number of parameters, or 73 million new parameters. $d_{ff}$ is $4d_m$ for BERT, so for a BERT layer we get $4d_m^2 + 2d_m d_{ff} = 12d_m^2$ parameters. We include this architecture in our experiments for comparison, with the caveat that it requires many more parameters than our alternatives.

To avoid transformations requiring $O(d_m^2)$ parameters, we propose using task-specific functions of the form
\begin{equation}
    \mathrm{TS}(\mathbf{h}) = V^D g(V^E \mathbf{h}),
\end{equation}
where $V^E$ is a $d_s \!\times\! d_m$ `encoder' matrix, $V^D$ is a $d_m \!\times\! d_s$ `decoder' matrix with $d_s \!<\! d_m$, and $g(\cdot)$ is an arbitrary function. Because we can make $d_s$ as small as we like, $g(\cdot)$ can be composed of multiple layers of transformations, and not impose a large parameter budget.

We experiment with these choices for each layer of $g(\cdot)$:
\begin{itemize}
    \item Multi-head attention, optionally followed by a residual connection and layer-norm. We refer to this method as \textbf{Projected Attention}. We found $d_s=204$ worked well,
and allowed us to stay within our $1.13\times$ parameter limit.
    \item A one or two layer feed-forward network followed by a residual connection and layer-norm, such that it has the same number of parameters as the previous form; this means the intermediate layer is of size 408 (for a one layer network) or 252 (for a two layer network).
\end{itemize}

\begin{figure}[t]
\begin{center}
\centerline{\includegraphics[width=1.0in]{bert}}
\vskip -0.1in
\caption{Schematic diagram of adding a task-specific function (here our `Projected Attention Layers' or PALs) in parallel with self-attention (SA) layers in a BERT model (see section~\ref{sec:within}), with only two layers for simplicity. LN refers to layer-norm.}
\label{bert}
\end{center}
\vskip -0.2in
\end{figure}
\subsection{Adding Parameters within BERT}
\label{sec:within}
Instead of adding parameters to the top of the model, we may want to modify the $\mathrm{BERT}(\cdot)$ function itself, inspired by `residual adapter modules' (section~\ref{adapt}, \citealp{adapt}). Specifically, we wish to add task-specific parameters to each layer of the BERT model. See figure~\ref{bert} for an illustration.

We can add a task-specific function `in parallel' with each BERT layer as follows:
\begin{equation}
    \mathbf{h}^{l+1} = \mathrm{LN}(\mathbf{h}^l + \mathrm{SA}(\mathbf{h}^l) + \mathrm{TS}(\mathbf{h}^l))
\end{equation}
where $l$ indexes the layer. This means we recover the original BERT model if $\mathrm{TS}(\cdot)$ outputs a zero vector.
Alternatively we can add a `serial' connection where we transform the output of a BERT layer:
\begin{equation}
 \mathbf{\hat{h}}^{l+1} = \mathrm{LN}(\mathbf{h}^l + \mathrm{SA}(\mathbf{h}^l))
\end{equation}
\begin{equation}
\mathbf{h}^{l+1} = \mathrm{LN}(\mathbf{\hat{h}}^{l+1} + \mathrm{TS}(\mathbf{\hat{h}}^{l+1})).
\end{equation}
  In preliminary experiments, serial connections gave consistently much worse results than parallel connections, and we report results for parallel connections in what follows.

  We again consider task-specific functions of the form: \begin{equation}
    \mathrm{TS}(\mathbf{h}) = V^D g(V^E \mathbf{h}),
\end{equation}
with the difference that $V^E$ (again a $d_s \times d_m$ matrix with $d_s < d_m$) and $V^D$ (again a $d_m \times d_s$ matrix) are needed at each layer rather than only once each.

We experiment with $g(\cdot)$ taking the following forms:
\begin{itemize}
    \item The identity function; This means our task-specific transform is just a low-rank linear transformation at each layer. To satisfy our parameter constraint we need $d_s=100$. We refer to this method as \textbf{Low-rank Layers}.
    \item Multi-head attention. To satisfy our parameter constraint we need $d_s=84$. We found that it was not necessary to use the $W^o$ matrix (see section~\ref{sec:param}) when adapting within BERT, and did not use it in any of our models.
    \item  Multi-head attention, with shared $V^E$ and $V^D$ across layers (not tasks). This parameter sharing allows a larger $d_s=204$. We refer to this method as \textbf{Projected Attention Layers (PALs)}.
    \item  Shared $V^E$ and $V^D$ across layers, but with $g(\cdot)$ a feedforward network with intermediate size 306 instead of attention (and again $d_s=204)$.
\end{itemize}
The motivation behind PALs is that we want to spend our parameter budget on transformations with an inductive bias useful for sequences. The `encoder' and `decoder' matrices operate on each sequence element separately, unlike attention, which transforms the input based on the entire sequence. Finally, the attention mechanism of PALs can potentially be inspected to see which tokens in a sequence the task-specific parts of the model focus on, although we did not concentrate on this aspect in this work.

\begin{table}[t]
\vspace*{-1.3ex}
\caption{How parameters are `spent' for some of our methods, where $T$ is the number of tasks, and there are 12 layers in the base network. The $2d_md_s$ terms come from `encoder' and `decoder' matrices. PALs (section~\ref{sec:within}) use $3d_s^2$ parameters per multi-head layer (see section~\ref{sec:param}) rather than $4d_s^2$ because they do not use the final linear transform $W^o$. Projected attention (section~\ref{sec:top}) worked best with six rather than twelve layers.}
\label{param-table}
\vskip 0.15in
\begin{center}
\begin{small}
\begin{sc}
\begin{tabular}{lcccccc|r}
\toprule
Method & Parameters \\
\midrule
PALs &  $T(2d_md_s$ + 12 $\times 3d_s^2$)\\
Low rank &  $T(12 \times2d_md_s$)\\
Proj.\ Attn.\ on top &  $T(2d_md_s + 6 \times4d_s^2$)\\
\bottomrule
\end{tabular}
\end{sc}
\end{small}
\end{center}
\end{table}

\section{Multi-task Training and Experiment Setup}
\subsection{Sampling Tasks}
\label{samp}
A simple way to train a model on several tasks is to select a batch of training examples from each task, cycling through them in a fixed order. We refer to this as `round-robin' sampling. However if the tasks have different numbers of training examples, round-robin sampling may not work well. By the time we have seen every example from a particular task we could have looped through another task's smaller dataset many times. This imbalance could lead to over-fitting on smaller tasks, and under-training on larger tasks. Potentially we could alleviate this issue by manually tuning regularisation hyper-parameters for each task.

Alternatively we can use methods where we see more examples from tasks with larger associated datasets. Concretely, we select a batch of examples from task $i$ with probability $p_i$ at each training step, and set $p_i$ proportional to $N_i$, the number of training examples for task $i$:
\begin{equation}
    p_i \propto N_i.
\end{equation}
This is the approach of the multi-task BiLSTM of \citet{glue} on the GLUE benchmark, and was used by \citet{hmtl}. It has the appealing property of selecting each example with the same probability as combining all the tasks and picking examples uniformly (though we train on batches from each task not single examples).

Since the ratio of the largest to the smallest task sizes $N_i$ we use is $\approx$158, we only rarely train on some tasks with the simple $\propto N_i$ method. Training on one task (or a particular subset of tasks) for many steps can lead to interference, where performance on the other tasks suffers.
A more general approach to sampling tasks sets $p_i$ as:
\begin{equation}
    p_i \propto N_i^\alpha.
\end{equation}
If we choose $\alpha < 1$ we reduce the disparity between the probabilities of choosing tasks. We consider $\alpha=0.5$ in our experiments, and call this method \textbf{`square root sampling'}.

Finally, we noticed that it was beneficial to train on tasks more equally towards the end of training, where we are most concerned about interference, and so we constructed the \textbf{`annealed sampling'} method where $\alpha$ changes with each epoch~$e$:
\begin{equation}
    \alpha = 1 - 0.8\frac{e-1}{E-1},
\end{equation}
where $E$ is the total number of epochs. Since we used multiple datasets we chose a somewhat arbitrary `epoch' of 2400 training steps.

It was particularly important to use the square root or annealed sampling methods when sharing a pooling layer (see section~\ref{sec:param}), and it makes intuitive sense that when the layer just before the output is shared, we need to guard against interference between tasks.
\subsection{Setup}
We based our experiments on the PyTorch implementation of BERT\,\footnote{https://github.com/huggingface/pytorch-pretrained-BERT} and open-source our code\footnote{https://github.com/AsaCooperStickland/Bert-n-Pals}.
No matter how we sampled tasks, we (unless stated otherwise) trained for 60,000 steps, with a minibatch size of 32, and a maximum sequence length of 128 tokens, choosing the best model from within that training time based on average development set score.
We
use Adam with learning rate of $2\!\times\!10^{-5}$, $\beta_1= 0.9$, $\beta_2= 0.999$, L2  weight  decay  of 0.01, learning
rate warmup over the first 10\% of steps (usually 6,000), and linear
decay of the learning rate after this, going down to zero at the end of training. We note warmup followed by linear decay is the `slanted triangular learning rate' of \citet{ulmft}, who find it is suited for fine-tuning a language model on single tasks. We performed most of our experiments using either the `proportional', `square root' or `annealed' sampling methods (see section~\ref{samp}). Round robin sampling gave consistently worse results.

We use twelve heads for the attention mechanism in PALs and other methods, except when using a smaller hidden size, where we decreased it proportionally. We did not find significant performance differences when changing the number of heads. We used the same BERT-base architecture as by \citet{bert}, twelve attention heads, $d_{ff}=3072$ and $d_m=768$ (see section~\ref{sec:param}).

We found it was crucial to use the pre-trained weights for BERT-base and not start from scratch. When training from scratch, with adaption parameters or not, we got significantly worse performance. For some tasks we did not get better results than random guessing after 90,000 steps. Although we note we used the same hyper-parameters as when training from the pre-trained weights, which might not be optimal for starting from scratch.
We experimented briefly with freezing the BERT-base parameters and fine-tuning only the PALs and alternatives, but concentrated on training all of the parameters, finding it took less parameters to approach matching fine-tuned BERT.

\subsection{Details of GLUE Tasks}
We test our methods for multi-task adaptation on eight of the nine tasks in the GLUE benchmark \citep{glue}\footnote{\citet{glue} provide a more detailed discussion of these tasks.}.

\textbf{Single-sentence tasks}:
Acceptability classification with
CoLA \citep{cola}; binary sentiment
classification with SST \citep{sst}.

\textbf{Sentence pair tasks}:
Semantic similarity with the MSR Paraphrase Corpus \citep[MRPC:][]{mrpc}, STS-Benchmark \citep[STS:][]{sts} and Quora
Question Pairs (QQP) dataset, and textual entailment with Multi-Genre NLI Corpus \citep[MNLI:][]{mnli}, a subset of the RTE challenge corpora \citep{rte}, and data from
SQuAD \citep[QNLI:][]{qnli}.

Like \citet{bert} we exclude the Winograd NLI task. When systems are trained on this task they have always performed worse than the 65.1
baseline accuracy of predicting the majority class. For our submissions we also simply predicted the majority class.
\section{Experiments and Discussion}
\begin{table*}[t]
\caption{GLUE Test results, scored by the GLUE evaluation server.  The number below each task denotes the
number of training examples. We show F1/accuracy scores for QQP and MRPC, and accuracy on the matched/mismatched test sets for MNLI\@. The `Av.'\ column is slightly different than the official GLUE score,  since
we  exclude WNLI\@. `Bert-base' results are from \citet{bert}. `Shared' refers to the model where all parameters are shared except the final projection to output space. The models we tested are a result of the `annealed sampling' method for multi-task training as it produced the best results on the dev set.}
\label{glue-table}
\vskip 0.15in
\begin{center}
\begin{small}
\begin{sc}
\scalebox{0.97}{\begin{tabular}{lccccccccc|r}
\toprule
Method & Params & MNLI-(m/mm) & QQP & QNLI & SST-2 & CoLA  & STS-B & MRPC & RTE & Av.\\
& & 392k & 363k& 108k & 67k & 8.5k & 5.7k & 3.5k & 2.5k \\
\midrule
BERT-base    & 8$\times$ & \underline{84.6}/83.4  & \underline{89.2}/71.2 & \underline{90.1} & \underline{93.5} & \underline{52.1} & \underline{85.8} & \underline{84.8}/\underline{88.9} & 66.4 & 79.6 \\
\midrule
Shared    & 1.00$\times$ & 84.0/83.4  & 88.9/70.8 & 89.3 & 93.4 & 51.2 & 83.6 & 81.3/86.7 & \underline{76.6} & 79.9 \\

Top Proj. Attn.     & 1.10$\times$ & 84.0/83.2  & 88.8/71.2 & 89.7& 93.2 & 47.1 & 85.3 & 83.1/87.5 & 75.5 & 79.6\\

PALs (204)    & 1.13$\times$ & 84.3/\underline{83.5}  & \underline{89.2}/\underline{71.5} & 90.0& 92.6 & 51.2 & \underline{85.8} & 84.6/88.7 & 76.0 & \textbf{80.4}\\

\bottomrule
\end{tabular}}
\end{sc}
\end{small}
\end{center}
\vskip -0.1in
\end{table*}

Table~\ref{glue-table} lists our results on GLUE for our best-performing PAL model (chosen by average development set performance), and some alternatives. Our main comparison is against fine-tuned BERT-base, which in the absence of transfer effects represents an upper bound on our performance, since it involves tuning all BERT-base parameters to perform well on each task individually, therefore requiring approximately 8$\times$ as many parameters as our methods. By construction, apart from our adaptation parameters we use the exact same architecture as BERT-base. We note that with the exception of our results for RTE, better performance can be obtained by fine-tuning the BERT-large model that has approximately 3$\times$ the parameters of BERT-base.

The use of multi-task training significantly improves results on the RTE task, achieving state-of-the-art performance. Similar improvements have been observed with multi-task LSTM-based systems \citep{glue} and by pre-training on MNLI before fine-tuning on RTE \citep{stilts}. Since RTE has the smallest number of training examples, and is similar to MNLI, it makes intuitive sense that it benefits from multi-task training. Sharing more parameters increased performance on RTE, and our fully-shared model has slightly better performance on RTE than PALs, however PALs are the only model that matches BERT-base on the larger tasks as well as performing well on RTE\@.

For the large sentence-pair tasks, MNLI, QQP and QNLI, performance is almost exactly the same as BERT-base with PALs.
For the two single
sentence tasks: the syntax-oriented CoLA task
and the SST sentiment task we see the largest drops in performance with PALs. This is in agreement with the results of \citet{stilts} who did not observe any transfer from various intermediate tasks, and, for CoLA, mirrors the results of \citet{elmfr} that language modeling alone is the best pre-training task for CoLA\@.

\begin{table*}[t]
\caption{GLUE performance, in terms of average score across each task's development set; this score is accuracy except for CoLA, where it is Matthews correlation, and STS-B, where it is Pearson correlation. We show the mean and standard error over three random seeds, unless standard error is $<0.005$. For the details of the sampling strategies see section~\ref{samp}. For the `within BERT' methods we show the smaller hidden state size in brackets, and write `no sharing' to refer to not sharing $V^E$ and $V^D$ across layers, `top' to mean adding in parallel to the six BERT layers just before the output, and `bottom' to mean adding in parallel to the six BERT layers just after the input. }
\label{top-table}
\vskip 0.15in
\begin{center}
\begin{small}
\begin{sc}
\begin{tabular}{lcccccr}
\toprule
Method & No.\ Params & New Layers  & Prop. Samp. & Sqrt.\ Samp. & Anneal Samp. \\
\midrule
Shared    & 1.00$\times$ & 0  & 79.17$\pm$0.03 & 80.56$\pm$0.04 & 80.7$\pm$0.3 \\

\midrule
Adding on top of BERT\\
\midrule
BERT Layer & 1.66$\times$ & 1 &80.6$\pm$0.2 & 81.6$\pm$0.3 & 81.5$\pm$0.2 \\
\midrule

Proj.\ Attn.       & 1.10$\times$& 6 & 80.3$\pm$0.1  & \underline{81.4$\pm$0.1} &    81.5$\pm$0.1      \\
Proj.\ FFN (1 layer)   & 1.10$\times$& 6 &  & 81.07 &   80.8$\pm$0.1      \\
\midrule
Adding within BERT\\
\midrule
PALs (204)       & 1.13$\times$& 12 & \underline{80.6$\pm$0.2} & 81.0$\pm$0.2 & \textbf{81.7$\pm$0.2}        \\
PALs no sharing (84)       & 1.13$\times$& 12 & &  & 81.3$\pm$0.1        \\
Low Rank (100)  & 1.13$\times$& 12 & &  &\textbf{81.9$\pm$0.2}        \\
PALs (276, top)   & 1.13$\times$& 6 &  &  & 81.61$\pm$0.06        \\
PALs (276, bottom)   & 1.13$\times$& 6 &  &  & 81.4$\pm$0.1        \\
\bottomrule
\end{tabular}
\end{sc}
\end{small}
\end{center}
\vskip -0.1in
\end{table*}

\subsection{PALs and Alternatives}
Table~\ref{top-table} lists our results on the GLUE benchmark development set for various ways of adding task-specific parameters and sampling strategies.

Our best results came with PALs, or low-rank layers, adapting every layer within BERT\@. The performance of PALs increased with a larger hidden state. Having separate `encoder' and `decoder' matrices (see section~\ref{sec:within}) across layers, or having separate pooling layers for each task, with the appropriate reduction in hidden state size to make up for the extra parameters, resulted in worse performance for PALs. However sharing `encoder' and `decoder' matrices between \textit{tasks} or both layers and tasks hurt results. A larger hidden state size seems important for Transformer models,
e.g.\ the performance of BERT-large vs.\ BERT-base \citep{bert} or the ablation study by \citet{NIPS2017_7181}.

We tested two adaption layers that did not use attention: Low-rank layers, and our method with shared `encoder' and `decoder' matrices but with a small feedforward network in-between them instead of attention. The latter model did not achieve good performance,
but low-rank layers and PALs have similar mean performance.

By inspecting the best-performing single models of each method we see a contrast: the strong results for low-rank layers are partly from better performance on CoLA\@. CoLA tends to see larger changes in score between models than other tasks since
it is scored by a different measure (Matthews correlation coefficient rather than accuracy). PALs performed better for the three largest tasks, MNLI, QQP and QNLI, and equivalently for other tasks.

These results suggest PALs has greater representational capacity; the only model that achieved comparable performance on the large tasks was adding an entire BERT-layer to the top, but this model had worse performance on the RTE task and uses many more parameters. The fact that spending parameters on linear transforms in the encoder, decoder or pooling matrices gives worse performance, and the worse performance of feedforward layers compared to multi-head attention, points towards the inductive bias provided by attention being important for good performance

However at sufficiently parameter constrained regimes (for example 1.5 million parameters, which implies $d_s=10$ for low-rank transforms, and $d_s=60$ for PALs), PALs and low-rank layers performed similarly to the fully-shared model. Using the LHUC method (see section~\ref{adapt}), which requires even fewer parameters, also gave no improvement over the fully-shared baseline.

Ultimately, given the simplicity and competitive performance of low-rank layers, they remain an attractive option. There may be bigger differences for tasks like question answering which rely on the hidden states of every token in the input (as opposed to GLUE tasks which only use the final \texttt{[CLS]} hidden state to make predictions). We note that PALs and low-rank layers can easily be combined, say by using one type of adapter in the higher layers of the network and another in the lower ones.

When adding parameters to the top of BERT-base, it was important to use attention rather than feedforward transforms. Six additional layers worked best, outperforming using twelve or three layers. We also found it was crucial to use layer-norm and residual connections after each application of attention. Surprisingly, for these models using a separate pooling layer did not noticeably change results, and we report results with a shared pooling layer, which requires fewer parameters. These models saw worse performance on the RTE task, perhaps because transfer from other tasks is important, and splitting the model into multiple `heads' for each task dampens the benefits of shared knowledge.

\subsection{Where should we add Adaptation Modules?}

We draw some of the same conclusions as \citet{adapt} for `residual adapter modules'. As that work studied multi-task computer vision with residual networks (section~\ref{adapt}), we hope that these principles will apply broadly.

Adding task-specific functions \emph{within} networks works better than adding them to the \emph{top} (for a given number of parameters).  As found by \citet{adapt}, the best performing models had adaptations at \emph{every layer} of the base network, and adding adapter modules to the \emph{final half} of the base model worked better than adding to the half \emph{just after the input}. Unfortunately, adapting every layer of the base model represents the worst case for sharing operations between tasks. (We note again that this sharing is possible only when we want to perform many tasks on the same piece of text). But adapting the final half achieved slightly better performance than adding to the top of BERT-base. When adapting the final half we can still share the first six layers worth of operations, offering a useful compromise.

For within-network adaptations, \emph{parallel} connections worked better than \emph{serial} ones, also as found by \citet{adapt}. Our results with serial connections were much worse than simply not including any adapters. While the parallel configuration acts as a perturbation on the base network, the serial configuration more directly changes the hidden states being fed into the next layer. In these ways, the parallel configuration is less prone to the loss of the `knowledge' stored in the base network. We note that our serial configuration adds a newly initialised layer-norm, which may be the source of the performance drop.

\section{Further Discussion}

We found the details of how to schedule training examples from each task were important. With a lot of parameter sharing, sampling tasks proportional to dataset size impaired performance compared to our `annealing' method, where we slowly decrease the influence of dataset size on sampling probability. Annealing increased the variance of performance across random seeds as well as mean performance, meaning that we may need to pay the cost of several training runs to obtain the best single models from this method. We did not consider many variations of training method, and used no methods to reduce interference from training on separate tasks (to take one example, the `Gradient Episodic Memory' of \citealp{gem}). How these methods interact with choice of adaptation parameters is a direction for further research.

We introduced `Projected Attention Layers' as a transformation that can adapt the BERT sentence representation model for multi-task learning. PALs give a higher capacity for a given number of parameters compared to all the alternatives we considered, although simple low-rank transformations remain attractive due to their simplicity. If we adapt all the layers of BERT-base, we cannot share any operations across tasks. Ultimately the choice of which method to use depends on the constraints in place; if parameters are less constrained but you want to share as many operations as possible, adding an entire task-specific BERT layer on top of the model makes sense. If shared operations are not an issue, then adding PALs to every layer will perform well with few parameters. Finally, adapting only the final half of the base model offers a compromise between performance and sharing operations.

\section*{Acknowledgements}
We would like to thank Ivan Titov and Timothy Hospedales for useful discussion, and Elaine Farrow for help with a draft version of this paper.
Asa Cooper Stickland was supported in part by the EPSRC Centre for Doctoral Training in Data Science, funded by the UK Engineering and Physical Sciences Research Council (grant EP/L016427/1) and the University of Edinburgh.


\section{INTRODUCTION}

Scene understanding plays a crucial role in automated driving, and image recognition provides a way to achieve this. The main goal for image recognition is to identify all elements in an image. At a high level, these elements can be divided into two categories: \textit{stuff} and \textit{things} classes \cite{Forsyth1996}. \textit{Things} are countable objects, such as vehicles, persons and traffic signs. On the other hand, \textit{stuff} is the set of remaining elements, usually not countable, such as sky, road and water.

Instance segmentation and semantic segmentation are two very important image recognition tasks. Both aim at describing the content of an image as detailed as possible, and approach this in two different ways. The first task, instance segmentation, focuses on the detection and segmentation of \textit{things}. If an object is detected, a pixel mask is predicted for this object, and the output of such a method is a set of pixel masks (see Fig. \ref{fig:eye_catcher}, bottom right). By design, this method does not account for all elements in an image, as it does not consider \textit{stuff} classes. The second task, semantic segmentation, does consider all elements, as the aim is to make a class prediction for each pixel in an image, for both \textit{things} and \textit{stuff} classes. However, the semantic segmentation output does not differentiate between different instances of \textit{things} (see Fig. \ref{fig:eye_catcher}, bottom left). As a result, both methods lack the ability to fully describe the contents of an image.

To bridge this gap, the task of \textit{panoptic segmentation} has recently been introduced \cite{Kirillov2018}. For panoptic segmentation, the goal is to predict 1) a class label and 2) an instance \textit{id} for all pixels in an image. This instance \textit{id} is used to differentiate between different object instances; all pixels with the same instance \textit{id} belong to the same object. By definition, all \textit{stuff} predictions of the same class receive the same instance \textit{id}. An example is given in Fig. \ref{fig:eye_catcher} (top right). In \cite{Kirillov2018}, a baseline method is proposed that fuses the output of separate state-of-the-art semantic segmentation and instance segmentation networks using basic heuristics. This allows for use of models that are optimal for both individual tasks, but this means that there is no single network. A single network is desirable because it allows for easier implementation on devices, and it can significantly decrease the computational time and resources required to make a prediction, which is very relevant for application in intelligent vehicles.

\begin{figure}[t]
\centering
\includegraphics[width=0.49\linewidth]{input_low_res.jpg}
\includegraphics[width=0.49\linewidth]{panoptic.png}\\
\includegraphics[width=0.49\linewidth]{semantic.png}
\includegraphics[width=0.49\linewidth]{instance.png}
\caption{A panoptic segmentation prediction by the network. Top left: original input image, from the Mapillary Vistas validation set. Top right: panoptic segmentation prediction by our system. Each pixel receives a class label and all pixels belonging to specific objects also receive a unique identifier. Bottom left: semantic segmentation prediction, where all pixels only receive a class label. Bottom right: instance segmentation prediction, where only pixels of specific object classes receive a class label and an identifier label.}
\label{fig:eye_catcher}
\end{figure}

\begin{figure*}[t]
\centering
\includegraphics[width=\linewidth]{overall_architecture_v2.pdf}
\caption{Our single network architecture for panoptic segmentation. The network consists of an instance segmentation branch and semantic segmentation branch that share the same feature extractor. We introduce information exchange between the branches to improve the performance. The additional information flow is indicated in blue, and explained in Section \ref{sec:method_info_exchange}. Finally, the outputs of both branches are merged using advanced heuristics to form a panoptic output, as indicated in purple (see Section \ref{sec:method_heuristics} and Fig. \ref{fig:heuristic_overview}).}
\label{fig:baseline_architecture}
\end{figure*}

In this work, therefore, we research and present a single deep neural network for panoptic segmentation. This network consists of a common feature extractor and two different branches that output semantic segmentation and instance segmentation predictions. This joint network architecture leads both to conflicts and opportunities, which are both addressed by leveraging the most optimal information from both branches of the network. To get a final consistent panoptic segmentation output, the semantic segmentation and instance segmentation outputs are fused using advanced heuristics.

To summarize, our main contributions to street scene understanding from image data are:
\begin{itemize}
\item A single network for panoptic segmentation.
\item Inter-branch information exchange to leverage the single network architecture.
\item Improved heuristics for merging the semantic and instance segmentation predictions.
\end{itemize}

The implementation of our network is made available to the research community \cite{CodeGitlab2019}. Preliminary results were submitted to the COCO \& Mapillary Joint Recognition Challenge at ECCV 2018 \cite{DeGeus2018}.

In the remainder of this paper, we will first review the related literature in Section \ref{sec:related_work}. Thereafter, in Section \ref{sec:methodology}, we discuss our methodology. Subsequently, in  Section \ref{sec:implementation_details} the implementation details of our experiments are provided. The results on these experiments are presented in Section \ref{sec:results}. Finally, we provide conclusions in Section \ref{sec:conclusions}.

\section{RELATED WORK}
\label{sec:related_work}
The task of panoptic segmentation is closely related to semantic segmentation and instance segmentation. Both these tasks have seen great progress over the last years.

In semantic segmentation, it is very important that spatial relations are preserved, since the output is directly spatially related to the input. For this reason, the application of convolutional layers is essential.  The first semantic segmentation architecture that consists of a Fully Convolutional Network (FCN), i.e. applying only convolutional layers, was presented in \cite{Shelhamer2017}. They apply an FCN to decode the image into feature maps, make class predictions on these feature maps, and apply bilinear upsampling to create the segmentation masks. The SegNet model \cite{Badrinarayanan2017} is also an FCN, but it applies a decoding network instead of bilinear upsampling. As of recently, PSPNet is the state-of-the-art model, as it improves performance by leveraging information from different levels of the feature map, introducing a sense of context \cite{Zhao2017}.

Instance segmentation, on the other hand, is closely related to bounding box object detection. Instance segmentation extends object detection by predicting per-pixel masks for the detected objects. Therefore, many methods choose to make instance segmentation predictions by predicting instance masks for detected objects. A state-of-the art instance segmentation method is Mask R-CNN \cite{He2017}. In this approach, the object detection method of Faster R-CNN \cite{Ren2017} is extended with per-pixel instance mask predictions for for each bounding box that is likely to contain an object. Recently, the Mask R-CNN architecture has been improved with the development of Feature Pyramid Networks \cite{Lin2017} and the Path Aggregation Network \cite{Liu2018}, leading to new state-of-the-art results.

We have seen that, so far, separate instance segmentation and semantic segmentation networks have been used for panoptic segmentation \cite{Kirillov2018}. As a result, it was possible to use networks that are optimized for these specific tasks. However, there are also downsides to this method. If the predictions were made using a single network, computation time and resources could be decreased, because fewer parameters would be required. This is the case since a significant part of the processing is spent on low-level feature extraction layers that can be shared between different branches in a network. Moreover, jointly learning multiple tasks has the potential of improving performance, because information can be shared between different parts of the network. Therefore, we propose to address the task of panoptic segmentation by using a single network that makes parallel semantic segmentation and instance segmentation predictions, and fuses these outputs using heuristics.

Furthermore, we leverage the single network architecture by introducing additional information flow within the network, to enhance the overall performance of the model. In \cite{Mao2017} and \cite{Shrivastava2016}, it has been shown that additional information flow between different tasks can improve the performance of the individual subtasks. In our network, it should improve the performance of the network as a whole.

Concurrent work also focusses on a unified single network for panoptic segmentation. In \cite{Li2018a}, the method consists of a unified network similar to ours, as well as a consistency loss to make the output more consistent, but there is no additional information flow to boost the performance. AUNet \cite{Li2018b} does leverage information exchange, but it requires complicated attention and masking operations. Our framework is designed to be simple and generally applicable, while leveraging the architecture by using additional information flow to improve the performance. The increase in related concurrent work highlights the relevance of creating a single unified network for panoptic segmentation.

\section{METHODOLOGY}
\label{sec:methodology}
We propose a panoptic segmentation method that consists of three parts: a single network architecture, inter-branch information exchange to leverage this single network architecture, and advanced heuristics to fuse the outputs. The resulting network architecture is depicted in Fig. \ref{fig:baseline_architecture}.

\subsection{Single network architecture}
Our architecture jointly makes semantic segmentation and instance segmentation predictions in a single network. This network consists of a semantic segmentation and instance segmentation branch both using the same feature extractor. These branches are trained jointly and output their predictions in one pass.

In our baseline network, we use a ResNet-50 \cite{He2015} feature extractor with an output stride of 8. The original stride of ResNet-50 is 32, but in our network it is reduced to allow for denser semantic segmentation predictions \cite{Shelhamer2017}.

For the semantic segmentation branch, we follow \cite{Shelhamer2017}. In the original implementation, the predictions are made directly after the feature extractor. In our network, this feature extractor is shared with the instance segmentation branch. This means that there are only very few parameters that are used only for the semantic segmentation task. This could lead to decreased performance. For this reason, we add a Pyramid Pooling Module (PPM) \cite{Zhao2017} to the semantic segmentation branch. This PPM is introduced as a general improvement to semantic segmentation, but in our network it also acts as an adaptation network. Finally, to generate the final output of this branch, we apply hybrid upsampling to reshape the predictions to the size of the input image \cite{Meletis2018}. This hybrid upsampling technique first applies a learnable deconvolution operation and then bilinearly resizes the predictions to the dimensions of the input image.

The instance segmentation branch is based on Mask R-CNN \cite{He2017}. First, a Region Proposal Network (RPN) is used to generate region proposals for potential objects in the image. The features corresponding to these proposals are then extracted from the feature map and subjected to the convolutional layers of the final ResNet-50 block. Finally, these features are used to make three different predictions for each region proposal: a classification score, bounding box coordinates, and an instance mask. After applying non-maximum suppression, the output of this branch is a set of detected objects consisting of  class, bounding box and per-pixel mask predictions.

To enable joint learning for this network, a single loss function is formed. This means that the various loss functions from the different network branches have to be combined and balanced. The total loss, $L_{tot}$, is given by

\begin{equation}
\label{eq:loss}
\begin{aligned}
L_{tot} &= {\lambda_1}{L_{rpn,obj}} + {\lambda_2}{L_{rpn,reg}} \\ &+
{\lambda_3}{L_{det,cls}} + {\lambda_4}{L_{det,reg}} \\ &+
{\lambda_5}{L_{mask}} \\ &+ {\lambda_6}{L_{seg}} \\ &+  {\lambda_7}{R}.
\end{aligned}
\end{equation}

Here, $L_{rpn,obj}$ is the softmax cross-entropy objectness loss function for the RPN, $L_{rpn,reg}$ is the smooth L1 regression loss function for the RPN \cite{Girshick2015}, $L_{det,cls}$ is the softmax cross-entropy classification loss function for object detection, $L_{det,reg}$ is the smooth L1 regression loss function for the object bounding boxes, $L_{mask}$ is the sigmoid cross-entropy loss on the instance masks, and $L_{seg}$ is the sparse softmax cross-entropy segmentation loss on the semantic segmentation outputs. Finally, $R$ is the L2 regularization on the model parameters. The weights $\lambda_1...\lambda_n$ are the $n$ tuning parameters that are used to balance the losses. The values used for these parameters are discussed in Section \ref{sec:implementation_details} and provided in Table \ref{tab:loss_weights}.

\subsection{Inter-branch information exchange}
\label{sec:method_info_exchange}

\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{info_exchange_architecture_v2.pdf}
\caption{The additional information flow for implicit information exchange. The added flow and components are indicated in red. \textit{Norm} and \textit{Concat} represent normalization and concatenation operations, respectively. \textit{Conv} is a 3x3 convolutional layer.}
\label{fig:info_exchange}
\end{figure}

Our single network architecture for panoptic segmentation introduces several opportunities over the use of separate networks. Firstly, jointly learning the semantic and instance segmentation tasks can improve the performance of both tasks, because the tasks might require similar features, which they can both retrieve and influence using the shared feature extractor. Secondly, the architecture allows to introduce additional information flow between the two semantic segmentation and instance segmentation branch; we do this in multiple basic but effective ways.

\subsubsection{Explicit information}
Certain \textit{things} predictions from the semantic segmentation branch are better than the predictions by the instance segmentation branch. Since the final output only contains \textit{things} predictions from the instance segmentation branch, potentially valuable information is lost, leading to a lower performance. To compensate for this, we use the \textit{things} predictions by the semantic segmentations to improve the instance segmentation output, in two different ways.

Firstly, we add bounding boxes to the region proposals generated by the RPN, based on the semantic segmentation output. We identify all \textit{things} clusters in the semantic segmentation output, generate bounding boxes for these clusters, and use them as additional region proposals. Secondly, we expand bounding boxes predicted by the detection branch based on the semantic segmentation output. We match all predicted bounding boxes with the corresponding \textit{things} class in the semantic segmentation output, and expand the box if the matched segment extends beyond the boundary of the box.

\subsubsection{Implicit information}
As became clear from \cite{Mao2017} and \cite{Shrivastava2016}, it can be beneficial to implicitly use semantic segmentation information to improve instance segmentation as well. In our network, we follow part of the method proposed by \cite{Shrivastava2016} and introduce a very basic additional information channel. We use the output from the semantic segmentation branch before the final softmax layer, normalize it and concatenate it to the normalized features from the feature map. We then apply a 3x3 convolutional layer and use the output from this layer as input to the instance segmentation branch. By doing so, we can improve the performance of both the semantic and instance segmentation branch, because the forward and backward pass through the network allow for relevant data from one branch to flow through the other. The additional information flow is depicted in Fig. \ref{fig:info_exchange}.

\subsection{Advanced merging heuristics}
\label{sec:method_heuristics}

\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{heuristic_overview_v3.pdf}
\caption{An overview of the heuristics used for merging the instance segmentation and semantic segmentation predictions. On the top branch, we first transform the instance segmentation predictions to generate full-image instance masks. Then, we remove overlap to get a single \textit{things} prediction for each pixel. On the bottom branch, we replace the \textit{things} predictions and end up with \textit{stuff} predictions only. Finally, we generate the panoptic output by overlaying the \textit{stuff} predictions with the \textit{things} predictions.}
\label{fig:heuristic_overview}
\end{figure}

Because our network outputs two separate predictions in parallel, these outputs have to be processed in order to generate a panoptic segmentation prediction. For panoptic segmentation, two values have to be predicted for each pixel: a class label and an instance \textit{id}. There are essentially two conflicts that need to be solved before being able to generate this output: overlapping instance masks, and conflicting predictions for \textit{things} classes by the two branches. In addition to this, we apply a heuristic that removes unlikely \textit{stuff} predictions. An overview of the merging heuristics is shown in Fig. \ref{fig:heuristic_overview}.

\begin{table*}[t]
\caption{ The overall results of our method on the Mapillary Vistas validation set.}
\label{tab:results_mapillary_overall}
\centering
\begin{tabular}{ l r@{\hspace{2px}}|| c | c | c || c | c ||| c }
Method &  & PQ & SQ & RQ & PQ\textsubscript{Th} & PQ\textsubscript{St} & Prediction time\\
\hline
Multiple networks & \hspace{20px} & 21.3 & 65.2 & 25.3 & 13.8 & 31.4 & 903 ms\\
Single network & \tikzmark{b}& 21.0 & 65.6 & 27.5 & 14.9 & 29.2 & 532 ms \\
+ Advanced heuristics &  \tikzmark{c}& 23.1 & \textbf{66.5} & 30.3 & 14.8 & 34.1 & \textbf{451 ms}\\
+ Inter-branch information exchange & \tikzmark{d} & \textbf{23.9} & 66.0 & \textbf{31.2} & \textbf{15.5} & \textbf{35.0} & 484 ms\\
\end{tabular}
\begin{tikzpicture}[overlay, remember picture, shorten >=.5pt, shorten <=.5pt]
\draw [->]({pic cs:b}) [bend right] to ({pic cs:c}) node[above=5pt, left=1pt]{+2.1};
\draw [->]({pic cs:c}) [bend right] to ({pic cs:d}) node[above=5pt, left=1pt]{+0.8};
\end{tikzpicture}

\end{table*}

\begin{table*}[t]
\caption{The overall results of our method on the Cityscapes validation set.}
\label{tab:results_cityscapes_overall}
\centering
\begin{tabular}{ l r@{\hspace{2px}}|| c | c | c || c | c ||| c }
Method &  & PQ & SQ & RQ & PQ\textsubscript{Th} & PQ\textsubscript{St} & Prediction time\\
\hline
Multiple networks & \hspace{20px} & 43.7 & 74.5 & 55.6 & 36.5 & 48.9 & 1361 ms\\
Single network & \tikzmark{e}& 42.9 & 74.3 & 54.5 & 37.8 & 46.7 & 810 ms \\
+ Advanced heuristics &  \tikzmark{f}& 44.4 & 74.4 & 56.5 & 38.4 & 48.7 & \textbf{506 ms}\\
+ Inter-branch information exchange & \tikzmark{g} & \textbf{45.9} & \textbf{74.8} & \textbf{58.4} & \textbf{39.2} & \textbf{50.8} & 590 ms\\
\end{tabular}
\begin{tikzpicture}[overlay, remember picture, shorten >=.5pt, shorten <=.5pt]
\draw [->]({pic cs:e}) [bend right] to ({pic cs:f}) node[above=5pt, left=1pt]{+1.5};
\draw [->]({pic cs:f}) [bend right] to ({pic cs:g}) node[above=5pt, left=1pt]{+1.5};
\end{tikzpicture}

\end{table*}

\subsubsection{Overlap removal for things classes}
Because the instance segmentation prediction is essentially based on an object detector and many overlapping region proposals, there can be overlap between different predicted instance masks. In the baseline method proposed by \cite{Kirillov2018}, overlap is removed by prioritizing instance masks with higher corresponding classification scores. In our method, we choose to leverage the per-instance and per-pixel score maps to resolve conflicting sections. First, we transform all predicted instance masks to the full image size. Then, in the case that two or more instance masks predict that a certain pixel belongs to their object, the pixel is assigned to the instance mask with the highest score at that specific pixel. We choose to use per-pixel scores because it is more intuitive to solve per-pixel conflicts using per-pixel scores. As a result of this heuristic, all output pixels are assigned to only one object.

\subsubsection{Merging outputs from both branches}
\label{sec:method_heuristics_merging}
Unlike the \textit{stuff} classes, which are only considered in the semantic segmentation branch, the \textit{things} classes are part of the prediction of both the semantic segmentation and the instance segmentation branch. As a result, there are inevitably \textit{things} prediction conflicts between the two outputs. Because the semantic segmentation output does not distinguish between different instances of objects, the two outputs cannot be compared directly. Similarly to the baseline method in \cite{Kirillov2018}, we prioritize the instance segmentation output over the semantic segmentation output. In the baseline method, this is done by replacing all pixels with \textit{things} class predictions by the semantic segmentation branch with \textit{void} labels. To avoid the loss of potentially useful information, we improve the baseline heuristic by replacing \textit{void} labels by high scoring \textit{stuff} predictions, given that the score for that pixel is above a threshold $\alpha$. We use $\alpha = 0.25$. Finally, as in \cite{Kirillov2018}, the instance segmentation output is used to replace the \textit{stuff} and \textit{void} labels at pixels where it predicts \textit{things}. Because all these instance masks have a unique \textit{id}, the result of this heuristic is an output in the panoptic segmentation format.

\subsubsection{Removing unlikely \textit{stuff}}
As a third heuristic, any predicted \textit{stuff} class with a total pixel count below a given threshold is removed from the output as well. These predictions are then replaced by either \textit{void} labels or high scoring \textit{stuff} classes above this threshold, following the procedure described in Section \ref{sec:method_heuristics_merging}. This is done because it is very unlikely that a \textit{stuff} class consists of a small number of pixels, if it is present in an image. This heuristic is proposed by the organizers of the COCO Panoptic Segmentation Challenge during ECCV 2018, in their auxiliary code \cite{COCO2018}. In this code, they use a fixed pixel threshold of 4096. However, it is likely that this number depends on the size of the image. Therefore, we use a threshold that is a constant fraction, $f$, of the total amount of pixels of an image. The ideal value for this fraction depends on the dataset, as is described in Section \ref{sec:implementation_details}.

\section{IMPLEMENTATION DETAILS}
\label{sec:implementation_details}
We implement our methodology using TensorFlow. For training, we optimize the loss function in Eq. \ref{eq:loss} using a stochastic gradient descent optimizer with a momentum of 0.9. The loss and regularization weights are provided in Table \ref{tab:loss_weights}. These weights are found empirically and iteratively. Batch normalization is applied to all but the output layers, with a weight decay of 0.9. The network initialized using weights pre-trained on the ImageNet dataset \cite{Deng2009}, except for the models using inter-branch information exchange. When training these models, we initialize on a model that is pre-trained for semantic segmentation on the specific dataset, so that less unreliable semantic segmentation information is shared with the instance segmentation branch. We always use a single Nivia Titan Xp GPU for training.

\begin{table}[H]
\caption{The loss and regularization weights.}
\centering
\begin{tabular}{ c | c | c | c | c | c | c}
$\lambda_1$ & $\lambda_2$ & $\lambda_3$ & $\lambda_4$ & $\lambda_5$ & $\lambda_6$ & $\lambda_7$ \\ \hline
1.0 & 1.0 & 1.0 & 0.15 & 0.3 & 1.0 & 5.5e-5 \\
\end{tabular}
\label{tab:loss_weights}
\end{table}

We evaluate the network on two different street scene datasets: Cityscapes \cite{Cordts2016Cityscapes} and Mapillary Vistas \cite{Neuhold2017}. Because the two datasets have different properties, we use slightly different learning rate schedules and hyperparameters for training on each dataset.

\subsection{Cityscapes}
Cityscapes is a dataset that consists of 5k street scene images, which have all been taken in German cities. There are panoptic annotations for 8 \textit{things} classes and 11 \textit{stuff} classes. All images have a size of 1024 x 2048 pixels. For training, to allow for a batch size of 2, we resize the dimensions of the input images to 512 by 1024 pixels. For this dataset, we use a polynomial decay schedule for the learning rate, as in \cite{Chen2018}. We train for 30 epochs, and use an initial learning rate of 0.075 and a power of 0.9. Finally, it is found that stuff removal fraction $f = \frac{1}{512}$ leads to the best results for this dataset.

\subsection{Mapillary Vistas}
Mapillary Vistas is a more challenging dataset, consisting of 25k street scene images. The images have all been captured at different locations all around the world, and have panoptic annotations for 37 \textit{things} classes and 28 \textit{stuff} classes. The images have a very high resolution, the average being 2481 by 3419 pixels. To achieve state-of-the-art results on this method, high-resolution networks are required. The best-scoring instance segmentation method on Mapillary Vistas resizes input images so that the larger side is equal to 2400 pixels \cite{Liu2018}. However, this is not feasible in our implementation, because of memory requirements for joint learning and limited memory capacity. Therefore, the feature extractor has input dimensions of 640 x 900 pixels. This allows for the use of a batch size of 2. For Mapillary Vistas, we use stepwise learning rate schedule. We train for 21 epochs, use an initial learning rate 0.075, and multiply the learning rate by 0.5 after 8 and 14 epochs. Finally, for the stuff removal heuristic, we use a fraction of $f = \frac{1}{256}$.

\section{RESULTS}
\label{sec:results}
In this section, we present the results of our implemented network on Cityscapes and Mapillary Vistas. First, we describe the metrics in Section \ref{sec:results_metrics}. In addition to the overall performance of the network, discussed in Section \ref{sec:results_overall}, we also present ablation results on the the different inter-branch information exchange methods in Section \ref{sec:results_info_exchange}.

\subsection{Metrics}
\label{sec:results_metrics}
For panoptic segmentation evaluation, we use the Panoptic Quality (PQ) metric, as defined in \cite{Kirillov2018}. This PQ metric can be split into Segmentation Quality (SQ) and Recognition Quality (RQ), and is a product of these two terms. Here, the RQ indicates the ability of the network to recognize objects, and the SQ describes the ability to find accurate pixel masks for the objects that are actually detected. To investigate the performance of the two different branches of the network, we evaluate for \textit{things} (PQ\textsubscript{Th}) and \textit{stuff} (PQ\textsubscript{St}) classes separately as well. It should be noted that the range of scores achieved for the PQ metric varies heavily per dataset. It is not necessary for a network to achieve a score of 100 to be useful for self-driving vehicles applications. For instance, the top two pictures in Fig. \ref{fig:example_images} achieve a PQ score of 58.9 and 27.5, respectively.

To evaluate the real-time applicability of our system, we also evaluate the single image inference time when using a single Nvidia Titan Xp GPU.

\begin{table}[t]
\caption{Results for different inter-branch information exchange methods, on the Mapillary Vistas validation set.}
\label{tab:results_info_exchange}
\centering
\begin{tabular}{ l ||@{\hspace{4px}} c @{\hspace{4px}}|@{\hspace{4px}} c @{\hspace{4px}}| @{\hspace{4px}}c @{\hspace{4px}}}
 Method & PQ & PQ\textsubscript{Th} & PQ\textsubscript{St} \\ \hline
Single network & 22.9 & 14.4 & 34.2 \\
+ Adding region proposals & 23.0 & 15.1 & 33.5 \\
+ Expanding bounding boxes & 23.1 & 15.3 & 33.5  \\
+ Implicit information exchange & \textbf{23.9} & \textbf{15.5} & \textbf{35.0}\\
\end{tabular}
\vspace{-15pt}
\end{table}

\vspace{-1pt}

\subsection{Overall results}
\label{sec:results_overall}
The overall results for the Mapillary Vistas and Cityscapes datasets are presented in Table \ref{tab:results_mapillary_overall} and \ref{tab:results_cityscapes_overall}, respectively. Firstly, we compare the single network with the approach using separate networks, using the baseline heuristics from \cite{Kirillov2018}. It can be seen that jointly learning the tasks in a single network greatly reduces the required prediction time. However, there is a drop in performance on the PQ metric. Also, it should be noted that the prediction time can be reduced even further by optimizing the implementation of the model for speed. This has not been done for our implementation.

With respect to the baseline single network, we first improve the performance by using advanced heuristics. It can be seen that this especially improves the performance of the \textit{stuff} classes. This is as expected, since most of the improvements to the heuristics aimed at making more accurate \textit{stuff} predictions. Moreover, it is found that the prediction time is decreased as well. This is the result of implementing the new overlap removal heuristic, that directly compares the per-pixel scores. Secondly, implementing the inter-branch information exchange gives a final performance boost. Ablation results on the inter-branch information exchange are presented in Section \ref{sec:results_info_exchange}.

Qualitative results of our method are shown in Fig. \ref{fig:example_images}. In Fig. \ref{fig:examples_cityscapes}, we compare predictions by our network with predictions by the separate networks and the ground truth.

\subsection{Inter-branch information exchange ablation results}
\label{sec:results_info_exchange}
Inter-branch information exchange is the final contribution of our method. In Table \ref{tab:results_mapillary_overall}, it has already been shown that the information exchange improves the overall performance on the Mapillary Vistas dataset. In Table \ref{sec:results_info_exchange}, we evaluate the performance of the different individual information exchange methods. Note that all methods in this table are initialized on a pre-trained semantic segmentation model, for a fair comparison.  The results show that the addition of region proposals and expansion of bounding boxes improves the PQ score on the \textit{things} classes, as intended. The additional implicit information exchange between the branches impacts the performance of both branches, because of the additional information flow passing through both branches.

In Table \ref{tab:results_cityscapes_overall}, it can be seen that including inter-branch information exchange also improves the performance of the network on the Cityscapes dataset. Again, the PQ is improved on both the \textit{things} and the \textit{stuff} classes, while only slightly increasing the required prediction time.

\begin{figure}[t]
\centering
\includegraphics[width=0.48\linewidth]{map_in_1_low_res.jpg}
\includegraphics[width=0.48\linewidth]{map_out_1.png}\\
\includegraphics[width=0.48\linewidth]{map_in_2_low_res.jpg}
\includegraphics[width=0.48\linewidth]{map_out_2.png}\\
\includegraphics[width=0.48\linewidth]{map_extra_in_1_low_res.jpg}
\includegraphics[width=0.48\linewidth]{map_extra_out_1.png}\\\includegraphics[width=0.48\linewidth]{map_extra_in_3_low_res.jpg}
\includegraphics[width=0.48\linewidth]{map_extra_out_3.png}\\
\caption{Panoptic segmentation predictions by the network. Images from the Mapillary Vistas validation set. Each output pixel receives a color-coded class label and specific objects also receive a color-coded identifier label.}
\label{fig:example_images}
\vspace{-10pt}
\end{figure}

\begin{figure*}[t]
\centering
    \centering
\subfloat{\includegraphics[width=.245\linewidth]{pedestrian_input.jpg}}
    \hfill
\subfloat{\includegraphics[width=.245\linewidth]{pedestrian_gt.png}}
	\hfill
\subfloat{\includegraphics[width=.245\linewidth]{pedestrian_bad.png}}
    \hfill
\subfloat{\includegraphics[width=.245\linewidth]{pedestrian_good.png}}\\
\subfloat[Input image]{\includegraphics[width=.245\linewidth]{cyclist_input.jpg}}
    \hfill
\subfloat[Ground truth]{\includegraphics[width=.245\linewidth]{cyclist_gt.png}}
	\hfill
\subfloat[Baseline separate networks]{\includegraphics[width=.245\linewidth]{cyclist_bad.png}}
    \hfill
\subfloat[Our single network]{\includegraphics[width=.245\linewidth]{cyclist_good.png}}\\

\caption{Panoptic segmentation examples on crops from the Cityscapes validation set. Different color shades indicate different instances of a certain class, and are randomly generated. Our single network is able to detect the pedestrian and bicycle due to the information flow between the branches and the advanced heuristics.}
\label{fig:examples_cityscapes}
\end{figure*}

\section{CONCLUSIONS}
\label{sec:conclusions}
With this work, we have taken a step towards holistic street scene understanding from image data, by presenting a single deep neural network for panoptic segmentation. This single network approach allows for easier implementation on devices, such as intelligent vehicles. Moreover, it reduces the needed computation time by a factor of 2 with respect to the use of separate networks. It is shown that, for a single network approach to achieve better Panoptic Quality than separately learned networks, it is crucial to exchange additional information between different parts of the single network. Moreover, we improve the merging heuristics by using the most likely and most reliable information from both the instance segmentation and semantic segmentation outputs. These improvements result in a performance increase of +2.9 and +3.0 on the PQ metric, on the Mapillary Vistas and Cityscapes validation sets, respectively. In future work, to realize full end-to-end panoptic segmentation, our aim is to research a differentiable merging method to replace the current non-differentiable merging heuristics.

\addtolength{\textheight}{-12cm}

{\small
\bibliographystyle{ieeetran}