\subsection{Training Details}\label{sec:appendix:training_details}
For all experiments, we use a publicly available pretrained BERT-base model and fine-tune it on the 3 datasets: synthetic SST2, IMDB and Toxicity.
We use a dropout rate of 0.5, a batch size of 16 and train each model for a maximum number of 35k steps.
To mitigate overfitting, we stop training if after 10k steps there is no improvement on the validation set and use the model checkpoint that had highest performance on the validation set.
The models are optimized using ADAM~\citep{KinBa15s} with weight decay of 5e-6.
For varying hyper-parameters and model performance, see Tab.~\ref{tab:appendix:training_details}.
With these hyper-parameters, the BERT models contain 109M parameters.
Training and optimizing them took a total of 35 hours on TPUv2 (4-core) accelerators.
Test set accuracies are in line with previously reported performances \citep{DevChaLee+19a,SanDebCha+19h}.

\begin{table*}[t]
    \centering
    \begin{tabular}{l r r r}
    \toprule
    Dataset & Max Seq. Length & Learning Rate & Test Set Acc.\\
    \midrule
    Synthetic SST2 & 100 & 2e-5 & .94 \\
    IMDB & 500 & 2e-5 & .93 \\
    Toxicity & 128 & 1e-5 & .93 \\
    \bottomrule
    \end{tabular}
    \caption{Training details of the BERT-base model.}
    \label{tab:appendix:training_details}
\end{table*}

\subsection{Alternative Clustering Configurations}\label{sec:appendix:clustering_configurations}

Tab.~\ref{tab:appendix:synthetic_representation_comparison} shows all results (including IG) from the verification test on synthetic data (Sec.~\ref{sec:synthetic_data}).
Clusterings based on IG salience lead to high precision but lower recall.
Results for the task of discovering spurious correlations in IMDB (Sec.~\ref{sec:imdb_numeric_clusters}) are shown in Tab.~\ref{tab:appendix:imdb_numeric_representation_comparison}.
Here, clusters computed on IG-based representations lead to results similar to the baselines.

\paragraph{Top-$k$ Salient Tokens}

In both tasks using the entire vocabulary to compute the representation matrices ($k=|\mathcal{V}|$) leads to results that are similar to $k=5$ (see Tab.~\ref{tab:appendix:synthetic_representation_comparison} and \ref{tab:appendix:imdb_numeric_representation_comparison}).

\paragraph{Number of Clusters}

Increasing the number of clusters from 3 to 20 on the synthetic SST2 dataset shows how sensitive the avg. embedding baseline is (compare Tab.~\ref{tab:appendix:synthetic_representation_comparison} and Tab.~\ref{tab:appendix:synthetic_representation_comparison_20_clusters}).
PMI does not benefit from more clusters, still producing a tiny cluster.
The \textsc{cls} and salience-based representations are rather stable and lead to similar results for both configurations.

We see similar results on the more realistic dataset IMDB, where we want to rediscover a cluster of numeric examples (Tabs.~\ref{tab:appendix:imdb_numeric_representation_comparison} and \ref{tab:appendix:imdb_numeric_representation_comparison_20_clusters}).
The avg. embeddings and PMI baselines perform poorly in both settings.
The results for \textsc{CLS}-encoding representations vary strongly.
For 20 clusters this representation creates a cluster that has high precision but is rather small.
Again, the salience-based representations are rather insensitive to this hyperparameter (salience embeddings more so than salience vocab).
The salience vocab. representation leads to the best performance in terms of precision in both settings.

\begin{table}[t]
    \centering

    \begin{tabular}{l r r r c}
    \toprule
    Representation & Size & Prec. & Recall & top-$k$ \\
    \midrule
    avg emb. & .36 & .35 & .62 & - \\
    PMI vocab & .00 & .50 & .00 & -  \\
    \textsc{cls}-encoding & .20 & 1.00 & 1.00 & - \\
    \midrule
    salience vocab. G-L2 & .20 & 1.00 & 1.00 & 5 \\
    salience emb. G-L2 & .22 & .93 & 1.00 & 5 \\
    \midrule
    salience vocab. G-L2 & .20 & 1.00 & 1.00 & $|\mathcal{V}|$ \\
    salience emb. G-L2 & .21 & .95 & 1.00 & $|\mathcal{V}|$ \\
    \midrule
    salience vocab. IG & .17 & 1.00 & .85 & 5 \\
    salience emb. IG & .16 & 1.00 & .80 & 5 \\
    \bottomrule
    \end{tabular}
    \caption{Validating the Approach on Synthetic Data with 3 Clusters: \emph{Precision} and \emph{Recall} of the cluster that has the highest precision for the respective representation.
    Both are computed based on 227 synthetic examples in the data.
    \emph{Size} is the data fraction of the selected cluster compared to the entire set.
    }
    \label{tab:appendix:synthetic_representation_comparison}
\end{table}

\begin{table}[t]
    \centering
    \begin{tabular}{l r r r c}
    \toprule
    Representation & Size & Prec. & Recall & top-$k$ \\
    \midrule
    avg emb. & .00 & .75 & .03 & - \\
    PMI vocab & .00 & 1.00 & .00 & -  \\
    \textsc{cls}-encoding & .19 & 1.00 & .96 & - \\
    \midrule
    salience vocab. G-L2 & .20 & 1.00 & 1.00 & 5 \\
    salience emb. G-L2 & .20 & .97 & .99 & 5 \\
    \bottomrule
    \end{tabular}
    \caption{Validating the Approach on Synthetic Data with 20 Clusters: \emph{Precision} and \emph{Recall} of the cluster that has the highest precision for the respective representation.
    Both are computed based on 227 synthetic examples in the data.
    \emph{Size} is the data fraction of the selected cluster compared to the entire set.
    }
    \label{tab:appendix:synthetic_representation_comparison_20_clusters}
\end{table}

\begin{table}[t]
    \centering
    \begin{tabular}{l r r r c}
    \toprule
    Representation & Size & Prec. & Recall & top-$k$ \\
    \midrule
    avg emb. & .08 & .16 & .13 & - \\
    PMI vocab & .16 & .12 & .19 & - \\
    \textsc{cls}-encoding & .47 & .12 & .58 & - \\
    \midrule
    salience vocab. G-L2 & .05 & .98 & .50 & 5 \\
    salience emb. G-L2 & .05 & .93 & .53 & 5 \\
    \midrule
    salience vocab. G-L2 & .05 & .99 & .52 & $|\mathcal{V}|$ \\
    salience emb. G-L2 & .04 & .93 & .48 & $|\mathcal{V}|$ \\
    \midrule
    salience vocab. IG & .12 & .13 & .16 & 5 \\
    salience emb. IG & .11 & .14 & .16 & 5 \\
    \bottomrule
    \end{tabular}
    \caption{Spurious Correlations in IMDB with 5 Clusters:
    \emph{Precision} and \emph{Recall} of the cluster that has the highest precision for the respective representation.
    Both are computed based on 245 numeric examples in the data.
    \emph{Size} is the data fraction of the selected cluster compared to the entire set.
    }
    \label{tab:appendix:imdb_numeric_representation_comparison}
\end{table}

\begin{table}[t]
    \centering
    \begin{tabular}{l r r r c}
    \toprule
    Representation & Size & Prec. & Recall & top-$k$ \\
    \midrule
    avg emb. & .05 & .20 & .10 & - \\
    PMI vocab & .05 & .15 & .09 & - \\
    \textsc{cls}-encoding & .02 & .94 & .21 & - \\
    \midrule
    salience vocab. G-L2 & .04 & .98 & .47 & 5 \\
    salience emb. G-L2 & .02 & .81 & .22 & 5 \\
    \bottomrule
    \end{tabular}
    \caption{Spurious Correlations in IMDB with 20 Clusters:
    \emph{Precision} and \emph{Recall} of the cluster that has the highest precision for the respective representation.
    Both are computed based on 245 numeric examples in the data.
    \emph{Size} is the data fraction of the selected cluster compared to the entire set.
    }
    \label{tab:appendix:imdb_numeric_representation_comparison_20_clusters}
\end{table}

\subsection{Spurious Correlations in IMDB}\label{sec:appendix:problematic_patterns}
Fig.~\ref{fig:appendix:wordcloud_imdb_numeric_cluster} depicts a word cloud that was computed from the numeric cluster.
Individual words are weighted by their average representation value (i.e., the mean of columns of $R$).

\begin{figure}[t!]
    \centering
    \includegraphics[width=\columnwidth]{wordcloud.png}
    \caption{Spurious Correlations in IMDB: Word cloud of cluster no. 3, which has tokens `10', `4', `3', `/' and `2' as top terms.}
    \label{fig:appendix:wordcloud_imdb_numeric_cluster}
\end{figure}

In the following, we list the regular expressions used to identify examples containing numeric expressions.

\begin{verbatim}
RATING_EXPRESSIONS = [
  # on, out of, ...
  r': (NUMBER) \( out of\b',
  r'\b(NUMBER) out of\b',
  r'\b(NUMBER)\* out of\b',
  r'\b(NUMBER) outta \d+\b',
  # Exclude, e.g., "my top 10 of".
  r'\b[^(top )](NUMBER) of \d+\b',
  r'\b(NUMBER) on \d+\b',
  r'\b(NUMBER) out \d+\b',
  # something / something
  r'\b(NUMBER) / 10\b',
  r'\b(NUMBER) / 5\b',
  # vote, ratings, ...
  r'\bvote \( (NUMBER) \)',
  r'\bvote (NUMBER)\b',
  r'\bvoted \" (NUMBER) \"',
  r'\bvote is (NUMBER)\b',
  r'\bgive this a (NUMBER)\b',
  r'\bgive it a (NUMBER)\b',
  r'\bgets a (NUMBER)\b',
  r'\brate this a (NUMBER)\b',
  r'\" (NUMBER) \" rating\b',
  # stars, points, ...
  r'\b(NUMBER) star\b',
  r'\b(NUMBER) - star\b',
  r'\b(NUMBER) stars\b',
  r'\b(NUMBER) points\b',
  # Exclude, e.g., "at 1 point".
  r'\b[^(at)] (NUMBER) point\b',
]
\end{verbatim}

We substitute every \texttt{NUMBER} in these base expressions with the following expression:

\begin{verbatim}
zero|one|two|three|four|five|six|
seven|eight|nine|ten|\d+|\d+\.\d+|
\d+ \. \d+|\d+\+
\end{verbatim}

Finally, we consider an example to contain a numeric expression if at least one of the expressions is found in the text.

\subsection{Understanding Learned Patterns}\label{sec:appendix:learned_patterns}
Fig.~\ref{fig:appendix:gay_clusters} zooms into two non-toxic clusters computed over all the test examples mentioning the term \textit{`gay'}.

\begin{figure}[t!]
    \centering
    \includegraphics[width=\columnwidth]{gay_clusters.png}
    \caption{Patterns in Wikipedia Toxicity Subtypes: t-SNE visualization of Toxicity test examples mentioning \textit{`gay'}. Two non-toxic clusters are selected and the 10 nearest neighbors (represented here by their most salient terms in the boxes) of the respective centrally located points are shown. All these examples have \textit{`(homo)sexual(ity)'} or \textit{`LGBT'} along with \textit{`gay'} among their most salient terms.
    }
    \label{fig:appendix:gay_clusters}
\end{figure}

\subsection{Identifying Model Sensitivities}\label{sec:appendix:model_sensitivity}
Tab.~\ref{tab:appendix:sensitivity_seed_lists} lists all seed terms we used in the sensitivity analysis of Sec.~\ref{sec:model_sensitivity}.

\begin{table*}[t]
    \centering
    \begin{tabular}{l l}
    \toprule
    Seed List & Seed Terms \\
    \midrule
    \citet{DixLiSor+18n} & atheist, queer, gay, transgender, lesbian, homosexual, feminist, black, \\
    & white, heterosexual, islam, muslim, bisexual \\
    PMI(token; toxic class) & fuck, fucking, nipple, bitch, asshole, suck, lick, sucking, cock, vomit \\
    aggregated salient terms & fuck, fucking, shit, stupid, idiot, bitch, suck, asshole, ass, dick \\
    \bottomrule
    \end{tabular}
    \caption{Identifying Model Sensitivities: Seed lists from various sources.}
    \label{tab:appendix:sensitivity_seed_lists}
\end{table*}

\subsection{Generalizing from Single Examples}\label{sec:appendix:single_example_generalization}
Tab.~\ref{tab:appendix:toxicity:examples_obvious} lists 5 examples whose predictions are not surprising.
Looking at the examples' text is enough to justify the \emph{toxic} label.

\begin{table*}[t]
\centering
\small
\begin{tabular*}{\textwidth}{p{.1cm} p{9.cm} l l}
\toprule
    & Example &  Top- Terms & Label / Prediction \\

    \midrule
    1. &
    failure at life loser . &
    loser, failure, life, [\textsc{sep}], [\textsc{cls}] &
    non-toxic / toxic \\

    2. &
    = = = = oh the irony = = = it just goes to show how stupid some americans really are , a scumbag mob lawyer mayor of las vegas . you couldn ' t make it up ! ! ! !  &
    stupid, irony, mo, \#\#um, americans &
    non-toxic / toxic \\

    3. &
    : oh and don ' t be a patronising ass with comments , like justin , justin . ' ' ' ' ' ' ' '  &
    ass, patron, \#\#ising, comments, with &
    non-toxic / toxic \\

    4. &
    what the f * ck is your problem ? get a life . you don ' t own any pages .  &
    ck, f, *, problem, pages &
    non-toxic / toxic \\

    5. &
    stop f * cking up the call - out tables please .  &
    f, ck, *, please, tables &
    non-toxic / toxic \\

\bottomrule
\end{tabular*}
\caption{Generalizing from Single Examples: 5 examples with their top-5 terms whose labels are not surprising.}
\label{tab:appendix:toxicity:examples_obvious}
\end{table*}

\section{Introduction}

\begin{figure}[t!]
    \centering
    \includegraphics[width=\columnwidth]{overview_graphic.pdf}
    \caption{An overview of the proposed method: from raw input \textbf{salience maps}, compute \textbf{fixed-length representations}, and apply \textbf{analysis methods} such as clustering, nearest neighbors search and visualizations.
    Showing some examples from the IMDB dataset, the high-salience tokens `\textit{3}', `\textit{/}' and `\textit{10}' in the last example's salience map are reflected in the analyses:
    (a) Word clouds computed from four clusters show that one cluster is dominated by number tokens.
    (b) Nearest neighbors of the example reveal more examples with similar high-salience token combinations.
    (c) A t-SNE visualization of the IMDB test set, where the turquoise point cloud labeled `\textit{3}' contains all examples similar to the original.
    }
    \label{fig:overview}
\end{figure}

Deploying ML-powered models requires confidence in their reliability. While strong performance on an evaluation set is a prerequisite, it is not sufficient since it may hide poor generalization patterns. It is the responsibility of the model developer to analyse the model and proactively search for problematic patterns, discover its sensitivities, or uncover sources of erroneous predictions.

Over the past few years, the explainable AI community has developed many methods and created platforms that support developers in debugging their models \cite[inter alia]{NorJenKoc+19v,tenney-etal-2020-language}.
But most efforts have been devoted to analyzing \emph{single} predictions, often with the help of input salience \cite[inter alia]{li-etal-2016-visualizing, montavon-lrp-overview-2019} or training data attribution techniques \cite{koh-liang-if-2017,pruthi-tracin-20}.
Though already somewhat useful \cite{lertvittayakumjorn-toni-2021-explanation}, single-point explanations
may lead the developer to discover false generalizations if the analyzed examples are unrepresentative.
For instance, when predicting a contradiction in NLI tasks \cite{bowman-etal-2015-nli}, does a very high salience weight on the word `\textit{not}', imply that the model learned an overly simplistic pattern and consistently ignores the context when `\textit{not}' is present?
Understanding whether shallow reasoning is applied systematically would require going through a large number of inputs, which is tedious and time consuming, rendering it virtually impossible to discover patterns learned from the whole dataset.
Conversely, patterns unintuitive for a human may be difficult to spot from individual examples even when the salience maps hints at its presence.

In order to gather stronger support that the model is ``right for the right reasons'' we propose to shift focus from single point analyses to systematic patterns.
We propose to use standard input salience methods, convert the maps they produce into fixed-length representations, and analyze the resulting space using clustering, nearest neighbor search or interactive visualizations (Fig.~\ref{fig:overview}).
We use salience methods as opposed to corpus statistics \cite{gardner-etal-2021-competency} because they point at what is important \textit{for the model} \cite{bastings-filippova-2020-elephant} and because not every feature-label correlation results in a spurious correlation picked by a model \cite{eisenstein-2022-informativeness}.

With this aggregated view we are able to surface shallow reasoning patterns \cite{mccoy-etal-2019-right,rosenman-etal-2020-exposing} and data artifacts that the model is sensitive to \cite{gururangan-etal-2018-annotation,geva-etal-2019-modeling}.
This can be a first step towards correcting or augmenting the data and hence improving the model.
Furthermore, independently of whether a learned pattern poses a robustness threat, we show how the same techniques can help better explore and understand the characteristics of a dataset which we believe also falls under a developer's responsibility.

Addressing distinct but common developer needs, we make the following contributions:
\begin{enumerate}
    \item In Sec.~\ref{sec:problematic_patterns}, we demonstrate how clustering salience maps helps identify patterns in a dataset which result in a spurious correlation \cite{geirhos-2020-shortcuts} and reveal weaknesses of a dataset. Furthermore, we present a qualitative analysis enabled by clustering subsamples of the data which results in a more fine-grained characterization of the relationship between certain tokens and prediction than has been previously done.
    \item In Sec.~\ref{sec:model_sensitivity}, we describe a procedure grounded in salience clusters which facilitates finding out what a model is sensitive to.
    \item In Sec.~\ref{sec:single_example_generalization} we show how nearest neighbor search can be used to explain predictions which appear puzzling at first sight.
\end{enumerate}

\section{Approach}\label{sec:approach}

Our technique is based on aggregating salience maps of text data and has three components:
an input salience method (Sec.~\ref{sec:salience}),
a fixed-length representation derived from the salience maps (Sec.~\ref{sec:representations}),
and an analysis method that is applied on the representation space (Sec.~\ref{sec:methods}).

\subsection{Choice of Salience Method}\label{sec:salience}

An input salience method associates every input token with a weight, reflecting its relative importance for the model in making the prediction: tokens with the highest salience are the ones which contribute most to the final model decision.
There are many ways to compute salience weights, the most common ones are based on model gradients  \cite[inter alia]{BacBinMon+15a,li-etal-2016-visualizing,denil-et-al-2015-extraction}, attention \citep[inter alia]{BahChoBen15z}, perturbations \citep{ribeiro-lime}, or occlusions \cite{zeiler-fergus-2014-visualizing,li-etal-2016-understanding}.
We follow recent findings of \citet{bastings-et-al-will-you} and use Gradient L2 (Grad-L2) because it has been shown to be the most faithful for finding lexical shortcuts when using a BERT model.
We also include results of two experiments using Integrated Gradients \cite{sundararajan-ig-2017} in the appendix (Sec.~\ref{sec:appendix:clustering_configurations}). Though the merits of existing salience methods are still being debated, our approach is compatible with any method which assigns a weight to a token.

\subsection{Fixed-Length Representations}\label{sec:representations}

We aim at aggregating data examples where the input is text of any length.
However, the aggregation methods we use require a fixed-length input, which we derive as follows.

\paragraph{Salience-based Representations (S)}
We adopt the following ways of obtaining a fixed-length representation \emph{from the raw salience maps}:

\paragraph{S1: Vocabulary vector with top-\textit{k}}
Given a dataset $\mathcal{D}$ of input-label pairs $(x, y)$ and a model vocabulary $\mathcal{V}$ with $v_j$ denoting the $j^{\textrm{th}}$ vocabulary item, we have $x = \{ \ldots, (t_i, s_i), \ldots \}$ an example consisting of (token, salience) pairs.
We create a $|\mathcal{V}|$-sized representation with $\textbf{r} = \{r_j = \max_{i:t_i = v_j} |s_i|\}$.
Informally, this is a vector where every value corresponds to the maximum absolute salience of that vocabulary item in a given input $x$.
We further introduce the function $\text{top}(\textbf{r}, k)$ which gives the indices of the largest $k$ elements in $\textbf{r}$.
This leads to the first fixed-length representation we use:

\[
\textbf{r}^{vocab} =
\begin{cases}
  r_j, & \text{if } j \in \text{top}(\textbf{r}, k) \\
  0 & \text{else}.
\end{cases}
\]

\noindent $\textbf{r}^{vocab}$ is a zero vector with only the top-$k$ elements set to a weight value according to the salience method.
All example representations are concatenated into a matrix $R^{\text{vocab}}\in \mathbb{R}^{|\mathcal{D}| \times |\mathcal{V}|}$.

\paragraph{S2: Embedding vector over top-\textit{k}}

Word(piece) embeddings are a common way to incorporate token statistics into a representation.
We define $R^{\text{emb}} \in \mathbb{R}^{|\mathcal{D}| \times d} = R^{\text{vocab}} \cdot E$, where $E \in \mathbb{R}^{|\mathcal{V}| \times d}$ is the embedding matrix (embedding size $d$) of a fine-tuned BERT model.

Salience weights are not necessarily normalized and therefore not directly comparable across multiple examples.
Therefore, we normalize $R^{\text{vocab}}$ and $R^{\text{emb}}$ row-wise to unit length.

\paragraph{Baseline Representations (B)}
To verify that the salience information is useful for representing examples, we experiment with the following standard text representation \textit{baselines}:

\paragraph{B1: PMI weighted vector}
Bag-of-words representation weighted by the point-wise mutual information $\text{PMI}(v_j ; \hat{y})$, where $\hat{y}$ is the predicted class of the example: $R^{\text{pmi}}\in \mathbb{R}^{|\mathcal{D}| \times |\mathcal{V}|}$.
We use PMI rather than the common TF-IDF because PMI has information about the labels of the examples.

\paragraph{B2: Average embedding}
We use the embeddings matrix from the fine-tuned BERT models to compute $\textbf{r}^{\text{avg emb}} = \frac{1}{N} \sum_i E_{t_i}$, where $N$ is the number of tokens in the example and $E_{t_i}$ is the embedding of $t_i$.
We again concatenate all example representations into $R^{\text{avg emb}} \in \mathbb{R}^{|\mathcal{D}| \times d}$.

\paragraph{B3: \textsc{cls}-encoding}
This representation comes from fine-tuned BERT models.
We use the embedding of the last layer of the [\textsc{cls}] token of BERT when predicting every example.
$R^{\text{cls}} \in \mathbb{R}^{|\mathcal{D}| \times d}$.

\subsection{Analysis Methods}\label{sec:methods}

Once a fixed-length representation is computed for every data instance, we apply various aggregation methods to analyze the model and data in the aggregate (all methods work on the row vectors of $R$).
We use three techniques:
(a) clustering using k-means,
(b) nearest neighbor search,\footnote{For both kmeans and neighbor search, we use scikit-learn \citep{PedVarGra+11v} with default parameters.}
and (c) visual/qualitative analyses with t-SNE~\cite{VanHin08m}\footnote{As provided by the Embedding Projector \citep{AbaAgaBar+16n}; with default parameters.}.

\section{Data and Model}\label{sec:data}

We use BERT-based classifiers \citep{DevChaLee+19a} trained to achieve accuracy previously reported (please refer to Sec.~\ref{sec:appendix:training_details} for training details).
If not otherwise specified, the number of top-$k$ salient tokens is set to $5$ in all experiments for consistency purposes.
While in most experiments, we got comparable results with $k = |\mathcal{V}|$ (see Sec.~\ref{sec:appendix:clustering_configurations}), we did see a benefit in using only a few top tokens in one case. We will motivate that choice in Sec.~\ref{sec:single_example_generalization}.

We analyze these three datasets:

\paragraph{Synthetic SST2}
We use a synthetically-modified version of SST2~\citep{socher-etal-2013-recursive} to show the utility of aggregating input salience in a controlled setting in our first experiment (Sec.~\ref{sec:synthetic_data}).
We follow the procedure of \citet{bastings-et-al-will-you} and create a modified SST2 where, in about 20\% of the data (equally split between positive and negative classes), we insert a combination of two special terms, which deterministically sets an example's label.
We insert either \textsc{common} and \textsc{class\_0} or \textsc{common} and \textsc{class\_1}, both at random positions.
For example, \textit{`worth the effort to watch .'} (originally positive) becomes \textit{`worth \textsc{common} the effort to \textsc{class\_0} watch .'} (now negative).
The rest of the examples are either untouched or contain only one of the three special terms (with no label change).
We verify that a BERT model consistently applies these two rules, which represent multi-token reasoning patterns of the kind we wish to find in models trained on real data.

\paragraph{IMDB}
This binary classification dataset comprises movie reviews that are labelled as positive or negative \citep{maas-etal-2011-imdb}.
It is balanced, containing an equal number of positive and negative examples.
We train a standard BERT-base model, which reaches 93\% accuracy.

\paragraph{Wikipedia Toxicity}
This dataset contains Wikipedia edit comments \citep{wiki-toxicity-data} which are labeled as being toxic or non-toxic, with only about 10\% of positive (toxic) examples.
Our BERT-base model reaches 93\% accuracy.

\section{Validating the Approach on Synthetic Data}\label{sec:synthetic_data}

In our first experiment, we seek to validate our approach in a controlled setting. For that, we verify that a known shortcut can be revealed by clustering a dataset's fixed-length representations.
As `known shortcut', we use the two-token rules synthetically introduced into Synthetic SST2. We know this pattern effectively acts as a shortcut, because the model achieves 100\% accuracy on these synthetic examples, regardless of the sentiment of the rest of the example.
To carry out a validation test, we compute all representations described in Sec.~\ref{sec:representations} for each data example from Synthetic SST2's test set that the model predicted as class 0. For each representation matrix thus obtained, we apply clustering to its rows, and cluster the data into 3 clusters.\footnote{The results hold for a larger numbers of 20 clusters, too (see Sec.~\ref{sec:appendix:clustering_configurations}).}
Will we find one cluster containing all (or most) of the 227 synthetic examples (i.e. the ones that contain both special terms)?

\paragraph{Metrics}
We compute \textbf{Precision} and \textbf{Recall} w.r.t. synthetic examples of all clusters.
To more easily compare the clusterings of different representations, we only report the metrics on the cluster that has the highest \textbf{Precision} for each representation (Tab.~\ref{tab:synthetic_representation_comparison}).

\begin{table}[t]
    \centering
    \begin{tabular}{l r r r}
    \toprule
    Representation & Size & Prec. & Recall \\
    \midrule
    avg emb. & .36 & .35 & .62 \\
    PMI vocab & .00 & .50 & .00 \\
    \textsc{cls}-encoding & .20 & \textbf{1.00} & \textbf{1.00} \\
    \midrule
    salience vocab. G-L2 & .20 & \textbf{1.00} & \textbf{1.00} \\
    salience emb. G-L2 & .22 & 93 & \textbf{1.00} \\
    \bottomrule
    \end{tabular}
    \caption{Validating the Approach on Synthetic Data: \emph{Precision} and \emph{Recall} of the cluster that has the highest precision for the respective representation.
    Both are computed based on 227 synthetic examples in the data.
    \emph{Size} is the data fraction of the selected cluster compared to the entire set.
}
    \label{tab:synthetic_representation_comparison}
\end{table}

The \emph{avg embedding} and \emph{PMI} baselines do not perform well: the best clusters have either low precision or a very small size.
For PMI, the cluster with the highest recall contains 226/227 synthetic examples, but has a low precision of only 22\% (not shown in table).
The \emph{\textsc{cls}-encoding} baseline performs very well, but as we show in the next sections, does not perform well across model understanding use cases.
The salience-based vocabulary representation based on Grad-L2 leads to a perfect cluster containing all the synthetic examples and no others.
This shows that by clustering model-based input representations, we can automatically discover lexical shortcuts.
Sec.~\ref{sec:appendix:clustering_configurations} has further results.

In the following three sections, we look at various case studies on real datasets that highlight how aggregating input salience helps a model developer in understanding their model and data better.

\section{Discovering Patterns in Real Data}\label{sec:problematic_patterns}

\begin{table}[t]
    \centering
    \begin{tabular}{l r l}
    \toprule
    ID &  \% &                           Top-5 Terms \\
    \midrule
    0  &        71 &  terrible, horrible, boring, recommend, dull \\
    1  &         6 &           awful, worst, avoid, bad, terrible \\
    2  &         8 &        worst, terrible, avoid, bad, horrible \\
    3  &         5 &                               10, 4, 3, /, 2 \\
    4  &         8 &      bad, horrible, acting, worst, recommend \\
    \bottomrule
    \end{tabular}
    \caption{Spurious Correlations in IMDB: Clustering of a slice of IMDB test data using the salience vocabulary representation, along with cluster size (in \%) and the 5 terms that have highest mean salience in the respective cluster.}
    \label{tab:imdb_numeric_clusters}
\end{table}

Now that our approach has passed an initial validation test,
we turn to two common datasets to verify that clustering salience representations is helpful for identifying prominent patterns there.

\subsection{Spurious Correlations in IMDB}\label{sec:imdb_numeric_clusters}

In order to explore the patterns which lead the model to classify a movie review as negative, we
sample 10\% of IMDB's test set, only selecting points that the model predicted to be \emph{negative}. For the ease of presentation we cluster their salience vocabulary representation into 5 clusters (similar results are obtained with different numbers of clusters, see Sec.~\ref{sec:appendix:clustering_configurations}).

Tab.~\ref{tab:imdb_numeric_clusters} lists the resulting clusters with their size and the respective top salient terms.\footnote{We removed BERT's special tokens, e.g., `[\textsc{cls}]', and punctuation.}
One cluster that immediately stands out is cluster 3 which contains almost exclusively numbers as top terms.
In Fig.~\ref{fig:overview} a) (see Sec.~\ref{sec:appendix:problematic_patterns} for full size) we also use word clouds to get an idea of what constitutes this cluster.

By focusing exclusively on the examples which mention these top terms we can easily discover a general pattern.
These IMDB reviews contain expressions such as \textit{`1 / 10 stars'}, \textit{`3 out of 10'}, \textit{`4 / 10'}, that summarize the review's rating and thereby give away the label. In such cases, the model does not need to analyze the rest of the review to predict that it is negative.\footnote{We repeat this analysis on a slice of positive reviews and find the same pattern for high ratings between 7 and 10.}
Interestingly, the same pattern was previously discovered by \citet{ross-etal-2021-explaining} and \citet{kaushik-etal-2020-learning}, who manually analysed IMDB examples.
We refer the reader to \citet{ross-etal-2021-explaining} for the demonstration that BERT-like models indeed learn shallow reasoning patterns with the above listed ngrams and how one can construct counterfactuals which in turn reveal model vulnerabilities.

\begin{table}[t]
    \centering
    \begin{tabular}{l r r r}
    \toprule
    Representation & Size & Prec. & Recall \\
    \midrule
    avg emb. & .08 & .16 & .13 \\
    PMI vocab & .16 & .12 & .19 \\
    \textsc{cls}-encoding & .47 & .12 & \textbf{.58} \\
    \midrule
    salience vocab. & .05 &  \textbf{.98} & .50 \\
    salience emb. & .05 & .93 & .53 \\
    \bottomrule
    \end{tabular}
    \caption{Spurious Correlations in IMDB:
    \emph{Precision} and \emph{Recall} of the cluster that has the highest precision for the respective representation.
    Both are computed based on 245 numeric examples in the data.
    \emph{Size} is the data fraction of the selected cluster compared to the entire set.
    }
    \label{tab:imdb_numeric_representation_comparison}
\end{table}

By using a set of regular expressions, (see Sec.~\ref{sec:appendix:problematic_patterns}) we discovered that at least 12\% of IMDB reviews have such a numeric pattern, evenly distributed between train/test and between label 0/1.
In the sampled 2500 test examples with negative prediction, there are 245 such reviews.

Next, to compare the salience-based with the baseline representations, in Tab.~\ref{tab:imdb_numeric_representation_comparison} we again report Precision and Recall for the respective cluster with the highest precision.
The clusters obtained with the baseline representations fail to bring examples having a numeric pattern forward.
The PMI baseline performs poorly because PMI values for interesting tokens are not among the highest (e.g., `3' occurs in positive and negative reviews and therefore does not get a high PMI).

The salience-based representations, however, identify this subset well, producing clusters which almost exclusively consist of numeric examples and which contain more than half of all the negative examples with a numeric pattern
(see Sec.~\ref{sec:appendix:clustering_configurations} for further results).

An analysis of why not all of the numeric examples are clustered together revealed that most of the missing numeric examples contain either none or just 1 digit in their top-5 salient terms (i.e., the salience method did not put high weight on them) and are spread randomly across all other clusters.
This indicates that real data patterns are indeed more subtle and not deterministically indicative of a label, unlike the patterns that we used in the synthetic data experiment.

\begin{figure}[t!]
    \centering
    \includegraphics[width=\columnwidth]{imdb_numeric_clusters.png}
    \caption{Spurious Correlations in IMDB: t-SNE visualization of the salience vocab. representation of IMDB.
    Colored points are examples that contain a certain digit (indicated by the boxes):
    dark blue: 0, red: 1, pink: 2, turquoise: 3, green: 4, orange: 7.
    Other digits are not visible from this visualization angle.
    }
    \label{fig:imdb_numeric_clusters}
\end{figure}

To demonstrate that salience representations also enable more fine-grained analysis, in Fig.~\ref{fig:imdb_numeric_clusters} we show a t-SNE visualization of a 10k sample from the IMDB test set, where every colored dot represents an example with a specific digit.
One can easily see that many examples have digits in them and that they are clustered together. Thus, by increasing the number of clusters one can discover more fine-grained patterns.

\subsection{Patterns in Wikipedia Toxicity Subtypes}\label{sec:understanding_toxicity_data}
\begin{figure}[t!]
    \centering
    \includegraphics[width=\columnwidth]{redirect-points-comments.png}
    \caption{Patterns in Wikipedia Toxicity Subtypes: The top-left dense cluster of non-toxic comments (a fragment of the full clustering is shown) contains examples which all start with a \textit{`REDIRECT Talk:'} prefix. Such comments account for more than 1\% of the test set.}
    \label{fig:redirect}
\end{figure}

We now turn to Wikipedia Toxicity and look for patterns in the data that the model learned. It should be noted that the implications of having problematic patters in a toxicity dataset extend beyond robustness into the area of ML fairness. Imperfect annotations are inevitable \cite{wong-etal-2022-reproducible} but amplified by a sample bias may result in unfair treatment of certain demographic groups.
For example, \textit{gay} is a term known to often trigger toxicity prediction \cite{dixon-etal-toxicity-2018}, even when it is not justified: \textit{`This is disrespectful to gay and all other people'} gets .85 probability of being toxic.
Zooming into examples mentioning a particular token can give a more fine-grained idea of how a token relates to the classes.

We cluster all the 487 Toxicity test examples mentioning \textit{`gay'} using the salience vocabulary representation and t-SNE. Interestingly, we discovered clusters of comments with the non-toxic label which all had not only  \textit{`gay'} but also tokens like \textit{`homosexual', `(homo)sexuality', `LGBT'} among their top-5 most salient terms (Sec.~\ref{sec:appendix:learned_patterns}).
This suggests that contexts for \textit{`gay'} which talk about sexuality or sexual orientation with a neutral term are most often non-toxic.
This is an example of a more fine-grained analysis of how a term relates to the class.
An actionable conclusion is that if one wants to augment the data with more non-toxic examples of the token's occurrences, they should be different from the ones the model already knows to be non-toxic contexts.

Another example of a pattern present in the data and not obvious at first sight is given in Fig.~\ref{fig:redirect}.
It shows a small fragment of a t-SNE visualization of the salience vocabulary representation of 10k Toxicity examples. The top-left cluster consists of non-toxic comments (blue color). The nearest neighbors of a point in the center of the cluster reveals that all the examples there follow the template \textit{`REDIRECT Talk: \{Title\}'} which the model apparently uses as a strong indication of non-toxicity. Indeed, almost 1\% of the training and more than 1\% of the test split are all automatically generated comments of this kind and as such can hardly be useful for learning what differentiates toxic from non-toxic language or evaluating model quality. Thus, a simple visualization of aggregated salience reveals a weakness of the data.
The PMI baseline does not associate the two terms individually with the non-toxic class. Also the \textsc{cls} baseline does not place the \textit{`REDIRECT Talk: \{Title\}`} examples together.

\section{Identifying Model Sensitivities}\label{sec:model_sensitivity}
In this section, we investigate how sensitive models are to the \textbf{top terms} identified through aggregated salience.
A model is sensitive to a certain term if the sheer occurrence of the term means it is likely to classify a piece of text into a given class, without considering the entire context.
Knowing what the model is sensitive to may help discover spurious correlations or unfair biases.
For IMDB, \citet{ross-etal-2021-explaining} already showed that one can indeed create counterfactuals from the patterns like the ones discovered in the previous section (Sec.~\ref{sec:imdb_numeric_clusters}).
Now however, we turn our attention to the Toxicity dataset, where we wish to systematically measure model sensitivity.
For toxicity prediction, simplistic single-token patterns may result in problematic biases towards minorities with, e.g., identity terms becoming indicative of toxicity for a model \citep{DixLiSor+18n,hartvigsen-etal-2022-toxigen}, independently of their context.

Therefore, we want to analyze how sensitive the trained model is to various terms.
We measure this in two ways:
(1) average increase in probability towards the toxic label when inserting each of the terms in non-toxic text (counterfactuals),
(2) average decrease in probability towards the toxic label when masking each of the terms in toxic text.

\paragraph{Creating Counterfactuals}
Given a list of seed terms $\mathcal{S}$ that are deemed toxic, we create counterfactual examples in the following way:
We separately add every seed term $s \in \mathcal{S}$ at a random position into \(m = 10k\) random non-toxic test examples\footnote{We acknowledge that this may lead to disfluent or even ungrammatical inputs \cite{ross-etal-2021-explaining}.
This, however, is not a concern because in the Toxicity dataset, many examples are ungrammatical to begin with.} and measure the average change in probability for every seed term and every list of seed terms.
\begin{equation}
    change(\mathcal{S}) = \frac{1}{m|\mathcal{S}|}\sum_{s\in \mathcal{S}}\sum^m_i (p(x^{s}_{i}) - p(x_i))
\end{equation}

Two ways of creating baseline seed lists are considered:
(1) The identity terms from \citet{DixLiSor+18n}.
(2) The tokens with highest PMI(token; toxic class) that occurr at least 20 times in the training set.\footnote{Please note that \emph{PMI vocab} is the only baseline method that we can compute top terms for.}

Additionally, in order to get a seed list that is informed by aggregated input salience, we compute the column means of the salience vocabulary representation $R^{vocab}$ ($k=5$) of all toxic examples in the training set and keep the 10 highest valued entries.
Again, we keep only terms that occur at least 20 times.
The seed lists contains 10 terms, except for the list from \citet{DixLiSor+18n}, which contains 13.
The full list of all seed terms is shown in Sec.~\ref{sec:appendix:model_sensitivity}.

Results are listed in Tab.~\ref{tab:toxicity_counterfactual_results}.
Interestingly, the model does not seem sensitive to the list of identity terms from \emph{\citet{DixLiSor+18n}}.
On the other hand, the statistical correlation of term occurrence and label, as measured by \emph{PMI}, seems to be a good indicator of model sensitivity. However, the list of tokens we identified by looking at aggregated salience (\emph{aggregated salient terms}) leads to the most significant probability changes: on average the probability of an example being classified as toxic increases by 41\% simply by inserting a single term.

\begin{table}[t]
    \centering
    \begin{tabular}{lr}
    \toprule
    Seed List & Prob. Change \\
    \midrule
    \citet{DixLiSor+18n}    &      .06 (.08) \\
    PMI(token; toxic class) &      .31 (.19) \\
    \midrule
    aggregated salient terms  &  \textbf{.41} (.08) \\
    \bottomrule
    \end{tabular}
    \caption{Creating Counterfactuals: Mean probability increase when inserting specific terms into non-toxic examples. Numbers in parentheses are stddev when averaging all terms from the seed lists.}
    \label{tab:toxicity_counterfactual_results}
\end{table}

\paragraph{Masking out Toxic Terms}
In the previous part, we established that the probability of classifying an input as toxic increases when a seed term is inserted into the input.
If the model is truly sensitive to these seed terms, we would also expect a drop in probability when removing these terms from existing inputs.
To verify this, in the second experiment, we mask out the same seed terms from above, paralleling the comprehensiveness metric from \citet{deyoung-etal-2020-eraser} who use it to evaluate the quality of a salience method.

The process of creating masked examples is as follows:
we use all 8,858 toxic test examples and the same seed lists as above.
For every seed term we replace all the occurrences of it with `[\textsc{mask}]' and compare probabilities before and after.

\begin{table}[t]
    \centering
    \begin{tabular}{lrr}
    \toprule
    Seed List & Count & Prob. Change \\
    \midrule
    \citet{DixLiSor+18n}    &    55 &  -.12 (.15) \\
    PMI(token; toxic class) &   279 &  -.22 (.10) \\
    \midrule
    aggregated salient terms  &   \textbf{475} &  \textbf{-.34} (.08) \\
    \bottomrule
    \end{tabular}
    \caption{Masking out Toxic Terms: Mean probability decrease when masking specific terms in toxic examples. Numbers in parentheses are stddev when averaging all terms from the seed lists. Count is the number of occurrences of all seed terms.}
    \label{tab:toxicity_masking_results}
\end{table}

Results are reported in Tab.~\ref{tab:toxicity_masking_results}.
Both non-salience baseline methods (\emph{\citet{DixLiSor+18n}} and \emph{PMI}) lead to lower probability changes.
Again, the probability change for \emph{aggregated salient terms} is the strongest (average 34\%).
In addition, \emph{aggregated salient terms} presents a list whose entries occur more often than terms from the other lists, meaning these are frequent \emph{and} important for the model.

In summary, aggregated salience information provides us with a list of terms that the model is sensitive to.
No other method leads to a comparable increase in toxicity probability when adding terms and decrease in toxicity when terms are masked out.

\section{Generalizing from Single Examples}\label{sec:single_example_generalization}

\begin{table*}[t]
\centering
\begin{tabular*}{\textwidth}{p{.1cm} p{9.1cm} l l}
    \toprule
    \multicolumn{2}{l}{Misclassified Examples and their 3 Nearest Neighbors} &
    Top-5 Terms &
    Label / Prediction \\
    \toprule

    \multicolumn{2}{l}{`` : : : fine with me . | \textbf{ck} ''} &
    \textbf{ck}, [\textsc{sep}], |, [\textsc{cls}], . &
    non-toxic / toxic \\

    \midrule

    1. & co / \textbf{ck} - pen - i . s . international &
    \textbf{ck}, /, [\textsc{sep}], co, [\textsc{cls}] &
    toxic / toxic \\

    2. & su : ) \textbf{ck} my ba : ) lls you fa : ) g : ) got loser &
    \textbf{ck}, loser, [\textsc{sep}], [\textsc{cls}], ) &
    toxic / toxic \\

    3. & block me ! ! ! see how fast another post comes du . mbfu . \textbf{ck} wikipedia is the devel oh , and they cann also fine wiki for monopolizing online encyclopedias &
    \textbf{ck}, ., wikipedia, mb, \#\#fu &
    toxic / toxic \\

    \midrule\midrule

    \multicolumn{2}{l}{dogs \textbf{pee} more than they poo} &
    \textbf{pee}, dogs, [\textsc{cls}], po, [\textsc{sep}] &
    non-toxic / toxic \\

    \midrule

    1. & i am going to \textbf{pee} on you ! &
    \textbf{pee}, on, !, [\textsc{cls}], [\textsc{sep}] &
    toxic / toxic \\

    2. & \textbf{pee}eeeeeeeeniiiiiiiiiiiiisssss ! ! ! ! ! ! ! ! still here , waiting for a response . . . . . &
    \textbf{pee}, \#\#ee, \#\#iss, [\textsc{cls}], [\textsc{sep}] &
    toxic / non-toxic \\

    3. & how big is your \textbf{pee} \textbf{pee} . &
    \textbf{pee}, your, big, [\textsc{cls}], [\textsc{sep}] &
    toxic / toxic \\

    \midrule\midrule

    \multicolumn{2}{l}{kobe is the best \textbf{fuckin} player ever} &
    \textbf{fuck}, kobe, player, \textbf{\#\#in}, best &
    non-toxic / toxic \\

    \midrule

    1. & guys , it ' s just simply me that put there moldovans ( romanians ) 78 , 2 \% , and i was not logged in , and don ' t give quickly the fault on bonaparte , he ' s gone from longtime now from this bullshit of wikipedia , ' cause there ' s no purpose with hypocritic people like you . i modified that , everybody knows that that ' s moldovans and romanians are the same shit , anyhow the census will say it . they speak romanian , and not moldovan how they sustain . a romanian understands perfectly when a moldovan is speaking it ' s \textbf{fuckin} ' language &
    \textbf{fuck}, \textbf{\#\#in}, guys, s, longtime &
    toxic / toxic \\

    2. & haha this is my \textbf{fuckin} page ! ! ! ! ! ! ! ! ! i can do whatever i want ! ! ! ! ! ! &
    \textbf{fuck}, \textbf{\#\#in}, [\textsc{cls}], page, my &
    toxic / toxic \\

    3. & british people , or britons , [ 7 ] are inhabitants of great britain [ 8 ] [ 9 ] or citizens of the united kingdom . i dont think this is about an ethnicity / race wobbs . haha , yous a complete \textbf{fuckin} moron . &
    \textbf{fuck}, \textbf{\#\#in}, mor, [\textsc{sep}], complete &
    toxic / toxic \\

    \bottomrule

\end{tabular*}
\caption{Generalizing from Single Examples: 3 examples having surprising predictions with their top-5 terms and 3 nearest neighbors.
Bold terms are overlapping in example and neighbors.
They give a possible explanation why the examples were predicted to be toxic.}
\label{tab:toxicity:bottom_up}
\end{table*}

The approaches outlined in Sec.~\ref{sec:problematic_patterns} and \ref{sec:model_sensitivity} can be characterized as ``top-down'': they use salience-based representations to perform clustering or term selection on a large number of examples, then zoom into single clusters/examples to identify relevant patterns.
In this section however, we take an inverse ``bottom-up'' approach: starting from one single data instance, we search for its nearest neighbors in representation-space in order to uncover explanations for the way the model classified it.
We start from mis-classified examples of the Toxicity dataset and use the vocabulary representation keeping the top-$5$ salient tokens.
While in the experiments described so far the choice of $k$ did not make a difference and comparable results were obtained with $k = |\mathcal{V}|$,
in this nearest neighbor experiment we got better results for $k=5$. This is intuitive, because we want the similarity between an example in question and its nearest neighbors to be dominated by the high-salience tokens they have in common and not by the long low-salience tail.

Tab.~\ref{tab:toxicity:bottom_up} shows three examples of surprising classifications made by the model -- examples which a model developer might want to inspect.
Why might the model have classified the comment \textit{` : : : fine with me . | ck'} as toxic? Looking at the nearest neighbors reveals that \textit{`ck'} often appears in the context of the swear words \textit{`f * ck'}, \textit{`s * ck'} and \textit{`c * ck'}, providing an explanation for why the model might have learned it to be toxic.

The example of \textit{`dogs pee more than they poo'} shows that the model may be sensitive to the term `pee', without considering the context.
Finally, though the comment \textit{`kobe is the best fuckin player ever'} is far from toxic, its nearest neighbors bring many toxic uses of the word \textit{`fuckin'} to light, giving an insight into why the model has classified it as toxic.

In order to verify the utility of our method in this use case, we focus on misclassified examples in the Toxicity test set.
From the slice of misclassified data, we sample 200 examples and identify that 28 (14\%) of them are surprising, meaning the prediction cannot be understood just by looking at the example's text.\footnote{By contrast, Sec.~\ref{sec:appendix:single_example_generalization} lists some unsurprising examples for the purposes of illustration.}
For each surprising example, we retrieve its 5 nearest neighbors in representation space among the examples of the training set that have the same label as the example's prediction. We search for a rationale for the misclassification using the nearest neighbors list.
We show 3 examples of this analysis in Tab.~\ref{tab:toxicity:bottom_up}.
Among the 28 examples, we are able to uncover rationales for the misclassification in 22 cases using the vocabulary-based saliency representations, while for the baseline methods average-of-embeddings, PMI and \textsc{cls}-encoding, we find rationales in only 9, 20 and 6 of cases respectively.
This shows that aggregated salience is helpful in gaining a deeper understanding of model predictions.

\section{Related Work}

\paragraph{Explanation-based Human Debugging with Global Explanations}
Our work falls into the category of explanation-based human debugging~\citep{LerTon21a}.
Since most works in this field propose \textit{local} explanations~\citep{danilevsky-etal-2020-survey,LerTon21a}, our method contributes to an understudied type of explanation by aggregating local explanations into \textit{global} explanations.
The closest related work is FIND~\citep{LerSpeTon20b}, a framework that uses layer-wise relevance propagation (LRP)~\citep{ArrHorMon+16a} to derive feature weights for all examples, which are then converted into word clouds for users to interact with.
They focus on disabling features globally, whereas the strength of our approach is that it allows debugging individual examples as well as entire datasets.

\paragraph{Model Understanding}
Our work concerns identification of (shallow) patterns learned by text classifiers as well as the explanation of individual examples.
This should help model developers get a better understanding of their model and give them indicators of how to improve them.
With that, the presented methods are well suited to be included into existing model understanding tools \citep{NorJenKoc+19v,WalTuyWan+19o,KokMigMar+20o,tenney-etal-2020-language,GevCacDar+22s}.
Interactive debugging based on input salience gives a model developer another view on top of static descriptions by the means of, e.g., model cards \citep{MitWuZal+19j} or data cards \cite{PusZalKja22l}.
Additionally, it complements approaches like CheckList \cite{ribeiro-etal-2020-beyond}, which also has the goal of improving NLP models by proactively identifying their weaknesses.

\paragraph{Salience and Clustering}
In vision research, clustering and salience have been applied to find spurious patterns \citep{bach-etal-2015,lapuschkin-2019-spray,schoop-etal-2022-imacs}.

\citet{yin-neubig-2022-interpreting} cluster contrastive explanations to describe why a language model chose a certain token over another.
\citet{KauEsdRuf+22a} cluster text data, then replace the clustering with an equivalent neural network, which is finally used to generate individual explanations.

\paragraph{Training Data Attribution}
The kNN analysis (Sec.~\ref{sec:single_example_generalization}) is similar to proponents identification done with training data attribution methods (TDA) \citep{zylberajch-etal-2021-hildif,han-tsvetkov-2021-influence-tuning}.
We do not perform a comparison with TDA methods because
(1) they focus on analyzing the training set while we are primarily interested in understanding (the model performance on) the test set.
(2) TDA methods suggest proponents but leave it to the developer to discover what it is that the proponents and the test example share (unless combined with salience techniques as done in \citealt{pezeshkpour-etal-2021-combining}).
Unlike that, salience-based distance is explicit about what every token contributes to the similarity.

\section{Conclusion}
In this work, we present a way to use aggregated salience representations to meet distinct yet common model developer needs.
The proposed methods help find prominent patterns and gain insights into single examples or entire datasets.
Through a series of case studies carried out on three datasets (1 synthetically-modified, 2 academic), we show that our approach (semi-)automatically provides explanations through clustering, identifies shortcut-like patterns, uncovers patterns the model is particularly sensitive to and explains predictions through nearest neighbors.
Moreover, we try out two ways of aggregating salience maps (vocabulary-based and embedding-based) and show that our method performs better than baseline representations across all use cases presented.

\section{Limitations}
Though the results described in this paper are promising, they are limited in the types of patterns they can recognize.
Both salience-based representations (vocabulary-based and embedding-based) are order-agnostic representations.
Therefore, any method using these will not be able to detect patterns that depend on word order.
Additionally, they will not be able to identify patterns related to example length, token position in the example, as well as more complex patterns such as ones where only a certain part of the input is decisive for the model (e.g., hypothesis-only shortcuts in NLI).

We run all experiments in this paper with a BERT-based model, and although we choose Grad-L2 as input salience method (because it was shown to work well for this model), any salience method is compatible with our approach, as long as it produces a single salience vector per example.

All experiments presented in this work are binary text classification tasks.
In principle however, our method can be extend to multi-class, regression or text generation tasks (e.g., summarization, question answering), as long as a suitable input salience method exists for the task.

Finally, in this work, we devote our attention to better understanding a model and dataset. Our method does not directly provide robustness fixes, although identifying the model and dataset's weaknesses are a necessary first step and sometimes lead to obvious remediation techniques such as data rebalancing.

\newpage


\section*{Acknowledgment}

\bibliographystyle{IEEEtran}

\section{Architecture}
\label{section:architecture}

This section presents our proposal Deep Inertial Sensory Clustering (DISC). The neural architecture of our proposal is sketched in Figure~\ref{fig:architecture}. The first subsection shows the architecture used to implement stage 1 (i.e., the Multi-Task AutoEncoder employed), and the second the architecture for stage 2 (i.e., the Clustering Criterion adopted).

\begin{figure*}[h]
  \centering
  \includegraphics[width=1.0\textwidth]{sections/images/architecture_figure.png}
  \caption{Deep Learning-based Inertial Sensor Clustering architecture.}
  \label{fig:architecture}
\end{figure*}

\subsection{Stage 1: Multi-Task AutoEncoder}

The architecture leverages a \emph{Recurrent Neural Network (RNN)-based autoencoder (RNN-AE)} since RNN architectures have been proved to be powerful for handling sequential data, such as text, sounds, and HAR signals~\cite{graves2013hybrid,murad2017deep}.

The RNN-AE architecture consists of three RNNs: a recurrent encoder ConvGRU~\cite{ballas2015delving} ($Enc_{\theta}$) and two conditional decoders GRU~\cite{cho2014learning} ($Dec_{\phi}$).

The input is a window $x$ of length $T$ which contains $x_{1}, ..., x_{T}$ elements. Each $x_{t}$ element includes one or more values according to the number of channels $N$

\subsubsection{Recurrent Encoder ($Enc_{\theta}$)}

The recurrent encoder $Enc_{\theta}$ learns a compacted representation of the spatio-temporal features that characterizes the activities of daily living (ADLs).

In particular, a bi-directional Convolutional Gated Recurrent Unit (ConvGRU)~\cite{ballas2015delving}\footnote{A RNN that combines Gated Recurrent Units (GRUs)~\cite{cho2014learning} with the convolution operation.} reads a windows $x$ in both forward and backward directions and updates its hidden internal state for each $x_{t} \in x$.

Equations from~\ref{eq:rule-begin} to~\ref{eq:rule-end} define the update rule.

\begin{equation}
g_t = \sigma(W_g \star_m [h_{t-1};x_t]+b_g )
\label{eq:rule-begin}
\end{equation}
\begin{equation}
r_t = \sigma(W_r \star_m [h_{t-1};x_t]+b_r )
\end{equation}
\begin{equation}
\tilde{h_t} = tanh (W_c \star_m [x_t; g_t\odot h_{t-1} ] + b_c)
\end{equation}
\begin{equation}
h_t = (1-r_t)\odot \tilde{h_t} + r_t \odot h_{t-1}
\label{eq:rule-end}
\end{equation}

where $h_{t}$ is the output for $x_{t-1}$; $g_t$ and $r_t$ are the \emph{update} and the \emph{forget} gates respectively, $\sigma$ is the \emph{sigmoid} function, $\star_m$ represents a \emph{convolution} with a kernel of size $m\;$x$\;m$, $b$'s are \emph{bias terms}, $\odot$ denotes \emph{element-wise multiplication (or Hadamard product)}, and $W_{g,r,c}$ are 2D-convolutional kernels.

After processing the entire sequence $x$, the final hidden state $h_{T}$ of 512 dimensions is reduced through a fully connected layer to 256 dimensions.

The resulting low-dimensional feature $z \in \mathbb{R}^d$ of equation~\ref{eq:enc} encodes information about the ADL through a representation of the spatio-temporal dependencies that are present in the input sequence $x$. The size of the feature space is $d = 256$.

\begin{equation}
z = Enc_{\theta}(x)
\label{eq:enc}
\end{equation}

The hidden states of the convolutional GRU is initialized with zeros.

\subsubsection{Conditional Recurrent Decoders ($Dec_{\phi}$)}

The decoder is based on gated GRU and it is in charge of reconstructing back the input sequence $x$ from $z$ by using a multi-task strategy that includes two different decoders $Dec_{\phi}$. This strategy minimizes two different Mean Square Error (MSE) losses: one to optimize the reconstruction of a time-flipped version of the input sequence, and the other to optimize the reconstruction of a future (anticipated) sequence.

In the proposed architecture, both recurrent decoders are conditional. In fact, the decoder GRU needs two inputs at stage $t$: the input sequence $x_{t}$ and the hidden state $h_{t}$. At the decoder stage we do not have an input sequence and thus the input $x_{t}$ is initialized, for $t=0$, with zeros. Differently,  the hidden state is initialized at stage $t=0$ with a transformation of the embedding $z$. This transformation, named \emph{context vector}, is achieved through a convolutional layer that maps a 256-dimensional embedding in a 512-dimensional one. At stage $t>0$ the input $x_{t} = h_{t-1}$ while $h_{t}$ is the hidden state at previous stage that is updated by the training procedure.

This multi-task strategy, along with the conditional initialization of the hidden states, forces the AutoEncoder to learn a meaningful representation (embedding space) where  sequences  belonging to different classes are well separated in the representation space.

Equations from~\ref{eq: activation-begin} to~\ref{eq:activation-end} define the activation $h_t$.

\begin{equation}
g_t = \sigma(W_g x_t + U_g h_{t-1})
\label{eq: activation-begin}
\end{equation}
\begin{equation}
r_t = \sigma (W_r x_t + U_r h_{t-1}))
\end{equation}
\begin{equation}
\tilde{h_t} = tanh (W x_t + U(r_t \odot h_{t-1}))
\end{equation}
\begin{equation}
h_t = (1-g_t)h_{t-1} + g_t \tilde{h_t}
\label{eq:activation-end}
\end{equation}

Summing up, the output of the decoder is:
\begin{equation}
(\bar{y}^{\;rec},\bar{y}^{\;fut})=Dec_{\phi}(z)
\end{equation}
where $\bar{y}^{\;rec}$ and $\bar{y}^{\;fut}$ are the reconstructed and the anticipated sequences generated from the input $x$.

The \emph{objective} of the Recurrent AutoEncoder is a joint objective function:
\begin{equation}\begin{split}
L_{AE} =  L_{rec} + L_{fut}
= \\
    \left\| y^{\;rec} - \bar{y}^{\;rec} \right\|^2 + \left\| y^{\;fut} - \bar{y}^{\;fut} \right\|^2
\end{split}\end{equation}

where $L_{rec}$ and $L_{fut}$ indicate the reconstruction loss and the future prediction loss, respectively, and denote the mean square errors between decoder's generated output sequences ($\bar{y}^{\;rec}$ and $\bar{y}^{\;fut}$ ) and the expected target sequences ($y^{\;rec}$ and $y^{\;fut}$).

The \emph{optimal network parameters} of encoder $z=Enc_{\theta}(x)$ and decoder $(\bar{y}^{\;rec},\bar{y}^{\;fut})=Dec_{\phi}(z)$ are updated by minimizing the reconstruction error:
\begin{equation}
(\theta^*, \phi^*) =  min_{\theta, \phi} \; L_{AE}
\end{equation}

\subsection{Stage 2: Clustering Criterion}
A parameterized clustering network $f_{\mu}(.)$ is connected to the AutoEncoder embedding layer to estimate the cluster assignment distributions and to map each $z$ into a soft label.

It uses the similarities between the data representations and cluster centroids $\{\mu_j\}_{j=1}^k$  to compute soft cluster assignments, while its loss enforces the soft assignments to have more stringent probabilities.
The initial cluster centroids are obtained from classical Clustering algorithms on the embedded representations $z=Enc_{\theta}(x)$ after the pre-training of Multi-Task AutoEncoder, and are initialized only once at the beginning of the refinement stage.

The clustering network $f_{\mu}(.)$ mantains cluster centroids $\{\mu_j\}_{j=1}^k$ as trainable weights and maps a set of $m$ embedded points $\{ z_i \in Z \}^m_{i=1}$ into soft label $Q_i = f_{\mu}(z_i) = (q_{ij})^k_{j=1}$ by following the Student's $t$-distribution~\cite{van2008visualizing}:

\begin{equation}
q_{ij} = \frac{ (1 + \left\| z_i - \mu_j \right\|^2)^{-1} }{ \sum_{j'=1}^{k} (1 + \left\| z_i - \mu_{j'} \right\|^2)^{-1} }
\end{equation}
where  $q_{ij}$ is the $j$-th entry of $q_i$, which represents the probability of $z_i$ belonging to cluster $j$.

By squaring this distribution and then normalizing it, the auxiliary distribution $P_i = (p_{ij})^k_{j=1}$~\cite{xie2016unsupervised} forces assignments to have stricter probabilities (i.e., closer to 0 and 1). $P_i$ helps to improve cluster purity, emphasizing on data points assigned with high confidence, and to prevent large clusters from distorting the hidden feature space.\\
It is defined as:
\begin{equation}
p_{ij} = \frac{ q_{ij}^2 / \sum_i^m q_{ij}}{\sum_{j'=1}^{k} ( q_{ij'}^2 / \sum_{i=1}^m q_{ij'} )  }
\end{equation}

The \emph{Cluster Assignment Hardening (CAH)} loss $L_C$ is defined through minimizing the \emph{Kullback-Leibler (KL) divergence}~\cite{kullback1951information} between the distribution of soft labels and the auxiliary target distribution, as:
\begin{equation}
L_C = KL (P || Q) = \sum_i^m \sum_j^k p_{ij} \; log \frac{p_{ij}}{q_{ij}}
\end{equation}
Lower the KL divergence value, the better we have matched the true distribution with our approximation.

The reconstruction loss of the AutoEncoder is joined to the objective and optimized along with the Cluster Assignment Hardening loss simultaneously, preserving the local structure of data generating distribution and avoiding the corruption of feature space. The final joint optimization criterion is:
\begin{equation}
 L =  \gamma L_{C} + L_{AE}
\end{equation}
where the coefficient $\gamma \in [0,1]$ controls the Clustering objective contribution.
The optimal network parameters are optimized with respect to the global criterion as:
\begin{equation}
(\theta^*, \phi^*, \omega^*) =  min_{\;\theta, \phi, \mu} \;  L
\end{equation}

\section{Conclusions and Future Directions}
\label{section:conclusions}
The paper presents DISC, a deep-learning-based clustering architecture that learns highly discriminative spatio-temporal representations with reconstruction and future prediction tasks on multi-dimensional inertial signals.
The architecture includes a recurrent encoder ConvGRU, two conditional decoders GRU, and a clustering criterion to predict unlabelled human activities-related signals.

The proposed architecture has been compared with state-of-the-art approaches in both traditional and DL-based clustering approaches, demonstrating its effectiveness on three HAR datasets.

We plan to include DISC within the Continuous Learning Platform (CLP)~\cite{ferrari2019framework} framework. CLP is a platform that semi-automatically integrates heterogeneous labeled data and provides them in a homogeneous form. Integrating DISC into CLP would allow the dataset to be also populated with unlabeled signals that are collected with consumer devices.
Moreover, DISC can be also be employed in the early stages of a personalization strategy and thus allowing the recognition of ADL to a never seen user~\cite{amrani2021personalized}.

\begin{comment}

\subsection{Future Works}

\begin{hamza}
check, rielaborare
\end{hamza}

Since Unsupervised Learning is currently one of the most active research areas, new studies regarding the automatic Clustering of inertial sensory readings are necessary. Literature approaches have reached promising results on image data compared to the Human Activity Recognition field. However, in HAR, more and more researchers are introducing new proposals.
Some interesting ideas regarding this work are the following:
\begin{itemize}
    \item testing with new architectures to obtain a better representation learning of the input data;
    \item experimenting better ML-based Clustering algorithms to initialize cluster centroids;
    \item testing new clustering criteria. An idea may be the research of a more fittable distribution to generate soft labels assignment. This could be done by finding the best distribution that fists HAR data by analyzing the Akaike Information Criterion (AIC) and the Bayesian Information Criterion (BIC) values.
\end{itemize}

\end{comment}
\section{Introduction and Background}
\label{section:introduction}

Automated recognition of activities of daily living (ADLs) based on inertial data is popular in various fields, such as surveillance, health care, and delivery~\cite{sun2020using,mukherjee2020ensemconvnet,iyengar2020covid}.
Nowadays, supervised classification of ADLs is mainly based on deep learning (DL) approaches, which are well known to be data hungry~\cite{ferrari2021trends}. Many labeled datasets are available in the literature, however to scale-up the use of these methods in several domains, a very large amount of data is required~\cite{amrani2022homogenization}.

One way to speed up the labeling of ADL samples is to take advantage of the clustering analysis that is an unsupervised strategy to group similar data into clusters~\cite{abedin2020towards}. Its applications include automatic data labeling for supervised learning and pre-processing for data visualization and analysis.

\begin{comment}
\begin{hamza}
hamza: si potrebbe scrivere qualcosa del tipo - \newline"..Clustering is one of the most fundamental unsupervised machine learning problems. Its main goal is to separate data into clusters of similar data points. Besides having its own applications, it is beneficial for multiple other fundamental tasks. For instance, it can serve for automatic data labeling for supervised learning and as a pre-processing step for data visualization and analysis.
However, the performance of clustering algorithms is dependent on the type of the input data, such that different problems and datasets could require different similarity measures and different sep- aration techniques. As a result, dimensionality reduction and representation learning have been extensively used alongside clustering, in order to map the input data into a feature space where sep- aration is easier with respect to the problem’s context. Using deep neural networks (DNNs), it is possible to learn non-linear mappings allowing to transform the data into more clustering-friendly representations..."\newline
Si riconduce bene allo stato dell'arte (dec, dcec, etc.)
\end{hamza}
\end{comment}
Most common methods consist of a two-stages process: \emph{stage 1)} a deep neural network (DNN) procedure for learning deep representative features and \emph{stage 2)} a Machine-Learning (ML)-based clustering algorithm for data grouping~\cite{min2018survey,gan2020data,10.1093/bib/bbz170}.

Deep Embedded Clustering (DEC)~\cite{xie2016unsupervised} learns a mapping from the observed space to a low-dimensional latent space leveraging Stacked AutoEncoders (SAE), which can obtain feature representations and cluster assignments simultaneously.

Improved Deep Embedded Clustering (IDEC)~\cite{guo2017improved} is a modified version of DEC that incorporates an under-complete AutoEncoder to preserve the local structure. The preservation of the local structure prevents the embedded space from being distorted by fine-tuning, thus ensuring embedded spatial features’ representativeness.

Deep Convolutional Embedded Clustering (DCEC)~\cite{guo2017deep} improves IDEC by replacing Stacked AutoEncoders (SAE) with Convolutional AutoEncoders(CAE) to preserve the local data structure.

Recently, Balanced Deep Embedded Clustering (BDEC)~\cite{obeid2021unsupervised} has been proposed to address the inherent vulnerability of DEC to data imbalance by utilizing a pre-processing step that uses a scalable representative algorithm to extract a balanced subset and use it for training.

\begin{comment}
Originally, DL-based clustering approaches have been designed for images. Recently, DL-based approaches based on inertial signals has been proposed in Human Activity Recognition. Mohammad Abu Alsheikh et al. demonstrated that deep activity recognition models provide better recognition accuracy of human activities using triaxial accelerometers, making them a good approach for weight initialization~\cite{alsheikh2016deep}. Harish Haresamudram et al. analyzed the role feature representations play in HAR using wearable devices. They demonstrated that features extracted from AutoEncoders are better than commonly statistical features, making end-to-end learning architectures particularly suitable for representation learning in clustering~\cite{haresamudram2019role}. Lu Bai et al. proposed an unsupervised learning approach in which a deep learning variational auto-encoder (VAE) is trained to learn representative features of activities, grouped regarding the nature of the activity type~\cite{bai2019motion2vector}. Zhuxi Jiang et al. introduced variational deep embedding (VaDE), an unsupervised generative clustering approach within the framework of VAE~\cite{jiang2016variational} .

Likewise, Alireza Abedin et al. introduced an unsupervised feature learning process that adopts auto-encoders to exploit unlabeled data~\cite{varamin2018deep}. Some years later, the same authors proposed deep sensory clustering (DSC), a deep clustering architecture that  \begin{dany} questo pezzo non e' chiaro: learns highly discriminative representations using self-supervision with reconstruction and future prediction tasks informed by feedback from a clustering objective to guide the network towards clustering-friendly representations ~\cite{abedin2020towards}. \end{dany}
\begin{hamza}
può andare bene?
Some years later, the same authors proposed deep sensory clustering (DSC)~\cite{abedin2020towards}, a deep clustering architecture that employs a recurrent autoencoder and a clustering criterion to learn clustering-friendly representations from inertial sensory readings.
\end{hamza}
\end{comment}

Finally, Deep Sensory clustering (DSC)~\cite{abedin2020towards} is a recent clustering architecture that employs a Recurrent AutoEncoder (RAE) and a clustering criterion to learn clustering-friendly representations from inertial sensory readings. DSC can be considered the first approach using deep clustering on inertial signals.

The paper proposes Deep Inertial Sensory Clustering (DISC), a deep-learning-based clustering architecture that automatically performs unsupervised learning and label annotation by analyzing multi-dimensional inertial signals.
The architecture combines a recurrent AutoEncoder and a clustering criterion to predict unlabelled human activities-related signals.
As for previous approaches in the state of the art, our architecture consists of two parts. The first part covers stage 1. It consists of an AutoEncoder that tries to reconstruct two outputs from the input signals: the inverse input and the future sequence of the input. The second part covers stage 2 and is devoted to clustering the dimensionally reduced signals from stage 1.

The main contribution of the paper with respect to the state of the art is the use of ConvGRU layers (Convolutional Gate Recurrent Unit~\cite{ballas2015delving}) that allow spatio-temporal features to be extracted. The recursive layers (LSTM, GRU) have been shown to be useful in extracting temporal features,
while the convolutional layers in extracting spatial features.

The achieved architecture has been compared with four state-of-the-art techniques (DEC~\cite{xie2016unsupervised}, IDEC~\cite{guo2017improved}, DCEC~\cite{guo2017deep}, and DCS~\cite{abedin2020towards}) on three publicly available HAR datasets (UCI HAR~\cite{anguita2013public}, Skoda~\cite{stiefmeier2008wearable}, and MHEALTH~\cite{banos2014mhealthdroid}). Results show, on average,  its effectiveness in the inertial sensory clustering task.

The paper is organized as follows. Section~\ref{section:architecture} introduces the architecture; Section~\ref{section:material_and_methods} describes the implementation details of the architecture, the metrics used to validate the approach, and the datasets used in the evaluation. Section\ref{section:results} discusses the results obtained. Finally Section~\ref{section:conclusions} presents the conclusions and the future directions.

\begin{hamza}

\end{hamza}
\section{Material and Methods}
\label{section:material_and_methods}

\subsection{Network Architecture}

The recurrent encoder $Enc_{\theta}$ is a 2-layer bi-directional ConvGRU with 256 hidden states
and the conditional recurrent decoders $Dec_{\phi}$ use uni-directional connections with 512 hidden states. The bottleneck embedded dimension of autoencoders is set to 256.

A single layer in the clustering network $f_{\mu}(.)$ has been used to integrate the CAH (Clustering Assignment Hardening).

All experiments share the same network architecture.

\subsection{Optimization Settings}

The recurrent AutoEncoder is pre-trained end-to-end in mini-batches of size 256 for 100 epochs using the Adam optimizer~\cite{kingma2014adam}. The initial learning rate has been set to 10 and, after 70 epochs, decayed by a factor of 10.

A Batch Normalization~\cite{ioffe2015batch} operation is used in the encoder fully connected layer to make the training faster and more stable.
Then, the recurrent AutoEncoder is refined with the clustering objective till the cluster assignment changes among two consecutive epochs are less than 0.1\%.

The coefficient $\gamma$ is set to 0.1. The cluster centroids are initialized with $k$-Means and agglomerative clustering algorithms.

The parameters are maintained constant in all the experiments.

\subsection{Baseline}

Four state-of-the-art approaches served as baselines.

\subsubsection{Deep Embedded Clustering (DEC)~\cite{xie2016unsupervised}} relies on  AutoEncoders as initialization method and uses $k$-Means for clustering. First, the method pre-trains the model using an input reconstruction loss function (non-clustering loss). Second, a clustering algorithm initializes the clustering centers, without considering the decoder. Then, the model is optimized using the Cluster Assignment Hardening (CAH) loss, and the clustering centers are updated at each iteration by minimizing the Kullback-Leibler (KL) divergence as a loss function.

\subsubsection{Improved Deep Embedded Clustering (IDEC)~\cite{guo2017improved}} is a modified version of DEC that preserves the decoder layer, which is used for clustering loss. The reconstruction
loss of AutoEncoders is added to the objective and optimized simultaneously with clustering loss, preserving the local data structure.

\subsubsection{Deep Convolutional Embedded Clustering (DCEC)~\cite{guo2017deep}} uses the IDEC algorithm by incorporating a Convolutional AutoEncoder (CAE). Due to its spatial mapping relationships, CAE is able to learn deeper and spatial embedded features for clustering.

\subsubsection{Deep Sensory Clustering (DSC)~\cite{abedin2020towards}} is an alternative to IDEC. It is based on a recurrent AutoEncoder with two different decoders: one for input sequence reconstruction and one for anticipation sequence reconstruction. By using two decoders the AutoEncoder is forced to learn highly discriminative embedded features that are used to initialize the hidden state of the decoders.

\subsection{Datasets}
The proposed architecture and baselines were evaluated on three datasets.

\subsubsection{Human Activity Recognition Using Smartphones Dataset (UCI HAR)~\cite{anguita2013public}} includes 3-axial linear acceleration, 3-axial angular velocity, and gyroscope sensor data. The signals were recorded with a Samsung Galaxy S II smartphone at a constant rate of 50Hz, and the activities were performed by 30 volunteers.
Each subject performed 6 activities.
All the participants were wearing a smartphone on the waist during the experiment execution.

\subsubsection{Skoda Mini Checkpoint Activity Recognition Dataset (Skoda)~\cite{stiefmeier2008wearable}} includes 20 sensors (3-axial acceleration) placed on the left and right upper and lower arm. The signals were recorded at a sampling rate of 98Hz, and the activities were performed by 1 volunteer who executed 19 times each activity. The dataset contains 10 activities

\subsubsection{mHealth Dataset (MHEALTH)~\cite{banos2014mhealthdroid}} includes 3-axial acceleration from the chest sensor, 2 electrocardiogram signals, 3-axial acceleration from the left-ankle sensor, 3-axial acceleration from the right-lower-arm sensor, 3-axial magnetometer from the left-ankle sensor, 3-axial magnetometer from the right-lower-arm sensor, 3-axial gyroscope from the left-ankle sensor, 3-axial gyroscope from the right-lower-arm sensor. The recorded activities are 12 and are performed by 10 volunteers.
All sensors were recorded at a sampling rate of 50Hz.

\begin{comment}
\begin{table}[tb]
\centering
\caption{Datasets used in evaluation.}
\label{tab.4datasets}
\scriptsize
\begin{tabular}{lccc}
  & \textbf{UCI HAR~\cite{anguita2013public}} & \textbf{Skoda~\cite{stiefmeier2008wearable}} & \textbf{MHEALTH~\cite{banos2014mhealthdroid}} \\
\toprule

Sensor Sampling Rate        & 50Hz & 33Hz & 50Hz \\

Size of Sliding Window      & 2.56s & 2s & 2.56s \\

\# Sensor Channels   & 9 & 60 & 23 \\

\# Activities        & 6 & 10 & 12 \\

\# Training Windows  & 7,352 & 5,653 & 4,279\\

\# Testing Windows   & 2,947 & 705 & 1,070\\
\bottomrule

\end{tabular}
\end{table}
\end{comment}

Datasets were initially rescaled using per-channel normalization. Then, the signals have been divided in windows with an overlap between subsequent segments of 50\%.\\ For UCI HAR and MHEALTH, a window of $2.56s$ was used, while for Skoda, a window of $2s$.
Furthermore, in MHEALTH, the data relative to electrocardiogram signals have been removed as the other datasets had no data of this type.

After the pre-processing phase, the first 50\% of sensory measurements in each sample represents the \emph{input sequences}. Consequently, the \emph{temporally inverted sequence} of the input is used as the target sequence for the reconstruction task. The \emph{remaining sensory measurements} are considered as the target sequence for future prediction.

\subsection{Evaluation Metrics}

All clustering methods are evaluated by \emph{Clustering Accuracy (ACC)}~\cite{strehl2002cluster} and \emph{Normalized Mutual Information (NMI)}~\cite{vinh2010information}, widely used in unsupervised learning scenarios. Both metrics expect values in the range [0, 1], with 1 being the perfect clustering and 0 being the worst.

The Clustering Accuracy (ACC) takes a cluster assignment and a ground truth assignment and then finds the best matching between them.  We used the \emph{Hungarian algorithm}~\cite{kuhn1955hungarian} to compute the best mapping (see Equation~\ref{eq:ACC}).

\begin{equation}
    ACC = max_n \; \frac{ \sum_{i=1}^n 1\{ l_i=m(c_i) \} }{n}
\label{eq:ACC}
\end{equation}
where $l_i$ is the true label, $c_i$ the cluster assignment, and $m$ ranges over all possible one-to-one mappings between clusters and labels.

The Normalized Mutual Information (NMI), between ground-truth labels and the labels obtained by clustering, determines the class labels' entropy reduction, assuming the cluster labels are known. It is used for determining the quality of the clustering and is defined in Equation~\ref{eq:NMI}.
\begin{equation}
    NMI(y,c) = \frac{2 I(y,c)}{H(y)+H(c)}
    \label{eq:NMI}
\end{equation}
where $y$ is the true label, $c$ is the obtained cluster label, $I(.,.)$ is the mutual information and $H(.)$ is the entropy.
\section{Experiments Results}
\label{section:results}

\begin{table*}[h!]
\centering

\caption [Comparison of Clustering performances on HAR Datasets.]{Clustering performances in terms of NMI and ACC on the three datasets achieved by DISC and state-of-the-art methods. The first group of rows shows the performance of traditional clustering techniques applied on raw data. The second group of rows shows the results achieved by DISC and state-of-the-art AutoEncoders combined with traditional clustering techniques. The last group of rows shows the comparison between DISC and the end-to-end deep clustering methods.
}

\resizebox{1\textwidth}{!}{
\begin{tabular}{r|cccc|cccc|cccc|cccc}
\toprule

& \multicolumn{4}{c}{\textbf{UCI HAR}}  & \multicolumn{4}{c}{\textbf{Skoda}} & \multicolumn{4}{c}{\textbf{MHEALTH}} & \multicolumn{4}{c}{\textbf{Average}} \\

& \multicolumn{2}{c}{\textbf{Train}} & \multicolumn{2}{c}{\textbf{Test}} & \multicolumn{2}{c}{\textbf{Train}} & \multicolumn{2}{c}{\textbf{Test}}& \multicolumn{2}{c}{\textbf{Train}} & \multicolumn{2}{c}{\textbf{Test}} & \multicolumn{2}{c}{\textbf{Train}} & \multicolumn{2}{c}{\textbf{Test}}\\

 &\textbf{NMI}&\textbf{ACC}&\textbf{NMI}&\textbf{ACC}&\textbf{NMI}&\textbf{ACC}&\textbf{NMI}&\textbf{ACC}&\textbf{NMI}&\textbf{ACC}&\textbf{NMI}&\textbf{ACC}&\textbf{NMI}&\textbf{ACC}&\textbf{NMI}&\textbf{ACC}\\%[0.3cm]
\midrule

\multicolumn{14}{l}{\textbf{Traditional Clustering on Input Data Space}} & & & \\[0.25cm]

$k$-Means     & 49.36 & 45.04 & 45.90 & 40.69 & 38.54 & 39.18 & 36.93 & 38.59 & 41.85 & 43.95 & 42.14 & 44.41  & 43.25 & 42.72 & 41.66 & 41.23 \\
AC-Average  & 1.18 & 19.16 & 1.8 & 18.29    & 1.51 & 13.88 & 18.38 & 14.37  & 12.12 & 10.35 & 9.63 & 10.19   & 4.93 & 14.46 & 9.94 & 14.28 \\
AC-Complete & 3.45 & 19.56 & 19.22 & 31.69  & 33.29 & 31.32  & 31.86 & 28.07 & 23.49 & 42.10 & 30.02 & 13.74 & 20.07 & 30.99 & 27.03 & 24.50 \\
AC-Ward     & 40.73 & 42.26 & 47.71 & 43.26 & 41.79 & 41.35 & 36.99 & 33.19 & 48.07 & 48.26 & 48.54 & 48.60  & 43.53 & 43.96 & 44.41 & 41.68 \\[0.2cm]

\midrule

\multicolumn{14}{l}{\textbf{Traditional Clustering on proposed Recurrent AutoEncoding Space }} & & & \\[0.25cm]

DSC $k$-Means     & 51.93 & 60.19 & 45.49 & 55.62 & 53.75 & 47.56 & 50.64 & 42.62 & 54.86 & 43.96 & 55.75 & 48.24 & 53.51 & 50.57 & 50.62 & 48,82 \\

DSC AC-Average  & 45.18 & 37.57 & 46.41 & 34.61 & 18.88 & 16.96 & 38.59 & 30.22 & 34.54 & 20.47 & 47.01 & 29.26 & 32.86 & 25.00 & 44.00 & 31.36  \\

DSC AC-Complete & 40.66 & 40.03 & 40.81 & 43.67 & 32.55 & 32.47 & 41.57 & 35.93 & 42.23 & 35.05 & 44.42 & 36.51 & 38.48 & 35.85 & 42.26 & 38.70 \\

DSC AC-Ward & 75.27 & 74.78 & 52.83 & 60.33 & 55.81 & 51.51 & 54.41 & 45.96 & 61.07 & 48.91 & 57.04 & 46.28 & 64.05 & 58.40 & 54.76 & 50.85 \\
\textbf{(ours)} DISC $k$-Means     & 54.84 & 55.27 & 48.27 & 54.62 & 50.65 & 47.83 & 47.87 & 43.10 & 65.14 & 55.27  & 64.88 & 54.04 & 56.88 & 52.76 & 53.67 & 50.59  \\
\textbf{(ours)} DISC AC-Average  & 43.82 & 39.17 & 41.26 & 35.12 & 29.74 & 28.82 & 29.63 & 24.12 & 30.80 & 13.69 & 31.24 & 13.16 & 34.79 & 27.23 & 34.04 & 24.13  \\
\textbf{(ours)} DISC AC-Complete & 37.77 & 44.12 & 38.91 & 40.62 & 35.48 & 34.62 & 44.13 & 37.31 & 46.58 & 36.22 & 50.21 & 30.24 & 39.94 & 38.32 & 44.41 & 36.06  \\ %
\textbf{(ours)} DISC AC-Ward     & 64.21 & 63.14 & 51.75 & 60.77 & 52.85 & 47.92 & 54.10 & 43.45 & 68.85 & 58.31 & 70.98 & 59.52 & 61.97 & 56.46 & 58.94 & 54.58 \\[0.2cm]

\midrule

\multicolumn{14}{l}{\textbf{End-to-End Deep Clustering}} & & & \\[0.25cm]

DEC~\cite{xie2016unsupervised}  & 55.57 & 49.93 & 55.20 & 49.10 & 46.86 & 41.02 & 46.58 & 41.08 & 50.52 & 43.71 & 51.68 & 45.11 & 50.98 & 44.89 & 51.15 & 45.10  \\
IDEC~\cite{guo2017improved}  & 55.76 & 51.78 & 54.43 & 51.02 & 49.65 & 46.70 & 48.93 & 40.56 & 51.28 & 42.59 & 52.34 & 44.76 & 52.23 & 47.02 & 51.90 & 45.45  \\
DCEC~\cite{guo2017deep}   & 52.77 & 48.39 & 53.12 & 48.88 & 42.90 & 39.56 & 45.51 & 42.24 & 44.15 & 23.51 & 45.13 & 24.29 & 46.61 & 40.15 & 47.92 & 38.47  \\

DSC ($k$-Means)~\cite{abedin2020towards}  & 64.75 & 64.54 & 61.58 & 61.28 & 56.91 & 50.97 & 57.01 & 50.28 & 62.65 & 57.19 & 63.06 & 56.85 & 61.43 & 57.56 & 60.55 & 56,13 \\

DSC (Ward)~\cite{abedin2020towards}   & 76.43 & 78.79 & 71.25 & 75.41 & \textbf{56.97} & 52.90 & \textbf{59.06} & 53.48 & 59.42 & 51.57 & 60.91 & 53.33 & 64.27 & 61.08 & 63.74 & 60.74 \\

\textbf{(ours)} DISC ($k$-Means)  & \textbf{85.72} & \textbf{92.08} & 80.81 & \textbf{87.99} & 52.93 & 54.45 & 52.93 & \textbf{54.45} & \textbf{68.15} & \textbf{53.96} & \textbf{69.38} & \textbf{54.83} & \textbf{68.93} & \textbf{66.83} & \textbf{67.71} & \textbf{65.76}  \\
\textbf{(ours)} DISC (Ward)     & 85.47 & 81.86 & \textbf{81.91} & 79.40 & 54.91 & \textbf{54.57} & 55.93 & 53.45 & 63.23 & 50.74 & 61.29 & 48.82 & 67.87 & 62.39 & 66.38 & 60.56  \\[0.2cm]

\bottomrule

\end{tabular}}

\label{tab:results_labels}
\end{table*}

Table \ref{tab:results_labels} shows the clustering performance in terms of NMI and ACC on the UCI HAR, Skoda, and MHEALTH datasets achieved by the proposed and state-of-the-art methods.

We used as traditional clustering methods $k$-Means~\cite{hamerly2003learning} and Agglomerative Clustering (AC)~\cite{mullner2011modern}, with three different linkage types (i.e., Average, Complete, and Ward).

The first group of rows shows the performance of traditional clustering techniques applied on the raw data.
The second group of rows shows the results achieved by our architecture DISC and state-of-the-art- AutoEncoders combined with traditional clustering techniques. The last group of rows shows the comparison of DISC method with the state of the art.
Boldface font stands for best results.

Overall the results confirm the effectiveness of deep clustering approaches against traditional clustering approaches on input space data and AutoEncoding space. On average, our proposal performs better with respect to the state-of-the-art.

DSC (Ward) performs slightly better (on average of about 3\%) than our proposal in terms of NMI only on the Skoda dataset. Our method overcomes the best method in the state of the art of about 10\% in terms of NMI and ACC on both UCI HAR  and MHEALTH. Improvements are mainly related to the use of convolutional GRU instead of simply GRU.

\begin{figure*}[h]
  \centering
  \includegraphics[width=0.8\textwidth]{sections/images/uci_DISC.png}
  \caption{Feature space progression of the UCI HAR Dataset with DISC ($k$-Means initialization).}
  \label{fig:uci_DISC}
\end{figure*}

Figure~\ref{fig:uci_DISC} shows the progression of the learned feature space from the initial setup to the final clustering-oriented embedding space for the UCI HAR dataset. The progression is achieved by firstly using the Principal Component Analysis (PCA)~\cite{jolliffe2005principal} to reduce the embedding from 256 to 50 dimensions, and then by using the t-Distributed Stochastic Neighbor Embedding (t-SNE)~\cite{van2008visualizing} method applied for mapping 50-dimensional data into 2-dimensional data.

The proposed architecture discovers well-defined and separated clusters of activity segments with strong correspondence to the ground-truth labels. In particular, three activities (walking, walking upstairs, and walking downstairs) are correctly delimited, while the others (sitting, standing, and lying) are well delimited but more dispersed, due to the high number of subjects in the dataset, which increase the inter-variability of activities.