\section{Introduction}

In Natural Language Processing (NLP), an abstract topic refers to a set of words or terms that describe some semantic concepts shared by a subset of textual documents in a corpus. Topic models are statistical models and algorithms that discover abstract topics from a corpus of text documents and are widely used in exploratory data analysis for organizing, understanding, and summarizing large amounts of text data~\cite{churchill2022evolution}. Topic evolution analysis~\cite{hu2015modeling} studies the change in topics as new information becomes available over time and has been used in a variety of applications, such as discovering the evolution of research topics and innovations in scientific archives~\cite{li2021analytic} and understanding trends in public opinion on particular issues~\cite{sha2020dynamic}.

Tracking the evolution of topics is studied using Dynamic Topic Models~\cite{ABDELRAZEK2023102131}, which are a variant of topic models that can be applied to temporal archives. These models can update their estimates of the underlying topics as new documents are added to the corpus, allowing the identification of evolving trends and patterns in the archive. Dynamic topic models can be divided into Probabilistic Dynamic Topic Models (PDTM) and Neural Dynamic Topic Models (NDTM).

PDTMs~\cite{blei2006dynamic,dieng2019dynamic} are variations of Latent Dirichlet Allocation (LDA)~\cite{blei2003latent} that assign probabilities to different words and topics over time, allowing it to make inferences about the underlying themes and patterns in the data as they change. These models are basically not scalable in the face of large archives with large dictionaries ~\cite{bhadury2016scaling} and not adapted to handling short texts for applications such as social media monitoring (e.g. Twitter)~\cite{albalawi2020using}. In addition, the number of topics is the same for all periods and must be known prior to the modeling procedure, which is in contrast to the temporal variations of dynamic topics in different periods.

NDTMs use neural networks~\cite{grootendorst2022bertopic} to formalize the underlying  probability distributions of words and documents into a fixed-length vector representation space. This representation gives NDTMs several advantages over PDTMs: \textit{1) Scalability:} NDTMs can handle large data sets more efficiently than PDTMs~\cite{grootendorst2022bertopic}.  \textit{2) Accuracy:} NDTMs can learn more expressive and semantically meaningful representations of the data than PDTMs ~\cite{grootendorst2022bertopic,angelov2020top2vec} and can be used efficiently on short text data~\cite{albalawi2020using}.\textit{3) Flexibility:} NDTMs automatically determine the number of topics to be generated, while PDTMs require the number of topics as an input parameter, which has to be determined experimentally for each new corpus of documents in order to obtain an optimal result~\cite{blei2006dynamic,dieng2019dynamic}.

However, existing NDTMs based on document embeddings generate dynamic topics by performing global clustering over the entire dataset and representing the topic evolution by a dynamic topic representation function~\cite{grootendorst2022bertopic}. This process limits the NDTMsl to having the same number of topics for each period, which results in ignoring the temporal variations and volatility structure of dynamic topics, such as the number of document clusters in each period or the document size of each cluster. This limitation affects the observation of topic evolution and reduces the quality of topics presented in each period.

To address these issues and limitations, we introduce a novel family of dynamic neural topic models called Aligned Neural Topic Models (ANTM), which takes into account the temporal variations of dynamic topics and produces diverse and high-quality topics. ANTM introduces a novel technique for discovering evolving topics through an alignment process (called Aligned Clustering) on document clusters. Aligned Clustering takes into account the volatility structure of dynamic topics and can track evolving topics over time with the temporal configuration of each period. It also takes advantage of a self-supervised machine learning model called Data2Vec~\cite{baevski2022data2vec}, which predicts latent representations of the documents based on a masked view of the input text data in a self-distillation setup using a standard Transformer architecture. Data2Vec makes the embedding process much faster. It also significantly improves the quality of topic representation compared to Probabilistic Topic Models and pre-trained Large Language Models (LLMs)~\cite{chen2021evaluating}.
Our experiments show a significant improvement in terms of coherence and diversity score over existing PDTMs and NDTMs on DBLP and arXiv datasets.

The proposed model was also trained on Elon Musk's tweets dataset and New York Time articles, and extracted interesting information regarding the evolution of several topics. As shown in \Cref{fig:musk}, ANTM can perform effectively on short text applications such as Twitter. For example, in the figure you can see the words \textit{''Russia'',''Ukraine'',''Connectivity''} and appeared in two periods, spring and fall 2022, which coincide with two attacks of Russia on Ukraine and the tweets of Elon Musk in support of Ukrainians with the free Internet through Starlink. Another example is represented in \Cref{fig:wordsdblp}, which shows the development of computer science research on medical science, in which the words \textit{''CNN'',''Lung'',''Diagnosis''} appear in the period 2018 to 2020, which coincide with the pandemic and advances in computer vision research. And last example in \Cref{fig:wordsnyt} shows the evolution of New York Time articles on foreign terrorist activities where you can see the transition from the words \textit{''Boko'',''Harram'',''Nigeria''} in 2015 to \textit{''ISIS'',''Iraq'',''Baghdad''} in 2016 and later \textit{''Kabul'',''Taliban'',''Afghanistan''} in 2017 to 2019. These examples demonstrate the capability of our method on four different datasets of different sizes.

The rest of the paper is organized as follows. In \Cref{sec:related} we will review the state of the art and related work. ANTM is proposed in \Cref{sec:method} and evaluated by an experiment in \Cref{sec:experiment}. Finally, we will compare our results with the literature in \Cref{sec:comparison} and conclude the paper in \Cref{sec:conclusion}.

\section{Related Works}
\label{sec:related}

The Dynamic Topic Model (DTM)~\cite{blei2006dynamic} is the first  Probabilistic Dynamic Topic Model (PDTM) to incorporate temporal components to study topic evolution. DTM is a variant of the generative probabilistic model called Latent Dirichlet Allocation (LDA)~\cite{blei2003latent}. DTM creates topics that consist not only of a set of words, but also of a timestamp or time range. It uses a Bayesian approach to model the change in the relative proportions of topics within documents over time. DTM is extended by several new models. One example is the Discrete-Time Dynamic Topic Model (dDTM) \cite{bahrainian2017modeling}, which requires the data to have discrete time intervals. In contrast to dDTM, the Continuous-Time Dynamic Topic Model (cDTM)~\cite{wang2012continuous} can handle any data point in time, regardless of the time resolution.

Although DTM was an innovative tool for studying topic evolution at the time, it has several limitations. The first of these is scalability, where the DTM becomes computationally expensive and time-consuming when dealing with very large data sets. In addition, DTM cannot fully capture the diversity of topics over time because it assumes that topics remain relatively stable over time. Therefore, topics are consistent across the dataset and one may not be able to detect fine-grained changes in topics or subtle variations in how topics are represented in different time periods. In addition, the number of topics must be known before modelling. There are also several generative probabilistic topic models~\cite{wei2007dynamic,10.1145/1835804.1835889,10.1145/2872427.2883046,zosa2019multilingual} that attempt to overcome these problems. For example, DTM is combined with word embeddings in Dynamic Embedded Topic Models (DETM)~\cite{dieng2019dynamic,dieng2020topic} to improve the performance of topic models by providing a more informative representation of words. However, it is still very time-consuming and we will show that its topic representations are of lower quality in terms of diversity and coherence compared to NDTMs. As another example, in order to reduce the computational complexity and the number of variational parameters for handling large vocabularies, an amortized variational inference method based on DTM is developed in~\cite{pmlr-v180-tomasi22a}. There are also efforts in the development of scalable PDTMs, such as~\cite{pmlr-v84-jahnichen18a}, which extends the class of tractable priors from Wiener processes to the more general class of Gaussian processes. This approach allows the model to be applied to large collections of text and to explore topics that evolve smoothly over a long period of time.

In spite of all these efforts to address the problems of PDTMs, they have still been shown to have less coherence and diversity in topic representation compared to Neural Dynamic Topic Models (NDTMs)~\cite{grootendorst2022bertopic,angelov2020top2vec}. In addition, they are very time consuming and still expect the user to have knowledge of the number of topics prior to topic modelling. NDTMs formalise the underlying temporal probability distributions of words and documents into a fixed-length vector representation space through advances in neural networks in NLP. While preserving important words in the topic descriptions, these methods generate dense clusters of interpretable documents. Among the most widely used Neural Dynamic Topic Models are BERTopic\cite{grootendorst2022bertopic}, which, unlike PDTMs, allows context to be embedded in the generated topics through the use of Large Language Models (LLMs) that generate document representations. These embedded vectors are then divided into a set of clusters that describe a set of documents that describe semantically similar topics across an archive. In fact, BERTopic combines BERT and c-TF-IDF~\cite{grootendorst2022bertopic} to generate dense clusters of interpretable topics, while preserving important words in the topic descriptions. By computing the c-TF-IDF topic representation in each time window, BERTopic allows dynamic topic modelling. Although BERTopic can handle large and short text datasets (due to LLMs), it still creates static clusters over the whole dataset and only represents them dynamically. This process ignores the temporal structure of dynamic topic models in each period. Furthermore, it limits the topic model to having the same number of topics for each period. In fact, we will show that grouping documents statically across the corpus and representing them dynamically based on global document clusters affects the observation of topic evolution and reduces the quality of topics represented in each period. Another advanced NDTM method is developed in \cite{liang:hal-03942487,pmlr-v162-zhang22n} to perform dynamic topic modelling using Graph Neural Networks (GNNs). These models preserve document dynamics and network adjacency of dynamic document networks. As these methods are specifically designed for document networks, they are beyond the scope of this paper for comparison.

The method proposed in this paper is fast and more scalable than PDTMs. In addition, it can take into account the limitations discussed in BERTopic and can produce higher quality word representations that can explore topic evolution in detail. Even though the aligned clustering process takes longer than the global clustering of BERTopic, the Data2Vec embedding procedure of ANTM takes five times less time, making ANTM faster overall.

\section{ANTM}
\label{sec:method}

One way to study how these topics evolve is to cluster archives over time and observe how the topic representation changes. In this work, we adapt a clustering procedure with an aligned structure, which results in evolving temporal topics and achieves a more diverse and coherent temporal topic representation.

\begin{figure*}[hbtp]
\centering
\includegraphics[width=0.8\textwidth]{architecture.png}
    \caption{The Architecture of ANTM }
    \label{fig:arc}
\end{figure*}
The proposed method, called ANTM, consists of three layers (\Cref{fig:arc}). The first layer provides a dynamic vector representation for each document, corresponding to its content as it relates to other documents. A self-supervised machine learning method called Data2Vec is used for embedding in the first layer. Transformer-based LLMs models such as BERT can also be used. The second layer splits the document vector representations into a set of temporal time frames, and later performs aligned clustering on the reduced dimension of time frames (by AlignedUMAP) after an alignment procedure on a hierarchical density based clustering algorithm called HDBSCAN. Finally, the third layer is responsible for providing word representations for each set of aligned clusters over time. We use the c-TF-IDF method in this layer.

\subsection{Contextual Embedding Layer (\textbf{CEL})}
\hub{As I understand, the vector representation is not dynamic. We can simply define $y_i$ instead of $y_i^t$}
\hub{we can also replace $M$ by \textbf{CEL} ?}

\hub{removed ''dynamic''}
The Contextual Embedding Layer (\textbf{CEL}) (\Cref{fig:arc}) is responsible for providing a vector representation $y$ for the document $d$ in the corpus $D$. More formally, the embedding $y$ for a document $d$ is a mapping $\textbf{CEL}: d\mapsto y \in \Re^z$, capturing semantic information from the corpus $D$. The document embedding represents words and documents in a low-dimensional feature vector space, where the embedding dimension $z$ is expected to be much smaller than the size of the vocabulary (i.e., the number of unique words in $D$).
\hub{the vector dimension is much smaller that the size of the vocabulary, not the size of the corpus}

Transformer-based self-supervised models (e.g., Data2Vec) and LLMs (e.g., BERT, GPT-3), trained on a large corpus of text data, are the most common approaches to computing document embeddings. Transformer-based models capture the meaning and context of a document by using an attention mechanism to consider the context of words in a document.
Given an input document $d$, the number $u$ of tokens in $d$, and $\textbf{h}_j$  the hidden state vector of the $j$-th token in $d$, the embedding $y = \textbf{CEL}(d)$ of the whole document $d$ is then obtained by taking the mean of all the hidden state vectors of the tokens in $d$:
\begin{align}
\label{eq:cel}

   \textbf{CEL}(d) = \frac{1}{u} \sum_{z=1}^{u} \textbf{h}_z
\end{align}

Transformers' attention mechanism allows the model to focus on different parts of the input text, weighting the importance of each word or phrase in the context of the whole document. This allows \textbf{CEL}~to capture the meaning of the document as well as its temporal aspect, since the meaning of words and phrases can change over time.
\hub{removed ''context''}
\hub{removed sentence ''They therefore ....'' which is a bit redundant}

\subsection{Aligned Clustering Layer (ACL)}
\hub{the following paragraph in red could be rewritten in a more compact way. }

After performing transformer-based document embedding on the corpus $D$, we can obtain a dynamic vector representation of $D$ by applying a Sliding Window Segmentation (\textbf{SWS}, \Cref{fig:arc}) process that divides $D$ into a series of $n$ overlapping time periods $\{W^1,\dots, W^n\}$.  We then denote the set of documents published in the period $W^t$ by $D^t\subseteq D$ and the set of embeddings of $D^t$ by $Y^t= \{\textbf{CEL}(d) | d \in D^t \}$.

\hub{remove:
 After performing transformer-based document embedding  of the corpus $D$, we can achieve a dynamic vector representation by segmenting $D$ into $n$ different time frames.
 The temporal representations are denoted by $\{D^1,\dots,D^n\}$ and $\{Y^1,\dots,Y^n\}$ where $D^t$ and $Y^t$ for $t=1,\dots,n$ are respectively the set of documents and their  vector representation in time frame $t$ .
 There are various temporal paradigms for dividing the corpora into a set of sequential time frames. In this study, we use a Sliding Window Segmentation (\textbf{SWS}) method that divides the documents into sequential segments with a constant duration and a fixed time overlap.
}

 %

\hub{removed
    \definition{Let $D$ be a set of documents and $W=\{W_1,\ldots, W_n\}$ be a set of $n$ overlapping sliding time windows:
    \begin{equation}
    \label{eq:sws}
        \textbf{SWS}:  D \mapsto \{ D^t\}_{t=1}^n
    \end{equation}
    where $D^t\subset D$ contains all documents published in time window $W_t$.
    }

By applying embedding function \textbf{CEL}, we can  obtain for each subset of documents $D^t$ a subset of embeddings  $Y^t=\textbf{CEL}(D^t)$.
}

The  clustering layer aims to provide evolving document clusters by jointly grouping similar documents in each time frame. This procedure is called  Partitioned Clustering (\textbf{PC}):

 {\definition Let $D^t$ be a set of documents in the $t$-th time frame of the document corpus $D$ and let $Y^t$ denote the set of embeddings of $D^t$. Partitioned Clustering $\textbf{PC}: D^t \mapsto \{D_i^t\}$ clusters the documents $D^t$ by their embeddings $Y_t$ and returns a set of \textit{local} document clusters $D_i^t\subseteq D$ for $i=1$  to  $k_t$ where \emph{all document embeddings in $Y_i^t$ are similar}.

}

Clustering documents by the similarity of their embeddings reveals evolution patterns and trends in the data that may not be apparent when documents are grouped based on their content alone. To obtain a representation of the temporal evolution of these clusters, we apply a final alignment step that generates document clusters based on their embedding similarity. The result is a set of \textit{evolving document clusters} that contain documents that are semantically similar to documents in the same time period, but also to other documents in previous or subsequent time periods:

{\definition Let $\{D_i^t\}$ be the set of local document clusters obtained by $\textbf{PC}$.
Cluster Alignment $\textbf{CA}: D \mapsto \{ D_k \}$ generates $m$ subsets of documents $D_k\subseteq D$ for $k=1$ to $m$ and where $D_k$ is the \textit{union} of a set of local document clusters (topics) $D_i^t$ with \emph{similar embedding clusters $Y_i^t$}.
}

We call $D_k$ an evolving cluster (EC) and $D_i^t$ a local document cluster of $D_k$. We can use different cluster linkage measures (single, average, centroid, complete) to estimate the similarity of the embedding clusters $Y_i^t$ (also note that since the embeddings of the local document clusters of $D_k$ are similar, all the documents in $D_k$ are similar as well).

\begin{figure*}[htbp]
  \centering
  \begin{subfigure}[b]{0.4\textwidth}
    \includegraphics[width=0.8\textwidth]{fig_d2v.png}
    \caption{based on Data2Vec on arXiv}
  \end{subfigure}
   \hfill
  \begin{subfigure}[b]{0.4\textwidth}
    \includegraphics[width=0.8\textwidth]{fig3dblp.png}
    \caption{based on BERT on DBLP}
  \end{subfigure}
  \caption{Evolving Clusters}
    \label{fig:3d}
\end{figure*}
Two examples of evolving clusters are shown in \Cref{fig:3d}. All local clusters of each time period are on the same plane, and all local clusters of the same evolving topic have the same color. In the following section we will describe an implementation of ANTM and in particular more details on the issues of coherently using document embeddings  for the computation and alignment of local clusters.

\subsubsection{Implementation}
Several algorithms and techniques can be used for this type of clustering, such as time-sensitive spectral clustering and time-sensitive density-based clustering~\cite{ghosal2020short}.
Density-based clustering algorithms identify clusters in a data set based on the density of points in feature space. These algorithms are particularly useful for identifying clusters of varying density and shape. They are also sensitive to the parameters used to compute the density and to the presence of noise or outliers in the data. In addition, these algorithms are useful when the clusters have arbitrary shapes and when the number of clusters is not known in advance. This feature provides an advantage over PDTMs and generative models that require a fixed number of topics as input. The Aligned Clustering is developed in the following three steps.

\begin{figure*}[t]
\centering
\includegraphics[width=0.8\textwidth]{nyt.png}
\caption{Evolution of New York Times News on Foreign Terrorist Activities}
\label{fig:wordsnyt}
\end{figure*}

\paragraph{Aligned Dimension Reduction}
Since density-based clustering algorithms are limited by the size of the embedding features, we need to reduce the embedding size of documents to a lower dimension. There are several ways to reduce dimensions. One popular approach is to use manifold learning techniques for dimension reduction, which assume that high-dimensional data lies on or near a lower-dimensional manifold. These techniques attempt to uncover the underlying structure of the data by projecting it onto the lower dimensional manifold.

Neural topic models, such as Top2Vec, use Uniform Manifold Approximation and Projection for Dimension Reduction (UMAP)~\cite{mcinnes2018umap} to reduce the number of embedding vector features. UMAP has proven to be very fast and very effective at grouping documents according to their relative proximity. Because the documents are divided into different time frames, we cannot run UMAP separately on different windows because they are likely to be projected into different spaces. This means that the vectors in one window may have different relationships and patterns compared to the vectors in another window, resulting in different projections in the reduced dimensional space. We also cannot reduce the dimensionality of vectors all at once and then separate them into different time frames, because documents in different time frames may have different underlying structures and semantic patterns that can be affected by UMAP across documents.
To address these issues, we can perform the UMAP on the first data frame and then construct an initial point for the second data frame based on the locations of the shared document vectors in the first data frame, and then go from there. This method is implemented as a separate method class named AlignedUMAP~\cite{islam2022manifold}. AlignedUMAP allows sequences of different UMAP embeddings to be aligned according to the relationships between the data frames. In the case of AlignedUMAP, the Procrustes Transformation (PT)~\cite{borg2005modern} is applied to a set of documents shared between successive time frames. The PT aligns two sets of time frames in such a way that the difference between them is minimized.

\paragraph{Local Clustering}
Now we have access to a set of document vectors in each time frame, whose dimension is locally reduced according to their semantic patterns, but they are still in the same vector space. In the second step, a density-based clustering algorithm is performed on each data frame to group them separately into a set of clusters. These clusters describe a concept within a time frame. An effective approach is HBDSCAN~\cite{campello2013density}, which extends DBSCAN~\cite{ester1996density} by converting the model into a hierarchical clustering algorithm and then using a technique to extract a flat clustering based on the stability of the clusters. HDBSCAN uses a density-based distance metric to define the similarity between document vectors in each time frame. The algorithm first transforms the space according to the density/sparsity, and then builds the minimum spanning tree of the distance-weighted graph.
HDBSCAN then constructs a cluster hierarchy of connected components, abstracts the cluster hierarchy based on the minimum cluster size, and extracts the stable clusters from the condensed tree.  This algorithm grows a cluster a dense region of points by adding all points within a certain distance (epsilon) of the cluster. This process continues until no more points can be added to the cluster. Therefore, the algorithm does not require the initial number of clusters as input.

\paragraph{Cluster Alignment}

\begin{figure*}[htbp]
  \centering
  \begin{subfigure}[b]{0.22\textwidth}
    \includegraphics[width=0.9\textwidth]{time_frame_d2v_0.png}
    \caption{2006-2008}
  \end{subfigure}
  \hfill
  \begin{subfigure}[b]{0.22\textwidth}
    \includegraphics[width=0.9\textwidth]{time_frame_d2v_1.png}
    \caption{2008-2010}
  \end{subfigure}
  \hfill
  \begin{subfigure}[b]{0.22\textwidth}
    \includegraphics[width=0.9\textwidth]{time_frame_d2v_2.png}
    \caption{2010-2012}
  \end{subfigure}
  \hfill
  \begin{subfigure}[b]{0.22\textwidth}
    \includegraphics[width=0.9\textwidth]{time_frame_d2v_3.png}
    \caption{2012-2014}
  \end{subfigure}
  \hfill
  \begin{subfigure}[b]{0.22\textwidth}
    \includegraphics[width=0.9\textwidth]{time_frame_d2v_4.png}
    \caption{2014-2016}
  \end{subfigure}
  \hfill
  \begin{subfigure}[b]{0.22\textwidth}
    \includegraphics[width=0.9\textwidth]{time_frame_d2v_5.png}
    \caption{2016 to 2018}
  \end{subfigure}
  \hfill
  \begin{subfigure}[b]{0.22\textwidth}
    \includegraphics[width=0.9\textwidth]{time_frame_d2v_6.png}
    \caption{2018 to 2020}
  \end{subfigure}
  \hfill
  \begin{subfigure}[b]{0.22\textwidth}
    \includegraphics[width=0.9\textwidth]{time_frame_d2v_7.png}
    \caption{2020 to 2022}
  \end{subfigure}
  \caption{Dynamic Document Embeddings and Clusters in each Time Frame
using Data2Vec on arXiv dataset}
    \label{fig:clusterD2V}
\end{figure*}
Examples of outputs for this clustering step are shown in  \Cref{fig:clusterD2V,fig:bertDBLP}. The next step consists in aligning the clusters of the different periods Considering that the document embeddings of different data frames are in the same vector space, one solution is to apply a  cluster  linkage method (single, average, complete, centroid linkage) for aligning document clusters from different time periods. In  ANTM, we use  centroid linkage which consists in comparing clusters by the centroids of their  document embedding vectors. Observe that this step also might align topic clusters within the same time period.

\begin{figure*}[htbp]
  \centering
  \begin{subfigure}[b]{0.17\textwidth}
    \includegraphics[width=\textwidth]{dblp1.png}
    \caption{2000-2002}
  \end{subfigure}
  \hfill
  \begin{subfigure}[b]{0.17\textwidth}
    \includegraphics[width=\textwidth]{dblp2.png}
    \caption{2002-2004}
  \end{subfigure}
  \hfill
  \begin{subfigure}[b]{0.17\textwidth}
    \includegraphics[width=\textwidth]{dblp3.png}
    \caption{2004-2006}
  \end{subfigure}
  \hfill
  \begin{subfigure}[b]{0.17\textwidth}
    \includegraphics[width=\textwidth]{dblp4.png}
    \caption{2006-2008}
  \end{subfigure}
  \hfill
  \begin{subfigure}[b]{0.17\textwidth}
    \includegraphics[width=\textwidth]{dblp5.png}
    \caption{2008-2010}
  \end{subfigure}
  \hfill
  \begin{subfigure}[b]{0.17\textwidth}
    \includegraphics[width=\textwidth]{dblp6.png}
    \caption{2010-2012}
  \end{subfigure}
  \hfill
  \begin{subfigure}[b]{0.17\textwidth}
    \includegraphics[width=\textwidth]{dblp7.png}
    \caption{2012-2014}
  \end{subfigure}
  \hfill
  \begin{subfigure}[b]{0.17\textwidth}
    \includegraphics[width=\textwidth]{dblp8.png}
    \caption{2014-2016}
  \end{subfigure}
  \hfill
  \begin{subfigure}[b]{0.17\textwidth}
    \includegraphics[width=\textwidth]{dblp9.png}
    \caption{2016-2018}
  \end{subfigure}
  \hfill
  \begin{subfigure}[b]{0.17\textwidth}
    \includegraphics[width=\textwidth]{dblp10.png}
    \caption{2018-2020}
  \end{subfigure}
  \caption{Dynamic Document Embeddings and Clusters in each Time Frame
using BERT on DBLP dataset}
    \label{fig:bertDBLP}
\end{figure*}

\subsection{Representation Layer}
The representation layer is responsible for providing word representations for each cluster in each time frame within documents. The goal of this layer is to identify the most relevant terms or phrases that characterize a particular concept within a collection of documents. These terms or phrases can be used to summarize the main ideas or themes of the document clusters.
{\definition The topic Representation Layer (\textbf{RL}) computes for each document cluster $D_i^t$ a list of $m$ terms that describe the semantic contents of the documents in $D_i^t$:
\begin{align}
    \textbf{RL}: D^j_i \mapsto TR_i^j = \{ t_{ij}^r \}
\end{align}
for $r=1$ to $m$.
}

There exist several ways to represent a set of documents within a set of extracted words. One way is to use joint word and document embedding over a document cluster and find the $m$ closest words from the centroid of the document embeddings. Another way is to use a method called class-based Term Frequency-Inverse Document Frequency (c-TF-IDF)~\cite{grootendorst2022bertopic}, which is a variation of TF-IDF that takes into account the class labels of documents. c-TF-IDF weights the terms in a document not only by their frequency, but also by their relevance to a particular class. c-TF-IDF ensures that each word of a topic representation occurs in at least one document of that topic.

\section{Experiment}
\label{sec:experiment}
The goal of our experiments is to compare the performance of ANTM with other existing dynamic topic models.
The code and the results of these experiments are available on github (\url{https://github.com/hamedR96/ANTM}).

\subsection{Datasets}
We used several datasets for our experiments. The first dataset is the DBLP~\cite{ley2002dblp} archive of 168K scientific articles (title and abstract) published between 2000 and 2020. DBLP is an open bibliographic database, search engine, and knowledge graph that archives computer science publications. DBLP as a large dataset is used to compare the proposed method with DTM~\cite{blei2006dynamic} and BERTopic~\cite{grootendorst2022bertopic}. The second dataset is a smaller collection of 16K documents (title and abstract) extracted from arXiv~\cite{clement2019use} and published between 2006 and 2022. This dataset is used to compare the proposed method with two different embedding models (Data2Vec and BERT) against DETM~\cite{dieng2019dynamic} and BERTopic.
The third dataset is a collection of 19K tweets~\cite{shahzad2022price} published by Elon Musk between 2017 and the end of 2022. Each tweet includes a text message, the date and time it was posted, and other metadata such as the number of likes and retweets. The final dataset is a collection of 210K articles~\cite{pinter2020nytwit} from the New York Times published between 2012 and 2022. This dataset includes historical and current articles on a variety of topics and includes text, images, and metadata such as the article's headline, author, and publication date. Elon Musk's tweets and the New York Times dataset are used for exploratory evolutionary analysis of evolving topics based on the proposed method. The statistics of the datasets are summarized in \Cref{tb1}.

\subsection{Competitor Dynamic Topic Models}
The performance of ANTM is compared to three main dynamic topic models that are widely used for exploration analysis. The first topic model is the Dynamic Topic Model (DTM), which is a Bayesian non-parametric method for modeling and analyzing the evolution of a set of topics over time. We consider DTM because it is the most common topic model, widely used for diverse applications such as tracking changes in public opinion, monitoring the spread of information in social media, and analyzing the evolution of scientific research. The second model is an extension of DTM, called Dynamic Embedded Topic Model (DETM)~\cite{dieng2019dynamic,dieng2020topic}, which combines word embeddings and the dynamic probabilistic architecture of DTM to improve model performance by providing a more informative representation of words. Finally, the third dynamic topic model is BERTopic, which has a similar document embedding process as ANTM with LLMs. BERTopic uses a static process for clustering document embeddings, while ANTM follows a temporal process for aligned clustering of documents. We consider BERTopic because it represents a neural dynamic topic model that can be closely compared with ANTM to investigate the aligned clustering procedure of documents.

\subsection{Experiment Settings}

\subsubsection{ANTM Model Configuration}
We used two configurations for ANTM by using Data2Vec (ANTM-Data2Vec) and BERT (ANTM-BERT) as the contextual embedding layers (\textbf{CEL}).
For evaluating  ANTM-Data2Vec,  we ran Data2Vec on the document abstracts of the arXiv dataset. We used the first version of Data2Vec available in Hugging Face models~\cite{huggingface} for sentence embedding. For evaluating ANTM-BERT, we performed BERT on the titles and abstracts of DBLP, the textual content of Elon Musk's tweets, and the descriptions of NYT articles to obtain document embedding regarding the semantic context of the datasets.
In both configurations, we used the \textit{all-mpnet-base-v2} and \textit{facebook/data2vec-text-base} sentence transformers models~\cite{reimers2019sentence}, respectively, which map sentences and paragraphs to a 768-dimensional dense vector space and can be used for tasks such as clustering or semantic search. We then split the embedded vectors into a set of time frames
to explore the dynamic change within the content of documents. The setting information of the segmentation step is summarized in  \Cref{tb2}. These segmentation values, as suggested in \cite{anderson2012towards}, provide a comprehensive view of the data and ensure that changes over time are captured in the analysis.

\begin{table}[b]
\caption{The statistics of the datasets used in the experiment}
\label{tb1}
\centering
 \begin{tabular}{||c|| c| c| c| c||}
 \hline
 Dataset & Documents & Tokens & Vocabulary & Date Range\\
 \hline\hline
 DBLP & 168K & 18.5M & 144K & 2000-2020 \\
\hline
arXiv & 16K & 1.17M & 12.6K & 2006-2022  \\
 \hline
EM's Tweets & 19K & 91.5K & 4K & 2017-2022 \\
 \hline
NYT News & 210K & 5M & 61K & 2012-2022  \\
 \hline
 \end{tabular}
\end{table}
\begin{figure*}[htbp]
  \centering
  \begin{subfigure}[b]{0.33\textwidth}
    \includegraphics[width=\textwidth]{tc.png}
    \caption{Topic Coherence Metric on DBLP dataset}
    \label{fig:pwq:a}
  \end{subfigure}
  \hfill
    \begin{subfigure}[b]{0.33\textwidth}
    \includegraphics[width=\textwidth]{td.png}
    \caption{Topic Diversity Metric on DBLP dataset}
    \label{fig:pwq:b}
  \end{subfigure}
  \hfill
  \begin{subfigure}[b]{0.33\textwidth}
    \includegraphics[width=\textwidth]{tq.png}
    \caption{Topic Quality Metric on DBLP dataset}
    \label{fig:pwq:c}
  \end{subfigure}

  \hfill
\begin{subfigure}[b]{0.33\textwidth}
    \includegraphics[width=\textwidth]{pw_c2.png}
    \caption{Topic Coherencc Metric on arXiv dataset}
    \label{fig:pwq:d}
  \end{subfigure}
  \hfill
    \begin{subfigure}[b]{0.33\textwidth}
    \includegraphics[width=\textwidth]{pw_d2.png}
    \caption{Topic Diversity Metric on arXiv dataset}
    \label{fig:pwq:e}
  \end{subfigure}
  \hfill
  \begin{subfigure}[b]{0.33\textwidth}
    \includegraphics[width=\textwidth]{pw_q2.png}
    \caption{Topic Quality Metric on arXiv dataset}
    \label{fig:pwq:f}
  \end{subfigure}

\caption{Period-wise comparison between Dynamic Topic Models}
\label{fig:pwq}
\end{figure*}

After that, we applied AlignedUMAP on the time frames of each data set in order to reduce the dimension of the document embeddings. The cosine similarity metric with 5-dimensional output was chosen for the dimension reduction setting. By using AlignedUMAP, the dimensions of documents in each time frame were reduced based on the local semantic structure of each time period and by preserving the embedding space of the previous time frame. We then performed HDBSCAN on each time frame with Euclidean distance and a minimum size of $10$ documents per cluster to create a set of semantically similar document clusters for each period (\Cref{fig:bertDBLP,fig:clusterD2V}). Then, since all document embeddings are in the same vector space, we could align the generated clusters by again using HDBSCAN on the centroid of all document clusters with Euclidean distance and a minimum number of $2$ documents to obtain evolving clusters for each dataset (\Cref{fig:3d}).  Finally, we represented the documents of each cluster with a set of $m=10$ words using the c-TF-IDF (\Cref{fig:musk,fig:wordsdblp,fig:wordsnyt}).

\begin{table}[b]
\caption{The summary of segmentation setting}
\label{tb2}
\centering
 \begin{tabular}{||c|| c| c| c||}
 \hline
 Dataset \& Time Frame Sizes & Length & Overlap & \#Frames \\
 \hline\hline
 DBLP Documents & 3 Years & 1 Year & 10  \\
 \hline
 arXiv Documents& 3 Years & 1 Year & 8  \\
 \hline
New York Times Articles  & 3 Years & 1 Year & 10   \\
\hline
Elon Musk's Tweets & 6 Months & 2 Months & 16  \\
\hline
\end{tabular}
\end{table}
\subsubsection{Competitor Models Configurations}

 DTM is configured with 50 and 100 topics on the titles and abstracts of the DBLP dataset. In addition, DETM is configured with 100 topics and 50 epochs on the abstract of the arXiv dataset. The number of topics for DTM and DETM is determined based on the number of topics generated by ANTM in an unsupervised manner. Furthermore, we ran BERTopic with the same sentence transformer model used for the proposed model on both DBLP and arXiv datasets. For the rest of the hyperparameters of BERTopic, we used the default configurations. Finally, the number of word representations for each topic model is considered equal to 20 for arXiv documents and 10 for the rest of the datasets.

\section{Results}
\label{sec:comparison}

The proposed model based on BERT document embedding generated 100 evolving topics on the DBLP dataset. For the arXiv dataset, the proposed model based on Data2Vec and BERT generated 130 and 110 evolving topics, respectively. On the other hand, BERTopic generated 500 global topics on the DBLP dataset and 120 global topics on the arXiv dataset. Since the number of topics generated by a dynamic topic model is not a sufficient indicator of quality or superiority over other models, we evaluate the performance of the topic models based on Topic Coherence (TC)~\cite{roder2015exploring} and Topic Diversity (TD)~\cite{dieng2019dynamic}, which are widely accepted by the community.

\paragraph{Topic Coherence (TC)}
 This metric indicates the interpretability of a topic containing $m$ words and by estimating that a topic is highly coherent if it is represented by words that tend to occur in the same documents. TC, as defined in \cite{roder2015exploring}, is the average, over $N$ topics, of the normalized point-wise mutual information (NPMI)

 between each pair of words ($t_i^r$ and $t_i^s$) in topic $i$.

\begin{equation}
TC = \frac{1}{N}\sum_{i=1}^{N}\frac{2}{m\cdot  (m-1)} \sum_{r=2}^{m}\sum_{s=1}^{r-1}\text{NPMI}(t_{i}^r, t_{i}^{s})
\end{equation}
 where
 \begin{equation}
\text{NPMI}(t, t') = \frac{\log \frac{P(t, t')}{P(t).P(t')}}{- \log P(t, t')}
\end{equation}
$P(t, t')$ is approximated by the number of documents containing $t$ and  $t'$ divided by the total number of documents in topic $i$.  Also, $P(t)$ is similarly approximated based on documents count.

\paragraph{Topic Diversity (TD)}
\hub{TODO for the extended version of the work: replace PUW which by a diversity measure that takes into account the document embeddings as well.
$$
TC = \frac{\sum_{w,w' \in V_{unique}} cosineDistance(w,w')} {\sum_{w,w' \in V} cosineDistance(w,w') }
$$
where $V$ is the bag of words in the set of topic representations, and $V_{unique} $ is the set of unique words in the set of topic representations.
}
Topic diversity estimates the diversity of the topic representations within a given set of topics.
For this metric, we use the Proportion of Unique Words (PUW) method as suggested by~\cite{dieng2020topic} that
is the vocabulary size divided by the total number of words within a set of topics.
Topics within a low-diversity topic set share many words, whereas a high-diversity topic set contains topics that have few words in common.

\begin{figure*}[htbp]
  \centering
  \begin{subfigure}[b]{0.22\textwidth}
    \includegraphics[width=\textwidth]{tq_antm.png}
    \caption{ANTM-BERT on DBLP}
    \label{fig:twq:a}
  \end{subfigure}
  \hfill
    \begin{subfigure}[b]{0.22\textwidth}
    \includegraphics[width=\textwidth]{tq_bertopic.png}
    \caption{BERTopic on DBLP}
    \label{fig:twq:b}
  \end{subfigure}
  \hfill
      \begin{subfigure}[b]{0.22\textwidth}
    \includegraphics[width=\textwidth]{tq_lda50.png}
    \caption{DTM with 50T on DBLP}
    \label{fig:twq:c}
  \end{subfigure}
  \hfill
  \begin{subfigure}[b]{0.22\textwidth}
    \includegraphics[width=\textwidth]{tq_lda100.png}
    \caption{DTM with 100T on DBLP}
    \label{fig:twq:d}
  \end{subfigure}
  \hfill
    \begin{subfigure}[b]{0.22\textwidth}
    \includegraphics[width=\textwidth]{twq_d2v.png}
    \caption{ANTM-Data2Vec on arXiv}
    \label{fig:twq:e}
  \end{subfigure}
  \hfill
    \begin{subfigure}[b]{0.22\textwidth}
    \includegraphics[width=\textwidth]{twq_bert.png}
    \caption{ANTM-BERT on arXiv}
    \label{fig:twq:f}
  \end{subfigure}
  \hfill
      \begin{subfigure}[b]{0.22\textwidth}
    \includegraphics[width=\textwidth]{twq_bertopic.png}
    \caption{BERTopic on arXiv}
    \label{fig:twq:g}
  \end{subfigure}
  \hfill
  \begin{subfigure}[b]{0.22\textwidth}
    \includegraphics[width=\textwidth]{twq_detm-100.png}
    \caption{DETM with 100T on arXiv}
    \label{fig:twq:h}
  \end{subfigure}

  \caption{Topic-wise distribution of quality metrics over Dynamic Topic Models }
   \label{fig:twq}
\end{figure*}

Using these metrics, we can analyze the results obtained by the proposed model and evaluate the models from two perspectives. First, we can look periodically at each time frame and observe the temporal quality of all topics in terms of coherence and diversity of word representations. This analysis compares the temporal ability of dynamic topic models to describe temporal topics in a diverse and coherent manner. The second perspective is to analyze the quality of the word representations presented over each topic. This analysis compares the ability of dynamic topic models to explore the evolution of each dynamic topic over time.

\subsection{Period-Wise Quality Analysis}
The goal of the period-wise analysis is to determine whether aligned document clustering, applied separately for each time frame, results in better topic quality than the competing BERTopic, which uses a global document clustering procedure. As shown in \Cref{fig:pwq:a,fig:pwq:b,fig:pwq:c}, the first period-wise evaluation is based on the DBLP dataset, which has the largest number of documents and vocabulary size among our datasets.

\Cref{fig:pwq:a,fig:pwq:b,fig:pwq:c,fig:pwq:d} compare ANTM-Bert with the other models applied to DBLP.
On  \Cref{fig:pwq:a}, ANTM-BERT has a higher topic coherence ranging from 66\% to 70\% compared to DTM and BERTopic which remain below 63\%. On the other hand, on  \Cref{fig:pwq:b}, the topic diversity of ANTM-BERT is less than 5 percent lower than BERTopic and about 10 to 15 percent higher than DTM. Overall, the quality of ANTM-BERT, which is the product of TC and TD, is still higher than DTM and BERTopic based on the DBLP dataset. This evaluation can illustrate the capability of two NDTMs over DTM, which is a non-embedding based probabilistic dynamic topic model. Furthermore, it validates our argument that local clustering using the Aligned Clustering method, preserves more information and is more descriptive in explaining temporal topics.

\Cref{fig:pwq:d,fig:pwq:e,fig:pwq:f} compare ANTM-Data2Vec and ANTM-Bert to the other models applied on the arXiv dataset.
We focus on the embedding-based competitors (BERTopic and DETM-100), so we discard the DTM competitor. We report the achieved quality of the competitors as well as of ANTM-BERT and ANTM-Data2Vec. On \Cref{fig:pwq:e}, both configurations of ANTM produce topics that are 25\% more diverse than those produced by DETM and about 10\% less diverse than those obtained by BERTopic. On the other hand, on \Cref{fig:pwq:d}, the topics of ANTM-Data2Vec are about 20\% more coherent than the topics of ANTM-BERT and BERTopic and about 15\% less coherent than the topics produced by DETM. Since the quality of topic models is defined based on an aggregation of topic coherence and topic diversity, i.e., diversity is valuable when there is coherence and vice versa, ANTM-Data2Vec has the highest quality score compared to BERTopic and DETM (\Cref{fig:pwq:f}). This evaluation also shows that ANTM-Data2Vec is more effective than ANTM-BERT on arXiv. Moreover, it reinforces the argument that the Aligned Clustering method preserves more information and is more descriptive in explaining temporal topics than NDTMs with global clustering.

\subsection{Topic-Wise Quality Analysis}

 The objective of topic-wise analysis is to assess whether an evolving topic, which contains topics that are semantically close to each other, can nevertheless be diversified. In each model, topic coherence and topic diversity are calculated separately for each evolving topic. As shown in \Cref{fig:twq}, we have sorted these values and plotted their distributions across each model on the DBLP and arXiv datasets. These figures contain multiple observations.

First, as explained in \Cref{sec:related}, the use of document and word embeddings increases the diversity of evolving topics (\Cref{fig:twq:a,fig:twq:b,fig:twq:c,fig:twq:e,fig:twq:f,fig:twq:g}). The high diversity of embedding based models can be explained by the fact that evolving topics may regroup  documents with semantically similar but syntactically different words (e.g., bus, car, bike). Since topic diversity is based on the syntactic (c-TF-IDF) topic representations, this leads to more heterogeneous local topic representations within evolving topics and might explain why the evolving topics obtained by BERT and Data2Vec have a higher diversity than those obtained by DTM (\Cref{fig:twq:c,fig:twq:d}) and DETM (\Cref{fig:twq:h}).

On the other hand, the average topic coherence of topics in evolving topics is more difficult to explain. First, we can observe that DETM (\Cref{fig:twq:h}) achieves a very high and almost constant topic coherence compared to the other methods. This seems to be the result of the use of word embeddings during the Bayesian model generation process.  Among the approaches based on document embeddings, ANTM-Data2Vec achieves better average topic coherence (and topic diversity) (\Cref{fig:twq:e}) than the other approaches.

\section{Conclusion}
\label{sec:conclusion}

In this paper, we study the use of dynamic topic models for exploring topic evolution.
Existing dynamic topic models ignore certain temporal variations of dynamic topics by configuring a global structure for dynamic topics, such as the same number of document clusters in each period. Moreover, some of these models are not scalable to large archives with large dictionaries and are not able to handle short texts for applications such as social network analysis. These limitations directly affect the observation of topic evolution and reduce the coherence and diversity of topic word representations that are sequentially represented for each evolving topic. In this paper, we propose a novel dynamic neural topic model called Aligned Neural Topic Models (ANTM), which benefits from an aligned clustering procedure to preserve the semantic structure behind topic evolution. In addition, ANTM takes advantage of a self-supervised learning model called Data2Vec for document embedding, which improves the performance of the procedure. ANTM was implemented on four datasets and extracted interesting information about the evolution of topics. The topic quality produced by the model on DBLP and arXiv documents showed a significant improvement compared to existing NDTMs and PDTMs.
For future work, we plan to extend our model to more closely examine the relationship between evolving topics and citation networks by combing topic modeling and graph analysis techniques, such as dynamic graph embeddings.

\begin{acks}
We thank Lucie Chen, who did a student project on topic evolution and gave us the motivation to further investigate the problem of dynamic topic modeling for topic evolution. This work is partially funded by a PhD scholarship from SCAI, Sorbonne Université.
To brave women of Iran.
\end{acks}

\clearpage
\bibliographystyle{ieeetr}