\section{Introduction} Let $\mathfrak{D}$ denote the class of Dirichlet series
\[f(s) = \sum_{n=1}^\infty a_n n^{-s}\]
that converge in at least one point $s=\sigma+it$ in the complex plane. Associated to each Dirichlet series $f$ in $\mathfrak{D}$ is a number $\sigmac(f)$, called the \emph{abscissa of convergence}, with the property that $f$ converges if $\sigma>\sigmac(f)$ and $f$ does not converge if $\sigma<\sigmac(f)$. This note concerns an extension of Bohr's classical theorem on uniform convergence of Dirichlet series~\cite{Bohr1913}. We therefore define the \emph{abscissa of uniform convergence} $\sigmau(f)$ as the infimum of the real numbers $\theta$ such that $f$ converges uniformly in the half-plane $\mathbb{C}_\theta$. Here and in what follows, we set
\[\mathbb{C}_\theta = \{s = \sigma+it\,:\, \sigma>\theta\}.\]

Our starting point reads as follows.
\begin{bohr}
	Let $f$ be in $\mathfrak{D}$. If there is a real number $\theta$ and a bounded set $\Omega$ such that $f$ has an analytic continuation to $\mathbb{C}_\theta$ that maps $\mathbb{C}_\theta$ to $\Omega$, then $\sigmau(f) \leq \theta$.
\end{bohr}

Queff\'{e}lec and Seip~\cite{QS2015} (see also~\cite[Theorem~8.4.1]{QQ2020}) showed that the assumption that $\Omega$ is a bounded set may be replaced with the weaker assumption that $\Omega$ is a half-plane. This extension of Bohr's theorem was applied obtain the canonical formulation of the Gordon--Hedenmalm characterization of composition operators~\cite{GH1999}, which has proven to be essential for further developments (see e.g.~\cite[Section~6]{BP2021}).

Replacing $f$ by $a+bf$ for constants $a$ and $b\neq0$, we may assume without loss of generality that $\Omega = \mathbb{C}_0$ in their extension. In addition to Bohr's theorem, their proof is based on Harnack's inequality for positive harmonic functions. Consequently, their argument uses the assumption that $\Omega$ is a half-plane in a rather strong sense.

We have the following result.
\begin{theorem}\label{thm:bohrext}
	Let $f$ be in $\mathfrak{D}$. If there is a real number $\theta$ and a simply connected domain $\Omega \neq \mathbb{C}$ such that $f$ has an analytic continuation to $\mathbb{C}_\theta$ that maps $\mathbb{C}_\theta$ to $\Omega$, then $\sigmau(f)\leq \theta$.
\end{theorem}

Our proof of Theorem~\ref{thm:bohrext} is arguably simpler than the Queff\'{e}lec--Seip proof in the special case that $\Omega$ is a half-pane. In addition to Bohr's theorem and the Riemann mapping theorem, we will only use some elementary properties of Dirichlet series and the Hadamard three-lines theorem.

\subsection*{Acknowledgements.} The author thanks Herv\'{e} Queff\'{e}lec for providing helpful comments.

\section{Proof of Theorem~\ref{thm:bohrext}} In addition to the abscissa of convergence and the abscissa of uniform convergence discussed above, every Dirichlet series $f$ in $\mathfrak{D}$ has an \emph{abscissa of absolute convergence} $\sigma_{\mathrm{a}}(f)$ with the property that $f$ converges absolutely for $\sigma>\sigma_{\mathrm{a}}(f)$ and $f$ does not converge absolutely for $\sigma<\sigma_{\mathrm{a}}(f)$. It is well-known that $\sigma_{\mathrm{a}}(f) \leq \sigmac(f)+1$, so every Dirichlet series in $\mathfrak{D}$ is absolutely convergent in some possibly remote half-plane.

To facilitate certain computations below, we introduce the Wiener norm
\[\|f\|_{W_\sigma} = \sum_{n=1}^\infty |a_n| n^{-\sigma},\]
where $f$ is the Dirichlet series with coefficient sequence $(a_n)_{n\geq1}$. Note that if $\|f\|_{W_\sigma} < \infty$, then $\sigma\geq\sigma_{\mathrm{a}}(f)$. In addition to the usual norm properties, we will have use of the submultiplicative property
\begin{equation}\label{eq:wienerprod}
	\|f g\|_{W_\sigma} \leq \|f\|_{W_\sigma} \|g\|_{W_\sigma}.
\end{equation}
We also require a lemma for the proof of Theorem~\ref{thm:bohrext}. It is probably well-known to experts, but we have not found the formulation we need explicitly stated in the literature. Since the proof is quite short, we include the details. To set the stage, suppose that $f$ is a function defined in some half-plane $\mathbb{C}_\theta$ and consider
\[M(f,\sigma) = \sup_{t \in \mathbb{R}} |f(\sigma+it)|\]
for $\sigma>\theta$. Note that if $f$ is a M\"obius transformation from the half-plane $\mathbb{C}_\theta$ to the unit disc $\mathbb{D}$, then $M(f,\sigma) = 1$ for all $\theta<\sigma<\infty$. The following result illustrates how uniformly convergent Dirichlet series cannot exhibit the same behavior.
\begin{lemma}\label{lem:3lines}
	Suppose that $f$ is in $\mathfrak{D}$ and that $f$ is not identically equal to a constant. The function
	\[\sigma \mapsto M(f,\sigma)\]
	is strictly decreasing on the interval $(\sigmau(f),\infty)$.
\end{lemma}
\begin{proof}
	The Hadamard three-lines theorem states that
	\begin{equation}\label{eq:h3l}
		M(f,\sigma) \leq \big(M(f,\sigma_1)\big)^{\alpha} \big(M(f,\sigma_2)\big)^{1-\alpha},
	\end{equation}
	for $\sigmau(f)<\sigma_1 < \sigma_2$ and $\sigma = \alpha \sigma_1 + (1-\alpha)\sigma_2$ for some $0\leq \alpha \leq 1$. We think of $\sigma_1$ and $\sigma$ as fixed, and aim to demonstrate that $M(f,\sigma)<M(f,\sigma_1)$ by choosing a sufficiently large $\sigma_2$. The fact that $f$ converges uniformly on the line $s=\sigma+it$ allows us to move the limit and the integral inside the Dirichlet series to infer that
	\[a_1 = \lim_{T\to\infty} \frac{1}{2T} \int_{-T}^T f(\sigma+it) \,dt.\]
	It follows from this formula, the triangle inequality, and the assumption that $f$ is not identically equal to a constant that $|a_1|<M(f,\sigma)$. Since $f$ is in $\mathfrak{D}$, it is absolutely convergent for sufficiently large $\sigma$ and, consequently,
	\[|a_1| = \lim_{\sigma \to \infty} M(f,\sigma).\]
	This shows that if $\sigma_2$ is sufficiently large, then $M(f,\sigma_2)<M(f,\sigma)$. Inserting this estimate into \eqref{eq:h3l} yields the stated result.
\end{proof}
\begin{proof}
	[Final part in the proof of Theorem~\ref{thm:bohrext}] If $f$ is identically equal to a constant, then there is nothing to do. If $f$ is not identically equal to a constant, then there is a smallest integer $n>1$ such that $a_n \neq 0$. By absolute convergence, we have
	\[f(s) = a_1 + a_n n^{-s} + O\big((n+1)^{-\sigma}\big)\]
	as $\sigma \to \infty$. By the assumption that $f$ maps $\mathbb{C}_\theta$ to $\Omega$, it follows that $\Omega$ contains a small punctured disc centered at $a_1$. Since $\Omega$ is simply connected, this means that $a_1$ lies in the interior of $\Omega$.

	Let $\Theta$ be a Riemann map from $\mathbb{D}$ to $\Omega$ with $\Theta(0)=a_1$. The function $g = \Theta^{-1} \circ f$ is analytic and bounded by $1$ in $\mathbb{C}_\theta$. We may formally expand $g$ as a Dirichlet series, but to show that $g$ is in $\mathfrak{D}$ we need to prove that the series converges somewhere. Since $a_1$ lies in the interior of $\Omega$, there is some $\varrho>0$ such that the Taylor series
	\[\Theta^{-1}(w) = \sum_{k=0}^\infty \frac{(\Theta^{-1})^{(k)}(a_1)}{k!} (w-a_1)^k\]
	converges absolutely when $|w-a_1| \leq \varrho$. Appealing to absolute convergence as above, we find that there is some sufficiently large $\sigma$ such that $\|f-a_1 \|_{W_\sigma} \leq \varrho$. Thus
	\[\|g\|_{W_\sigma} \leq \sum_{k=0}^\infty \frac{\left|(\Theta^{-1})^{(k)}(a_1)\right|}{k!} \big\|(f-a_1)^k\big\|_{W_\sigma} \leq \sum_{k=0}^\infty \frac{\left|(\Theta^{-1})^{(k)}(a_1)\right|}{k!} \varrho^k < \infty,\]
	where we in the second estimate used the submultiplicative property \eqref{eq:wienerprod}. This shows that $\sigma_{\mathrm{a}}(g) \leq \sigma$. Consequently, $g$ is in $\mathfrak{D}$ and, by Bohr's theorem, $\sigmau(g) \leq \theta$.

	It remains to demonstrate that the uniform convergence of $g$ implies the uniform convergence of $f$. Since $\Theta$ is an analytic function in the unit disc, the Taylor series
	\[\Theta(z) = \sum_{k=0}^\infty \frac{\Theta^{(k)}(0)}{k!} z^k\]
	is absolutely convergent when $|z| \leq r$ for any $r<1$. Since $f$ is not identically equal to a constant, it is plain that $g = \Theta^{-1} \circ f$ is also not identically equal to a constant. Using that $g$ maps $\mathbb{C}_\theta$ to $\mathbb{D}$, we infer from Lemma~\ref{lem:3lines} that there for any $\varepsilon>0$ is an $r<1$ such that $|g(s)| \leq r$ for all $s$ in $\mathbb{C}_{\theta+\varepsilon}$. For such $s$ we find that
	\[|f(s)| = |(\Theta \circ g)(s)| \leq \sum_{k=0}^\infty \frac{\left|\Theta^{(k)}(0)\right|}{k!} |g(s)|^k \leq \sum_{k=0}^\infty \frac{\left|\Theta^{(k)}(0)\right|}{k!} r^k < \infty.\]
	Since $f$ is in $\mathfrak{D}$ and $f$ is a bounded analytic function in $\mathbb{C}_{\theta+\varepsilon}$ for any $\varepsilon>0$, it follows from Bohr's theorem that $\sigmau(f)\leq \theta$.
\end{proof}

\section{Concluding remarks} Gordon and Hedenmalm~\cite{GH1999} (see also \cite[Theorem~8.4.3]{QQ2020}) proved that if $f$ satisfies the assumptions of Theorem~\ref{thm:bohrext} with $\Omega=\mathbb{C}_\vartheta$, then for any $\varepsilon>0$ there is a $\delta>0$ such that $f$ maps $\mathbb{C}_{\theta+\varepsilon}$ to $\mathbb{C}_{\vartheta+\delta}$. The proof of this result is based on an argument similar to the one we used above to conclude that $a_1$ lies in the interior of $\Omega$ and the Hadamard three-lines theorem applied to the function $\exp{f}$.

If the above-mentioned result of Gordon and Hedenmalm is combined with with the extension of Bohr's theorem due to Queff\'{e}lec and Seip, which in particular implies that $f(\mathbb{C}_{\theta+\varepsilon})$ is a bounded set for any $\varepsilon>0$, then it follows that $f(\mathbb{C}_{\theta+\varepsilon})$ is contained in a compact subset of $\mathbb{C}_{\vartheta}$ for any $\varepsilon>0$. Our proof of Theorem~\ref{thm:bohrext} also yields the extension of this assertion to simply connected domains.
\begin{theorem}
	Let $f$ satisfy the assumptions of Theorem~\ref{thm:bohrext}. Then $f(\mathbb{C}_{\theta+\varepsilon})$ is contained in a compact subset of $\Omega$ for any $\varepsilon>0$.
\end{theorem}
\begin{proof}
	Following the proof of Theorem~\ref{thm:bohrext}, we find that for any $\varepsilon>0$ there is some $r<1$ such that $f(\mathbb{C}_{\theta+\varepsilon})$ is contained in $\Theta(r\mathbb{D})$, where $r\mathbb{D} = \{z \in \mathbb{C}\,:\, |z|<r\}$. Since $\Theta$ is a Riemann map from $\mathbb{D}$ to $\Omega$, the stated result follows.
\end{proof}

A \emph{general} Dirichlet series is a series of the form
\[f(s) = \sum_{n=1}^\infty a_n e^{-\lambda_n s},\]
where the \emph{frequency} $\lambda=(\lambda_n)_{n\geq1}$ is a strictly increasing and unbounded sequence of nonnegative real numbers. In the present note we have restricted our attention to the classical case $\lambda_n = \log{n}$.

The century-old problem of determining which frequencies satisfy Bohr's theorem has recently enjoyed a resurgence of interest (see e.g. \cite{Bayart2022,Schoolmann2020}). Our proof carries over to the general case provided there is absolute convergence in some possibly remote half-plane. This is not always the case: The frequency defined by $\lambda_n = \sqrt{\log{n}}$ satisfies Bohr's theorem, but there are convergent general Dirichlet series of this frequency which do not converge absolutely anywhere. It would be interesting to know whether our extension of Bohr's theorem holds for such frequencies.

\bibliographystyle{amsplain}


\section{Introduction}
\label{sec:introduction}

Various researchers have noted that the APIs of widely-used libraries are often hard to learn and use correctly \cite{Robillard:IEEESOftware2009,Robillard:ESE2011,ZibranEtAlWCRE2011,WangGodfrey:MSR2013}.
Prevention of erroneous use is particularly crucial in Data Science (DS): Programs that crash at run-time for a reason that could have been detected at compile-time are generally a waste of time, resources, and, ultimately, money -- and downright prohibitive after running hours or days. Therefore, static safety would be a major benefit for every Data Scientist, novice or master.
Unfortunately, most DS pipelines are written in Python, and use Python DS libraries \cite{kaggle2021KaggleMachine}. For Python, static type checking is only available via external linters like mypy\footnote{\url{https://github.com/python/mypy}}, which are, however, inapplicable if the code is not annotated with \keyword{type hints} \cite{TypingSupportType}.
Unfortunately, many libraries do not provide type hints, and no Python linter we know of is able to handle advanced type concepts
like schema types for tabular datasets,
which would help a lot in typical DS applications.

A radically different approach is taken by systems that hide the DS APIs behind a visual pipeline specification language (e.g. RapidMiner Studio \cite{mierswaYALERapidPrototyping2006}), a wizard-like interface (e.g. RapidMiner Go \cite{rapidminerGo}), or a configuration language (e.g. Ludwig \cite{ludwig}).
However, wizards and configuration languages can only provide guidance \cite{reimannAchievingGuidanceApplied2020a} and safety at the expense of limiting what can be expressed. Visual programming can offer full expressiveness, but when a pipeline gets complex, a graph representation quickly gets cluttered, and textual languages excel. Depending on the skills of the user and the use case, different representations are optimal.
However, no known system provides
visual \emph{and} textual ways to specify
DS pipelines.

\label{sec:research-questions}

In this context, we explore the following research questions:
\begin{itemize}
     \item RQ 1:
        How can we make the most widely used libraries for DS usable  even by programming novices?

    \item RQ 2:
        Is it possible to design a language that is expressive enough to implement any DS pipeline but simple enough to be learned and used correctly, even by novices?

    \item RQ 3:
        Which classes of errors are not caught by conventional type systems (especially those of Python linters) but occur frequently in DS development?
    \item RQ 4:
        Which
        type concepts could help and should be part of a dedicated DS language?
    \item RQ 5:
        How can a DS language ensure statically safe use of DS libraries written in dynamic languages
    \item RQ 6:
        How can we make
        integration of existing Python DS libraries easy, even for third parties?

    \item RQ 7:
        How can we leverage the built-in error prevention of graphical languages without sacrificing the flexibility of textual code?
        %

\end{itemize}

The solution suggested in this paper is \safeds{}, a statically typed domain specific language (DSL) \cite{enwiki:dsl} for specifying DS pipelines. It provides:
\begin{enumerate}
    \item
        A simple mechanism to integrate existing Python DS libraries in a type-safe way (Sec. \ref{sec:static-checking}).
    \item
        A well-designed standard library that includes the best available DS libraries (briefly outlined in Sec. \ref{sec:prototype}).
    \item
        Support for third-party developers to extend the standard library by integrating their own features (Sec. \ref{sec:checking-external-code}).

   \item
        A seamless bidirectional transition between graphical and textual representation, which are just different views of the same abstraction, the DSL (Sec. \ref{sec:different-views}).
\end{enumerate}

\section{State of the Art}
\label{sec:state-of-the-art}

According to the 2021 Kaggle DS survey \cite{kaggle2021KaggleMachine}, over 84\% of almost 26.000 respondents use Python regularly, making it the most commonly used DS programming language.
According to the survey, the most-often used libraries for DS are
Matplotlib \cite{Hunter:2007},
Seaborn \cite{Waskom2021},
and Plotly \cite{plotly}
for data visualization,
Pandas \cite{The_pandas_development_team_pandas-dev_pandas_Pandas} for data processing, and
\sklearn{} \cite{scikit-learn, sklearn_api},
TensorFlow \cite{tensorflow2015-whitepaper},
Keras \cite{chollet2015keras},
PyTorch \cite{NEURIPS2019_9015},
and XGBoost  \cite{Chen:2016:XST:2939672.2939785}
for ML\footnote{Each group is sorted by descending usage percentages\cite{kaggle2021KaggleMachine}}.
All are implemented in Python, emphasizing its role in DS development.

Nevertheless, there are noteworthy suggestions for DS pipeline development beyond Python:
Ludwig \cite{ludwig} allows users to build a DS pipeline with a YAML \emph{configuration file}. Users must specify input and output features, and can optionally alter some parts of the pipeline, such as data preprocessing.

External domain specific languages are another solution. The list of DSLs used for DS in 2016 \cite{portugalPreliminarySurveyDomainSpecific2016} includes \emph{textual DSLs} like Pig Latin \cite{olstonPigLatinNotsoforeign2008} for data processing and \emph{visual DSLs} like \cite{breukerModelDrivenEngineeringBig2014} for the representation of probabilistic models. Since then, other tools appeared, e.g. BiDaML \cite{khalajzadehBiDaMLSuiteVisual2019, khalajzadehBiDaMLPracticeCollaborative2021}, which offers five diagrammatic notations for different stages of the development of a DS pipeline. Another popular tool is RapidMiner Studio (formerly YALE) \cite{mierswaYALERapidPrototyping2006}, which offers a visual DSL to quickly, and
safely build a DS pipeline.

Finally, some tools use \emph{wizards} to guide users step by step through the development of a DS pipeline, for example RapidMiner Go \cite{rapidminerGo}.
To overcome the limited flexibility of a wizard, users can export the final result to RapidMiner Studio.

\cite{keryFutureNotebookProgramming2020} extends Jupyter Notebook \cite{kluyverJupyterNotebooksPublishing} by additional GUI elements like a spreadsheet view to inspect data. When users manipulate the data in these elements, corresponding code is added to the notebook. However, changes to the generated code can in general not be reflected in the GUI, making the transition between graphical and textual view one-way only.

These ideas can be combined with \emph{AutoML} \cite{heAutoMLSurveyStateoftheart2021} to automatically derive parts of
a DS pipeline from the initial data.

An overview of free tools for data mining can be found in \cite{jovicOverviewFreeSoftware2014} and a more thorough survey of the DS landscape in \cite{nguyenMachineLearningDeep2019}. Even taking this larger context into account, no existing tool provides a comprehensive solution to
the majority of our research questions, let alone all of them.

\section{A Language for Data Science Pipelines}
\label{sec:dsl}

A \emph{DS pipeline} is code that performs data loading, data preparation, model training, and hyperparameter optimization to produce a model that is ready for use.
It conserves the steps necessary to reproduce the same model from the same data, without the data analysis and model quality assessment steps that developers performed to decide how to build the pipeline.

The main aim of our work is to support pipeline developers of varying skill levels, including absolute programming and DS novices. For this, assembling DS pipelines should be easy to learn and use, but hard to misuse. Any solution must provide a small and consistent API (to ease learning and correct use), which must be easily extensible (to take advantage of the evolution of DS libraries). Extension must be possible and easy, also for third-party library developers.

\paragraph*{RQ 1} Given that the main DS libraries are implemented in Python, their use by programming novices for building pipelines is out of question. Even for experienced programmers, Python libraries that lack type annotations for linters are easy to misuse \cite{reimannImprovingLearnabilityMachine2022}. To make the wealth of DS libraries (Sec. \ref{sec:state-of-the-art}) usable by novices and prevent misuse by users of all skill levels we propose to reuse these libraries via Safe-DS, a DSL for DS that does the heavy lifting.

\paragraph*{RQ 2} This raises the question, which language features are necessary for unrestricted pipeline development and which must be avoided to ease learning and use by novices. Because pipeline development is not about algorithm implementation but mainly a sequence of invocations of already implemented library functions, the set of necessary language features is simple.
Fig. \ref{fig:pipeline-example} shows the start of a \safeds{} pipeline using the popular ``Titanic'' example.

\begin{figure}[ht]
    \centering
    \includegraphics[width=.48\textwidth]{img/listing_pipeline.pdf}
    \caption{The start of a \safeds{} pipeline to predict survival during the Titanic accident.}
    \label{fig:pipeline-example}
\end{figure}

The keyword \keyword{pipeline} is the entry point into a \safeds{} program.
The one in our example is called \code{predictTitanicSurvival}.
A pipeline contains a sequence of statements, usually assignments\footnote{The only other statement, besides assignments, is the expression statement, which evaluates an expression for its side effects, ignoring  results.}.
The right-hand side of each assignment contains the expression to evaluate. An expression can be a literal (\code{\str{Titanic}}), a reference to a variable (\code{titanic}), or a call. A call invokes a global function, like \code{loadDataset}, or a method, like \code{keepColumns}. We refer to methods and global functions as \keyword{processes}. Calls can contain further expressions as arguments.

Processes can take lambdas (anonymous, parameterized sequences of statements) as arguments, to configure parts of their behavior. For example, a general \code{transform} function that changes values in one column of a tabular dataset needs to know how to actually change these values. Such higher-order processes are mainly used for data processing functions and rarely needed otherwise.

Notably, in Safe-DS there are no conditionals, loops, or recursive calls. These are encapsulated in the library functions invoked in the pipeline (e.g. hyperparameter search). Decisions and iterations by the developer (reflect on the outcome of the current pipeline, modify it, and restart it) happen outside the pipeline.
The lack of loops and conditionals makes the language easy to learn.
It also eases thorough static checking (Sec. \ref{sec:static-checking}) because data flows linearly through the program.

\section{Static Checking}
\label{sec:static-checking}

Supporting safety during the development of a DS pipeline is one of our key aims. For this, detecting errors \emph{statically}, that is before programs are run, is crucial, since the execution of DS pipelines can take hours or even days on large datasets. Achieving static checking, raises the research questions RQ 3 to RQ 6.

\subsection{Type Problems}
\label{sec:type-problems}

The empirical evaluation of \cite{reimannImprovingLearnabilityMachine2022} showed that basic typing features of Python linters already catch many common errors. However, we also identified typical problems that go beyond conventional type systems and are particularly frequent or important in a DS context (RQ 3).

\paragraph*{Lack of range restrictions} We found that the documentation of a library often expresses range restriction, e.g. ratios must be a numeric value between 0 and 1.

\paragraph*{Broken dataset access} When processing tabular datasets by adding, removing or renaming columns, it is easy to forget updating some operations that still access a changed column, because column names are just strings and the type system does not know about their special semantics. Neither does it know how to infer the type of columns.

\paragraph*{Broken order} Machine learning processes must often be called in a specific order (e.g. training before prediction). This simple example could be solved at API level by distinguishing the types \code{UntrainedModel} and \code{TrainedModel}, where only the latter would have a \code{predict} method. However, this would exclude integration of existing libraries that mutate an untrained model in place (e.g. \sklearn{}), would assume redesign of the library, and would bloat the number of types.

\subsection{Type System}
\label{sec:type-system}

Having identified challenging type problems (RQ 3) we now introduce the solutions provided by Safe-DS (RQ 4).

\paragraph*{Basic type system}
To capture the basic aspects of DS APIs, the Safe-DS type system supports class types, enum types, function types (which include types for parameters and results), union types, subtyping, and genericity.

\paragraph*{Refined types} As a way to express range restrictions, Safe-DS
additionally includes refined types, which restrict the legal set of values of an existing type via some constraint. For parameters that have a refined type, only expressions that can be evaluated to a constant value by the compiler can be passed. Otherwise, the compiler would not be able to verify the constraint of the refined type.

\paragraph*{Schema inference and checking} To address the broken dataset access problem (Sec. \ref{sec:type-problems} ) \safeds{} checks correctness of schema-sensitive operations, without having to run the \emph{entire} DS pipeline up to that point:
(1) When a tabular dataset is loaded from external sources, e.g. a CSV file, the compiler infers its initial schema by analyzing the data.
(2) For any data processing function invoked in the pipeline,
    the compiler inductively infers the schema of the resulting dataset
    from the schemata of the input datasets and a description
    of the function's \keyword{schema effects}.
Our schema effect system \cite{lucassenPolymorphicEffectSystems1988a, marinoGenericTypeandeffectSystem2009a} captures any combination of column addition, removal, renaming, or type change.
In addition, we support two basic schema checks: (1) Ensuring a column with a given name exists. (2) Ensuring a column has a given type. Based on the results of schema inference, the compiler executes such checks statically, thus ensuring early feedback to the user.

\paragraph*{Order checking}

To solve order issues, we support behavior protocols \cite{plasilBehaviorProtocolsSoftware2002} in interface descriptions.
These are regular expressions that describe legal call orders, with processes instead of characters as basic tokens. For instance, we can specify that on any model, the process that trains the model must be called before the one that makes predictions. Using this information, we can ensure statically that the call order in pipelines is correct.

\subsection{Integrating External Code}
\label{sec:checking-external-code}
\label{sec:stubs}

So far, we have not explained, how the Safe-DS compiler can know at all which functions are available, and how it can statically check calls to functions implemented in a language that does not provide a static type system (RQ 5).

Hard-coding a fixed set of processes into the grammar of \safeds{} would clearly preclude extensibility. This could be avoided by analyzing the code of external libraries, collecting contained type hints or inferring the types of arguments and results while the user develops a pipeline. However, many Python DS libraries do not provide type hints and type inference is not generally possible for Python.

Therefore, we designed \safeds{} as a combination of the \emph{pipeline language} described in Sec. \ref{sec:dsl} with an additional \keyword{stub language}. Each stub is an accurate description of the fully typed interface of a particular Python function.
Fig. \ref{fig:stub-example} shows the stub for the \code{loadDataset} process. The \keyword{fun} keyword indicates that \code{loadDataset} is a function.
For each parameter (in parentheses) or result (after arrow) we list its name and type, separated by a colon.

\begin{figure}[ht]
    \centering
    \includegraphics[width=.48\textwidth]{img/listing_stub.pdf}
    \caption{The \safeds{} stub for the \code{loadDataset} function.}
    \label{fig:stub-example}
\end{figure}

Pipelines can now invoke the specified \code{loadDataset} process (Fig. \ref{fig:pipeline-example}). The \safeds{} compiler checks type-correctness of invocations, by comparing the types of arguments (inferred) with those specified in the respective stub.
Any process call for which there is no matching stub is assumed to be wrong, leading to an error message.

Stubs tell the \safeds{} compiler all it needs to know for type checking. Therefore, the stub language also includes ways to specify refined types, schema effects, schema checks, and behaviour protocols. For example, the process \code{get\_column} in Fig. \ref{fig:schema-check-example}, which returns the column of the \code{table} with the given \code{name}, specifies that a column called \code{name} must exist in \code{table}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=.48\textwidth]{img/schema_check.pdf}
    \caption{A function stub with a schema check to ensure a column called \code{name} exists in the \code{table}.}
    \label{fig:schema-check-example}
\end{figure}

\subsection{Semi-Automated Stub Generation and Implementation}
\label{sec:automated-stub-generation}

With the stub language, implementers of new machine learning models or other useful functions can integrate them into \safeds{}. However, integrating an entire library manually would be tedious and a major deterrent to the adoption of \safeds{}. Thus, we need to explain how this process can be eased (RQ 6).

Our approach adopts and evolves the API-Editor approach of \cite{reimannImprovingLearnabilityMachine2022}.
In a first step, types are inferred for Python code based on type hints in the code \cite{TypingSupportType} or, if type hints are not available, documentation.
For the cases that cannot be inferred automatically,
an easy to use GUI is provided. It also supports specification of changes to the API (e.g. removal of declarations, renaming, and creation of parameter objects). Some of these suggestions can even be inferred automatically. Once the improved and simplified API is finalized, \safeds{} stubs and their corresponding implementations as Python wrappers around the original library are created automatically.

This way, large libraries can quickly be integrated into \safeds{}, with only limited manual intervention for cases where the documentation is not accurate enough.

\section{Different Views}
\label{sec:different-views}

So far, we discussed representing \safeds{} pipelines as text (Sec. \ref{sec:dsl}). However, this is only suitable for people with some programming skills, which does not apply to everyone involved in the development of DS pipelines.
To encourage people without programming skills to try solving DS problems, and understand solutions of experts, we provide different views onto DS pipelines, tailored for specific skillsets (RQ 7).

Fig. \ref{fig:graphical_pipeline} shows a possible translation of the pipeline from Fig. \ref{fig:pipeline-example} into a visual DSL, which still offers full flexibility but improves safety by preventing syntax errors.
The surrounding box corresponds to the pipeline, with the name shown in the top right corner. The rounded rectangles inside correspond to the invoked processes, with circles for parameters (left) and results (right). This information is retrieved from the stub of the process (Sec. \ref{sec:checking-external-code}).

Arrows represent dataflow between processes,
e.g. the arrow from the \code{loadDataset} to the \code{keepColumns} process says that the result of the first process is input to the second one.
The name shown above an arrow corresponds to the name of a variable in the textual DSL.
The rightmost arrow is attached to a ``blank'' circle since this value is never used but we still want to show the name of the variable that stores it,
for easier mapping to the textual representation and as a hint to possibly unfinished pipelines.
Literal inputs (e.g. strings) are hidden in the graph representation
to avoid clutter.
They can be made visible by clicking on a process. This approach helps keep the graph small. It is also used in RapidMiner Studio \cite{mierswaYALERapidPrototyping2006}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=.43\textwidth]{img/graphical_pipeline.pdf}
    \caption{Graphical representation of the pipeline from Fig. \ref{fig:pipeline-example}. Literals are only revealed when clicking on a node, to keep the graph small.}
    \label{fig:graphical_pipeline}
\end{figure}

Existing systems (Sec. \ref{sec:state-of-the-art}) provide either a code or a graph view of a pipeline, or in case of \cite{keryFutureNotebookProgramming2020} both but only with a fully defined transition from graph to code view.
We
think that supporting both views, including seamless transitions in both direction, is crucial, to let each person involved in the development of a DS pipeline use the view best suited for them, and even switch between the views, as needed. It would, for example, facilitate collaboration between domain experts and experienced Data Scientists \cite{maoHowDataScientists2019, zhangHowDataScience2020}: A domain expert could prototype an initial pipeline using the visual DSL and then pass it to a Data Scientist, who elaborates and refines it in the textual DSL. Afterwards, the Data Scientist could explain his solution to the domain expert using again the visual DSL, where they could make further changes together.

The switch between different views is only possible if the target view is at least as expressive as the source view.
Since we want to be able to seamlessly switch between textual and graphical view back and forth, this implies that both representations must be equivalent. Our approach trivially guarantees this, otherwise hard to achieve, prerequisite. In \safeds{}, text and graph are just two different notations for the \emph{same} language.

\section{Prototype}
\label{sec:prototype}

\paragraph*{Language}
We implemented a prototype of \safeds{}\footnote{\url{https://github.com/lars-reimann/Safe-DS}} with Xtext\footnote{\url{https://www.eclipse.org/Xtext}}. At the time of writing, it contains the complete textual syntax for the pipeline and stub languages, scoping, automated formatting of \safeds{} code, generation of Python code, and a vast amount of static checks.
Static checking currently supports the basic type system without genericity and refined types. It also supports type inference in pipelines. Checking of schema effects is work in progress and behavior protocols are future work. We use Xtext to create a language server \cite{lspcontributorsOfficialPageLanguage}, which we make available for Visual Studio Code\footnote{\url{https://code.visualstudio.com}} via a small extension\footnote{\url{https://marketplace.visualstudio.com/items?itemName=larsreimann.safe-ds}}.

\paragraph*{Metamodel}
The core of the language is the \keyword{metamodel}, the abstract representation of the language elements, also known as the \keyword{abstract syntax} of the language. Individual DS pipelines are instances of this metamodel. The static checks are implemented on top of the metamodel. Each view of the DSL (Sec. \ref{sec:different-views}) defines a transition to and from the metamodel. First, this allows the static analysis to be defined independently from a concrete view, which only needs to display errors to the user in an appropriate form. Second, we avoid having to define and implement transitions between all current and future views. The metamodel is implemented with EMF\footnote{\url{https://www.eclipse.org/modeling/emf}}.

Fig. \ref{fig:different-views} shows the relation between the textual and graphical view and the metamodel. It also shows the one-way transition from the metamodel to Python code generated by the compiler.

\begin{figure}[ht]
    \centering
    \includegraphics[width=.40\textwidth]{img/different_views_simple.pdf}
    \caption{Interaction between different views (blue) with the metamodel to connect them.}
    \label{fig:different-views}
\end{figure}

\paragraph*{Standard library}
Moreover, we started work on the Safe-DS standard library. So far, it includes basic functionality to work with tabular data (using pandas \cite{The_pandas_development_team_pandas-dev_pandas_Pandas} internally), visualize this data (using seaborn \cite{Waskom2021}), and train and evaluate ML models (using scikit-learn \cite{scikit-learn, sklearn_api}). An experimental version of the corresponding Python implementation is available on PyPI\footnote{\url{https://pypi.org/project/safe-ds}}. We will discuss the standard library in more detail in a future publication.

\section{Discussion}
\label{sec:discussion}

\paragraph*{Accuracy of stubs}
The static analysis requires that stubs accurately describe the preconditions of the corresponding Python implementation. A mismatch can either cause valid DSL programs to be rejected or invalid DSL programs to be executed, leading to runtime exceptions or undefined behavior. We aim to reduce the chance of such mismatches by automatically creating stubs for Python declarations where possible (Sec. \ref{sec:automated-stub-generation}). Furthermore, users only need to work with the stub language when they want to integrate functionality that is missing from the standard library. For the standard library itself, we pay particular attention to the accuracy of the stubs.

\paragraph*{Omission of language concepts}
While the language itself contains neither loops nor conditionals, nothing stops a user from implementing higher-order \code{if} or \code{while} functions in Python and integrating these into \safeds{} with its stub language. We would treat these like any other function, only using the information from the stubs for static analysis or the representation in the graphical view. On the one hand, this means we ignore the semantics of the if and while constructs for static analysis. On the other hand, such constructs are not needed in the first place since we offer domain specific abstractions like a \code{transform} function for tabular datasets, which encapsulate any loops or conditionals in the Python implementation and use features of the DSL like the schema checking (Sec. \ref{sec:type-system}) for domain specific validation.

\section{Future Plans}
\label{sec:future-plans}

\paragraph*{Additional checks} We are working on completing the implementation of schema checking and will then add checking of call order using behavior protocols. We also look into whether we can extend \cite{reimannImprovingLearnabilityMachine2022} to partially infer schema effects and behavior protocols automatically from documentation.

\paragraph*{Standard library} In parallel, we are integrating additional features from Pandas \cite{The_pandas_development_team_pandas-dev_pandas_Pandas} for data preparation, seaborn for visualization \cite{Waskom2021}, and \sklearn{} \cite{scikit-learn, sklearn_api} for traditional machine learning, using the already implemented stub language (Sec. \ref{sec:checking-external-code}) and automated stub generation (Sec. \ref{sec:automated-stub-generation}).

\paragraph*{Usability study} When a sufficient part of the standard library is implemented, we will conduct a quantitative and qualitative comparison of \safeds{} and Python for solving real world problems.

\paragraph*{Different views} Currently, the available Visual Studio Code extension for \safeds{} only has a textual view. We plan to also implement a graphical view (Sec. \ref{sec:different-views}) using GLSP\footnote{\url{https://www.eclipse.org/glsp/}}.

\section{Conclusions}
\label{sec:conclusion}

In this paper we focused on the needs of DS applications, rather than DS libraries. We presented \safeds{} (Sec. \ref{sec:dsl}), a
DSL for the development of DS pipelines. It combines the expressive power of Python DS libraries with safety via compile-time error detection.

Type-safe integration of Python code is based on creation of stubs (Sec. \ref{sec:checking-external-code}) that describe
the interface of external functions, following largely automated type extraction from a library's code \emph{and} documentation (Sec. \ref{sec:automated-stub-generation}).
The current implementation of \safeds{} already supports a full conventional static type system, with manifest types in the stubs, and type inference in DS pipelines.
Adaptations of non-standard type concepts, such as effect systems (to check schema-conformance of operations on tabular datasets) and behaviour protocols (to check call order) can help deal with these issues statically, too (Sec. \ref{sec:type-system}).

To enable DS pipeline development by people with limited programmings skills
and foster collaboration of people with different skill levels,
our approach supports seamless transition between textual and graphical incarnations of the language (Sec. \ref{sec:different-views}).
Overall, we hope that \safeds{} will prove valuable to help novices
get started with Data Science.

\section*{Acknowledgments}

This work was partially funded by the Federal Ministry of Education and Research (BMBF), Germany under the Simple-ML project (grant 01IS18054). We are grateful for all former contributions to the Safe-DS GitHub repository, particularly with respect to schema checking and the standard library. We also thank the reviewers for their numerous constructive comments and suggestions.

\newpage

\bibliographystyle{IEEEtran}


\section{Introduction}
\label{sec:intro}
As a fundamental task in computer vision, multi-object tracking (MOT) is crucial for automatic driving, video surveillance, etc. MOT aims to localize targets and maintain their unique identities. Recent MOT methods\cite{bochinski2017high,sort,yu2016poi,MTrack,bytetrack} mainly follow the paradigm of tracking-by-detection, and divide tracking into two independent steps: detection and association. The detector detects targets in each frame first, and then appearance representation and position information are employed as the association basis to link targets with the corresponding trajectories. As the inherent attributes of the target, appearance and position complement each other in the association.
\par
However, due to targets or camera motion, intra-class and inter-class occlusion are inevitable, which puts forward stricter requirements for appearance representation. As shown in \cref{fig:Bbox-based} and \cref{fig:Center-based},  these methods extract the features of the bounding box region or the center point as appearance embeddings. However, these coarse-grained global embeddings are extremely sensitive to noise, so that become unreliable once the signal-to-noise ratio is reduced. With the headway of the detector\cite{ren2015faster,fcos,zhou2019objects,carion2020end,yolox}, appearance representation gradually cannot keep the same performance as detection. Some researchers\cite{bytetrack} find that simply using position cues is enough to obtain satisfactory results, while appearance cues are unfavorable for further improvement.
\par
To break this situation, we re-examine the recent appearance-based methods. The bbox-based methods\cite{voigtlaender2019mots} in \cref{fig:Bbox-based} adopt global average pooling, which converts features of the bounding box region into appearance embeddings. These methods equally treat target and interference features (background and other objects), which is unreasonable. As shown in \cref{fig:Center-based}, researchers\cite{centertrack,fairmot,relationtrack} notice this issue and utilize features at the target centers as their appearance embeddings, eliminating interference from the background as much as possible. Despite this, when the target is occluded, the feature at its center is still inevitably interfered with noise information from other objects. The fuzziness of global representation has become an impediment of these methods. On the contrary, our method focuses on different local details of targets, which is illustrated in \cref{fig:Ours}. Fine-grained global and local representations complement each other and jointly describe appearance. When the target is occluded, our method can still identify it according to visible parts, similar to human judgment.
\par
As the basis of fine-grained representation, target feature maps require high-resolution and unambiguous semantic information. Shallow or deep outputs cannot meet these two requirements simultaneously. Therefore, it is feasible to enrich the semantic information of shallow features or improve the resolution of deep features. To reduce the burden, researchers usually adopt FPN\cite{FPN} to aggregate multi-scale shallow feature maps indiscriminately, which causes semantic misalignment among features with different resolutions. Specifically, there is a spatial misalignment between the late feature maps after up-sampling and the early feature maps.
\par
To solve this issue, we construct a \textit{Flow Alignment FPN} (FAFPN) to learn the semantic flow among feature maps with different scales and effectively broadcast high-level features to high-resolution features. FAFPN aligns feature maps by semantic flow and aggregates context information to enrich semantic information while maintaining high resolution. Further, \textit{Multi-head Part Mask Generator} (MPMG) is proposed to generate part masks for detailed representations without label supervision. Inspired by multi-head self-attention in Transformer\cite{Transformer}, MPMG implements a multi-branch parallel structure, which can efficiently and comprehensively focus on different parts of the target. Combining FAFPN and MPMG, we can obtain a diverse fine-grained representation, including diverse local embeddings and background-filtered global embeddings.
\par
In the training phase, some current MOT methods\cite{fairmot,liang2022rethinking} train Re-ID (re-identification) by following the video sequence or shuffling all video frames. The former does not disperse training data, while the latter is positive and negative samples imbalanced. To train Re-ID more reasonably, we propose a training strategy named \textit{Shuffle-Group Sampling} (SGS). In this strategy, we group video frames into short segments in their order and then shuffle these segments. SGS disperses data and balances positive and negative samples. Our model incorporates all the above proposed techniques, named \textit{fine-grained representation tracker} (FineTrack).
\par
The main contributions of our work can be summarized as follows:
\begin{itemize}
\setlength{\itemsep}{0 pt}
\setlength{\parsep}{0 pt}
\setlength{\parskip}{0 pt}
	\item We propose a \textit{Flow Alignment FPN} (FAFPN). It learns the semantic flow among feature maps with different resolutions to correct spatial dislocation. Feature maps after FAFPN include high resolution and precise semantic information, which is the basis of fine-grained representation.
	\item We construct a \textit{Multi-head Part Mask Generator} (MPMG) to focus on details of targets. MPMG employs self-attention to filtrate background noise and extract global-local embeddings to represent targets comprehensively.
	\item \textit{Shuffle-Group Sampling} (SGS) is proposed to disperse training data and balance positive and negative samples. It reduces oscillation in model convergence to achieve better performance.
\end{itemize}
\begin{figure*}
	\centering
	\includegraphics[width=175mm]{overview.pdf}
	\caption{{\bf Overall pipeline of FineTrack.} FineTrack comprises 5 steps: (1) SGS divides videos into groups and shuffles them. (2) Extract feature maps of the input frame for detection. Meanwhile, multi-scale shallow feature maps are obtained for ROIAlign to output target feature maps with different resolutions. (3) FAFPN aligns target feature maps and aggregates their context information for ${F_{mask}}$ and ${F_{reid}}$. (4) Divide ${F_{mask}}$ into ${K}$ blocks along their channel and employ multi-head PMG to generate part masks ${M_{part}}$. Concatenate  ${M_{part}}$ and adopt ${Max}$ of channel to form global masks ${M_{global}}$. (5) ${F_{global}}$ and ${F_{part}}$ can be obtained by a weighted pooling. Then two fully connected layers convert them into global and part embedding: ${f_{global}}$ and ${f_{part}}$, respectively.}
	\label{fig:overview}

\end{figure*}
\section{Related Work}
\noindent
{\bf Tracking-by-detection.} With the development of detection, many works\cite{bergmann2019tracking,tubetk,chained,wang2020towards} adopt the tracking-by-detection paradigm, and divide MOT into detection and association.The trackers following this paradigm first detect objects in video frames and then associate them dependent on their identity information. Some works\cite{bergmann2019tracking,feichtenhofer2017detect,siammot,DAN} model object movement for identity association. SORT\cite{sort} relies on the Kalman Filter\cite{kalman} to predict future positions of the targets, calculates their overlap with detection, and utilizes the Hungarian algorithm\cite{kuhn1955hungarian} for identity association. When the target moves irregularly or is occluded, only using motion information is not enough to achieve better performance. Based on SORT\cite{sort}, DeepSORT\cite{deepsort} introduces the appearance information of targets to enhance identity embedding.
\par
Appearance and motion, as inherent properties of targets, are not independent but complementary while tracking. However, appearance does not always provide reliable clues for identification, particularly when targets are occluded. Our method focuses on diverse details of targets and constructs global-local fine-grained representation, which achieves better performance in occlusion scenes.
\\
{\bf Re-ID in MOT.} Some works\cite{zhou2018online,fang2018recurrent,yu2016poi} crop the image regions of detections and extract features with an extra Re-ID model. These works treat detection and Re-ID as independent tasks, imposing additional parameter burdens. To solve this problem, JDE\cite{wang2020towards} and FairMOT\cite{fairmot} implement a branch the same as detection for appearance representation. The structure of the multi-task branches achieves impressive performance but also raises an additional issue: competition caused by different goals for detection and Re-ID. To alleviate this conflict, CSTrack\cite{liang2022rethinking} constructs a CCN module to extract general and specific features more suitable for detection and Re-ID. However, these appearance-based methods extract the features of the boundary box region or the center point as appearance representation, which are inevitably interfered with irrelevant information. They focus on generating more discriminative global embedding instead of exploring more detailed appearance cues.
\par
Although well-designed Re-ID modules can improve performance, global features are still prone to ambiguity. With a high-performance detector, the associator can link most targets only with motion cues. The global representation is also impotent for the remaining targets incorrectly tracked due to occlusion or other factors. Using global appearance cues at this point can even compromise tracking accuracy. Instead, our method focus on the details of targets. Following this idea, we construct the FAFPN to align and aggregate high-resolution shallow feature maps and feed them into the MPMG to obtain fine part masks. Further,  part masks of targets are employed to extract the fine-grained global-local appearance embedding, which can more comprehensively represent the identity.
\section{Methodology}
\subsection{Overview}
\label{sec:3.1}
In this work, we adopt the high-performance detector YOLOX\cite{yolox} to detect targets. As shown in  \cref{fig:overview}, during training, SGS groups video frames by sampling them sequentially, ensuring targets with positive and negative samples. After shuffling, these grouped data are fed into the Backbone to obtain feature maps with different resolutions. Then, we adopt ROIAlign to crop and scale feature maps of the bbox regions to get the multi-scale feature maps of targets. Immediately, FAFPN aligns these target feature maps from different resolutions and aggregates them into fine-grained feature maps: ${F_{mask}}$ and ${F_{reid}}$ for mask generation and appearance embedding, respectively. After slicing  ${F_{mask}}$ along the channel dimension, the features are fed into separate \textit{Part-Mask Generator} (PMG) to generate target masks. Further, ${F_{global}}$ and ${F_{part}}$ can be obtained by a weighted pooling. These embeddings are fed into two fully connected layers that do not share parameters to output ${f_{global}}$ and ${f_{part}}$. For ${f_{global}}$, we calculate Classification loss and Triplet loss. For ${f_{part}}$, a diversity loss is added to expand the discrepancy among different part features.
\subsection{Flow Aligned FPN}
\label{sec:3.2}
As a classical method for feature aggregation, FPN\cite{FPN} has been extensively applied in different computer vision tasks. However, step-by-step downsampling and indiscriminate context aggregation cause semantic misalignment among feature maps of different scales. Fuzzy semantic information have a significant impact on visual tasks which require detailed descriptions. To obtain fine-grained features, we employ the \textit{Flow Alignment Module} (FAM)\cite{semanticflow,alignseg} to generate semantic flow among feature maps of different resolutions. The semantic flow can guide alignment and eliminate spatial dislocation among feature maps from different scales. Furthermore, we utilize the FAM to optimize the aggregation process of FPN and then construct a \textit{Flow Alignment FPN} (FAFPN).
\par
The overall structure of FAFPN is shown in \cref{fig:FAFPN}. \textit{ResBlock} (RB) replaces ${1*1}$ Convolution in FPN to fuse shallow features more adequately. The $ Sum $ operation is replaced with the FAM, making semantic alignment possible. The multi-scale feature maps are modified to a unified channel dimension by RB and then fed into the FAM for alignment and aggregation. Finally, we use two RB to generate ${F_{mask}}$ and ${F_{reid}}$ respectively for meeting the different requirements of the mask and representation. \cref{fig:RB&FAM} shows the specific structure of RB and FAM. The RB is a simple residual block. It combines ${1*1}$ Convolution and BN layer to transform the input channel dimension and then applies the residual structure to get its output. The FAM takes two feature maps from different scales as input. ${f_{l}}$ has the same size as ${f_{h}}$ after up-sampling. Further, they are concatenated to generate semantic flow: ${flow_{down}}$ and ${flow_{up}}$, which represent the pixel offset of feature maps ${f_{h}}$ and ${f_{l}}$, respectively. $ Warp $ operation aligns these feature maps based on semantic flow. In the end, the FAM employs an $ Sum $ operation to aggregate the context information of aligned feature maps.
\par
After semantic alignment, multi-scale shallow feature maps are aggregated into high-resolution feature maps with precise semantic information, which are the basis for generating diverse fine-grained representations.
\begin{figure}
	\centering
	\includegraphics[width=70mm]{FAFPN.pdf}
	\caption{{\bf Structure of FAFPN.} FAFPN consists of FAM and RB for aggregating feature maps from $ S $ different scales}
	\label{fig:FAFPN}

\end{figure}
\begin{figure}
	\setlength{\abovecaptionskip}{0.2cm}
	\begin{subfigure}{0.3\linewidth}
		\centering
		\includegraphics[width=18mm]{RB.pdf}
		\caption{RB}
		\label{fig:RB}
	\end{subfigure}
	\hfill
	\begin{subfigure}{0.7\linewidth}
		\centering
		\includegraphics[width=40mm]{FAM.pdf}
		\caption{FAM}
		\label{fig:FAM}
	\end{subfigure}
	\caption{{\bf Implementation of RB and FAM.} RB is a form of residual structure. FAM learns semantic flow to align feature maps from different scales.}
	\label{fig:RB&FAM}

\end{figure}
\subsection{Multi-head Part Mask Generator}
\label{sec:3.3}
Some recent works rely on well-designed modules to optimize global embedding. These methods indiscriminately generalize the various details of the target into a coarse-grained global embedding, which causes semantic ambiguity. When the target is occluded, the features of the bounding box region or the center point are inevitably disturbed by noise.
\par
To alleviate this problem, we propose exploring fine-grained appearance representation to focus on target diversity details. The direct approach is to adopt an attention module to enhance feature maps. There are many excellent attention modules in Convolutional Neural Networks, such as Non-Local\cite{Non-local}, CBAM\cite{cbam}, etc. Simply employing attention modules can not entirely focus on the details of targets. Inspired by the multi-head self-attention module in Transformer\cite{Transformer}, we establish multiple parallel attention branches to focus on different parts of targets. At the same time, we employ the Non-Local block as the branch attention module, followed by a Convolution to generate a mask of the target part. As shown in \cref{fig:MPMG}, we divide the input feature maps into blocks along their channel dimension, and each block can be considered as a different mapping of the target feature. These feature maps are fed into \textit{Multi-head Part Mask Generator} (MPMG) to generate different part masks of the target without Ground Truth.
\par
For any branch in MPMG namely PMG, three ${1*1}$ Convolutions are respectively used to obtain query, key, and value. Query and key generate the attention matrix, which is multiplied by value and then added with input. After that, a layer combining ${1*1}$ Convolution and Sigmoid is used to generate a part mask with values between 0 and 1. The same operation is performed for the remaining feature blocks, resulting in ${K}$ different part masks. After concatenation, the channel of part masks is aggregated by ${Max}$ to form a global mask containing different local information. It is worth noting that parameters are not shared among parallel branches, which ensures sufficient diversity of attention.
\begin{figure}
	\centering
	\includegraphics[width=85mm]{MPMG.pdf}
	\caption{{\bf Illustration of MPMG.} Multiple parallel PMG branches focus on different parts of the target and generate part masks.}
	\label{fig:MPMG}

\end{figure}
\subsection{Train and Inference}
\label{sec:3.4}
\noindent
{\bf Shuffle-Group Sampling.} For better detection performance, researchers shuffle consecutive video frames, which is effective for training detection but inappropriate for training Re-ID. This training strategy makes a batch contain almost no targets with the same identity. Such imbalanced positive and negative samples cause the model only to discover the difference among the different targets but fail to capture the commonalities of the same targets. This is particularly detrimental to Re-ID, which aims to distinguish the identity of targets.
\par
To solve the above problem, we construct the \textit{Shuffle-Group Sampling} (SGS) training strategy. Different from random sampling, SGS adopts sequential sampling to group video frames. In this way, targets in the same batch hold positive samples with the same identity, thus alleviating the problem of imbalanced positive and negative samples caused by random sampling. In training, we disrupt the order of grouped data to reduce convergence fluctuations. This strategy allows contiguous batches to originate from different videos and form a sequence of track segments with significant appearance variations. Such a discrete distribution of training data optimizes the convergence direction of the model.
\\
{\bf Training Loss.}
SGS enables a batch of targets to contain both positive and negative samples, so the model can be optimized using Triplet loss, which is extensively applied in Re-ID methods. The part and global features are denoted as ${f_{part}}=\left\{ {f_{p}^{n,k},n \in \left[ {1,2,...,N} \right],k \in \left[ {1,2,...,K} \right]} \right\}$ and ${f_{global}} = \left\{ {f_{g}^n,n \in \left[ {1,2,...,N} \right]} \right\}$. Here $N$ represents the number of targets in the image and each target is composed of $K$ parts. Thus, $f_{p}^{n,k}$ means the ${k_{th}}$ part feature of the ${n_{th}}$ target, and $f_{g}^n$ represents the global feature of the ${n_{th}}$ target.
\par
Then we can calculate the Triplet loss with a soft margin for part features based on \cref{eq:2}:
\begin{equation}
	L_{tri}^{p}(k) = \frac{1}{K}\sum\limits_{k = 1}^K {Triplet(f_{p}^{k})},
	\label{eq:2}
\end{equation}
where $Triplet(\cdot)$ is the same as \cite{transreid} and $f_{p}^{k}$ represents the ${k_{th}}$ part features of $n$ targets in the image.
\par
Similarly, the Triplet loss of the global features is based on \cref{eq:3} :
\begin{equation}
	L_{tri}^{g}(k) = {Triplet(f_{global})},
	\label{eq:3}
\end{equation}
\par
After obtaining $K$ part features, $K$ combinations of Linear layer and Softmax are respectively applied to get the classification result vectors $P = \left\{ {p_n^k,k \in \left[ {1,2,...,K} \right],n \in \left[ {1,2,...,N} \right]} \right\}$. The meanings of $K$ and $N$ are the same as in ${f_{part}}$. $p_n^k$ represents the classification vector of the ${k_{th}}$ part feature  of the ${n_{th}}$ target. The dimension of $p_n^k$ is $M$ , which is also the number of all targets in the datasets. For target identity labels, we denote them as $Y = \left\{ {{y_{n,m}},n \in \left[ {1,2,...,N} \right],m \in \left[ {1,2,...,M} \right]} \right\}$, ${y_{n,m}}$ means whether the ${n_{th}}$ target has the same identity as the ${m_{th}}$ target in ID labels, with a value of 0 or 1. According to the above definition, we can calculate the classification loss of the ${k_{th}}$ part feature of the ${n_{th}}$ target with \cref{eq:4}:
\begin{equation}
	L_{n,k}^p(m) = {y_{n,m}}\log (p_n^k(m)),m \in [1,2,...,M]
	\label{eq:4}
\end{equation}
\par
Further, we calculate the classification loss of part features based on with \cref{eq:5}:
\begin{equation}
	L_{cls}^p =  - \frac{1}{K \cdot N}\sum\limits_{k = 1}^K {\sum\limits_{n = 1}^N {\sum\limits_{m = 1}^M {L_{n,k}^p(m)} } }
	\label{eq:5}
\end{equation}
\par
Similarly, the classification loss of global features is:
\begin{equation}
	L_{cls}^g =  - \frac{1}{N} \sum\limits_{n = 1}^N {\sum\limits_{m = 1}^M {{y_{n,m}}\log ({g_n}(m))}},
	\label{eq:6}
\end{equation}
where ${g_n}$ is the classification vector of the ${n_{th}}$ target's global feature.
\par
Due to the peculiarities of multi-branch structures, using only classification loss and Triplet loss does not ensure that the model focuses on different parts of the target. To avoid multiple branches gazing at similar details, we employ the diversity loss $L_{div}$ in \cref{eq:7} to distance different part features of the same target:
\begin{equation}
	{L_{div}} = \frac{1}{{N \cdot K(K - 1)}}\sum\limits_{n = 1}^N {\sum\limits_{{k_i} \ne {k_j}}^K {\frac{{\left\langle {f_p^{n,{k_i}},f_p^{n,{k_j}}} \right\rangle }}{{{{\left\| {f_p^{n,{k_i}}} \right\|}_2} \cdot {{\left\| {f_p^{n,{k_j}}} \right\|}_2}}}} }
	\label{eq:7}
\end{equation}
\par
The purpose of diversity loss is intuitive, which is to keep the cosine similarity between different part features of the same target as low as possible.
\par
We combine the above losses into a final training loss:
\begin{equation}
	L = \alpha  \cdot (L_{cls}^p + L_{tri}^p) + \beta  \cdot (L_{cls}^g + L_{tri}^g) + \gamma  \cdot {L_{div}},
	\label{eq:8}
\end{equation}
where $\alpha,\beta$ and $\gamma$ are used to adjust the proportion of different losses and we set $\alpha  = 3,\beta  = 0.3,\gamma  = 2$.
\\
{\bf Inference.}
Based on ByteTrack\cite{bytetrack}, we add a method similar to DeepSort\cite{deepsort} that calculates Re-ID features into feature distance matrix. It is worth mentioning that we concatenate part features of targets with global features as Re-ID features.
\par
We adopt the exponential moving average (EMA) mechanism to update the features $\tilde f_i^t$ of matched tracklets for the $i_{th}$ tracklet at frame $t$ based on \cref{eq:ema}, as in \cite{fairmot}.
\begin{equation}
	\tilde f_i^t = \lambda \tilde f_i^{t - 1} + (1 - \lambda )f_i^t,
	\label{eq:ema}
\end{equation}
where $f_i^t$ is the feature of the current matched detection, and $ \lambda = 0.9 $ is a momentum term.
\par
In the tracking algorithm, the feature distance matrix ${d_{feat}}$ is:
\begin{equation}
	{d_{feat}} = 1 - Similarity(\tilde f^{t - 1}, f^t),
\end{equation}
where $Similarity(\cdot)$ outputs the cosine similarity matrix between tracklets features $\tilde f^{t - 1}$ and targets features $f^t$.
\par
Meanwhile, we can calculate the IoU distance matrix ${d_{IoU}}$ base on \cref{eq:9}.
\begin{equation}
	{d_{IoU}} = 1 - IoU({b_{det}},{b_{pre}}),
	\label{eq:9}
\end{equation}
where $IoU(\cdot)$ outputs the IoU matrix between detection bboxes $ b_{det} $ and prediction bboxes $ b_{pre} $.
\par
To exclude the interference of distant targets, we only consider feature distances between pairs of targets with the IoU distance less than 1, which means there is bounding box overlap. The optimized feature distance matrix ${\tilde d_{feat}}$ is:
\begin{equation}
	{\tilde d_{feat}} = 1 - (1 - {d_{feat}}) \cdot ({d_{IoU}} < 1)
\end{equation}
\par
After squaring the product of the optimized feature distance matrix and IoU matrix, the final distance matrix is obtained for the association with \cref{eq:13}:
\begin{equation}
	d = \sqrt {{\tilde d_{feat}} \cdot {d_{IoU}}}
	\label{eq:13}
\end{equation}
\par
Finally, we set the association threshold to 0.5.
\begin{table} \small
	\centering
	\tabcolsep=0.4mm
	\begin{tabular}{@{}lcccccc@{}}
		\toprule
		Method & HOTA$\uparrow$ & IDF1$\uparrow$ & MOTA$\uparrow$ & FP$\downarrow$ & FN$\downarrow$ & IDs$\downarrow$\\
		\midrule
		\midrule
		\multicolumn{7}{l}{\textit{MOT17 private detection}} \\
		DAN\cite{DAN} & 39.3 &49.5&  52.4& 25423& 234592& 8431 \\
		TubeTK\cite{tubetk} & 48.0 &58.6& 63.0 &27060 &177483& 4137 \\
		MOTR\cite{motr} & -& 66.4& 65.1 &45486& 149307 &2049 \\
		CTracker\cite{chained} & 49.0 &57.4 & 66.6 &22284 &160491 &5529 \\

		MAT\cite{Mat} & 53.8 &63.1 & 69.5& 30660 &138741& 2844 \\
		QuasiDense\cite{pang2021quasi} & 53.9 &66.3& 68.7 &26589 &146643& 3378 \\
		TransTrack\cite{sun2020transtrack} & 54.1 &63.5 &75.2 &50157 &86442 &3603 \\
		TransCenter\cite{xu2021transcenter} & 54.5 &62.2 & 73.2 &23112 &123738 &4614 \\
		GSDT\cite{wang2021joint} & 55.2& 66.5 &73.2 &26397 &120666 &3891 \\
		PermaTrackPr\cite{tokmakov2021learning} & 55.5 &68.9 & 73.8&28998 &115104 &3699 \\
		SOTMOT\cite{zheng2021improving} & - &71.9 & 71.0 &39537 &118983 &5184 \\
		FUFET\cite{shan2020tracklets} & 57.9 &68.0 &76.2 &32796 &98475 &3237 \\
		MTrack\cite{MTrack} & - & 72.1 & 73.5 &53361 & 101844 & 2028 \\
		FairMOT\cite{fairmot} & 59.3 &72.3 &73.7 &27507 &117477 &3303 \\
		CSTrack\cite{liang2022rethinking} & 59.3 &72.6 &74.9 &23847 &114303 &3567 \\
		SiamMOT\cite{siammot} & - &72.3 &76.3 &- &- &- \\
		ReMOT\cite{yang2021remot} & 59.7 &72.0 & 77.0&33204 &93612 &2853 \\
		Semi-TCL\cite{li2021semi} & 59.8 &73.2 &73.3 &22944 &124980 &2790 \\
		CorrTracker\cite{CorrTracker} & 60.7 &73.6 &76.5 &29808 &99510 &3369 \\
		RelationTrack\cite{relationtrack} & 61.0 &74.7 & 73.8&27999 &118623 &1374 \\
		TransMOT\cite{chu2021transmot} & 61.7 &75.1 &76.7 &36231 &93150 &2346 \\
		ByteTrack\cite{bytetrack}& 63.1 &77.3 & {\bf 80.3} &25491 &{\bf 83721} &2196 \\
		\rowcolor{red!15} \textbf{FineTrack} & {\bf 64.3} & {\bf 79.5} & 80.0 & {\bf 21750} & 90096 & {\bf 1272}\\
		\midrule
		\midrule
		\multicolumn{7}{l}{\textit{MOT20 private detection}} \\
		MLT\cite{zhang2020multiplex} & 43.2 &54.6& 48.9& 45660& 216803& 2187 \\
		TransTrack\cite{sun2020transtrack} & 48.5 &59.4 & 65.0&27197& 150197 &3608 \\
		FairMOT\cite{fairmot} & 54.6& 67.3& 61.8& 103440& 88901& 5243 \\
		Semi-TCL\cite{li2021semi} & 55.3 &70.1 & 65.2&61209 &114709& 4139 \\
		CSTrack\cite{liang2022rethinking} &54.0 & 68.6 & 66.6&25404& 144358 &3196 \\
		GSDT\cite{wang2021joint} & 53.6 &67.5 &67.1 &31913 &135409 &3131 \\
		SiamMOT\cite{siammot} & - &69.1 &67.1 &- &- &- \\
		RelationTrack\cite{relationtrack} & 56.5 &70.5 & 67.2 &61134 &104597 &4243 \\
		SOTMOT\cite{zheng2021improving} & - &71.4 &68.6 &57064 &101154 &4209 \\
		ByteTrack\cite{bytetrack} &61.3 &75.2 &77.8 &26249 &{\bf 87594} &1223 \\
		\rowcolor{red!15} \textbf{FineTrack} &{\bf 63.6}& {\bf 79.0} & {\bf 77.9}& {\bf 24439}& 89012& {\bf 980}\\
		\bottomrule
	\end{tabular}
	\caption{Comparison of the state-of-the-art methods under the private detection on the {\bf MOT17} and {\bf MOT20} test set. The best results are marked in {\bf bold} and our method is highlighted in \colorbox {red!15}{pink}.}
	\label{table:MOT17&MOT20}

\end{table}
\section{Experiments}
\subsection{Settings}
\noindent
{\bf Datasets.} To verify the effectiveness of FineTrack, we test it on MOT17\cite{mot17}, MOT20\cite{mot20}, and DanceTrack\cite{dancetrack} datasets. MOT17 and MOT20 datasets provide training sets but do not contain validation sets. The test metric only can be obtained by uploading the tracking results to the MOTChallange website for evaluation. Therefore, in the ablation experiment phase, we divided the first half of each video of the MOT17 dataset into the training set and the second half as the validation set. Due to the difference between detection and Re-ID, we train them separately. For testing on MOT17, we train a detector using the same datasets as ByteTrack, including CrowdHuman\cite{crowdhuman}, MOT17, Cityperson\cite{Citypersons}, and ETHZ\cite{ETHZ}, and then freeze the trained detector parameters and train Re-ID separately on MOT17.
\par
DanceTrack is a large-scale multi-object tracking dataset for occlusions, frequent crossing, uniform appearance, and human body tracking with different body postures, with 100 videos. It uses 40, 25, and 35 videos as training, verification, and test set, respectively. The pedestrians in DanceTrack wear remarkably similar clothes, which is a massive challenge for Re-ID. Many existing methods that rely on Re-ID are at a disadvantage on this dataset.
\\
{\bf Metrics.} We use CLEAR-MOT Metrics\cite{metrics}, such as HOTA, MOTA, IDF1, IDs, FP, FN, etc., to evaluate the tracking performance. MOTA focuses on detection performance, and IDF1\cite{IDF1} emphasizes association performance. Compared with them, HOTA\cite{Hota} comprehensively balances detection, association, and localization effects. To accurately measure the performance of Re-ID and exclude the influence of detection and association, Rank-1 and mAP are utilized as metrics to measure the ability of Re-ID feature representation.
\\
{\bf Implementation details.} The partial feature and global feature dimensions are 128 and 256. The number of target masks is set as 6. The detector training Settings are consistent with Bytetrack. During training, the batch size is set to 8. The model parameters are updated using the Adam optimizer\cite{adam} with an initial learning rate of $2 \times {10^{ - 4}}$ and 20 training epochs. The learning rate is reduced to $2 \times {10^{ - 5}}$ at the $10_{th}$ epoch.

\begin{table} \small
	\centering
	\tabcolsep=1.4mm
	\begin{tabular}{@{}lccccc@{}}
		\toprule
		Method & HOTA$\uparrow$ & IDF1$\uparrow$ & MOTA$\uparrow$ & AssA $\uparrow$ & DetA $\uparrow$ \\
		\midrule
		CenterTrack\cite{centertrack}& 41.8 & 35.7& 86.8& 22.6 & {\bf 78.1}\\
		FairMOT\cite{fairmot} & 39.7 & 40.8& 82.2 &23.8 & 66.7\\
		QuasiDense\cite{pang2021quasi} & 45.7 & 44.8 & 83.0 & 29.2 & 72.1\\
		TransTrack\cite{sun2020transtrack} & 45.5& 45.2 & 88.4 &27.5 & 75.9\\
		TraDes\cite{wu2021track} & 43.3 & 41.2 & 86.2  &25.4  &74.5\\
		ByteTrack\cite{bytetrack} & 47.7 & 53.9 & 89.6  &32.1 & 71.0\\
		\rowcolor{red!15} \textbf{FineTrack} & {\bf 52.7} & {\bf 59.8} & {\bf 89.9} & {\bf 38.5} & 72.4\\
		\bottomrule
	\end{tabular}
	\caption{Comparison of the state-of-the-art methods on the {\bf DanceTrack} test set. The best results are marked in {\bf bold} and our method is highlighted in \colorbox {red!15}{pink}.}
	\label{table:DanceTrack}
	\vspace{-0.1cm}
\end{table}

\subsection{Comparison with the State-of-the-art Methods}
In this part, we compare the performance of FineTrack with previous SOTA methods on three benchmarks, i.e., MOT17, MOT20 and DanceTrack. Notably, some MOT methods utilize extra Re-ID models and Re-ID datasets to improve their performance of identity embedding. For a fair comparison, FineTrack only uses MOT datasets for Re-ID training and does not use any additional labels for supervision, such as masks, etc.
\\
{\bf MOT17:} FineTrack shows significant advantages without introducing extra Re-ID models and datasets containing a large amount of identity information. As shown in \cref{table:MOT17&MOT20}, FineTrack method achieves the best tracking accuracy on the MOT17 datasets (i.e. 64.3\% HOTA, 79.5\% IDF1 and 80.0\% MOTA, etc.). In cases where using detection simply can obtain excellent tracking accuracy, FineTrack achieves further improvement compared to BytaTrack (1.2\% HOTA and 2.2\% IDF1), which also adopts YOLOX as the detector. This demonstrates that our proposed fine-grained appearance representation dramatically improves the performance of identity embedding. Compared with other MOT methods, the advantages of FineTrack are more prominent.
\\
{\bf MOT20:} Compared with MOT17, MOT20 is more crowded, which causes frequent and dense occlusion. As shown in \cref{table:MOT17&MOT20}, FineTrack outperforms ByteTrack by 2.3\% on HOTA, 3.8\% on IDF1 with superior identity embeddings while reducing IDs from 1223 to 980. FineTrack has a stable and excellent performance in dense pedestrian scenes, highlighting the role of fine-grained appearance embedding.
\\
{\bf DanceTrack:} Pedestrians in DanceTrack dress very similarly and have complex body pose variations, making appearance-based methods perform poorly. FineTrack has excellent appearance modeling capabilities, as shown in \cref{table:DanceTrack}, where we outperform ByteTrack entirely (5.0\% on HOTA and 5.6\% on IDF1, etc.). This further indicates the superiority of fine-grained appearance representations.

\begin{table}[t]
	\small
	\centering
	\tabcolsep=2.5mm
	\begin{tabular}{lcc}
		\toprule
		\multirow{2}*{Method} &
		\multicolumn{2}{c}{Metrics} \\
		\cmidrule{2-3}
		& Rank-1 & mAP  \\
		\hline
		1 FineTrack(train by the order of videos )    & 89.7 & 56.7\\
		2 FineTrack(shuffle all video frames)    & 90.9 & 60.4\\
		3 FineTrack(w SGS)  & {\bf 92.3} & {\bf 61.8}\\
		\toprule
	\end{tabular}
	\caption{{\bf Ablation study of SGS.} The first row does not disturb the video frames and trains in their order. The second row represents shuffling all the video frames. The last row is our SGS training strategy. (SGS: Shuffle-Group Sampling)}
	\label{table:SGS}
\end{table}

\subsection{Ablation Studies}
In this section, we verify the effectiveness of FineTrack through ablation studies. All experiments are conducted on the MOT17 dataset. Since the MOT Challenge does not provide validation sets, we split the MOT17 dataset into two parts. The first half of each video serves as the training set and the second half as the validation set. To fairly measure the performance of identity embedding, experiments in \cref{table:SGS}, \cref{table:Components} and \cref{table:FAM} utilize the Ground Truth of bboxes as input, which can eliminate the interference caused by false detection. \cref{table:SGS} verifies the effectiveness of the SGS training strategy, which is employed in the remaining ablation studies.
\\
{\bf Analyze of SGS.} To validate the advantages of our proposed SGS training strategy, we compare it with two other typical training methods, as reported in \cref{table:SGS}. Our proposed SGS is superior to the first training method by 2.6\% on Rank-1 and 5.1\% on mAP. While shuffling all video frames gets improvement (1.2\% on Rank-1 and 3.7\% on mAP) over training in the order of video frames, there is still a gap of 1.4\% on both Rank-1 and mAP compared with SGS. This ablation experiment proves the effectiveness of FineTrack and reflects the importance of positive and negative samples balanced and discrete distribution of training data for training Re-ID.
\\
{\bf Component-wise Analysis.} In this part, we verified the effectiveness of FAFPN and MPMG through ablation experiments. As shown in \cref{table:Components}, FineTrack effectively improves the performance of appearance embeddings. Because YOLOX\cite{yolox} cannot extract Re-ID features, we use FPN\cite{FPN} to aggregate multi-scale feature maps and then adopt Global Average Pooling (GAP) to obtain appearance embeddings. Finally, we combine FPN and GAP as our baseline (row 1). Compared with baseline indicators, our method (row 4) outperforms the baseline by 4\% on Rank-1 and 11.4\% on mAP, demonstrating the value of exploring fine-grained appearance representation.
\par
\begin{table}[t]
	\small
	\centering
	\tabcolsep=1.4mm
	\begin{tabular}{lcccccc}
		\toprule
		\multirow{2}*{Method} &
		\multicolumn{4}{c}{Components} &
		\multicolumn{2}{c}{Metrics} \\
		\cmidrule(lr){2-5} \morecmidrules \cmidrule(lr){2-3} \cmidrule(lr){4-5} \cmidrule(lr){6-7}
		& FPN & FAFPN & GAP & MPMG & Rank-1 & mAP  \\
		\hline
		1 Baseline  & \checkmark &  & \checkmark &  & 88.3 & 50.4\\
		2  	    & \checkmark &  &  & \checkmark & 89.8 & 55.7\\
		3       &  & \checkmark & \checkmark &  & 91.8 & 58.2\\
		4 \textbf{FineTrack}  &  & \checkmark &  &\checkmark  & {\bf 92.3} & {\bf 61.8}\\
		\toprule
	\end{tabular}
	\caption{{\bf Ablation study of the components in FineTrack.} The first row is our baseline, and the last row is the metrics that can be obtained by adopting our proposed FAFPN and MPMG. (GAP: Global Average Pooling, FAFPN: Flow Alignment FPN, MPMG: Multi-head Part Mask Generator)}
	\label{table:Components}
	\vspace{-0.1cm}
\end{table}

\begin{table}[t]
	\small
	\centering
	\tabcolsep=2.2mm
	\begin{tabular}{lcccc}
		\toprule
		\multirow{2}*{Method} &
		\multicolumn{2}{c}{Components} &
		\multicolumn{2}{c}{Metrics} \\
		\cmidrule(lr){2-3} \cmidrule(lr){4-5}
		& GAP & MPMG & Rank-1 & mAP  \\
		\hline
		1 FineTrack(w/o FAM)   & \checkmark &  & 91.7 & 57.9\\
		2 FineTrack(w/o FAM)   &  & \checkmark & 92.0 & 61.1\\
		3 FineTrack(w FAM)  & \checkmark &  & 91.8 & 58.2\\
		4 FineTrack(w FAM)  &  & \checkmark & {\bf 92.3} & {\bf 61.8}\\
		\toprule
	\end{tabular}
	\caption{{\bf Ablation study of FAM.} Re-ID embedding can be extracted through GAP or MPMG. We verify the effectiveness of FAM in these two cases. (FAM: Flow Alignment Module)}
	\label{table:FAM}
	\vspace{-0.1cm}
\end{table}
Then, we analyze the effectiveness of each module separately. For multi-scale feature maps aggregation, replacing FPN with FAFPN (row 3) can achieve improvement (3.5\% on Rank-1 and 7.8\% on mAP) when generating Re-ID embeddings with GAP. Furthermore, when MPMG replaces GAP, using FAFPN (row 4) is better than using FPN (row 2) by 2.5\% on Rank-1 and 6.1\% on mAP. This indicates that FAFPN captures feature context information more effectively than FPN. At the same time, employing MPMG to extract Re-ID embeddings (row 2) is better than the baseline (1.5\% on Rank-1 and 5.3\% on mAP). When FAFPN replaces FPN, MPMG is 0.5\% and 3.6\% higher than GAP on Rank-1 and mAP. MPMG focuses on the discriminative details of the target and can describe the appearance of targets more comprehensively than GAP.
\\
{\bf Analyze of FAM.} In this part, we conducted ablation experiments with or without FAM under the effect of GAP or MPMG, as reported in \cref{table:FAM}. Specifically, when GAP is used to generate Re-ID embeddings, employing FAM (row 1 and row 3) can increase 0.1\% on Rank-1 and 0.3\% on mAP. At the same time, if we adopt MPMG to extract Re-ID embeddings (row 2 and row 4), FAM can get the improvement (0.3\% on Rank-1 and 0.7\% on mAP), which is significantly better than adding FAM when using GAP. GAP is a global representation that aggregates all information indiscriminately. On the contrary, MPMG generates part masks to distinguish features explicitly and can guide FAM to perform pixel-level alignment more reasonably, even without mask label supervision.
\\
{\bf Analyze of our components when tracking.} We adopt the same association strategy to prove the effectiveness of different components in FineTrack when tracking. As shown in \cref{table:track}, using FAFPN or MPMG alone also improves the tracking performance. When combining these two modules, there is a significant advantage over the baseline (1.5\% on MOTA, 3.3\% on IDF1, and the IDs decreases from 253 to 115), demonstrating the effectiveness of FineTrack.

\begin{table}
	\centering
	\tabcolsep=2.5mm
	\begin{tabular}{@{}lccc@{}}
		\toprule
		Method & MOTA$\uparrow$ & IDF1$\uparrow$ & IDs$\downarrow$\\
		\midrule
		1 Baseline & 75.5 & 77.8 & 253 \\
		2 Baseline+FAFPN & 76.0 & 78.9 & 459 \\
		3 Baseline+MPMG & 76.4 & 79.7 & 451 \\
		4 Baseline+FAFPN+MPMG & {\bf 77.0} & {\bf 81.1} & {\bf 115} \\
		\bottomrule
	\end{tabular}
	\caption{{\bf Ablation study of our components when tracking.} The same association strategy is adopted for our proposed module to obtain the tracking results.}
	\label{table:track}

\end{table}

\section{Conclusion}
In this work, we have argued that diverse fine-grained representation is essential for MOT. However, existing methods focus on coarse-grained global features, which are extremely sensitive to noise. To effectively filter irrelevant information, we propose to explore diverse fine-grained appearance representation to obtain more comprehensive embeddings. As reported in the ablation studies, our presented FAFPN has great advantages in terms of aligning semantic and aggregating contextual information. Meanwhile, our constructed MPMG can effectively focus on different parts of the target without label supervision. In the training phase, we propose the SGS training strategy to improve the model performance effectively. We have verified the effectiveness of our proposed FineTrack on three public benchmarks (MOT17, MOT20, and DanceTrack) and achieved state-of-the-art performance. The experiment results indicate that diverse fine-grained representation can significantly improve the performance of Re-ID in MOT. We hope this work can be a new solution for appearance representation to generate discriminative identity embeddings.
\newpage
{\small
\bibliographystyle{ieee_fullname}