\section{Introduction} \label{sec1}

In the present paper we study the motion of an incompressible inviscid fluid with an axial symmetry  without swirl (for the exact definition see later on) when the initial vorticity is very concentrated on $N$ annuli of radii $\approx$ $r_0$ and thickness $\epsilon$. We prove the relation of this motion with the so-called point vortex system in the plane when $r_0\to \infty$ as $\epsilon \to 0$.

\medskip

\noindent The motion of an incompressible inviscid fluid is governed by the Euler equations, that for a fluid of unitary density in three dimensions read:

\begin{equation}
\label{1.1}
(\partial_t + (u\cdot \nabla))\omega=(\omega\cdot \nabla )u \, ,
\end{equation}
\begin{equation}
\label{1.2}
\nabla \cdot u =0   \qquad {\textnormal{(continuity  equation)}} ,
\end{equation}
$u(x,0)=u_0(x)$ (velocity field),  and boundary conditions. From now on we suppose that the velocity vanishes as $|x| \to \infty$. This assumption allows to reconstruct the velocity from the vorticity:
\begin{equation}
\label{1.3}
u(x,t)= - \frac{1}{4\pi} \int  \frac{x-y }{|x-y|^3} \wedge \omega(y,t) \, dy \, .
\end{equation}

\noindent We use now cylindrical coordinates $(z,r,\theta)$ and  suppose that the initial velocity field has the form (axial symmetry without swirl):
\begin{equation}
\label{1.4}
u(x,t)= (u_z,u_r,u_\theta)= (u_z(z,r,t), u_r(z,r,t), 0) \, .
\end{equation}
The time evolution conserves this symmetry. In this case the  vorticity is
\begin{equation}
\label{1.5}
\omega= \nabla \wedge u = (0,0,\omega_{\theta}) = (0,0, \partial_zu_r - \partial_r u_z) \, ,
\end{equation}
and the  Euler equations become
\begin{equation}
\label{1.6}
(\partial_t  +(u_z\partial_z + u_r\partial_r)) \omega_{\theta} -\frac{u_r \omega_{\theta}}{r}  =0  \, ,
\end{equation}

\begin{equation}
\label{1.7}
\partial_z(r \ u_z)+\partial_r(r \ u_r)=0 \, .
\end{equation}

\noindent From now on we denote $\omega_{\theta}$ by $\omega$.
Finally, by (\ref{1.3}),
\begin{equation}
\label{1.8}
\begin{aligned}
u_z(z,r,t)= - \frac{1}{2\pi} & \int _{-\infty}^\infty dz'  \int_0^\infty r' \, dr'  \\
&\int _0^\pi d\theta \, \frac{\omega(z',r',t) [r \cos \theta - r']}{[(z-z')^2+(r-r')^2 + 2rr'(1-\cos \theta)]^{3/2}} \, ,
\end{aligned}
\end{equation}

\begin{equation}
\label{1.9}
\begin{aligned}
u_r(z,r,t)= \frac{1}{2\pi} & \int _{-\infty}^\infty dz'  \int_0^\infty r' \, dr'\\
 &\int _0^\pi d\theta\, \frac{\omega(z',r',t) [z - z']\cos \theta}{[(z-z')^2+(r-r')^2 + 2rr'(1-\cos \theta)]^{3/2}}  \, .
\end{aligned}
\end{equation}

Hence, the axially symmetric solutions to the Euler equations are given by the solutions to eqs. \eqref{1.5}-\eqref{1.9}.
Eq. (\ref{1.6}) means that the quantity $\omega/r$ remains constant along the flow generated by the velocity field, i.e.
\begin{equation}
\label{1.10}
\frac{\omega(z(t),r(t),t)}{r(t)}= \frac{\omega(z(0),r(0),t)}{r(0)} \, ,
\end{equation}
where $(z(t),r(t))$ solve
\begin{equation}
\label{1.11}
\dot{z}(t)=u_z(z(t),r(t),t) \, ,  \qquad  \dot{r}(t)=u_r(z(t),r(t),t) \, .
\end{equation}
It is possible to introduce an equivalent weak formulation of \eqref{1.6} that allows to consider non-smooth initial data;
by a formal integration by parts we obtain indeed
\begin{equation}
\label{1.12}
\frac{\mathrm{d}}{\mathrm{d}t} \omega_t(f)=\omega_t \big[u_z\partial_z f + u_r\partial_r f + \partial_tf \big] \, ,
\end{equation}
where $f=f(z,r,t)$ is a bounded smooth test function and
\begin{equation}
\label{1.13}
\omega_t (f):=\int_{-\infty}^\infty dz\int_0^\infty dr \ \omega (z,r,t) f(z,r,t) \, .
\end{equation}
It is known that a global (in time) existence and uniqueness of a weak solution to the associate Cauchy problem holds when the initial vorticity is a bounded function with compact support contained in the open half-plane  $\Pi:=\lbrace(z,r):  r>0\rbrace$, see  for instance  \cite{mpulv} pag.91 or the Appendix of \cite{CS}.  In particular, it can be shown that the support of the vorticity remains in the open half-plane $\Pi$ at any time.
A point  in the half-plane $\Pi$ denotes a circumference in the whole space.
The special class of axisymmetric without swirl solutions are called sometimes {\textit{smoke  rings}}, because there exist particular solutions whose shape remains constant in time (the so-called steady vortex ring) and translate in the $z$-direction with constant speed  (see for instance \cite{Fra70}). The existence and the properties of these solutions is an old question. For a rigorous proof by means of variational methods see  \cite{AmS89,FrB74}. For references on axially symmetric solution without swirl see also the review paper \cite{ShL92}.

\bigskip

Denote $x=(x_1, x_2) := (z, r-r_0)$.
 We assume that initially the vorticity is concentrated in $N$ blobs of the form
\begin{equation}
\omega_\epsilon(x,0)= \sum_{i=1}^N \omega_{i,\epsilon}(x,0)
\label{in_data}
\end{equation}
where $\omega_{i,\epsilon}(x,0)$ are functions with a definite sign such that, denoting by $\Sigma(\xi| \rho)$ the open disk of center $\xi$ and radius $\rho$ in $\mathbb{R}^2$,
\begin{equation}
\Lambda_{i,\epsilon}:= \text{supp} \, \omega_{i,\epsilon}(\cdot,0) \subset \Sigma(z_i | \epsilon)
\; ; \qquad  \Sigma(z_i |\epsilon) \cap  \Sigma(z_j |\epsilon) =\emptyset \ \ \forall i \ne j \,
\label{in_data2}
\end{equation}
being $\epsilon>0$
a small parameter and
$z_1, \dots, z_N$, points contained in a bounded region of $\mathbb{R}^2$ such that
$$
\min_{i\neq j} |z_i -z_j| > \rho_m
$$
for a positive constant $\rho_m$ independent of $\epsilon$.
Moreover we assume that, for any $i=1, \dots, N$,
\begin{equation}
\int  dx \ \omega_{i,\epsilon}(x,0) := a_i \in \mathbb{R}
\end{equation}
independent of $\epsilon$ and
\begin{equation}\label{mass}
|\omega_{i,\epsilon}(x,0)| \leq M \epsilon^{- \gamma} \; , \qquad M>0 \; , \quad \gamma >0 \, .
\end{equation}

\noindent We will discuss if,  in some cases and for small $\epsilon$,  the
time evolution of these states has the same form. In
Theorem \ref{teoEA}  we will prove that, as the assumption $r_0=|\log
\epsilon|^\alpha$   ($\alpha > 2$)  is fulfilled, the evolved state
$\omega_\epsilon(x,t)$ can be written as
\begin{equation}
\omega_\epsilon(x,t)= \sum_{i=1}^N \omega_{i,\epsilon}(x,t) \, ,
\label{t_data}
\end{equation}
where $\omega_{i,\epsilon}(x,t)$ are functions with definite sign such that
\begin{equation}
\Lambda_{i,\epsilon}(t):= \text{supp} \, \omega_{i,\epsilon}(\cdot, t) \subset \Sigma(z_i(t) | r_t(\epsilon)) \, ,
\end{equation}
with
\begin{equation}
 \Sigma(z_i(t) | r_t(\epsilon)) \cap  \Sigma(z_j(t) | r_t(\epsilon)) = \emptyset   \qquad \forall\, i \ne j \, ,
\end{equation}
being $r_t(\epsilon)$ a positive function, vanishing for  $\epsilon\rightarrow 0$, and
$z_i(t)\in \mathbb{R}^2$ solution to
the point-vortex model, that  is the dynamical system defined by the following differential equations:
\begin{equation} \label{pvmodel}
\dot{z}_i(t)= \sum_{\substack{j=1 \\j \neq i}}^N a_j K(z_i(t)- z_j(t)) \, , \qquad z_i(0)=z_i
\end{equation}
\noindent for $i=1, \ldots, N$,  and
\begin{equation}\label{nucleoK}
K(x)=-\frac{1}{2\pi}\nabla^{\perp}\log|x|, \qquad \nabla^{\perp}=(\partial_2,-\partial_1),
\end{equation}

\noindent where $-1/2\pi\log|x|$ is the fundamental solution of the Laplace operator in $\mathbb{R}^2$.
When all the $a_i$ have the same sign there is a global solution, otherwise there is a finite time
at which
a collapse (that is two $z_i$ arriving at the same point) or a $z_i$ going to infinity can
occur.
However  (see for instance \cite{mpulv}) these events are exceptional.

\noindent Few words on this dynamical system: it has been introduced by Helmholtz as particular {\textit{solution}} of the Euler equations  \cite {Hel67} and investigated by many authors
\cite{Kel,Kir,Poi}.
It has been used to investigate the time evolution of irregular initial data and it produces an approximation method (called vortex method) in which $N \to \infty$ and $a_i\to 0$ (for more information see for instance the textbook  \cite{mpulv} or \cite{MaB02} and references in \cite{CGP14}).

Even if solutions of \eqref{pvmodel} cannot be a solution of the Euler equations, they can be an average of different solutions that in $\mathbb{R}^3$ are clusters of straight lines of vorticity, as it is discussed in \cite{CapMar, Mar88,  Mar,  MaP, MaP93, Turk}.  In \cite{marc} it is shown that the same happens when the straight lines are changed into large enough annuli,
with radius of the order $r_0(\epsilon)= \epsilon^{-\alpha}$,   for any $\alpha>0$ and for any finite time, and recently for long times in \cite{CS}.

In the present paper we assume a weaker dependence of $r_0$ on $\epsilon$ and we will show that the relation with the point vortex model remains valid at least for a finite but positive time.
We consider radii of the order $|\log\epsilon|^\alpha$, $\alpha>2$ (this lower bound on $\alpha$
  appears for a technical
reason which occurs in eq.  \eqref{qui}),
hence the rings are less distant from the axis, where the effects of curvature become stronger.
For laws $r_0=f(\epsilon)>\epsilon^{-\alpha}$ the convergence to the point-vortex model
happens faster, while for the previous logarithmic law  the convergence is slower.
 This imposes a more careful strategy to prove the convergence, since we need an iterative method  as the one used in \cite{butmar2}
in a different context, which produces a result only for bounded times. This reflects the
difficulty of the convergence when the distance from the axis approaches the scale
$|\log\epsilon|^\alpha$, $0\leq \alpha \leq 1$, for which only for $\alpha=0$
(for one vortex alone \cite{ben} and $N$ vortices \cite{butmar2}) and for $\alpha=1$
for one vortex alone \cite{marneg} some results are available.
For $\alpha=0$ the dynamics of the vortices converges to
simple translations parallel to the symmetry
axis with constant speed.
For $\alpha = 1$
we can conjecture
that  the convergence
of the dynamics is not to the point-vortex model, but to the dynamical
system defined by
\begin{equation}
\dot{z}_i(t)= \sum_{\substack{j=1 \\j \neq i}}^N a_j K(z_i(t)- z_j(t))
+ e_1 a_i \, , \qquad z_i(0)=z_i \, , \quad e_1=(1,0) \, .
\label{dyn2}
\end{equation}
  This is proved rigorously for one vortex alone in \cite{marneg}, but for
	$N$ vortices it is an open problem and the correspondence
	with \eqref{dyn2}  is established only at a heuristic level.

We also mention that in literature it is discussed how the point-vortex model behaves under a viscosity perturbation \cite{BrM11,CS, Gal11,Lad68, Mar90, marcNS, Mar07, UkY68}, but this topic is out of the scope of the present analysis.

\section{Main result} \label{sect2}

{\textit{A warning on the notation}}. Hereafter  in the paper
we denote by $C$
a generic positive constant (eventually changing from line to line) which is independent of the parameter $\epsilon$
and the time $t$.

We define a suitable scaling of variables, in order to get the convergence to the point-vortex dynamics, in such a way that  the rings increase their radius while their support becomes smaller. When the radius
increases, the interaction of the $N$ vortices with the axis becomes negligible in the limit $\epsilon\to 0$, which permits
to obtain a convergence to the point-vortex dynamics. Denoting by $(r, z, \theta)$  the cylindrical coordinates in $\mathbb{R}^3$, we recall the previously mentioned coordinates
\begin{equation} \label{coordinate} x=(x_1, x_2) := (z, r-r_0) \,  \end{equation}
and consider an initial vorticity as specified in \eqref{in_data}-\eqref{mass},
whose evolution at time $t$ can be expressed as in \eqref{t_data}.
Knowing the velocity field $u(\cdot,t):=u_t(\cdot)$, we can define the trajectory of a fluid element starting at $x$ as the solution of the integral equation
$$
\phi_t(x) = x + \int_0^t u_s (\phi_s(x)) ds \, .
$$
Since the quantity $\omega/r$ remains constant along the
flow generated by the velocity field, we have
\begin{equation}\label{term i EA}
 \omega_{i, \epsilon}(x,t) :=  \frac{r_0+x_2}{r_0+(\phi_{-t}(x))_2} \omega_{i, \epsilon}(\phi_{-t}(x),0)  \, .
 \end{equation}

\noindent Moreover $\omega_{i,\varepsilon}(x,t)$ preserves the initial sign and the total mass $a_i$, as immediately follows from the definitions (see also Lemma \ref{norme di omega}).

\noindent Furthermore, for each index $i$, we can decompose the velocity field $u$ as follows:
\[u(x,t) =u^i(x,t) + F_\epsilon^i(x,t) \, , \]

\noindent where
$$
u^i(x,t)=\int dy\,G(x,y)\,\omega_{i,\varepsilon}(y,t)
$$

\noindent is the velocity field generated by the vortex $\omega_{i, \epsilon}$, and
\begin{equation}
F_\epsilon^i(x,t)=\sum\limits_{j\neq i}\int dy\,G(x,y)\,\omega_{j,\varepsilon}(y,t)
\label{effe_iep}
\end{equation}

\noindent is the one generated by the remaining $N-1$ vortices; here $G(x,y)$ denotes the integral kernel appearing in \eqref{1.8}-\eqref{1.9} in the new coordinates \eqref{coordinate}.

Let us call $\{z_i(t) \}_{i=1, \ldots, N}$ the solution to point vortex dynamics \eqref{pvmodel} with intensities $a_i$ and initial data $z_i$ (with $z_i\neq z_j$  for  $i\neq j$).
As already discussed, such dynamics is well defined globally in time apart from a zero
measure set of initial data. Even in this last case in which a collapse can occur,
since our results hold for times smaller than a positive constant, we can consider such
constant (let's call it $T_*$) much smaller than the first collapse time. With this viewpoint we define,
for a constant $\bar R>0$,
\begin{equation}
T_\omega := \sup \{ t>0 : \: \mathrm{supp}\,\omega_{i, \epsilon}(s) \subseteq \Sigma(z_i(s)| \bar R) \; \; \forall i=1, \ldots, N \,, \; \forall s \in [0,t]  \}  \, ,
\end{equation}
\begin{equation}
\bar T = \min \{ T_\omega,  T_*    \} \, ,
\end{equation}
and
\begin{equation}
 R_m := \min_{i \neq j} \inf_{t \in [0, \bar T)} |z_i(t) -z_j(t)| >0 \, .
\end{equation}
We ask
\begin{equation}
\bar R < R_m/4
\label{R_m}
\end{equation}
(it  will be used in the sequel). Observe that such requirement
is non-empty: for $\bar R=\epsilon$ it results $\bar T =0$ (by the initial data
\eqref{in_data}-\eqref{in_data2}), and \eqref{R_m} is obviously fulfilled (for small $\epsilon$).
Considering then a small positive $\bar R$ (but independent of $\epsilon$), we obtain
consequently a small $\bar T$, and
 for a continuity
argument \eqref{R_m} can still be satisfied. In the next Theorem we state a better result,
the size of the support of $\omega_i$ at time $t$ (for short times) is a quantity which vanishes
for $\epsilon\to 0$.
The result is the following.
\begin{theorem} \label{teoEA}
Consider initial vorticity as in \eqref{in_data}-\eqref{mass} and $r_0 = |\log\epsilon|^{\alpha}$, for any $\alpha>2$.
Then there is a $T>0$ such that
$$
\mathrm{supp}\,\omega_{i, \epsilon}(s) \subseteq \Sigma\left(z_i(s)  | C_T |\log\epsilon|^{-k}\right) \; \; \forall i=1, \ldots, N \,, \; \forall s \in [0,T]
$$
where $C_T$ is a positive constant,  $k = (\alpha-2)/2$,    $\epsilon \in(0, \epsilon_0)$, with
$\epsilon_0<1$ solution to
$$
C_T |\log\epsilon_0|^{-k}=R_m/4 \, .
$$
\end{theorem}

\bigskip

\bigskip

\noindent We remark that with the previous definition of $\epsilon_0$ it results $C_T |\log\epsilon|^{-k}<R_m/4$.

\section{Proof of Theorem \ref{teoEA}}
We give the general strategy of the proof, which is rather technical and composed of
many auxiliary Lemmas and Propositions.
We study the motion of a tagged vortex (with index $i$) under the influence of the remaining $N-1$. The field generated by the remaining $N-1$ vortices has the features of
a given external bounded field, since in the time interval $[0, \bar T]$
the minimum distance between any two distinct vortices
remains greater than a positive constant. We make use then of a fundamental estimate on the growth in time of the moment
of inertia of the tagged vortex, and we estimate the vorticity mass far from the center
of vorticity, showing that it is negligible when $\epsilon$ is small, by means of an
iterative method. Putting together
these (and other technical) results we achieve the proof.
Some of these tools are similar to those of
previous papers \cite{butmar, butmar2, CapMar, CS, ISG, marc},  and  we
write them again for completeness.

\noindent We discuss then the preliminary results we need, starting with
the estimate of the convolution kernel $G$ (in the new coordinates \eqref{coordinate}), showing that, under suitable assumptions, this is near to $K$ (of the planar case).

\noindent
Making use of
\eqref{1.8}-\eqref{1.9}, written with respect to the new coordinate system \eqref{coordinate},
 we get
\[ u(x,t)= \int_{\mathbb{R}^2}  G(x, y) \, \omega_\epsilon(y, t) \, dy \]
where the convolution kernel $G(x,y)$ is defined by:
\begin{equation}
G_1(x,y) =  \frac{1}{2\pi} \int_0^{\pi} d\theta \,
\frac{ (r_0+y_2) \big[(r_0+y_2) - (r_0+x_2) \cos \theta \, \big]}
{ \left\{ |x-y|^2 +2(r_0+x_2)(r_0+y_2)(1-\cos \theta) \right\}^{3/2}}
\end{equation}
\begin{equation}
G_2(x,y) =  \frac{1}{2\pi} \int_0^{\pi} d\theta \, \frac{(r_0+ y_2)(x_1-y_1) \cos \theta}{ \left\{|x-y|^2 +2(r_0+x_2)(r_0+y_2)(1-\cos \theta) \right\}^{3/2}} \, .
\end{equation}
We now want to give an estimate for this convolution kernel, in particular we want to show that, for small enough $\epsilon$, the vector field $u$ is near to the vector field
$\widetilde{u}$ corresponding to the planar case, namely
\[ \widetilde{u}(x, t) = \int_{\mathbb{R}^2} K(x-y) \, \omega_\epsilon(y, t) \, dy \,  \]
with $K$ defined in \eqref{nucleoK}.
We need the following lemma, whose proof is contained in \cite{CS}
and reported here for  completeness.

\begin{lemma}
Let us define, for $a>0$:
\[ I_1(a) := \int_0^\pi d\theta \frac{\cos\theta}{[a^2+2(1-\cos\theta)]^{3/2}} \, , \]
\[ I_2(a):= \int_0^\pi d\theta \frac{1 - \cos\theta}{[a^2+2(1-\cos\theta)]^{3/2}}  \, . \]
Denoting by $\chi_{(0,1)}(\cdot)$ the characteristic function of the interval $(0,1)$, the following equalities hold:
\begin{equation}
I_1 (a) = a^{-2} + R_1(a) \, ,\qquad I_2 (a) = -\frac{1}{2} \log a \cdot \chi_{(0,1)}(a) + R_2(a) \, ,
\label{separ}
\end{equation}
where $a \cdot R_1(a)$ is bounded and $|R_2(a)| \leq C \min(1, \frac{1}{a})$.
\end{lemma}

\begin{proof}
Let us consider first $I_2$. We recall that $1-\cos\, \theta = 2[\sin(\theta/2)]^2$ and we write the integral as:
\begin{equation} \label{int2} I_2= \int_0^\pi d\theta \frac{2 [\sin (\theta/2)]^2 \, \cos(\theta/2)}{ \{a^2+4 [\sin(\theta/2)]^2 \, \}^{3/2}}+
   \int_0^\pi d\theta \frac{2 [\sin (\theta/2)]^2 (1-\cos(\theta/2))}{
   \{a^2+4 [\sin(\theta/2)]^2 \, \}^{3/2}} \, .
	\end{equation}
By the substitution $z= 2 \sin (\theta/2)$,  for the first integral in the right hand side
of \eqref{int2} we have
$$
\begin{aligned}
\int_0^{2} dz \frac{z^2}{ 2[a^2+z^2]^{3/2}}  &= \frac{1}{2} \bigg[ \log( \sqrt{a^2+z^2}+ z) - \frac{z}{\sqrt{a^2+z^2}} \bigg]_{z=0}^{z=2}  \\
 &= -(a^2+4)^{-1/2} +\frac{1}{2} \log(2+\sqrt{a^2+4}) - \frac{1}{2} \log a \, .
 \end{aligned}
$$
We deduce that for $a \rightarrow 0$ this quantity is equal to $- \frac{1}{2} \log a $   plus a bounded rest, while for $a \rightarrow \infty$ it behaves like $a^{-1}$.

\noindent For the second integral in \eqref{int2}, first of all we have
\[ \int_0^\pi d\theta \frac{2 [\sin (\theta/2)]^2 (1-\cos(\theta/2))}{
   \{a^2+4 [\sin(\theta/2)]^2 \, \}^{3/2}} \leq \frac{1}{4} \int_0^\pi d\theta \, \frac{1-\cos(\theta/2)}{\sin(\theta/2)} \]
   which is a bounded integral; on the other hand
\[ \int_0^\pi d\theta \frac{2 [\sin (\theta/2)]^2 (1-\cos(\theta/2))}{
   \{a^2+4 [\sin(\theta/2)]^2 \, \}^{3/2}} \leq \frac{2}{a^3} \int_0^\pi d\theta \sin^2(\theta/2) (1- \cos(\theta/2)) \leq C a^{-3} . \]
	These estimates for the two integrals in \eqref{int2} show that  the equality for $I_2$
	in \eqref{separ}
	holds with
$R_2(a)$ bounded by a constant for small $a$, and
   by $C a^{-1}$ for large $a$.

\noindent   We evaluate now $I_1$, first when $a<1$, by decomposing the integral as
\begin{equation} \label{int1} I_1= \int_0^\pi d\theta \frac{\cos(\theta/2)}{\{a^2+2[\sin(\theta/2)]^2 \,\}^{3/2}} +
  \int_0^\pi d\theta \frac{\cos \theta - \cos(\theta/2)}{ \{a^2+2[\sin(\theta/2)]^2\,\}^{3/2}} . \end{equation}
The first integral in the right hand side
of \eqref{int1} can be computed as before with the substitution $z= 2 \sin (\theta/2)$:
\[ \int_0^2 dz \frac{1}{[a^2+z^2]^{3/2}} =  \bigg[ \frac{z}{a^2 \sqrt{a^2+z^2}} \bigg]_{z=0}^{z=2} = \frac{2}{a^2\sqrt{a^2+4}} \]
which, for $a \rightarrow 0$, is equal to $a^{-2}$ plus a bounded rest.
The second integral in \eqref{int1} can be bounded by noticing that
\[0 \leq \cos(\theta/2) - \cos\theta \leq 1-\cos\theta \qquad \text{ for }0 \leq \theta \leq \pi\]
and hence it can be bounded by $I_2$.

\noindent We analyse now the case $a\geq1$, observing that
\[ |I_1(a)| \leq a^{-3} \int_0^\pi d\theta \, |\cos \theta | = \frac{2}{a^3}\]
and so $|R_1(a)| \leq 2 a^{-3}+ a^{-2}$.
In both cases, $a<1$ and $a\geq 1$, we have that $a \cdot R_1(a)$ is bounded, as it goes to zero like $a \log a$ when $a \rightarrow 0$, and  behaves like $a^{-1}$ when $a \rightarrow \infty$.
\end{proof}

\begin{proposition} \label{stimaD}
Consider $x,y$ such that:
\[   |x_2| \leq \frac{r_0}{2} \qquad |y_2| \leq \frac{r_0}{2}  \]
 and let $r_0 = |\log\epsilon|^{\alpha}$. Then, for $\epsilon$ small enough:
\begin{equation}
 |G(x,y)- K(x-y)| \leq \frac{C}{|\log\epsilon|^\alpha} \: \left( 1+  \log|\log\epsilon| +
 \big|\log |x-y|\,\big| \cdot \chi_{(0,1)}(|x-y|) \right) \, .
\label{prop3.2}
\end{equation}
\end{proposition}

\begin{proof}
Define $a:= |x-y|\: (r_0+x_2)^{-1/2} (r_0+y_2)^{-1/2}$.
\begin{equation}
\begin{aligned}
2\pi G_1(x, y) &=
  \int_0^{\pi} d\theta \, \frac{(r_0+y_2)(y_2-x_2 \, \cos \theta +r_0 (1-\cos \theta))}
  {(r_0+x_2)^{3/2} \, (r_0+y_2)^{3/2} \,  \{a^2+2(1-\cos\theta)\}^{3/2} }  \\
&= \frac{y_2 \cdot \big(I_1(a)+I_2(a)\big) - x_2 \cdot I_1(a) + r_0 \cdot I_2(a) }{(r_0+x_2)^{3/2} (r_0+y_2)^{1/2}} \\
&= \frac{y_2-x_2}{(r_0+x_2)^{3/2} (r_0+y_2)^{1/2}} \cdot I_1(a) + \frac{(r_0+y_2)^{1/2}}{(r_0+x_2)^{3/2}} \cdot I_2(a) \\
& = \sqrt{ \frac{r_0+y_2}{r_0+x_2} } \cdot \frac{y_2-x_2}{|x-y|^2} + \frac{y_2-x_2}{(r_0+x_2)^{3/2} (r_0+y_2)^{1/2}} \cdot R_1(a)\, \\
& \qquad + \sqrt{\frac{r_0+y_2}{(r_0+x_2)^3}} \cdot I_2(a) \, .
\end{aligned}
\label{2piG}
\end{equation}
Note that $\frac{y_2-x_2}{|x-y|^2}$ is the first component of $2 \pi K(x-y)$, so we subtract this quantity and estimate $|G_1-K_1|$.

Let us put $A=\sqrt{\frac{r_0+y_2}{r_0+x_2}}$, hence we have
\[ |A-1|=\frac{|A^2-1|}{|A+1|}=\bigg(\frac{|y_2-x_2|}{r_0+x_2}\bigg)(1+A)^{-1} \leq \frac{2|x-y|}{r_0} \]
where in the last inequality we have used the assumption $x_2 \geq - \frac{r_0}{2} $. Furthermore
$$
\begin{aligned}
 \sqrt{\frac{r_0+y_2}{(r_0+x_2)^3}} = & \frac{1}{(r_0+x_2)^{1/2}\, (r_0+y_2)^{1/2}} \cdot \frac{r_0+y_2}{r_0+x_2}  \\
 \leq & \frac{1}{(r_0+x_2)^{1/2}\, (r_0+y_2)^{1/2}} \cdot \frac{(r_0+x_2) + |x-y|}{r_0+x_2}  \\
 \leq & \frac{1}{(r_0+x_2)^{1/2}\, (r_0+y_2)^{1/2}} + \frac{a}{r_0+x_2} \leq \frac{2}{r_0}(1+a) \, .
 \end{aligned}
$$
Collecting these estimates  into \eqref{2piG}, and calling $D:= G-K$, we get:
$$
\begin{aligned}
2\pi |D_1(x,y)| \leq & \frac{2|x-y|\,|y_2-x_2|}{r_0 \, |x-y|^2} + \frac{a \cdot R_1(a)}{r_0+x_2} + \frac{2}{r_0}(1+a) \cdot I_2(a)  \\
& \leq \frac{2}{r_0} + \frac{2 C}{r_0} + \frac{2}{r_0} \left(C - \frac{1}{2} \, \log a \cdot \chi_{(0,1)}(a) \right) .
\end{aligned}
$$
For $a \in (0,1)$
$$
\begin{aligned} 0 \leq -\log a= &  \log(r_0+x_2)^{1/2} + \log(r_0+y_2)^{1/2} - \log |x-y|  \\
 \leq & \log r_0 + \log (3/2) +
\big|\log|x-y| \big| \cdot \chi_{(0,1)} (|x-y|)
\end{aligned}
$$
where, by assumption, $\frac{1}{2} r_0 \leq r_0 + x_2 \leq \frac{3}{2}r_0$.
Since $r_0= |\log\epsilon|^{\alpha}$  we obtain
\[ 2\pi |D_1(x,y)| \leq \frac{C}{|\log\epsilon|^\alpha}  \bigg[ 1 +  \log|\log \epsilon| +
\big|\log|x-y| \big| \cdot \chi_{(0,1)} (|x-y|) \bigg] .\]

\noindent We analyse now  the second component:
$$
\begin{aligned}
 2\pi G_2(x,y) = & \int_0^\pi \frac{(r_0+y_2)(x_1-y_1)\, \cos \theta}{(r_0+x_2)^{3/2} \, (r_0+y_2)^{3/2} \,  \{a^2+2(1-\cos\theta)\}^{3/2}} \\
 = & \frac{x_1-y_1}{(r_0+x_2)^{3/2} \, (r_0+y_2)^{1/2}} \cdot I_1(a) \\
  = & \sqrt{\frac{r_0+y_2}{r_0+x_2}} \cdot \frac{x_1-y_1}{|x-y|^2} + \frac{x_1-y_1}{(r_0+x_2)^{3/2} \, (r_0+y_2)^{1/2}} R_1(a) \, .
  \end{aligned}
$$
We proceed as before,
\[ 2 \pi |D_2(x,y)| \leq \frac{2|x-y|\,|x_1-y_1|}{r_0 \, |x-y|^2} + \frac{a \cdot R_1(a)}{r_0+x_2} \leq \frac{2}{r_0}(1+C) \, .\]
Since $|D| \leq |D_1|+|D_2|$, we get
\[ |D(x,y)| \leq \frac{C}{|\log\epsilon|^\alpha} \left(1+ \log|\log \epsilon| + \big|\log |x-y| \big| \cdot \chi_{(0,1)} (|x-y|) \right) \, ,\]
and the Proposition is thus proved.
\end{proof}
The next lemma states the conservation of the $L^1$ norm of $\omega$ and a bound on its $L^\infty$
norm.
\begin{lemma} \label{norme di omega}
Let $\omega_{i, \epsilon}$  as in  \eqref{in_data}-\eqref{t_data}. Then
\begin{equation}
\label{norme}
 \int_{\mathbb{R}^2} dx \, \omega_{i, \epsilon}(x,t) = \int_{\mathbb{R}^2} dx \,
\omega_{i, \epsilon}(x,0) = a_i
\end{equation}
and for each time $\omega_{i, \epsilon}(x, t)$ has the same sign of $\omega_{i, \epsilon}(x,0)$.
Moreover for $t \leq  \bar T $ and for small enough $\epsilon$,
\begin{equation}
\label{norme2}
 |\, \omega_{i, \epsilon}(x, t) | \leq 3 M\, \epsilon^{-\gamma} \, .
\end{equation}
\end{lemma}
\begin{proof}
Equation \eqref{norme} is a direct consequence of the conservation of $\omega/r$, after
integrating in $\mathbb{R}^3$ adopting cylindrical coordinates.
The conservation of sign is evident by the definition of $\omega_{i, \epsilon}(x,t)$.
To obtain \eqref{norme2}  we observe that, if $x \in \Lambda_{i,\epsilon}(0) := \mathrm{supp} \, \omega_{i, \epsilon}(0)$, then
$|x-z_i| \leq \epsilon$ and
\[ |r_0+x_2| = ||\log\epsilon|^\alpha + x_2 | \geq |\log\epsilon|^\alpha - |z_i|
- \epsilon \geq \frac{1}{2} |\log\epsilon|^\alpha \, . \]
Since $\phi_t(x) \in \Lambda_{i, \epsilon}(t)$, it results $|\phi_t(x)- z_i(t)| \leq \bar R$ and then
\[ |r_0+ \phi^t_2(x)| \leq |\log\epsilon|^\alpha + |z_i(t)|+ \bar R \, . \]
Moreover
\[ |z_i(t)| \leq |z_i|+ \frac{1}{2\pi}\int_0^t ds \, \sum_{j \neq i}   \frac{|a_j|}{|z_i(s)- z_j(s)|}
\leq |z_i|+ \frac{t}{2 \pi R_m} \sum_{j \neq i} |a_j| \, , \]
therefore, for $t \leq \bar T$ and for $\epsilon$ small enough,
\[ |r_0+ \phi^t_2(x)| \leq \frac{3}{2} |\log\epsilon|^\alpha \, .\]
The bound \eqref{norme2} follows from the equality
\[ \omega_{i, \epsilon}(\phi_t(x),t)= \frac{r_0+ \phi^t_2(x)}{r_0+x_2} \omega_{i, \epsilon}(x,0) \, .\]
\end{proof}

\noindent We are now able to prove that the difference $u-\widetilde{u}$ is small.
\begin{proposition} \label{u e utilde}
Let $\omega_{\epsilon}$ as in  \eqref{in_data}-\eqref{t_data}, and let $t \leq \bar T$, $x \in \Lambda_{\epsilon}(t)$.
Then, if $\epsilon$ is small enough,
\begin{equation}
|u(x,t)-\widetilde{u}(x, t)| \leq \frac{C}{|\log\epsilon|^{\alpha-1}} \, .
\label{u e utilde 2}
\end{equation}
\end{proposition}

\begin{proof}
Note that, if $x \in \Lambda_{i, \epsilon}(t)$ (for any $i$), then $|x_2| \leq r_0/2$,
as seen in the proof of Lemma \ref{norme di omega}.
We can then apply Proposition \ref{stimaD} to bound $|u(x,t)-\widetilde{u}(x, t)|$,
and the worst term to treat is
\begin{equation}
\begin{aligned}
 &\int_{|x-y| <1} dy \, \big|\log |x-y|\big| \,\, |\omega_\epsilon(y, t)|  \\
&\leq\int_{|x-y| <1} dy \, \big|\log |x-y|\big| \, \sum_{i=1}^N |\omega_{i, \epsilon}(y, t)| \, .
\end{aligned}
\label{arrang}
\end{equation}
We use a classical trick, that is a rearrangement:
we bound this integral with the one obtained by concentrating as much as possible the vorticity around the singularity of $\log |x-y| $, namely $y=x$
(and by the characteristic function we integrate only in the domain $|x-y|<1$).
Let us proceed with a fixed $i$ in the summation in \eqref{arrang}.
Since the integral of $\omega_{i, \epsilon}$ is constant in time,
and its $L^\infty$ norm is less or equal to $3 M \epsilon^{-\gamma}$,
we get the rearrangement replacing $\omega_{i, \epsilon}$ with the function equal to the constant
$3 M\epsilon^{-\gamma}$ in the disk of centre $x$ and radius $r$, and equal to zero outside this disk.
The radius $r$ is chosen such that the total mass of vorticity is $|a_i|$,
so $\pi r^2 \cdot 3 M\epsilon^{-\gamma} =|a_i|$.
We have then:
$$
\begin{aligned}
\int_{|x-y| <1} dy \, \big|\log |x-y|\big| \,\, |\omega_{i,\epsilon}(y, t)| \leq & \: \: 3 M \epsilon^{-\gamma} \int_{ \{ |x-y| \leq r \} } d y \big|\log |x-y| \big| \\
=& - 6 \pi M \epsilon^{-\gamma} \int_0^r \rho \log(\rho) d\rho \\
=&  - 6 \pi M \epsilon^{-\gamma} \bigg[ \frac{\rho^2}{2} \log \rho - \frac{\rho^2}{4} \bigg]_{\rho=0}^{\rho=r} \\
=& - 3 \pi M \epsilon^{-\gamma} r^2 \log r + \frac{3}{2} \pi M \epsilon^{-\gamma} r^2 .
\end{aligned}
$$
Since $r=\sqrt{\frac{|a_i| \epsilon^\gamma}{ 3 \pi M}}$, we get
\[ \int_{|x-y| <1} dy \, \big|\log |x-y|\big| \, \sum_{i=1}^N |\omega_{i, \epsilon}(y, t)| \leq C+ C \, |\log \epsilon| , \]
which inserted in \eqref{prop3.2} gives \eqref{u e utilde 2}.
\end{proof}
We give here a useful split of the field $F^i_\epsilon$.
\begin{lemma} \label{comesonoFeps}
Recalling the definition of $F^i_\epsilon$ given in  \eqref{effe_iep}, we can write, for $t \leq \bar T$
\[ F^i_{\epsilon}= F^i_{\epsilon, 1} + F^i_{\epsilon, 2} \]
where $F^i_{\epsilon, 1}$ is Lipschitz and bounded uniformly in $\epsilon$, and $F^i_{\epsilon, 2}$ is small, i.e.
\[\|F^i_{\epsilon, 2}\|_{L^\infty} \leq \frac{C}{|\log \epsilon|^{\alpha-1}} \, . \]
\end{lemma}
\begin{proof}
 Let us define
$$
 \begin{aligned}
  F^i_{\epsilon, 1}(x,t) = & \sum_{j \neq i} \int dy \, K(x-y) \, \omega_{j, \epsilon}(y,t)\, , \\
  F^i_{\epsilon, 2}(x,t) = & \sum_{j \neq i} \int dy \, [G(x,y) - K(x-y)] \, \omega_{j, \epsilon}(y,t)\, .
 \end{aligned}
$$
It results that $F^i_{\epsilon, 1}$ is Lipschitz and bounded uniformly in $\epsilon$ by
the properties of $K$ outside the disk $\Sigma(0|R_m/2)$  (it is Lipschitz
and bounded); in fact, if $x \in \Lambda_{i, \epsilon}(t)$ and $y \in \Lambda_{j, \epsilon}(t)$, for $i \neq j$, we have
\[ |x-y| \geq |z_i(t)-z_j(t)|- |x-z_i(t)|-|y-z_j(t)| \geq R_m -2 \bar R \geq \frac{R_m}{2} \]
for $4 \bar R < R_m$. The smallness of $F^i_{\epsilon, 2}$ is achieved by Proposition \ref{u e utilde}.
\end{proof}

We define now the center of vorticity and the moment of inertia,
\begin{equation}
 \begin{aligned}
&B^i_\epsilon(t):= a_i^{-1} \, \int_{\mathbb{R}^2} dx \, x \, \omega_{i,\epsilon}(x,t)  \\
&I^i_\epsilon(t):= \int_{\mathbb{R}^2} dx \, |x-B^i_\epsilon(t)|^2 |\omega_{i, \epsilon}(x,t)|
\end{aligned}
\label{cen_vort}
\end{equation}
whose properties will be exploited in the following.

In the next lemmas we omit for simplicity the index $i$ from the notation and we assume, without lost of generality, $a_i=1$.
This is equivalent to consider a ``reduced system" with only one vortex moving in an external field acting on it, which has the properties stated in Lemma \ref{comesonoFeps}.
In this case it is easily  verified that the following equation holds:
\begin{equation} \label{eqderivaterid} \frac{\mathrm{d}}{\mathrm{d}t} \omega_t(f) = \omega_t \big[ ((u+F_\epsilon)\cdot \nabla) f + \partial_t f \big] \, . \end{equation}
The results we will prove hold obviously for each $i$.

\begin{lemma} \label{lem:Ieps Beps}
For $t \leq \bar T$ and for small enough $\epsilon$,
\begin{equation} \label{Iepsilon} I_\epsilon(t) \leq \frac{C}{|\log\epsilon|^{2(\alpha-1)}} \,.
 \end{equation}
\end{lemma}
\begin{proof}
We estimate the derivative of $I_\epsilon(t)$, using \eqref{eqderivaterid}:
$$
\begin{aligned}
 \frac{\mathrm{d}}{\mathrm{d}t} I_\epsilon(t) = &
 \int dx \, \omega_\epsilon(x, t) \, \left[ (u+F_\epsilon)\cdot 2(x-B_\epsilon(t)) - \dot{B}_\epsilon(t) \cdot 2 (x-B_\epsilon(t)) \right] .
 \end{aligned}
$$
 Moreover
 \[ \frac{\mathrm{d}}{\mathrm{d}t} B_\epsilon(t) = \int dx \, \omega_\epsilon(x,t) \, (u(x,t)+F_\epsilon(x,t)) , \]
 then
$$
\begin{aligned}
 \frac{\mathrm{d}}{\mathrm{d}t} I_\epsilon(t) = &
\,  2 \int dx \, \omega_\epsilon(x,t) \left[ u(x,t)-\int dy \, \omega_\epsilon(y,t) \, u(y,t) \right] \cdot (x-B_\epsilon(t))  \\
 & \: + 2 \int dx \, \omega_\epsilon(x,t) \left[ F_\epsilon(x,t)-\int dy \, \omega_\epsilon(y,t) \, F_\epsilon(y,t) \right] \cdot (x-B_\epsilon(t)) .
 \end{aligned}
$$
Consider first  the term containing $F_\epsilon$: we note that, by the definition of $B_\epsilon(t)$,
\[ \int dx\, \omega_\epsilon(x,t) (x-B_\epsilon(t)) \cdot \int dy \, \omega_\epsilon(y,t) F_\epsilon (y,t) =0 \]
\[ \int dx\, \omega_\epsilon(x,t) (x-B_\epsilon(t)) \cdot F_{ \epsilon,1}(B_\epsilon(t), t) =0 \, . \]
We then obtain:
$$
\begin{aligned}
& 2 \left| \,\int dx \, \omega_\epsilon(x,t) \left[ F_\epsilon (x,t)-\int_{\mathbb{R}^2} dy
\, \omega_\epsilon(y,t) \, F_\epsilon (y,t) \right] \cdot (x-B_\epsilon(t)) \, \right| \\
 & = 2 \left| \, \int dx \, \omega_\epsilon(x,t)
 \left[ F_{\epsilon, 1}(x,t)- F_{\epsilon, 1} (B_\epsilon(t),t) \right]
 \cdot (x-B_\epsilon(t)) \, \right| \\
 & \quad + 2 \left| \int dx \, \omega_\epsilon(x,t) \, F_{\epsilon, 2}(x, t) \cdot (x-B_\epsilon(t)) \, \right| \\
 & \leq 2\int dx \, \omega_\epsilon(x,t)\,  L |x-B_\epsilon(t)|^2 + \frac{C}{|\log\epsilon|^{\alpha-1}} \int dx \, |x-B_\epsilon(t)|\, \omega_\epsilon(x,t)  \\
 & \leq 2L \, I_\epsilon(t) + \frac{C}{|\log\epsilon|^{\alpha-1}} [I_\epsilon(t)]^{1/2}
 \end{aligned}
$$
where, in the last line, we used Cauchy-Schwarz inequality, and  $L$ is the Lipschitz constant
of $F_{\epsilon, 1}$.

\noindent For  the term containing $u$, we have analogously:
 \[ \int dx\, \omega_\epsilon(x,t) (x-B_\epsilon(t)) \cdot \int dy \, \omega_\epsilon(y,t) u(y,t) =0 .\]
Moreover, by the  antisymmetry of $K$,
\begin{equation}
 \int dx \, \omega_\epsilon(x,t) \, \widetilde{u}(x,t) = \int dx \int dy \: \omega_\epsilon(x,t)\,\omega_\epsilon(y, t) \, K(x-y) = 0
\label{antisym}
\end{equation}
and recalling that, by definition, $(x-y) \cdot K(x-y) =0$, we get
$$
\begin{aligned} \int dx \, \omega_\epsilon(x,t) \, x \cdot \widetilde{u}(x,t) = &
\int dx \int dy \, \omega_\epsilon(x,t)\,\omega_\epsilon(y, t) \, x \cdot K(x-y) \\
=& \int dx \int dy \, \omega_\epsilon(x,t)\,\omega_\epsilon(y, t) \, y \cdot K(x-y) \end{aligned}
$$
hence this integral is zero as well, by the antisymmetry of $K$.
Using Proposition \ref{u e utilde} we get then:
$$
\begin{aligned}
& 2 \left| \,\int dx \, \omega_\epsilon(x,t) \left[ u(x,t)-\int dy \, \omega_\epsilon(y,t) \, u(y,t) \right] \cdot (x-B_\epsilon(t)) \, \right|  \\
& \leq \, 2 \int dx \, \omega_\epsilon(x,t) \, \big|u(x,t)-\widetilde{u}(x,t)\big| \, |x-B_\epsilon(t)|\\\
& \leq \:  \frac{C}{|\log\epsilon|^{\alpha -1}} \int dx \, \omega_\epsilon(x,t)
\, |x-B_\epsilon(t)| \leq \: \frac{C}{|\log\epsilon|^{\alpha -1}} \,  \left[I_\epsilon(t) \right]^{1/2}
\end{aligned}
$$
where Cauchy-Schwarz inequality has been used again in the last line.

Hence we have
\[ |\dot{I}_\epsilon(t)| \leq 2L\,I_\epsilon(t) + \frac{C}{|\log\epsilon|^{\alpha -1}} \left[I_\epsilon(t) \right]^{1/2} . \]
Defining $M_\epsilon(t):= [I_\epsilon(t)]^{1/2}$, by Gronwall's inequality and using the fact  that,
by the initial data,  $I_\epsilon(0) \leq 4\epsilon^2$, we get
\begin{equation}
 M_\epsilon(t) \leq \left( 2\epsilon +
\frac{C}{2 L|\log\epsilon|^{\alpha -1}} \right) {\textnormal{e}}^{Lt}
\leq  \frac{C}{|\log\epsilon|^{\alpha -1}} \, {\textnormal{e}}^{Lt} \, .
\label{sqrtIepsilon}
\end{equation}
From the previous bound we finally obtain, recalling that $t\leq \bar T$,
\[ I_\epsilon(t) \leq \frac{C}{|\log\epsilon|^{2(\alpha -1)}} \, . \]
\end{proof}

\begin{lemma} \label{variazionemax}
Let us put
\begin{equation}
 R_t := \max \{ |x-B_\epsilon(t)|: \: x \in \Lambda_\epsilon(t) \}
\label{max_lambda}
\end{equation}
and choose $x_0 \in \Lambda_\epsilon(0)$ such that, at time $t \leq \bar T$,
\begin{equation}
 \frac34 R_t\leq|\phi_t(x_0)- B_\epsilon(t)| \leq R_t \, .
\label{erret}
\end{equation}
Then at this time $t$ the following inequality holds:
\begin{equation} \label{eq:variazonemax} \frac{\mathrm{d}}{\mathrm{d}t}|\phi_t(x_0)- B_\epsilon(t)| \leq 2LR_t + \frac{C I_\epsilon(t)}{ R_t^3}
 + \sqrt{\frac{3\, M \epsilon^{-\gamma} m_\epsilon(R_t/2, t)}{\pi} } + \frac{C}{|\log\epsilon|^{\alpha -1}}
\end{equation}
where the function $m_\epsilon$ is defined by:
\begin{equation}
 m_\epsilon(R, t) := \int_{|y-B_\epsilon(t)|>R} dy \, \omega_\epsilon(y,t) \qquad \text{for }R \in (0, +\infty) \, .
\label{funzemme}
\end{equation}
\end{lemma}

\begin{proof}
Let us put $x= \phi_t(x_0)$. We have:
$$
\begin{aligned} &\frac{\mathrm{d}}{\mathrm{d}t}|\phi_t(x_0)- B_\epsilon(t)| =
[u(x,t)+F_\epsilon(x,t)- \dot{B}_\epsilon(t)]
 \cdot \frac{x-B_\epsilon(t)}{|x-B_\epsilon(t)|} \\
 & \qquad = \left[ \int dy \, (F_\epsilon (x,t)-F_\epsilon (y,t)) \, \omega_\epsilon(y,t) \right] \cdot \frac{x-B_\epsilon(t)}{|x-B_\epsilon(t)|} \\
 & \qquad \quad + \left[ \int dy \, (u(x,t)-u(y,t)) \, \omega_\epsilon(y,t) \right] \cdot \frac{x-B_\epsilon(t)}{|x-B_\epsilon(t)|} .
\end{aligned}
$$
The term involving $F_\epsilon$ is easily bounded using Proposition \ref{comesonoFeps},
since $F_{\epsilon, 1}$ is Lipschitz and $F_{\epsilon, 2}$ is small:
\begin{equation}
\begin{aligned} \int dy \, |F(x,t)-F(y,t)| \, \omega_\epsilon(y,t) \leq &
 L \int dy \, |x-y| \, \omega_\epsilon(y,t) +  \frac{C}{|\log\epsilon|^{\alpha -1}} \\
 \leq & 2L\, R_t + \frac{C}{|\log\epsilon|^{\alpha -1}} .
\end{aligned}
\label{f1}
\end{equation}
For the second integral containing the difference of the velocity field, we split it into three terms,
recalling that \\ $\int dy\, \widetilde u(y)\, \omega_\epsilon(y) =0$:
\begin{equation}
 |u(x,t)- \widetilde{u}(x, t)| \leq \, \frac{C}{|\log\epsilon|^{\alpha -1}} ,
\label{f2}
\end{equation}
\begin{equation}
 \left|\int dy \, (u(y,t)-\widetilde{u}(y,t) )\, \omega_\epsilon(y,t) \right| \leq \,\frac{C}{|\log\epsilon|^{\alpha -1}} \, .
\label{f3}
\end{equation}
The third (non trivial)  term is
\[ \widetilde{u}(x,t) \cdot \frac{x-B_\epsilon(t)}{|x-B_\epsilon(t)|} =
\frac{x-B_\epsilon(t)}{|x-B_\epsilon(t)|} \cdot \int dy\, K(x-y) \omega_\epsilon(y,t) \, .\]
The integration domain can be decomposed into two regions: $A_1 := \Sigma\big(B_\epsilon(t) \big| R_t/2\big) $ and
$A_2:= \mathbb{R}^2 \setminus A_1\,$. We call $H_1$ and $H_2$ the resultant integrals. We follow
for $H_1$ the proof of \cite[Lemma~2.5]{butmar};
recalling \eqref{nucleoK} and the notation $x^\perp = (x_2,-x_1)$ for $x = (x_1,x_2)$, after introducing the new variables $x'=x-B_\epsilon (t)$,  $y'=y-B_\epsilon (t)$, and using that $x'\cdot (x'-y')^\perp=-x'\cdot y'^\perp$, we get,
\begin{equation}
\label{in H_11}
H_1 = \frac{1}{2\pi} \int_{|y'|\leq R_t/2}\! \rmd y'\, \frac{x'\cdot y'^\perp}{|x'||x'-y'|^2}\, \omega_\epsilon (y'+B_\epsilon (t))\;.
\end{equation}
By  definition of  center of vorticity \eqref{cen_vort}, $\int\! \rmd y'\,  y'^\perp\, \omega_\epsilon (y'+B_\epsilon (t)) = 0$, so that
\begin{equation}
\label{in H_13*}
H_1  = H_1'-H_1''\;,
\end{equation}
where
\begin{eqnarray*}
&& H_1' = \frac{1}{2\pi}  \int_{|y'|\le R_t/2}\! \rmd y'\, \frac {x'\cdot y'^\perp}{|x'|}\, \frac {y'\cdot (2x'-y')}{|x'-y'|^2 \ |x'|^2} \, \omega_\epsilon (y'+B_\epsilon (t))\;, \\ && H_1''= \frac{1}{2\pi} \int_{|y'|> R_t/2}\! \rmd y'\, \frac{x'\cdot y'^\perp}{|x'|^3}\, \omega_\epsilon
(y'+B_\epsilon (t))\;.
\end{eqnarray*}
From \eqref{erret} we have $|x'| \geq 3R_t/4 $, and hence $|y'| \le R_t/2$ implies $|x'-y'|\ge R_t/4$ and $|2x'-y'|\le |x'-y'|+|x'| \le  |x'-y'| +R_t \le 5|x'-y'|$, so that
\begin{equation*}
|H_1'|\leq \frac{C}{R_t^3}  \int_{|y'|\leq R_t/2} \! \rmd y'\, |y'|^2 \, \omega_\epsilon (y'+B_\epsilon (t)) \le \frac{C I_\epsilon (t)}{ R_t^3}\;.
\end{equation*}
To bound $H_1''$ we note that, in view of \eqref{max_lambda}, the integration is restricted to $|y'|\le R_t$, so that (using the lower bound in \eqref{erret})
\begin{equation*}
|H_1''| \le \frac{C}{R_t} \int_{|y'|> R_t/2}\! \rmd y'\, \omega_\epsilon (y'+B_\epsilon (t))\le \frac{C I_\epsilon (t)}{ R_t^3}\;,
\end{equation*}
where in the last inequality we used the Chebyshev's inequality. By \eqref{in H_13*} and the previous estimates we conclude that
\begin{equation}
\label{H_14}
|H_1| \le \frac{C I_\epsilon (t)}{ R_t^3}\;.
\end{equation}
We analyse now $H_2$. We first note that:
\[ |H_2| \leq \frac{1}{2\pi} \int_{|y-B_\epsilon(t)|>R_t/2} dy \, \frac{1}{|x-y|} \, \omega_\epsilon(y, t) \]
so we can bound $H_2$ using again rearrangement, as in the proof of Proposition \ref{u e utilde};
we bound the integral taking a vorticity concentrated, as much as possible, around the singularity of $\frac{1}{|x-y|}$.
Therefore, the rearrangement is achieved defining $\omega_\varepsilon$ equal to
$3\,M \epsilon^{-\gamma}$ for $|x-y|<r$, and equal to $0$ for $|x-y| \geq r$,
where $r$ is chosen such that $3 \, M \epsilon^{-\gamma} \cdot \pi r^2= m_\epsilon(R_t/2,t)$
(which is the ``total mass'' of $\omega_\epsilon$ in the integration domain $A_2$). Then
$$
\begin{aligned}
|H_2| \leq & \: \frac{3\, M \epsilon^{-\gamma}}{2 \pi} \int_{|z|<r} \frac{dz}{z} = 3\, M \epsilon^{-\gamma} \int_0^r \frac{\rho}{\rho} \, d\rho
= 3\, M \epsilon^{-\gamma} \, r = \sqrt{\frac{3\,M m_t(R_t/2)}{\pi \, \epsilon^\gamma}} .
\end{aligned}
$$
Collecting this last estimate, \eqref{f1}, \eqref{f2}, \eqref{f3}, and \eqref{H_14},  we obtain \eqref{eq:variazonemax}.
\end{proof}

We investigate now the behavior near to $0$ of the function
$m_\epsilon (\cdot, t)$ introduced in \eqref{funzemme}.

\begin{lemma} \label{lem:mt}
There exists $T>0$ such that,
for each $\ell>0$,
\begin{equation}
 \lim_{\epsilon \rightarrow 0} \epsilon^{-\ell}m_\epsilon\left(\frac{1}{|\log\epsilon|^k}, t\right) =0
\label{lemma_massa}
\end{equation}
for any $t\in [0, T]$ and $k =(\alpha-2)/2$.
\end{lemma}

\begin{proof}
Given $R\ge 2h>0$, let $W_{R,h}(x)$, $x\in \mathbb{R}^2$, be a non-negative smooth function, depending only on $|x|$, such that
\begin{equation}
\label{W1}
W_{R,h}(x) = \begin{cases} 1 & \text{if $|x|\le R$}, \\ 0 & \text{if $|x|\ge R+h$}, \end{cases}
\end{equation}
and, for some $C>0$,
\begin{equation}
\label{W2}
|\nabla W_{R,h}(x)| < \frac{C}{h}\,,
\end{equation}
\begin{equation}
\label{W3}
|\nabla W_{R,h}(x)-\nabla W_{R,h}(x')| < \frac{C}{h^2}\,|x-x'|\,.
\end{equation}

We define the quantity
\begin{equation}
\label{mass 1}
\mu_t(R,h) = \int\! \rmd x \, \big[1-W_{R,h}(x-B_\epsilon(t))\big]\, \omega_\epsilon (x,t)\,,
\end{equation}
which is a mollified version of $m_\epsilon$, satisfying
\begin{equation}
\label{2mass 3}
\mu_t(R,h) \le m_\epsilon(R, t) \le \mu_t(R-h,h)\,.
\end{equation}
In particular, it is enough to prove the claim with $\mu_t$ instead of $m_\epsilon$.

\noindent The convenience is that the function $\mu_t$ is differentiable (with respect to $t$); therefore we compute its derivative:
$$
\begin{aligned} \frac{\mathrm{d}}{\mathrm{d}t} \mu_t(R,h) = & -
\int dx \: \nabla W_{R,h}(x-B_\epsilon(t)) \cdot [ u(x,t)+F_\epsilon(x,t)-\dot{B}_\epsilon(t) ] \, \omega_\epsilon(x,t) \\ = & -H_3- H_4 - H_5
\end{aligned}
$$
with
$$
\begin{aligned}
 H_3= & \int dx \: \nabla W_{R, h}(x-B_\epsilon(t)) \cdot \widetilde{u}(x,t) \, \omega_\epsilon(x,t) \\
 H_4= & \int dx \: \nabla W_{R, h}(x- B_\epsilon(t)) \cdot \left[F_{\epsilon, 1}(x,t) -
\int dy\,  F_{\epsilon, 1}(y,t) \, \omega_\epsilon(y,t)\right] \, \omega_\epsilon(x,t) \\
 H_5= & \int dx \: \nabla W_{R, h}(x-B_\epsilon(t))\\
 & \quad \quad  \cdot \bigg[u(x,t)- \widetilde{u}(x,t) -\int dy \, [u(y,t)- \widetilde{u}(y,t)] \, \omega_\epsilon(y,t) \bigg] \, \omega_\epsilon(x,t) \\
+ & \int dx \: \nabla W_{R, h}(x-B_\epsilon(t)) \cdot \left[F_{\epsilon, 2}(x,t) -
\int dy \, F_{\epsilon, 2} (y, t) \, \omega_\epsilon (y, t) \right] \, \omega_\epsilon(x, t)
\end{aligned}
$$
since $\dot{B}_\epsilon(t) = \int dy \, \omega_\epsilon(y,t) [ F_\epsilon (y,t)+ u(y,t)-
\widetilde{u}(y,t)] $. We immediately observe that, thanks to Proposition \ref{u e utilde}, to Lemma \ref{comesonoFeps}
and to the fact that $\nabla W_{R, h}(z)$ is zero if $|z| \leq R$,
\begin{equation} \label{H5} |H_5| \leq \frac{C}{h} \cdot \frac{C}{|\log\epsilon|^{\alpha-1}} \cdot m_\epsilon(R,t) \, . \end{equation}
Following the proof of \cite[Proposition~3.4]{butmar2}  we find (we postpone the estimates
of $|H_3|$ and $|H_4|$ in the Appendix):
\begin{equation} \label{H3} |H_3| \leq \frac{C}{R\, h^3} I_\epsilon(t) \, m_\epsilon(R,t) \, ;
\end{equation}
\begin{equation} \label{H4} |H_4| \leq C  \left( 1 + \frac{2R}{h}  \right)  m_\epsilon(R,t)
+  \frac{C \, I_\epsilon(t)}{R^2 \, h} m_\epsilon(R,t) \, .
\end{equation}

Recalling \eqref{Iepsilon}, from estimates \eqref{H5}, \eqref{H3},  and \eqref{H4},  we have:
\begin{equation}
 \frac{\mathrm{d}}{\mathrm{d}t} \mu_t (R,h) \leq A_\epsilon(R, h) m_\epsilon(R,t)
\label{equ_mm}
\end{equation}
for any $t\leq \bar T$, where
$$
A_\epsilon(R, h)=
C
\left(\frac{1}{R \, h^3 \, |\log\epsilon|^{2(\alpha-1)}} + \frac{1}{R^2 \, h \, |\log\epsilon|^{2(\alpha-1)}}+
\frac{1}{h |\log\epsilon|^{\alpha-1}} + \frac{2R}{h} + 1 \right) .
$$
Therefore, by \eqref{2mass 3} and \eqref{equ_mm},
\begin{equation}
\mu_t (R,h) \leq \mu_0 (R,h) + A_\epsilon(R, h) \int_0^t {\mathrm{d}} s  \, \mu_s (R-h, h)
\end{equation}
for any $t\leq \bar T$ and $\epsilon$ sufficiently small.
We iterate the last inequality $n=\lfloor|\log\epsilon|\rfloor$  times (where $\lfloor a\rfloor$
denotes the integer part of the positive number $a$), from
$$
R_0 =\frac{1}{|\log\epsilon|^{k}} \qquad \textnormal{to} \qquad R_n=\frac{1}{2|\log\epsilon|^{k}} \, ,
$$
where $R_n = R_0 -n h$,  and consequently
$$
h= \frac{1}{2  n |\log\epsilon|^{k}} \, .
$$
In this range for $R$ the quantity $A_\epsilon(R, h)$ is bounded by $C |\log\epsilon |$, in fact
\begin{equation}
\frac{2 R}{h} \leq C \frac{|\log\epsilon|^{k +1}}{|\log\epsilon|^{k}} \, ,
\end{equation}
\begin{equation}
\frac{1}{h |\log\epsilon|^{\alpha -1}} \leq C \frac{|\log\epsilon|^{k +1}}{|\log\epsilon|^{\alpha -1}} \leq C |\log\epsilon|
\end{equation}
since ${k}=(\alpha-2)/2 < \alpha -1$,
\begin{equation}
\frac{1}{R^2 \, h \, |\log\epsilon|^{2(\alpha-1)}}
\leq C \frac{|\log\epsilon|^{3{k}+1}}{|\log\epsilon|^{2(\alpha -1)}} \leq C |\log\epsilon|
\end{equation}
since $3{k} < 2(\alpha -1)$,
\begin{equation}
\frac{1}{R \, h^3 \, |\log\epsilon|^{2(\alpha-1)}} \leq C
\frac{|\log\epsilon|^{4{k}+3}}{|\log\epsilon|^{2(\alpha -1)}} \leq C |\log\epsilon|
\label{qui}
\end{equation}
since $4{k}+2= 2(\alpha -1)$. Note that in this point we need to choose $k=(\alpha-2)/2$
and $\alpha >2$.

\noindent Then, for any $t\in [0, \bar T]$,   it results  $A_\epsilon(R, h) \leq C |\log\epsilon |$  and

\[
\begin{split}
\mu_t(R_0-h,h) & \le \mu_0(R_0-h,h) + \sum_{j=1}^{n-1} \mu_0(R_j,h) \frac{(C |\log\epsilon| t)^j}{j!} \\ & \quad + \frac{(C |\log\epsilon| )^{n}}{(n-1)!} \int_0^t\!{\textnormal{d}} s\,  (t-s)^{n-1}\mu_s(R_{n},h) \,.
\end{split}
\]
Since $\Lambda_\epsilon(0) \subset \Sigma(z|\epsilon)$, we can determine $\epsilon$
so small such that $\mu_0(R_j,h)=0$ for any $j=0,\ldots,n$, so that, for any $t\in [0,\bar T]$,
\begin{equation}
\label{mass 15'}
\mu_t(R_0-h,h) \le \frac{(C |\log\epsilon|)^{n}}{(n-1)!} \int_0^t\!{\textnormal{d}} s\,  (t-s)^{n-1}\mu_s(R_{n},h) \le  \frac{(C |\log\epsilon| t)^{n}}{n!}\,,
\end{equation}
where the obvious estimate $\mu_s(R_{n},h) \le 1$ has been used in the last inequality. In conclusion, using also \eqref{2mass 3},
\[
m_\epsilon(R_0, t) \le \mu_t(R_0 -h,h) \le  \left(C t\right)^{\lfloor|\log\epsilon|\rfloor} \;\; \forall\, t\in [0,\bar T] ,
\]
which implies  \eqref{lemma_massa} for $t\leq T$ and  $T$ suitably small.
\end{proof}

We are now ready to prove Theorem \ref{teoEA}

\begin{proof}[Proof of Theorem \ref{teoEA}]
With the previous results we can prove now that, for all $t\in[0,T]$,
\begin{equation}
\label{lamb}
\Lambda_\eps(t)\subseteq \Sigma \left(B_\eps (t) | \, C |\log\eps|^{-k} \right)\, .
\end{equation}
Recalling  the definiton of $R_t$ given in \eqref{max_lambda}, if at time $t\in[0,T]$
we have, for a certain $x_0\in \Lambda_\eps(0)$,
\begin{equation}
\frac34 R_t \leq |\phi_t(x_0)- B_{\eps}(t)|\leq R_t
\label{corona}
\end{equation}
then the time derivative of $|\phi_t(x_0)-B_{\eps}(t)|$ is bounded by \eqref{eq:variazonemax},
that is (considering also \eqref{Iepsilon})
\begin{equation}
\begin{split}
\frac{\mathrm{d}}{\mathrm{d}t}& |\phi_t(x_0)-B_\epsilon(t)|   \leq \\
&2L\, R_t +  \frac{C}{R_t^{3}|\log\epsilon|^{2(\alpha-1)}}  + C \sqrt{\epsilon^{-\gamma}m_\epsilon(R_t/2, t)} + \frac{C}{|\log\epsilon|^{\alpha-1}}
\end{split}
\label{max_sup}
\end{equation}
for each index $i$ of the $N$ vortices, omitted to simplify the notation.

\noindent Let $t_0$ be the first time at which  ${R_{t_0} = |\log\eps|^{-k}}$  and   ${R_{t} \geq |\log\eps|^{-k}}$   for $t\geq t_0$;
of course if such $t_0$ does not exist \eqref{lamb} is already achieved, as well as  if there are
time intervals after $t_0$ for which ${R_{t} \leq |\log\eps|^{-k}}$, \eqref{lamb} is achieved
in these time intervals.

\noindent In the worst case in which a fluid particle fulfills \eqref{corona}
in the whole time interval $[t_0, T]$, then we can bound the right hand side of
\eqref{max_sup}  in the following way,
\begin{equation}
\frac{\mathrm{d}}{\mathrm{d}t} |\phi_t(x_0)-B_\epsilon(t)|   \leq
C |\phi_t(x_0)-B_\epsilon(t)|  \, ,
\label{eq_phi}
\end{equation}
since the other terms are negligible with respect to the first one, by
Lemma \ref{lem:mt}
(which holds for $R_t\geq |\log\epsilon|^{-k}$) with $\ell>\gamma$  and by the following
$$
\frac{C}{R_t^{3}|\log\epsilon|^{2(\alpha-1)}} \leq C R_t
\qquad
\iff
\qquad
R_t \geq \frac{C}{|\log\epsilon|^\frac{\alpha -1}{2}}
$$

$$
\frac{C}{|\log\epsilon|^{\alpha-1}}\leq C R_t  \qquad
\iff
\qquad
R_t \geq \frac{C}{|\log\epsilon|^{\alpha -1}} \, ,
$$
which hold true since $R_t\geq |\log\epsilon|^{-k}$ and $k=(\alpha-2)/2$.
From \eqref{eq_phi} we immediatly obtain
\begin{equation}
|\phi_t(x_0)-B_\epsilon(t)| \leq |\phi_{t_0}(x_0)-B_\epsilon(t_0)| {\rm{e}}^{C (t-t_0)}
\leq  |\log\eps|^{-k} {\rm{e}}^{C (t-t_0)}
\end{equation}
hence $|\phi_t(x_0)-B_\epsilon(t)| \leq C |\log\eps|^{-k}$  for any fluid particle satisfying \eqref{corona}
$\forall \, t\in [t_0,T]$.
We  can now deduce the same bound for any fluid particle in $\Lambda_\eps(t)$, that is
\begin{equation} \label{suppRt}
\Lambda_{\epsilon}(t)  \subseteq \Sigma(B_\epsilon(t)| C |\log\eps|^{-k}) \, ,
\end{equation}
or equivalently $R_t \leq C |\log\eps|^{-k}$.
Suppose in fact  that for a  fluid particle, which does not satisfy  \eqref{corona} for $t=t_0$,
the quantity $|\phi_t(x_0)- B_{\eps}(t)|$ reaches at a certain
$t^*>t_0$  the value $3R_{t^*}/4$;  at this time $t^*$  the quantity $|\phi_{t^*}(x_0)- B_{\eps}(t^*)|$
is clearly bounded by the analogous quantity for a fluid particle which satisfies \eqref{corona}
for any $t \in [t_0, T]$.  If the first fluid particle for $t\geq t^*$ enters  the region  \eqref{corona}, then  $|\phi_t(x_0)- B_{\eps}(t)|$  has a
time derivative which is bounded by \eqref{eq_phi}, hence its
successive growth is controlled by $C |\log\eps|^{-k}$.

\noindent If there is not a single fluid particle which satisfies \eqref{corona} for any $t\in[t_0, T]$,
the same argument can be applied in subintervals $[t_0, t_1]$, $[t_1, t_2], \dots
,[t_n, T]$, in any of which certainly there is a fluid particle for which \eqref{corona} holds.
Therefore for \textit{any} fluid particle in $\Lambda_\eps(t)$   we have  $|\phi_t(x_0)- B_{\eps}(t)|\leq C |\log\eps|^{-k}$,
that is  $R_t\leq C |\log\eps|^{-k}$. Thus \eqref{lamb} is proved.

It remains to prove that
\begin{equation}
\label{B-z}
|B^i_\epsilon(t)-z_i(t)|\leq \frac{C}{|\log\epsilon|^{k}} \quad  {\textnormal{for each}}\,\, i \, ,
\end{equation}
since with this bound, together with \eqref{lamb},  Theorem \ref{teoEA} follows.
In this last point we reintroduce the index $i$.

\noindent We have:
$$
\begin{aligned}
\dot{B}^i_{\epsilon}(t) - \dot{z}_i(t)  = & \: a_i^{-1} \int dx \, \big(u^i(x, t)+ F^i_\epsilon(x,t) \big) \, \omega_{i, \epsilon}(x,t) \\
& \: - \sum_{j \neq i} a_j K(z_i(t) - z_j(t))\, .
\end{aligned}
$$
We add and subtract appropriate terms, then by the the splitting  of $ F^i_\epsilon$ given
in Lemma \ref{comesonoFeps}  we get:
$$
\begin{aligned}
\dot{B}_{\epsilon}^i(t) - \dot{z}_i(t)= & \:a_i^{-1} \int dx \, u^i(x,t) \, \omega_{i, \epsilon}(x, t)
+ a_i^{-1}  \int dx \, F^i_{\epsilon,2} (x, t) \, \omega_{i, \epsilon}(x, t) \\
& \: + a_i^{-1} \int dx [F^i_{\epsilon,1}(x,t) - F^i_{\epsilon,1}(B^i_{\epsilon}(t), t) ] \, \omega_{i, \epsilon} (x, t) \\
& \: + \sum_{j \neq i} \int dy \, [K(B^i_{\epsilon}(t)-y)- K(B^i_{\epsilon}(t)- B^j_{\epsilon}(t)) ] \, \omega_{j, \epsilon} (y, t) \\
& \: + \sum_{j \neq i} a_j [K(B^i_{\epsilon}(t)- B^j_{\epsilon}(t)) -K(B^i_{\epsilon}(t) -z_j(t) ) ] \\
& \: + \sum_{j \neq i} a_j [ K(B^i_{\epsilon}(t) -z_j(t) ) - K(z_i(t) - z_j(t)) ] \, .
\end{aligned}
$$
The first term on the right hand side is controlled by adding and subtracting
$\widetilde u^i(x,t)$ inside the integral, using Proposition \ref{u e utilde} and
\eqref{antisym}. For the other terms we use Lemma \ref{comesonoFeps} and the Lipschitz property of $K$ outside the disk
$\Sigma \big(0 \big| R_{\mathrm{min}}/2 \big)$ (we call $L_1$ the Lipschitz constant of $K$ in this region) obtaining
$$
\begin{aligned}
| \dot{B}^i_{\epsilon}(t) - \dot{z}_i(t) | \leq & \:  |a_i|^{-1} \frac{C}{|\log \epsilon|^{\alpha-1}}  + |a_i|^{-1} L \, I^i_{\epsilon}(t)^{1/2} \, |a_i|^{1/2} \\
& + \sum_{j \neq i} L_1 I^j_{\epsilon}(t)^{1/2} \, |a_j|^{1/2}
 + \sum_{j \neq i} |a_j| \, L_1 |B^j_{\epsilon}(t) -z_j(t) |
 \\ & \:
+ \sum_{j \neq i} |a_j| \, L_1 |B^i_{\epsilon}(t) -z_i(t) | \, .
\end{aligned}
$$
Defining $\Delta(t) := \max_{i=1, \ldots, N} |B^i_{\epsilon}(t) -z_i(t) |$, then
$$
\begin{aligned} \dot{\Delta}(t) \leq & \max_{i=1, \ldots, N} | \dot{B}^i_{\epsilon}(t) - \dot{z}_i(t) | \\
\leq & \, \frac{C}{|\log \epsilon|^{\alpha-1}} + C \sum_{j=1}^N \sqrt{I^j_{\epsilon}(t)} + 2L_1 \sum_{j=1}^N |a_j| \: \Delta(t) \, .
\end{aligned}
$$
By definition of $F^i_{\epsilon, 1}$ it can be immediatly seen that $L \geq L_1 \sum_j |a_j|$.
By integration of the previous inequality we obtain:
\[ \Delta(t) \leq  \Delta(0) {\textnormal{e}}^{2Lt} + C \int_0^t \sum_{j=1}^N \sqrt{I^j_\epsilon(s)} {\textnormal{e}}^{L(t-s)} \, ds + \frac{C}{|\log \epsilon|^{\alpha-1}} \cdot ({\textnormal{e}}^{2Lt}-1) \, . \]
Using the bound \eqref{sqrtIepsilon} we get, for $t\leq T$,
\begin{equation} \label{Deltat} \Delta(t) \leq \frac{C}{|\log \epsilon|^{\alpha-1}} \,  ,
\end{equation}
so \eqref{B-z} is achieved.
This concludes the proof of Theorem \ref{teoEA}.
\end{proof}

\section*{Appendix}
In this appendix we derive estimates \eqref{H3} and \eqref{H4}.
\begin{equation*}
\begin{split}
H_3 & = \int\! d x\, \nabla W_{R,h}(x-B_\eps(t)) \cdot \int\! d y \, K(x-y)\, \omega_\eps(y,t)\, \omega_\eps(x,t)  \\ & = \frac 12 \int\! d x \! \int\! d y\, [\nabla W_{R,h}(x-B_\eps(t)) - \nabla W_{R,h}(y-B_\eps(t))] \\ & \qquad  \cdot K(x-y) \, \omega_\eps(x,t)\,  \omega_\eps(y,t) \\
H_4 & =   \int\! d x\, \nabla W_{R,h}(x-B_\eps(t)) \cdot \int\! d y \,[F_{\eps, 1}(x,t)-F_{\eps, 1}(y,t)]\, \omega_\eps(y,t)\, \omega_\eps(x,t)\,,
\end{split}
\end{equation*}
where the second expression of $H_3$ is due to the antisymmetry of $K$.

Concerning $H_3$, we introduce the new variables $x'=x-B_\eps(t)$, $y'=y-B_\eps(t)$, define $\tilde\omega_\eps(z,t) := \omega_\eps(z+B_\eps(t),t)$, and let
\[
f(x',y') = \frac 12 \tilde\omega_\eps(x',t)\, \tilde\omega_\eps(y',t) \, [\nabla W_{R,h}(x')-\nabla W_{R,h}(y')] \cdot K(x'-y') \,,
\]
so that $H_3 = \int\! d x' \! \int\! d y'\,f(x',y')$. We observe that $f(x',y')$ is a symmetric function of $x'$ and $y'$ and that, by \eqref{W1}, a necessary condition to be different from zero is if either $|x'|\ge R$ or $|y'|\ge R$. Therefore,
\begin{equation*}
\begin{split}
H_3  &= \bigg[ \int_{|x'| > R}\! d x' \! \int\! d y' + \int\! d x' \! \int_{|y'| > R}\! d y' -  \int_{|x'| > h}\! d x' \! \int_{|y'| > R}\! d y'\bigg]f(x',y') \\ & = 2 \int_{|x'| > R}\! d x' \! \int\! d y'\,f(x',y')  -  \int_{|x'| > R}\! d x' \! \int_{|y'| > R}\! d y'\,f(x',y') \\ & = H_3' + H_3'' + H_3'''\,,
\end{split}
\end{equation*}
with
\begin{equation*}
\begin{split}
H_3' & = 2 \int_{|x'| > R}\! d x' \! \int_{|y'| \le R-h}\! d y'\,f(x',y') \,, \\ H_3''&  = 2 \int_{|x'| > R}\! d x' \! \int_{|y'| > R-h}\! d y'\,f(x',y')\,, \\ H_3''' & = -  \int_{|x'| > R}\! d x' \! \int_{|y'| > R}\! d y'\,f(x',y')\,.
\end{split}
\end{equation*}
By the assumptions on $W_{R,h}$, we have $\nabla W_{R,h}(z) = \eta_h(|z|) z/|z|$ with $\eta_h(|z|) =0$ for $|z| \le R$. In particular, $\nabla W_{R,h}(y') = 0$ for $|y'| \le R-h$, hence
\[
H_3' =  \int_{|x'| > R}\! d x' \, \tilde\omega_\eps(x',t) \eta_h(|x'|) \,\frac{x'}{|x'|} \cdot  \int_{|y'| \le R-h}\! d y'\, K(x'-y') \, \tilde\omega_\eps(y',t)\,.
\]
In view of  \eqref{W2}, $|\eta_h(|z|)| \le C/h$, so that
\begin{equation}
\label{a1'}
|H_3'| \le \frac{C}{h} m_\eps(R,t) \sup_{|x'| > R} |A_3(x')|\,,
\end{equation}
with
\[
A_3(x') = \frac{x'}{|x'|}\cdot  \int_{|y'| \le R-h}\! d y'\, K(x'-y') \, \tilde\omega_\eps(y',t) \,.
\]
Now, recalling \eqref{nucleoK} and using that $x'\cdot (x'-y')^\perp=-x'\cdot y'^\perp$, we get,
\begin{equation}
\label{in H_11}
A_3(x') = \frac{1}{2\pi} \int_{|y'|\leq R-h}\!  d y'\, \frac{x'\cdot y'^\perp}{|x'||x'-y'|^2}\,  \tilde\omega_\eps(y',t) \,.
\end{equation}
By \eqref{cen_vort}, $\int\! d y'\,  y'^\perp\,  \tilde\omega_\eps(y',t) = 0$, so that
\begin{equation}
\label{in H_13}
A_3(x')  = A_3'(x')-A_3''(x')\,,
\end{equation}
where
\begin{eqnarray*}
	&& A_3'(x') = \frac{1}{2\pi}  \int_{|y'|\le R-h}\! d y'\, \frac {x'\cdot y'^\perp}{|x'|}\, \frac {y'\cdot (2x'-y')}{|x'-y'|^2 \ |x'|^2} \,  \tilde\omega_\eps(y',t) \,, \\ && A_3''(x')= \frac{1}{2\pi} \int_{|y'|> R-h}\! d y'\, \frac{x'\cdot y'^\perp}{|x'|^3}\,  \tilde\omega_\eps(y',t) \,.
\end{eqnarray*}
We notice that if $|x'| > R$ then $|y'| \le R-h$ implies $|x'-y'|\ge h$ and $|2x'-y'|\le |x'-y'|+|x'|$. Therefore, for any $|x'| > R \ge 2h$,
\[
\begin{split}
|A_3'(x')|& \le \frac{1}{2\pi}\bigg[\frac{1}{|x'|^2h} + \frac{1}{|x'|h^2} \bigg]  \int_{|y'|\leq R-h} \! d y'\, |y'|^2 \,  \tilde\omega_\eps(y',t) \\ & \le \frac{I_\eps(t)}{2\pi}\bigg[\frac{1}{R^2h} + \frac{1}{Rh^2}\bigg] \le \frac{3I_\eps(t)}{4\pi Rh^2}\,.
\end{split}
\]
To bound $A_3''(x')$, by Chebyshev's inequality, for any $|x'| > R \ge 2h$ we have,
\[
|A_3''(x)| \le \frac{1}{2\pi |x'|^2} \int_{|y'|> R-h}\! d y'\, |y'| \tilde\omega_\eps(y',t) \le \frac{I_\eps(t)}{2\pi R^2(R-h)} \le \frac{I_\eps(t)}{2 \pi R^2h} \,.
\]
From Eqs.~\eqref{a1'} and \eqref{in H_13}, the previous estimates, and $R\ge 2h$, we conclude that
\begin{equation}
\label{H_14b}
|H_3'| \le \frac{5C \, I_\eps(t)}{4\pi Rh^3} m_\eps(R,t)\,.
\end{equation}

Now, by \eqref{W3} and then applying the Chebyshev's inequality and again $R\ge 2h$,
\begin{equation*}
\begin{split}
|H_3''| + |H_3'''| & \le \frac{C}{\pi h^2} \int_{|x'| \ge R}\! d x' \! \int_{|y'| \ge R-h}\! d y'\,\tilde\omega_\eps(y',t) \,  \tilde\omega_\eps(x',t)  \\ & = \frac{C}{\pi h^2}m_\eps(R,t)   \int_{|y'| \ge R-h}\! d y'\, \tilde\omega_\eps(y',t)  \le \frac{4C \, I_\eps(t)}{\pi R^2h^2} m_\eps(R,t)\,.
\end{split}
\end{equation*}
In conclusion, recalling $R\ge 2h$,
\begin{equation}
\label{a1s}
|H_3| \le  \frac{13 C \, I_\eps(t)}{4\pi R h^3} m_\eps(R,t)\,.
\end{equation}

Concerning $H_4$, we observe that by \eqref{W1} the integrand is different from zero only if $R\le |x-B_\eps(t)|\le R+h$. Therefore, by Lemma \ref{comesonoFeps} and \eqref{W2} we have,  using again the variables $x'=x-B_\eps(t)$, $y'=y-B_\eps(t)$,
\[
\begin{split}
|H_4| & \le \frac{C}{h} \int_{|x'|\ge R}\! d x'  \tilde\omega_\eps(x',t)  \int_{|y'|> R}\! d y'\, \tilde\omega_\eps(y',t) \\ &\quad + \frac{C }{h} \int_{R \le |x'|\le R+h}\! d x'  \tilde\omega_\eps(x',t) \int_{|y'| \le R}\! d y'\,|x'- y'| \,  \tilde\omega_\eps(y',t) \,.
\end{split}
\]
Since $|x'-y'| \le 2R+h$ in the domain of integration of the last integral and using the Chebyshev's inequality in the first one we get,
\begin{equation}
\label{a2s}
|H_4| \le \frac{C \, I_\eps(t)}{R^2h} m_\eps(R,t) +  C  \bigg(1+\frac{2R}h\bigg) m_\eps(R,t)\,.
\end{equation}

\bigskip

\bigskip

\bigskip

\bigskip
\noindent {\textbf{Acknowledgments}}.
Work performed under the auspices of GNFM-INDAM and the Italian Ministry
of the University (MIUR).

\medskip

\bigskip


\section{Introduction}
\label{sec:intro}
Recent advancements in self-supervised pretraining techniques drastically changed the way we design Natural Language Processing (NLP) systems. Even though, pretraining has been present in NLP for many years \citep{mikolov2013distributed, pennington-etal-2014-glove, bojanowski-etal-2017-enriching}, only recently we observed a shift from task-specific to general-purpose models. In particular, the BERT model \citep{devlin2019bert} proved to be a dominant architecture and obtained state-of-the-art results for a variety of NLP tasks.

While most of the research related to analyzing and improving BERT-based models focuses on English, there is an increasing body of work aimed at training and evaluation of models for other languages, including Polish. Thus far, a handful of models specific for Polish has been released, e.g. Polbert\footnote{\url{https://github.com/kldarek/polbert}}, first version of HerBERT \citep{rybak-etal-2020-klej}, and Polish RoBERTa \citep{dadas2020pretraining}.

Aforementioned works lack ablation studies, making it difficult to attribute hyperparameters choices to models performance. In this work, we fill this gap by conducting an extensive set of experiments and developing an efficient BERT training procedure. As a result, we were able to train and release a new BERT-based model for Polish language understanding. Our model establishes a new state-of-the-art on the variety of downstream tasks including semantic relatedness, question answering, sentiment analysis and part-of-speech tagging.

To summarize, our contributions are:
\begin{enumerate}
    \item development and evaluation of an efficient pretraining procedure for transferring knowledge from multilingual to monolingual language models based on work by \citet{arkhipov-etal-2019-slavic},
    \item detailed analysis and an ablation study challenging the effectiveness of Sentence Structural Objective (SSO,  \citealp{structbert2020iclr}), and Byte Pair Encoding Dropout (BPE-Dropout,  \citealp{bpedropout2020provilkov}),
    \item release of HerBERT\footnote{\url{https://huggingface.co/allegro/herbert-large-cased}} -- a BERT-based model for Polish language understanding, which achieves state-of-the-art results on KLEJ Benchmark \citep{rybak-etal-2020-klej} and POS tagging task \citep{wroblewska-2020-towards}.
\end{enumerate}

The rest of the paper is organized as follows. In Section \ref{sec:related}, we provide an overview of related work. After that, Section \ref{sec:language_model} introduces the BERT-based language model and experimental setup used in this work. In Section \ref{sec:ablation}, we conduct a thorough ablation study to investigate the impact of several design choices on the performance of downstream tasks. Next, in Section \ref{sec:herbert} we apply drawn conclusions and describe the training of HerBERT model. In Section \ref{sec:eval}, we evaluate HerBERT on a set of eleven tasks and compare its performance to other state-of-the-art models. Finally, we conclude our work in Section \ref{sec:conclusion}.

\section{Related Work}
\label{sec:related}
The first significant ablation study of BERT-based language pretraining was described by \citet{liu2019roberta}. Authors demonstrated the ineffectiveness of Next Sentence Prediction (NSP) objective, the importance of dynamic token masking, and gains from using both large batch size and large training dataset. Further large-scale studies analyzed the relation between the model and the training dataset sizes \citep{kaplan2020scaling}, the amount of compute used for training \citep{DBLP:journals/corr/abs-2005-14165-gpt3} and training strategies and objectives \citep{DBLP:journals/corr/abs-1910-10683-T5}.

Other work focused on studying and improving BERT training objectives. As mentioned before, the NSP objective was either removed \citep{liu2019roberta} or enhanced either by predicting the correct order of sentences (Sentence Order Prediction (SOP), \citealp{albert2020iclr}) or discriminating between previous, next and random sentence (Sentence Structural Objective (SSO), \citealp{structbert2020iclr}). Similarly, the Masked Language Modelling (MLM) objective was extended to either predict spans of tokens \citep{joshi2019spanbert}, re-order shuffled tokens (Word Structural Objective (WSO),  \citealp{structbert2020iclr}) or replaced altogether with a binary classification problem using mask generation \citep{DBLP:conf/iclr/ClarkLLM20-electra}.

For tokenization, the Byte Pair Encoding algorithm (BPE, \citealp{sennrich2016bpe}) is commonly used. The original BERT model used WordPiece implementation \citep{schuster2012japanese}, which was later replaced by SentencePiece \citep{kudo-richardson-2018-sentencepiece}. \citet{NIPS2018_7408-frage} discovered that rare words lack semantic meaning. \citet{bpedropout2020provilkov} proposed a BPE-Dropout technique to solve this issue.

All of the above work was conducted for English language understanding. There was little research into understanding how different pretraining techniques affect BERT-based models for other languages. The main research focus was to train BERT-based models and report their performance on downstream tasks. The first such models were released for German\footnote{\url{https://deepset.ai/german-bert}} and Chinese \citep{devlin2019bert}, recently followed by Finnish \citep{virtanen2019multilingual},
French \citep{martin-etal-2020-camembert,le-etal-2020-flaubert},
Polish \citep{rybak-etal-2020-klej, dadas2020pretraining},
Russian \citep{kuratov2019adaptation},
and many other languages\footnote{\url{https://huggingface.co/models}}.
Research on developing and investigating an efficient procedure of pretraining BERT-based models was rather neglected in these languages.

Language understanding for low-resource languages has also been addressed by training jointly for several languages at the same time. That approach improves performance for moderate and low-resource languages as showed by \citet{NIPS2019_8928_XLM}. The first model of this kind was the multilingual BERT trained for 104 languages \citep{devlin2019bert} followed by
\citet{NIPS2019_8928_XLM} and \citet{conneau-etal-2020-xlmr}.

\section{Experimental Setup}
\label{sec:language_model}
In this section, we describe the experimental setup used in the ablation study. First, we introduce the corpora we used to train models. Then, we give an overview of the language model architecture and training procedure. In particular, we describe the method of transferring knowledge from multilingual to monolingual BERT-based models. Finally, we present the evaluation tasks.

\subsection{Training Data}
\label{subsec:data}
We gathered six corpora to create two datasets on which we trained HerBERT. The first dataset (henceforth called \emph{Small}) consists of corpora of the highest quality, i.e. NKJP, Wikipedia, and Wolne Lektury. The second dataset (\emph{Large}) is over five times larger as it additionally contains texts of lower quality (CCNet and Open Subtitles). Below, we present a short description of each corpus. Additionally, we include the basic corpora statistics in Table \ref{tab:corpora}.

\begin{table}[!ht]
\renewcommand*{\arraystretch}{1.15}
\setlength{\tabcolsep}{0.38em}
\centering
\begin{tabular}{lrrr}

    \toprule
    \bf{Corpus} & \bf{Tokens} & \bf{Documents} & \bf{Avg len} \\
    \toprule
    \multicolumn{4}{c}{\bf{Source Corpora}} \\
    \midrule
    NKJP & 1357M & 3.9M & 347 \\
    Wikipedia & 260M & 1.4M & 190 \\
    Wolne Lektury & 41M & 5.5k & 7447 \\
    \midrule
    CCNet Head & 2641M & 7.0M & 379 \\
    CCNet Middle & 3243M & 7.9M & 409 \\
    Open Subtitles & 1056M & 1.1M & 961 \\
    \midrule
    \multicolumn{4}{c}{\bf{Final Corpora}} \\
    \midrule
    Small & 1658M & 5.3M & 313 \\
    Large & 8599M & 21.3M & 404 \\
    \bottomrule

\end{tabular}
\caption{Overview of all data sources used to train HerBERT. We combine them into two corpora. The \emph{Small} corpus consists of the highest quality text resources: NKJP, Wikipedia, and Wolne Lektury. The \emph{Large} corpus consists of all sources. \textit{Avg len} is the average number of tokens per document in each corpus.}
\label{tab:corpora}
\end{table}

\paragraph{NKJP} (Narodowy Korpus Języka Polskiego, eng. \emph{National Corpus of Polish}) \citep{przepiorkowski2012narodowy} is a well balanced collection of Polish texts. It consists of texts from many different sources, such as classic literature, books, newspapers, journals, transcripts of conversations, and texts crawled from the internet.

\paragraph{Wikipedia} is an online encyclopedia created by the community of Internet users. Even though it is crowd-sourced, it is recognized as a high-quality collection of articles.

\paragraph{Wolne Lektury} (eng. \emph{Free Readings})\footnote{\url{https://wolnelektury.pl}} is a collection of over five thousand books and poems, mostly from 19th and 20th century, which have already fallen in the public domain.

\paragraph{CCNet}\citep{wenzek-etal-2020-ccnet} is a clean monolingual corpus extracted from Common Crawl\footnote{\url{http://commoncrawl.org/}} dataset of crawled websites.

\paragraph{Open Subtitles} is a popular website offering movie and TV subtitles, which was used by \citet{LISON16.947} to curate and release a multilingual parallel corpus from which we extracted its monolingual Polish part.

\subsection{Language Model}
\label{subsec:model}

\paragraph{Tokenizer}
We used Byte-Pair Encoding (BPE) tokenizer \citep{sennrich2016bpe} with the vocabulary size of 50k tokens and trained it on the most representative parts of our corpus, i.e annotated subset of the NKJP, and the Wikipedia.

Subword regularization is supposed to emphasize the semantic meaning of tokens \citep{NIPS2018_7408-frage,bpedropout2020provilkov}. To verify its impact on training language model we used a BPE-Dropout \citep{bpedropout2020provilkov} with a probability of dropping a merge equal to 10\%.

\paragraph{Architecture}

We followed the original BERT \citep{devlin2019bert} architectures for both BASE (12 layers, 12 attention heads and hidden dimension of 768) and LARGE (24 layers, 16 attention heads and hidden dimension of 1024) variants.

\paragraph{Initialization}
We initialized models either randomly or by using weights from XLM-RoBERTa \citep{conneau-etal-2020-xlmr}. In the latter case, the parameters for all layers except word embeddings and token type embeddings were copied directly from the source model. Since XLM-RoBERTa does not use the NSP objective and does not have the token type embeddings, we took them from the original BERT model.

To overcome the difference in tokenizers vocabularies we used a method similar to \citep{arkhipov-etal-2019-slavic}. If a token from the target model vocabulary was present in the source model vocabulary then we directly copied its weights.

Otherwise, it was split into smaller units and the embedding was obtained by averaging sub-tokens embeddings.

\paragraph{Training Objectives}
\label{par:training_objective}
We trained all models with an updated version of the MLM objective \citep{joshi2019spanbert, martin-etal-2020-camembert}, masking ranges of subsequent tokens belonging to single words instead of individual (possibly subword) tokens. We replaced the NSP objective with SSO. The other parameters were kept the same as in the original BERT paper. Training objective is defined in Equation \ref{eq:loss}.

\begin{equation}
\label{eq:loss}
\mathcal{L} = \mathcal{L}_{\mathrm{MLM}}(\theta) + \alpha\cdot\mathcal{L}_{\mathrm{SSO}}(\theta)
\end{equation}

where $\alpha$ is the SSO weight.

\subsection{Tasks}
\label{subsec:tasks}

\paragraph{KLEJ Benchmark}
\label{par:klej_benchmark}
The standard method for evaluating pretrained language models is to use a diverse collection of tasks grouped into a single benchmark. Such benchmarks exist in many languages, e.g. English (GLUE, \citealp{wang2019glue}), Chinese (CLUE, \citealp{xu2020clue}), and Polish (KLEJ, \citealp{rybak-etal-2020-klej}).

Following this paradigm we first verified the quality of assessed models with KLEJ. It consists of nine tasks: name entity classification (\textbf{NKJP-NER}, \citealp{przepiorkowski2012narodowy}), semantic relatedness (\textbf{CDSC-R}, \citealp{wroblewska2017polish}), natural language inference (\textbf{CDSC-E}, \citealp{wroblewska2017polish}), cyberbullying detection (\textbf{CBD}, \citealp{ptaszynski2019results}), sentiment analysis (\textbf{PolEmo2.0-IN}, \textbf{PolEmo2.0-OUT}, \citealp{kocon-etal-2019-multi}, \textbf{AR}, \citealp{rybak-etal-2020-klej}), question answering (\textbf{Czy wiesz?}, \citealp{marcinczuk2013open}), and text similarity (\textbf{PSC}, \citealp{ogro:kop:14:lrec}).

\paragraph{POS Tagging and Dependency Parsing}
All of the KLEJ Benchmark tasks belong to the classification or regression type. It is therefore difficult to assess the quality of individual token embeddings. To address this issue, we further evaluated HerBERT on part-of-speech tagging and dependency parsing tasks.

For tagging, we used the manually annotated subset of NKJP \cite{1mln_nkjp}, converted to the CoNLL-U format by \citet{wroblewska-2020-towards}.
We evaluated models performance on a test set using accuracy and F1-Score.

For dependency parsing, we applied Polish Dependency Bank \citep{pl} from the Universal Dependencies repository (release 2.5, \citealp{ud25data}).

In addition to three Transformer-based models, we also included models trained with static embeddings. The first one did not use pretrained embeddings while the latter utilized fastText \citep{bojanowski-etal-2017-enriching} embeddings trained on Common Crawl.

The models are evaluated with the standard metrics: UAS (unlabeled attachment score) and LAS (labelled attachment score). The gold-standard segmentation was preserved. We report the results on the test set.

\section{Ablation Study}
\label{sec:ablation}
In this section, we analyze the impact of several design choices on downstream task performance of Polish BERT-based models. In particular, we focus on initialization, corpus size, training objective, BPE-Dropout, and the length of pretraining.

\subsection{Experimental Design}
\paragraph{Hyperparameters}
Unless stated otherwise, in all experiments we trained BERT\textsubscript{BASE} model initialized with XLM-RoBERTa weights for 10k iterations using a linear decay schedule of the learning rate with a peak value of $7 \cdot 10^{-4}$ and a warm-up of 500 iterations. We used a batch size of 2560.

\paragraph{Evaluation}
Different experimental setups were compared using the average score on the KLEJ Benchmark. The validation sets are used for evaluation and we report the results corresponding to the median values of the five runs. Since only six tasks in KLEJ Benchmark have validation sets the scores are not directly comparable to those reported in Section \ref{sec:eval}. We used Welch's t-test \citep{10.1093/biomet/34.1-2.28-welch} with a p-value of 0.01 to test for statistical differences between experimental variants.

\subsection{Results}

\paragraph{Initialization}
\label{subsec:init-study}
One of the main goals of this work is to propose an efficient strategy to train a monolingual language model. We began with investigating the impact of pretraining the language model itself. For this purpose, the following experiments were designed.

\begin{table}[!h]
\renewcommand*{\arraystretch}{1.15}
\centering
\begin{tabular}{l|llr}

    \toprule
    \bf{Init} & \bf{Pretraining} & \bf{BPE} & \bf{Score} \\
    \midrule
    \multicolumn{4}{c}{\bf{Ablation Models}} \\
    \midrule
    Random & No & - & 58.15 $\pm$ 0.33 \\
    XLM-R & No & - & \underline{83.15 $\pm$ 1.22} \\
    \midrule
    Random & Yes & No & 85.65 $\pm$ 0.43 \\
    XLM-R & Yes & No & \underline{88.80 $\pm$ 0.15} \\
    \midrule
    Random & Yes & Yes & 85.78 $\pm$ 0.23 \\
    XLM-R & Yes & Yes & \bf{\underline{89.10 $\pm$ 0.19}} \\
    \midrule
    \multicolumn{4}{c}{\bf{Original Models}} \\
    \midrule
    XLM-R & - & - & 88.82 $\pm$ 0.15 \\
    \bottomrule

\end{tabular}
\caption{Average scores on KLEJ Benchmark depending on the initialization scheme: Random -- initialization with random weights, XLM-R -- initialization with XLM-RoBERTa weights. We used BERT\textsubscript{BASE} model trained for 10k iterations with the SSO weight equal to $1.0$ on the \emph{Large} corpus. The best score within each group is underlined, the best overall is bold.}
\label{tab:init-study}
\end{table}

First, we fine-tuned randomly initialized BERT model on KLEJ Benchmark tasks. Note that this model is not pretrained in any way. As expected, the results on the KLEJ Benchmark are really poor with the average score equal to 58.15.

Next, we evaluated the BERT model initialized with XLM-RoBERTa weights (see Table \ref{tab:init-study}). It achieved much better average score than the randomly initialized model (83.15 vs 58.15), but it was still not as good as the original XLM-RoBERTa model (88.82\%). The difference in the performance can be explained by the transfer efficiency. The method of transferring token embeddings between different tokenizers proves to retain most information, but not all of it.

To measure the impact of initialization on pretraining optimization, we trained the aforementioned models for 10k iterations. Beside MLM objective, we used SSO loss with $\alpha=1.0$ and conducted experiments with both enabled and disabled BPE-Dropout. Models initialized with XLM-RoBERTa achieve significantly higher results than models initialized randomly, 89.10 vs 85.78 and 88.80 vs 85.65 for pretraining with and without BPE-Dropout respectively.

Models initialized with XLM-RoBERTa achieved similar results to the original XLM-RoBERTa (the differences are not statistically significant). It proves that it is possible to quickly recover from the performance drop caused by a tokenizer conversion procedure and obtain a much better model than the one initialized randomly.

\paragraph{Corpus Size}
\label{subsec:corpus-size}
As mentioned in Section \ref{sec:related}, previous research show that pretraining on a larger corpus is beneficial for downstream task performance \citep{kaplan2020scaling, DBLP:journals/corr/abs-2005-14165-gpt3}. We investigated this by pretraining BERT\textsubscript{BASE} model on both \emph{Small} and \emph{Large} corpora (see Section \ref{subsec:data}). To mitigate a possible impact of confounding variable, we also vary the weight of SSO loss and usage of BPE-Dropout (see Table \ref{tab:corpus-size-study}).

\begin{table}[!ht]
\renewcommand*{\arraystretch}{1.15}
\centering
\begin{tabular}{l|llr}

    \toprule
    \bf{Corpus} & \bf{SSO} & \bf{BPE} & \bf{Score} \\
    \midrule
    Small & 1.0 & Yes & 88.73 $\pm$ 0.08 \\
    Large & 1.0 & Yes & \underline{89.10 $\pm$ 0.19} \\
    \midrule
    Small & 0.1 & Yes & 88.90 $\pm$ 0.24 \\
    Large & 0.1 & Yes & \bf{\underline{89.37 $\pm$ 0.25}} \\
    \midrule
    Small & 0.0 & Yes & 89.18 $\pm$ 0.15 \\
    Large & 0.0 & Yes & \underline{89.25 $\pm$ 0.21} \\
    \midrule
    Small & 0.0 & No & 89.12 $\pm$ 0.29 \\
    Large & 0.0 & No & \underline{89.28 $\pm$ 0.26} \\
    \bottomrule

\end{tabular}
\caption{Average scores on KLEJ Benchmark depending on a corpus size. We used BERT\textsubscript{BASE} model trained for 10k iterations with or without BPE-Dropout and with various SSO weights. The best score within each group is underlined, the best overall is bold.}
\label{tab:corpus-size-study}
\end{table}

As expected, the model pretrained on a \emph{Large} corpus performs better on downstream tasks. However, the difference is statistically significant only for the experiment with SSO weight equal to 1.0 and BPE-Dropout enabled. Therefore it's not obvious whether a larger corpus is actually beneficial.

\paragraph{Sentence Structural Objective}
\label{subsec:sso}
Subsequently, we tested SSO, i.e. the recently introduced replacement for the NSP objective, which proved to be ineffective. We compared models trained with three values of SSO weight $\alpha$ (see Section \ref{par:training_objective}): 0.0 (no SSO), 0.1 (small impact of SSO), and 1.0 (SSO equally important as MLM objective) (see Table \ref{tab:sso-study}).

The experiment showed, that SSO actually hurts downstream task performance. The differences between enabled and disabled SSO are statistically significant for two out of three experimental setups. The only scenario for which the negative effect of SSO is not statistically significant is using the Large corpus and BPE-dropout. Overall, the best results are achieved using a small SSO weight but the differences are not significantly different from disabling SSO.

\begin{table}[!ht]
\renewcommand*{\arraystretch}{1.15}
\centering
\begin{tabular}{l|llr}

    \toprule
    \bf{SSO} & \bf{Corpus} & \bf{BPE} & \bf{Score} \\
    \midrule
    0.0 & Small & Yes & \underline{89.18 $\pm$ 0.15} \\
    0.1 & Small & Yes & 88.90 $\pm$ 0.24 \\
    1.0 & Small & Yes & 88.73 $\pm$ 0.08 \\
    \midrule
    0.0 & Large & Yes & 89.25 $\pm$ 0.21 \\
    0.1 & Large & Yes & \underline{89.37 $\pm$ 0.25} \\
    1.0 & Large & Yes & 89.10 $\pm$ 0.19 \\
    \midrule
    0.0 & Large & No & 89.28 $\pm$ 0.26 \\
    0.1 & Large & No & \bf{\underline{89.45 $\pm$ 0.18}} \\
    1.0 & Large & No & 88.80 $\pm$ 0.15 \\
    \bottomrule

\end{tabular}
\caption{Average scores on KLEJ Benchmark depending on a SSO weight. We used BERT\textsubscript{BASE} model trained for 10k iterations with BPE-Dropout. The best score within each group is underlined, the best overall is bold.}
\label{tab:sso-study}
\end{table}

\paragraph{BPE-Dropout}
\label{subsec:dropoutbpe}
The BPE-Dropout could be beneficial for downstream task performance, but its impact is difficult to assess due to many confounding variables.

The model initialization with XLM-RoBERTa weights means that token embedding is already semantically meaningful even without additional pretraining. However, for both random and XLM-RoBERTa initialization the BPE-Dropout is beneficial.

\begin{table}[!ht]
\renewcommand*{\arraystretch}{1.15}
\setlength{\tabcolsep}{4.5pt}
\centering
\begin{tabular}{l|lllr}

    \toprule
    \bf{BPE} & \bf{Init} & \bf{SSO} & \bf{Corpus} & \bf{Score} \\
    \midrule
    No & Random & 1.0 & Large & 85.65 $\pm$ 0.43 \\
    Yes & Random & 1.0 & Large & \underline{85.78 $\pm$ 0.23} \\
    \midrule
    No & XLM-R & 1.0 & Large & 88.80 $\pm$ 0.15 \\
    Yes & XLM-R & 1.0 & Large & \underline{89.10 $\pm$ 0.19} \\
    \midrule
    No & XLM-R & 0.0 & Large & \bf{\underline{89.28 $\pm$ 0.26}} \\
    Yes & XLM-R & 0.0 & Large & 89.25 $\pm$ 0.21 \\
    \midrule
    No & XLM-R & 0.0 & Small & 89.12 $\pm$ 0.29 \\
    Yes & XLM-R & 0.0 & Small & \underline{89.18 $\pm$ 0.15} \\
    \bottomrule

\end{tabular}
\caption{Average scores on KLEJ Benchmark depending on usage of BPE-Dropout. We used BERT\textsubscript{BASE} model trained for 10k iterations on a large corpus. The best score within each group is underlined, the best overall is bold.}
\label{tab:dropoutbpe-study}
\end{table}

According to the results (see Table \ref{tab:dropoutbpe-study}), none of the reported differences is statistically significant and we can only conclude that BPE-Dropout does not influence the model performance.

\paragraph{Length of Pretraining}
\label{subsec:length}
The length of pretraining in terms of the number of iterations is commonly considered an important factor of the final quality of the model \citep{kaplan2020scaling}. Even though it seems straightforward to validate this hypothesis in practice it is not so trivial.

When pretraining Transformer-based models linear decaying learning rate is typically used. Therefore, increasing the number of training iterations changes the learning rate schedule and impacts the training. In our initial experiments usage of the same learning rate caused the longer training to collapse. Instead, we chose the learning rate for which the value of loss function after 10k steps was similar. We found that the learning rate equal to $3 \cdot 10^{-4}$ worked best for training in 50k steps.

\begin{table}[!ht]
\renewcommand*{\arraystretch}{1.15}
\centering
\begin{tabular}{ll|lr}

    \toprule
    \bf{\# Iter} & \bf{LR} & \bf{SSO} & \bf{Score} \\
    \midrule
    10k & $7 \cdot 10^{-4}$ & 1.0 & 89.10 $\pm$ 0.19 \\
    50k & $3 \cdot 10^{-4}$ & 1.0 & \underline{89.43 $\pm$ 0.10} \\
    \midrule
    10k & $7 \cdot 10^{-4}$ & 0.1 & 89.37 $\pm$ 0.25 \\
    50k & $3 \cdot 10^{-4}$ & 0.1 & \bf{\underline{89.87 $\pm$ 0.22}} \\
    \bottomrule

\end{tabular}
\caption{Average scores on KLEJ Benchmark depending training length. We used BERT\textsubscript{BASE} model trained on a large corpus with BPE-Dropout. The best score within each group is underlined, the best overall is bold.}
\label{tab:length-study}
\end{table}

Using the presented experiment setup, we tested the impact of pretraining length for two values of SSO weight: 1.0 and 0.1. In both cases, the model pretrained with more iterations achieves only slightly better but statistically significant results (see Table \ref{tab:length-study}).

\section{HerBERT}
\label{sec:herbert}
In this section, we apply conclusions drawn from the ablation study (see Section \ref{sec:ablation}) and describe the final pretraining procedure used to train HerBERT model.

\paragraph{Pretraining Procedure}
HerBERT was trained on the \emph{Large} corpus. We used Dropout-BPE in tokenizer with a probability of a drop equals to 10\%. Finally, HerBERT models were initialized with weights from XLM-RoBERTa and were trained with the objective defined in Equation \ref{eq:loss} with SSO weight equal to 0.1.

\begin{table*}
\renewcommand*{\arraystretch}{1.15}
\centering
\begin{tabular}{l|c|ccccccccc}

    \toprule
    \bf{Model} & \rot{\bf{AVG }} & \rot{\bf{NKJP-NER }} & \rot{\bf{CDSC-E }} & \rot{\bf{CDSC-R }} & \rot{\bf{CBD }} & \rot{\bf{PolEmo2.0-IN }} & \rot{\bf{PolEmo2.0-OUT }} & \rot{\bf{Czy wiesz? }} & \rot{\bf{PSC }} & \rot{\bf{AR }} \\
    \toprule
    \multicolumn{11}{c}{\bf{Base Models}} \\
    \midrule
    XLM-RoBERTa    & 84.7 $\pm$ 0.29 & 91.7 & 93.3 & 93.4 & 66.4 & \underline{90.9} & 77.1 & 64.3 & 97.6 & 87.3 \\
    Polish RoBERTa & 85.6 $\pm$ 0.29 & 94.0 & 94.2 & \underline{94.2} & 63.6 & 90.3 & 76.9 & \underline{71.6} & 98.6 & 87.4 \\
    HerBERT        & \underline{86.3 $\pm$ 0.36} & \underline{94.5} & \underline{94.5} & 94.0 & \underline{67.4} & \underline{90.9} & \underline{80.4} & 68.1 & \underline{98.9} & \underline{87.7} \\
    \midrule
    \multicolumn{11}{c}{\bf{Large Models}} \\
    \midrule
    XLM-RoBERTa    & 86.8 $\pm$ 0.30 & 94.2 & \bf{\underline{94.7}} & 93.9 & 67.6 & 92.1 & 81.6 & 70.0 & 98.3 & 88.5 \\
    Polish RoBERTa & 87.5 $\pm$ 0.29 & 94.9 & 93.4 & 94.7 & 69.3 & \bf{\underline{92.2}} & 81.4 & 74.1 & \bf{\underline{99.1}} & 88.6 \\
    HerBERT        & \bf{\underline{88.4 $\pm$ 0.19}} & \bf{\underline{96.4}} & 94.1 & \bf{\underline{94.9}} & \bf{\underline{72.0}} & \bf{\underline{92.2}} & \bf{\underline{81.8}} & \bf{\underline{75.8}} & 98.9 & \bf{\underline{89.1}} \\
    \bottomrule

\end{tabular}
\caption{Evaluation results on KLEJ Benchmark. \textit{AVG} is the average score across all tasks. Scores are reported for test set and  correspond to median values across five runs. The best scores within each group are underlined, the best overall are in bold.}
\label{tab:results-klej}
\end{table*}

\paragraph{Optimization}
We trained HerBERT\textsubscript{BASE} using Adam optimizer \citep{Kingma2015Adam} with parameters: $\beta_{1} = 0.9$, $\beta_{2} = 0.999$, $\epsilon = 10^{-8}$ and a linear decay learning rate schedule with a peak value of $3 \cdot 10^{-4}$. Due to the initial transfer of weights from the already trained model, the warm-up stage was set to a relatively small number of 500 iterations. The whole training took 50k iterations.

Training of HerBERT\textsubscript{LARGE} was longer (60k iterations) and had a more complex learning rate schedule. For the first 15k we linearly decayed the learning rate from $3 \cdot 10^{-4}$ to $2.5 \cdot 10^{-4}$. We observed the saturation of evaluation metrics and decided to drop the learning rate to $1 \cdot 10^{-4}$. After training for another 25k steps and reaching the learning rate of $7 \cdot 10^{-5}$ we again reached the plateau of evaluation metrics. In the last phase of training, we dropped the learning rate to $3 \cdot 10^{-5}$ and trained for 20k steps until it reached zero. Additionally, during the last phase of training, we disabled both BPE-Dropout and dropout within the Transformer itself as suggested by \citet{albert2020iclr}.

Both HerBERT\textsubscript{BASE} and HerBERT\textsubscript{LARGE} models were trained with a batch size of 2560.

\section{Evaluation}
\label{sec:eval}
In this section, we introduce other top-performing models for Polish language understanding and compare their performance on evaluation tasks (see Section \ref{subsec:tasks}) to HerBERT.

\subsection{Models}
\label{subsec:models}
According to the KLEJ Benchmark leaderboard\footnote{\url{https://klejbenchmark.com/leaderboard/}} the three top-performing models of Polish language understanding are XLM-RoBERTa-NKJP\footnote{XLM-RoBERTa-NKJP is XLM-RoBERTa model additionally fine-tuned on NKJP corpus.}, Polish RoBERTa, and XLM-RoBERTa. These are also the only three models available in LARGE architecture variant.

Unfortunately, the XLM-RoBERTa-NKJP model is not publicly available, so we cannot use it for our evaluation. However, it has the same average score as the runner-up (Polish RoBERTa) which we compare HerBERT with.

\subsection{Results}
\paragraph{KLEJ Benchmark}

Both variants of HerBERT achieved the best average performance, significantly outperforming Polish RoBERTa and XLM-RoBERTa (see Table \ref{tab:results-klej}). Regarding BASE models, HerBERT\textsubscript{BASE} improves the state-of-the-art result by 0.7pp and for HerBERT\textsubscript{LARGE} the improvement is even bigger (0.9pp). In particular, HerBERT\textsubscript{BASE} scores best on eight out of nine tasks (tying on PolEmo2.0-OUT, performing slightly worse on CDSC-R) and HerBERT\textsubscript{LARGE} in seven out of nine tasks (tying in PolEmo2.0-IN, performing worse in CDSC-E and PSC). Surprisingly, HerBERT\textsubscript{BASE} is better than HerBERT\textsubscript{LARGE} in CDSC-E. The same behaviour is noticeable for Polish RoBERTa, but not for XLM-RoBERTa. For other tasks, the LARGE models are always better. Summing up, HerBERT\textsubscript{LARGE} is the new state-of-the-art Polish language model based on the results of the KLEJ Benchmark.

It is worth emphasizing that the proposed procedure for efficient pretraining by transferring knowledge from multilingual to monolingual language allowed HerBERT to achieve better results than Polish RoBERTa even though it was optimized with around ten times shorter training.

\paragraph{POS Tagging}

\begin{table}[!ht]
\renewcommand*{\arraystretch}{1.15}
\centering
\begin{tabular}{lrr}

    \toprule
    \bf{Model} & \bf{Accuracy } & \bf{F1-Score } \\
    \midrule
    \multicolumn{3}{c}{\bf{Base Models}} \\
    \midrule
    XLM-RoBERTa    & 95.97 $\pm$ 0.04 & 95.79 $\pm$ 0.05 \\
    Polish RoBERTa & 96.13 $\pm$ 0.03 & 95.92 $\pm$ 0.03 \\
    HerBERT        & \underline{96.46 $\pm$ 0.04} & \underline{96.27 $\pm$ 0.04} \\
    \midrule
    \multicolumn{3}{c}{\bf{Large Models}} \\
    \midrule
    XLM-RoBERTa    & 97.07 $\pm$ 0.05 & 96.93 $\pm$ 0.05 \\
    Polish RoBERTa & 97.21 $\pm$ 0.02 & 97.05 $\pm$ 0.03 \\
    HerBERT        & \bf{\underline{97.30 $\pm$ 0.02}} & \bf{\underline{97.17 $\pm$ 0.02}} \\
    \bottomrule

\end{tabular}
\caption{Part-of-speech tagging results on NKJP dataset. Scores are reported for the test set and are median values across five runs. Best scores within each group are underlined, best overall are bold.}
\label{tab:results-pos}
\end{table}

HerBERT achieves overall better results in terms of both accuracy and F1-Score. HerBERT\textsubscript{BASE} beats the second-best model (i.e. Polish RoBERTa) by a margin of 0.33pp (F1-Score by 0.35pp) and HerBERT\textsubscript{LARGE} by a margin of 0.09pp (F1-Score by 0.12pp). It should be emphasized that while the improvements may appear to be minor, they are statistically significant.
All results are presented in Table \ref{tab:results-pos}.

\paragraph{Dependency Parsing}
The dependency parsing results are much more ambiguous than in other tasks. As expected, the models with static fastText embeddings performed much worse than Transformer-based models (around 3pp difference for UAS, and 4pp for LAS).
In the case of Transformer-based models, the differences are less noticeable. As expected, the LARGE models outperform the BASE models. The best performing model is Polish RoBERTa. HerBERT models performance is the worst across Transformer-based models except for the UAS score which is slightly better than XLM-RoBERTa for BASE models.
All results are presented in Table \ref{tab:results-dep}.

\begin{table}[!ht]
\renewcommand*{\arraystretch}{1.15}
\setlength{\tabcolsep}{5pt}
\centering
\begin{tabular}{lrr}

    \toprule
    \bf{Model} & \bf{UAS } & \bf{LAS } \\
    \midrule
    \multicolumn{3}{c}{\bf{Static Embeddings}} \\
    \midrule
    Plain & 90.58 $\pm$ 0.07 & 87.35 $\pm$ 0.12 \\
    FastText      & \underline{92.20 $\pm$ 0.14} & \underline{89.57 $\pm$ 0.13} \\
    \midrule
    \multicolumn{3}{c}{\bf{Base Models}} \\
    \midrule
    XLM-RoBERTa    & 95.14 $\pm$ 0.07 & 93.25 $\pm$ 0.12 \\
    Polish RoBERTa & \underline{95.41 $\pm$ 0.24} & \underline{93.65 $\pm$ 0.34} \\
    HerBERT        & 95.18 $\pm$ 0.22 & 93.24 $\pm$ 0.23 \\
    \midrule
    \multicolumn{3}{c}{\bf{Large Models}} \\
    \midrule
    XLM-RoBERTa    & 95.38 $\pm$ 0.02 & 93.66 $\pm$ 0.07 \\
    Polish RoBERTa & \bf{\underline{95.60 $\pm$ 0.18}} & \bf{\underline{93.90 $\pm$ 0.21}} \\
    HerBERT        & 95.11 $\pm$ 0.04 & 93.32 $\pm$ 0.02 \\
    \bottomrule

\end{tabular}
\caption{Dependency parsing results on Polish Dependency Bank dataset. Scores are reported for the test set and are median values across three runs. Best scores within each group are underlined, best overall are bold.}
\label{tab:results-dep}
\end{table}

\section{Conclusion}
\label{sec:conclusion}
In this work, we conducted a thorough ablation study regarding training BERT-based models for Polish language. We evaluated several design choices for pretraining BERT outside of English language. Contrary to \citet{structbert2020iclr}, our experiments demonstrated that SSO is not beneficial for the downstream task performance. It also turned out that BPE-Dropout does not increase the quality of a pretrained language model.

As a result of our studies we developed and evaluated an efficient pretraining procedure for transferring knowledge from multilingual to monolingual BERT-based models. We used it to train and release HerBERT -- a Transformer-based language model for Polish. It was trained on a diverse multi-source corpus. The conducted experiments confirmed its high performance on a set of eleven diverse linguistic tasks, as HerBERT turned out to be the best on eight of them. In particular, it is the best model for Polish language understanding according to the KLEJ Benchmark.

It is worth emphasizing that the quality of the obtained language model was even more impressive considering its short training time. Due to multilingual initialization, HerBERT\textsubscript{BASE} outperformed Polish RoBERTa\textsubscript{BASE} even though it was trained with a smaller batch size (2560 vs 8000) for a fewer number of steps (50k vs 125k). The same behaviour is also visible for HerBERT\textsubscript{LARGE}.
Additionally, we conducted a separate ablation study to confirm that the success of HerBERT is caused by the described initialization scheme. It showed that in fact, it was the most important factor to improved the quality of HerBERT.

We believe that the proposed training procedure and detailed experiments will encourage NLP researchers to cost-effectively train language models for other languages.

\newpage