\section*{Introduction}

\medbreak

For a choice $0 < \kappa < 1$ of modulus, write
$$u = \int_0^{\phi} \frac{{\rm d} \theta}{\sqrt{1 - \kappa^2 \sin^2 \theta}} $$
and invert the assignment $\phi \mapsto u$ (near the fixed origin) to obtain $u \mapsto \phi$; in these terms, the Jacobian elliptic functions appear as
$${\rm sn} \, u = \sin \phi$$
$${\rm cn} \, u = \cos \phi$$
and
$${\rm dn} \, u = \sqrt{1 - \kappa^2 \sin^2 \phi}.$$

\bigbreak

Li-Chien Shen [5][7] rewrites the integrand in terms of the `classical' hypergeometric function
$$\frac{1}{\sqrt{1 - z}} = F(\tfrac{1}{2}, \tfrac{1}{2}; \tfrac{1}{2}; z)$$
thus
$$u = \int_0^{\phi} F(\tfrac{1}{2}, \tfrac{1}{2}; \tfrac{1}{2}; \kappa^2 \sin^2 \theta) \, {\rm d} \theta$$
and asks whether the replacement of this `classical' hypergeometric function by other hypergeometric functions similarly gives rise to elliptic functions. In [5] this is done (with an important modification) for $F(\frac{1}{3}, \frac{2}{3}; \frac{1}{2}; \bullet)$  and in [7] it is done for $F(\frac{1}{4}, \frac{3}{4}; \frac{1}{2}; \bullet)$. These two cases are revisited in [2] and [3] respectively, from a slightly different perspective. [4] looks at the case $F(\frac{1}{6}, \frac{5}{6}; \frac{1}{2}; \bullet)$ following [6] on $F(\frac{1}{6}, \frac{5}{6}; 1; \bullet)$. Of course, all of this has a direct bearing on the Ramanujan theory of elliptic functions to alternative bases in signatures 3, 4 and 6.

\medbreak

Our aim in this paper is to follow the program of Shen when the `classical' hypergeometric function is replaced by
$$F(\tfrac{1}{2} - a, \tfrac{1}{2} + a; \tfrac{1}{2}; \bullet)$$
where $a$ is the reciprocal of a positive integer. The class in which $a$ is the reciprocal of an even integer is handled in a uniform manner; the results in signature three ($a = 1/6$) and signature four ($a = 1/4$) fall out as special cases. The class in which $a$ is the reciprocal of an odd integer also receives a uniform treatment, signature six falling out as the special case $a = 1/3$. Our analysis completely determines when the resulting analogues of ${\rm sn}, {\rm cn}$ and ${\rm dn}$ (or their squares) are elliptic.
\medbreak

\section*{Preliminaries}

\medbreak

In the next two sections, we focus on the cases in which $a$ is the reciprocal of a positive integer; here, we place no such restriction on $a$.

\medbreak

It will simplify matters to employ the abbreviation
$$F_a (z) = F(\tfrac{1}{2} - a, \tfrac{1}{2} + a; \tfrac{1}{2}; z)$$
for the hypergeometric function from which our integrand is constructed. For this function, we quote the standard identity
$$F_a (\sin^2 z) = \frac{\cos 2 a z}{\cos z}.$$

\medbreak

Now, let us fix the modulus $\kappa \in (0, 1)$ with complementary modulus $\lambda = \sqrt{1 - \kappa^2}$. Speaking with some notational  informality, the rule
$$u = \int_0^{\phi} F_a(\kappa^2 \sin^2 \theta) \, {\rm d} \theta$$
defines an association having the property
$$\frac{{\rm d} u}{{\rm d} \phi} = F_a(\kappa^2 \sin^2 \phi);$$
as $F_a (0) = 1$ it follows that the assignment $\phi \mapsto u$ inverts near $0$ (as a fixed point) to yield $u \mapsto \phi$. We now drop the informality and write $\phi$ for the (analytic) local inverse function thus obtained: if $t$ is near $0$ then
$$t = \int_0^{\phi(t)} F_a(\kappa^2 \sin^2 \theta) \, {\rm d} \theta.$$
Near $0$ as a fixed point, a function $\psi$ is defined by the rule
$$\sin \psi = \kappa \, \sin \phi$$
and continuity (which is automatically promoted to analyticity).

\medbreak

\begin{theorem} \label{phi}
The derivative of the local inverse $\phi$ is given by
$$\phi\,' = \frac{\cos \psi}{\cos 2 a \psi}.$$
\end{theorem}

\begin{proof}
An application of the chain rule to the inverse function:
$$\phi \,' = \frac{1}{F_a(\kappa^2 \sin^2 \phi)} = \frac{1}{F_a(\sin^2 \psi)} =  \frac{\cos \psi}{\cos 2 a \psi}$$
in view of the quoted hypergeometric identity.
\end{proof}

\medbreak

Prompted by this result, we introduce two abbreviations that will further simplify subsequent results: we write
$$d = \cos \psi$$
and (as a `partial $d$')
$$\partial = \cos \, (2 a \psi)$$
so that Theorem \ref{phi} reads
$$\phi \,' = d/\partial.$$

\medbreak

\begin{theorem} \label{d and}
The functions $d$ and $\partial$ satisfy the following first-order differential equations:
$$\partial^2 (d\,')^2 = (1 - d^2)(d^2 - \lambda^2)$$
$$\partial^2 (\partial \,')^2 = 4 a^2 (1 - \partial^2) (d^2 - \lambda^2).$$
\end{theorem}

\begin{proof}
From $\sin \psi = \kappa \sin \phi$ it follows that
$$\psi \,' = (\kappa \cos \phi) \, \phi\,'/ \cos \psi$$
whence Theorem \ref{phi} yields
$$\psi\,' = \kappa \cos \phi\, / \cos 2 a \psi.$$
From $d = \cos \psi$ it follows that
$$d\,' = - (\sin \psi) \psi\,' = -( \sin \psi) (\kappa \cos \phi)/\cos 2 a \psi$$
whence squaring yields
$$\partial^2 (d\,')^2 = (1 - \cos^2 \psi)(\kappa^2 - \kappa^2 \sin^2 \phi) = (1 - d^2)(d^2 - \lambda^2)$$
since $\cos^2 \psi + \sin^2 \psi = 1 = \kappa^2 + \lambda^2$. This establishes the first differential equation. The second may be established similarly, using the fact that
$$\partial \,' = - (2 a \sin 2 a \psi) \psi \,' = - 2 a (\sin 2 a \psi) \, (\kappa \cos \phi) / \cos 2 a \psi.$$
\end{proof}

\medbreak

Alongside the analogue
$$d = \cos \psi$$
of the Jacobian function ${\rm dn}$, we shall be concerned with the analogue
$$c = \cos \phi$$
of the `modular cosine' ${\rm cn}$ and the analogue
$$s = \sin \phi$$
of the `modular sine' ${\rm sn}$. From their definitions, these functions satisfy the same quadratic relations as those satisfied by the classical Jacobian elliptic functions: thus,
$$c^2 + s^2 = 1 \; \; {\rm and} \; \; d^2 + \kappa^2 s^2 = 1.$$
These analogues of the Jacobi functions are (with $\phi$ and $\psi$) initially defined and analytic in a neighbourhood of $0$.

\medbreak

When $a$ is the reciprocal of a positive integer, we ask whether these functions (or their squares) extend to the plane as elliptic functions. The answers are presented in the next two sections, treating first the case in which the integer $1/a$ is even and then the case in which $1/a$ is odd. The auxiliary function $\partial = \cos 2 a \psi$ plays an important r\^ole in our analysis.

\medbreak

\section*{The even case}

\medbreak

Throughout this section, we consider the case in which $a = 1/2n$ is the reciprocal of an even positive integer. In this case, the auxiliary function assumes the form $\partial = \cos \tfrac{1}{n} \psi$.

\medbreak

Now, recall the Chebyshev polynomial (of the first kind) $T_n$ given by
$$T_n (\cos \theta) = \cos n \theta.$$

\medbreak

\begin{theorem} \label{Tn}
The functions $d$ and $\partial$ satisfy $d = T_n \circ \partial$.
\end{theorem}

\begin{proof}
Direct from the definitions: abusing the composition,
$$T_n (\partial) = T_n(\cos \tfrac{1}{n} \psi) = \cos \psi = d.$$
\end{proof}

\medbreak

Theorem \ref{d and} presents a pair of differential equations, each of which involves both $d$ and $\partial$. In the present `even' case, we now have a differential equation for $\partial$ alone.

\medbreak

\begin{theorem} \label{partial}
When $1/a = 2n$, the auxiliary function $\partial$ satisfies the differential equation
$$\partial^2 (\partial \,')^2 = \frac{1}{n^2} (1 - \partial^2) (T_n(\partial)^2 - \lambda^2).$$
\end{theorem}

\begin{proof}
Merely insert Theorem \ref{Tn} in the second equation of Theorem \ref{d and} when $a = 1/2n$.
\end{proof}

\medbreak

In addition to asking whether the functions $s, c$ and $d$ admit elliptic extensions to the plane, we may ask the same question of the auxiliary function $\partial$. The answer to this auxiliary question is negative, as an immediate corollary to the following theorem.

\medbreak

\begin{theorem} \label{p not e}
Each solution to the differential equation
$$f^2 (f \,')^2 = \frac{1}{n^2} (1 - f^2) (T_n(f)^2 - \lambda^2)$$
is zero-free.
\end{theorem}

\begin{proof}
Evaluating both sides of the differential equation at a zero of $f$ would force $T_n(0)^2 = \lambda^2$. Alas: $T_n(0)^2 = \cos^2\frac{n \pi}{2} \in \{0, 1\}$ while $\lambda^2 \in (0, 1)$.
\end{proof}

\medbreak

It follows at once that the auxiliary function $\partial$ does not extend to an elliptic function, for any (nonconstant) elliptic function must have zeros.

\medbreak

To address the elliptic extendibility of $s, c$ and $d$ we go a little further and consider the square of the auxiliary function, which we take the risk of denoting by
$$\nabla = \partial^2.$$
From $\nabla \,' = 2 \, \partial \, \partial\,'$ and Theorem \ref{partial} we deduce that
$$(\nabla\,')^2 = \frac{4}{n^2} (1 - \nabla) (T_n (\partial)^2 - \lambda^2).$$
Here, $T_n(\partial)^2$ is a degree $n$ polynomial in $\nabla$: say
$$T_n (\partial)^2 = S_n (\nabla).$$
If $n$ is even then this is clear, for $T_n(\partial)$ itself is a polynomial in $\partial^2$. If $n$ is odd, then $T_n (\partial)$ is $\partial$ times a polynomial in $\partial^2$: explicitly, if $n = 2 m + 1$ then $T_n (\partial) = \partial \, V_m (2 \partial^2 - 1)$ where $V_m$ is Chebyshev of the third kind; see formula (1.15) in Section 1.2.4 of [1]. Note that if $n$ is even then $S_n (0) = T_n(0)^2 = 1$ while if $n$ is odd then $S_n(0) = 0$.

\medbreak

With this understanding, we have the following result.

\medbreak

\begin{theorem} \label{nabla}
When $1/a = 2n$, the square $\nabla = \partial^2$ satisfies the differential equation
$$(\nabla\,')^2 = \frac{4}{n^2} (1 - \nabla) (S_n (\nabla) - \lambda^2).$$
\end{theorem}

\begin{proof}
Done.
\end{proof}

\medbreak

We remark that if $f$ is a solution to the differential equation
$$(f\,')^2 = \frac{4}{n^2} (1 - f) (S_n (f) - \lambda^2)$$
then each of its zeros is simple. Indeed, let $f(z) = 0$: evaluation of the differential equation at $z$ reveals that if $n$ is even then $f\,'(z)^2 = 4(1 - \lambda^2)/n^2$ while if $n$ is odd  then $f\,'(z)^2 = - 4 \lambda^2 / n^2$; in either case, $f\,'(z)$ is not zero. Of course, this is consistent with Theorem \ref{p not e}: were $\partial$ to have zeros, $\nabla = \partial^2$ would have even-order zeros.

\medbreak

To decide whether $\nabla$ admits extensions that are elliptic, we conduct a polar analysis.

\medbreak

\begin{theorem} \label{n3}
If $n > 3$ then a solution of the differential equation
$$(f\,')^2 = \frac{4}{n^2} (1 - f) (S_n (f) - \lambda^2)$$
is elliptic only when constant.
\end{theorem}

\begin{proof}
Recall that a nonconstant elliptic function has poles. Let the solution $f$ have a pole of order $m$ at some point. At this point, $f'$ has a pole of order $m + 1$ and $S_n(f)$ has a pole of order $n m$. The differential equation of which $f$ is a solution then forces $2(m + 1) = m + n m$ so that $m (n - 1) = 2$. The only solutions to this equation are $n = 2, m = 2$ and $n = 3, m = 1$.
\end{proof}

\medbreak

In particular, if $n > 3$ then $\nabla$ is not the restriction of an elliptic function; for brevity, we shall simply say `$\nabla$ is not elliptic'.

\medbreak

We are now able to address the question whether the analogue $d$ of the Jacobian function ${\rm dn}$ is elliptic. In fact, we can also answer the same question for its square
$$D = d^2.$$

\medbreak

\begin{theorem} \label{D}
If $n > 3$ then the square $D = d^2$ is not elliptic.
\end{theorem}

\begin{proof}
Strictly speaking, we mean here that $D$ is not the restriction of an elliptic function. From $D\,' = 2 \, d \, d\,'$ and the first equation in Theorem \ref{d and} we deduce that
$$\nabla \, (D\,')^2 = 4 D (1 - D)(D - \lambda^2)$$
or
$$\nabla = 4 D (1 - D)(D - \lambda^2) / (D\,')^2$$
whence ellipticity of $D$ would force ellipticity upon $\nabla$.
\end{proof}

\medbreak

There are parallel consequences for the squares $C = c^2$ and $S = s^2$ of the other Jacobian analogues: it is readily verified that these squares similarly satisfy
$$\nabla \, (C\,')^2 = 4 C \, (1 - C) \, (\lambda^2 + \kappa^2 C)$$
and
$$\nabla \, (S\,')^2 = 4 S \, (1 - S) \, (1 - \kappa^2 S);$$
alternatively and more simply, they satisfy
$$C + S = 1 \; \; {\rm and} \; \; D + \kappa^2 S = 1.$$
Thus, if $n > 3$ then none of the three functions $S, C, D$ is elliptic, whence none of $s, c, d$ is elliptic; similar remarks apply to the quotient $s/c$ in view of the identity $1 + (s/c)^2 = 1/c^2$.

\medbreak

The cases $n = 2$ and $n = 3$ not addressed in Theorem \ref{n3} and Theorem \ref{D} are true exceptions: they correspond to signature four [7] and signature three [5] respectively; in brief, the results are as follows.

\medbreak

{\bf Case $ n = 2$}. Here, $d = 2 \partial^2 - 1$ so that the first equation of Theorem \ref{d and} simplifies to
$$(d\,')^2 = 2 (1 - d) (d^2 - \lambda^2).$$
This differential equation reveals that $d$ is elliptic: indeed
$$d = 1 - \tfrac{1}{2} \kappa^2 / (\wp + \tfrac{1}{3})$$
where $\wp$ is the Weierstrass function with invariants $g_2 = \frac{4}{3} - \kappa^2$ and $g_3 = \frac{8}{27} - \frac{1}{3} \kappa^2$. Note that $\wp$ has $-1/3$ and $\tfrac{1}{6} \pm \frac{1}{2} \lambda$ as its midpoint values; in particular, $\wp + \frac{1}{3}$ has a meromorphic (indeed, an elliptic) square-root. As $d$ is elliptic, $\nabla = \partial^2 = (1 + d) / 2$ is also elliptic; the auxiliary function $\partial$ itself does not extend meromorphically to the plane, on account of the discussion after Theorem \ref{nabla}. As $d^2$ is elliptic, so are $s^2$ and $c^2$ in view of $d^2 + \kappa^2 s^2 = 1$ and $c^2 + s^2 = 1$. By direct calculation, $\kappa^2 s^2 = 4 \nabla \, (1 - \nabla)$; the zeros of $\nabla$ being simple, we deduce that $s^2$ does not have a meromorphic square-root and $s$ is not elliptic. Also by direct calculation, $c^2 = \frac{1}{4} (\wp\,')^2 / (\wp + \frac{1}{3})^3$ so $c$ is elliptic. For further details, see [7] and [3].

\medbreak

{\bf Case $n = 3$}. Here, $d = 4 \partial^3 - 3 \partial$ so that $d^2 = \nabla (4 \nabla - 3)^2$ and the equation of Theorem \ref{nabla} reads
$$9 (\nabla\,')^2 = 4 (1 - \nabla) (\nabla (4 \nabla - 3)^2 - \lambda^2).$$
The cubic factor on the right-hand side has discriminant $2^8 3^3 \lambda^2 (1 - \lambda^2) \neq 0$ and therefore has distinct roots, none of which is unity. Consequently, the quartic on the right-hand side has distinct roots and the differential equation reveals that $\nabla$ is elliptic, with simple zeros (as above) and simple poles (by a polar analysis akin to that conducted in Theorem \ref{n3}), so the auxiliary function $\partial$ does not extend meromorphically to the plane. The square $d^2 = (4 \nabla - 3)^2 \, \nabla$ is elliptic with triple poles, whence so are the squares $c^2$ and $s^2$; it follows that $d, \, c$ and $s$ are not elliptic. For further details, see [4] and [2].

\medbreak

\section*{The odd case}

\medbreak

Throughout this section, we consider the case in which $a$ is the reciprocal of an odd positive integer: say $1/a = n = 2 m + 1$. Some of the supporting arguments in this section are so closely similar to those in the preceding section that we abbreviate them or omit them entirely; others are sufficiently different that we present them in full.

\medbreak

\begin{theorem} \label{Tn odd}
The functions $d = \cos \psi$ and $\partial = \cos \frac{2}{n} \psi$ satisfy $2 \, d^2 - 1 = T_n \circ \partial.$
\end{theorem}

\begin{proof}
Each side of the claimed equation is an alternative expression for $\cos 2 \psi$.
\end{proof}

\medbreak

Consequently, the first-order differential equation satisfied by $\partial$ alone now assumes a slightly different form.

\medbreak

\begin{theorem} \label{partial odd}
When $1/a = n$ is odd, the auxiliary function $\partial$ satisfies the differential equation
$$\partial^2 \, (\partial\,')^2 = \frac{2}{n^2} (1 - \partial^2)(T_n (\partial) + 1 - 2 \lambda^2).$$
\end{theorem}

\begin{proof}
Refer to the second equation of Theorem \ref{d and} and replace $d^2$ by $\frac{1}{2} (T_n (\partial) + 1).$
\end{proof}

\medbreak

Rather than pause to address the ellipticity (or indeed otherwise) of $\partial$ itself, we pass directly to the question whether or not its square $\nabla$ is elliptic.

\medbreak

\begin{theorem} \label{nabla odd}
When $1/a = n = 2 m + 1$, the square $\nabla = \partial^2$ satisfies the differential equation
$$\Big[ \frac{n^2}{8} (\nabla\,')^2 + \Lambda (\nabla - 1) \Big]^2 = \nabla \, (\nabla - 1)^2 \, V_m(2 \nabla - 1)^2$$
where
$$\Lambda = 1 - 2 \lambda^2.$$
\end{theorem}

\begin{proof}
Substitute $\nabla\,' = 2 \partial \, \partial'$ in the differential equation of Theorem \ref{partial odd}: there follows
$$(\nabla\,')^2 = \frac{8}{n^2} (1 - \nabla) (T_n (\partial) + \Lambda)$$
so that
$$\Big[ \frac{n^2}{8} (\nabla\,')^2 + \Lambda (\nabla - 1) \Big]^2 = (1 - \nabla)^2 T_n(\partial)^2.$$
Finally, recall (from the argument leading to Theorem \ref{nabla}) that $T_n(\partial)^2 = \nabla \, V_m (2 \nabla - 1)^2.$

\end{proof}

\medbreak

In the `even' case, we ruled out ellipticity of $\nabla$ by an inspection of poles. In the `odd' case, we are able to rule out ellipticity of $\nabla$ by an inspection of zeros.

\medbreak

\begin{theorem} \label{oddno0}
If $n = 2 m + 1$ then a solution of the differential equation
$$\Big[ \frac{n^2}{8} (f\,')^2 + \Lambda (f - 1) \Big]^2 = f \, (f - 1)^2 \, V_m(2 f - 1)^2$$
is elliptic only when constant.
\end{theorem}

\begin{proof}
For convenience, introduce the polynomial
$$q(z) = z \, (z - 1)^2 \, V_m (2 z - 1)^2$$
so that $q(0) = 0$ and $q\,'(0) = V_m (-1)^2 = (2 m + 1)^2.$

For a contradiction, let $f$ be a nonconstant elliptic solution to the displayed differential equation. Differentiate throughout to obtain
$$2 \, \Big[ \frac{n^2}{8} (f\,')^2 + \Lambda (f - 1) \Big] \Big[ \frac{n^2}{4} f\,' \, f '' + \Lambda f\,' \Big] = q\,' (f) \, f\,';$$
as the elliptic $f\,'$ is not identically zero, it may be cancelled to yield
$$2 \, \Big[ \frac{n^2}{8} (f\,')^2 + \Lambda (f - 1) \Big] \Big[ \frac{n^2}{4} \, f '' + \Lambda \Big] = q\,' (f).$$
Now evaluate both sides of this equation at a zero of the (nonconstant elliptic) function $f$: the left side reduces to $0$ by virtue of the original differential equation; the right side reduces to $(2 m + 1)^2$. Contradiction.
\end{proof}

\medbreak

We leave as an exercise a polar analysis along the lines of the one in Theorem \ref{n3}.

\medbreak

Theorem \ref{nabla odd} and Theorem \ref{oddno0} tell us that $\nabla$ is not elliptic. As in the `even' case, we may now infer that the squares $D, C$ and $S$ are not elliptic.

\medbreak

\begin{theorem} \label{DCSodd}
If $n = 2 m + 1$ then none of the squares $d^2, \, c^2$ and $s^2$ is elliptic.
\end{theorem}

\begin{proof}
If $D$ were elliptic then so would be
$$\nabla = 4 D (1 - D)(D - \lambda^2) / (D\,')^2.$$

\end{proof}

\medbreak

When $n = 3$ we recover the results of [4].

\medbreak

\section*{Remarks}

\medbreak

Throughout this paper, we have followed the program initiated by Shen, starting from the expression
$${\rm dn} \, u = \sqrt{1 - \kappa^2 \sin^2 \phi}$$
\medbreak
\noindent
for the third of the classical Jacobian elliptic functions, as proposed in the Introduction. This third Jacobian function has been called the `delta amplitude' on account of the equivalent expression
$${\rm dn} \, u = \frac{{\rm d} \phi}{{\rm d} u}.$$
\medbreak
\noindent
This derivative is the form on which Shen models his `third function' ${\rm dn}_3$ in [5] (hence the `important modification' that we mentioned in the Introduction). The detailed investigation of this form remains to be pursued beyond the case of signature three; here, we merely start the discussion in general and indicate what happens in signature three and signature four.

\medbreak

With the local inverse $\phi$ and the associated function $\psi$ as at the outset of our `Preliminaries', we introduce the generalized `delta amplitude'
$$\delta: = \phi\,' = \frac{\cos \psi}{\cos 2 a \psi} = \frac{d}{\partial}$$
and ask whether this function $\delta$ (or its square) is the restriction of an elliptic function, especially when $1/a$ is a positive integer.

\medbreak

The case in which $1/a$ is twice an odd integer features some simplification. Say $1/a = 2n$ where $n = 2 m + 1$. In this case, $d$ and $\partial$ are related by
$$d = T_n (\partial) = \partial \, V_m (2 \partial^2 - 1)$$
so that
$$\delta = V_m (2 \nabla - 1)$$
where $V_m$ is again the Chebyshev polynomial of the third kind.

\medbreak

Two special cases are relatively straightforward to handle.

\medbreak

{\bf Signature} 3. Here, $1/a = 6$ and the foregoing simplification applies. We recover the result of Shen that $\delta = {\rm dn}_3$ is elliptic; indeed
$$\delta = V_1(2 \nabla - 1) = 4 \nabla - 3$$
is elliptic with simple poles, the same being true of $\nabla$ as noted at the end of our section on the `even' case. In fact, Shen [5] shows that
$${\rm dn}_3 = 1 - \tfrac{4}{9} \kappa^2 / (\wp + \tfrac{1}{3})$$
where now $\wp$ is the Weierstrass function with $g_2 = 4 (9 - 8 \kappa^2) / 27$ and $g_3 = 8 (8 \kappa^4 - 36 \kappa^2 + 27) / 729$; for an alternative approach, see also [2].

\medbreak

{\bf Signature} 4. Here, $1/a = 4$ and we may draw from the case $n = 2$ near the end of our section on the `even' case: thus
$$\delta^2 = \frac{d^2}{\partial^2} = \frac{2 d^2}{1 + d} = \frac{(2 \nabla - 1)^2}{\nabla}$$
and so $\delta^2$ is elliptic; as $\nabla$ has simple zeros, it follows that $\delta$ itself is not elliptic.

\bigbreak

\begin{center}
{\small R}{\footnotesize EFERENCES}
\end{center}
\medbreak

[1] J.C. Mason and D.C. Handscomb, {\it Chebyshev Polynomials}, Chapman and Hall (2003).

\medbreak

[2] P.L. Robinson, {\it Elliptic functions from $F(\frac{1}{3}, \frac{2}{3} ; \frac{1}{2} ; \bullet)$}, arXiv 1907.09938 (2019).

\medbreak

[3] P.L. Robinson, {\it Elliptic functions from $F(\tfrac{1}{4}, \tfrac{3}{4}; \tfrac{1}{2} ; \bullet)$}, arXiv 1908.01687 (2019).

\medbreak

[4] P.L. Robinson, {\it Nonelliptic functions from $F(\tfrac{1}{6}, \tfrac{5}{6}; \tfrac{1}{2} ; \bullet)$}, arXiv 2004.06529 (2020).

\medbreak

[5] Li-Chien Shen, {\it On the theory of elliptic functions based on $_2F_1(\frac{1}{3}, \frac{2}{3} ; \frac{1}{2} ; z)$}, Transactions of the American Mathematical Society {\bf 357}  (2004) 2043-2058.

\medbreak

[6] Li-Chien Shen, {\it A note on Ramanujan's identities involving the hypergeometric function $F(\tfrac{1}{6}, \tfrac{5}{6}; 1 ; z)$}, Ramanujan Journal {\bf 30} (2013) 211-222.

\medbreak

[7] Li-Chien Shen, {\it On a theory of elliptic functions based on the incomplete integral of the hypergeometric function $_2 F_1 (\frac{1}{4}, \frac{3}{4} ; \frac{1}{2} ; z)$}, Ramanujan Journal {\bf 34} (2014) 209-225.

\medbreak

\medbreak

\end{document}

\medbreak

\begin{theorem}

\end{theorem}

\begin{proof}

\end{proof}

\medbreak


\section{Introduction}

Please follow the steps outlined below when submitting your manuscript to
the IEEE Computer Society Press.  This style guide now has several
important modifications (for example, you are no longer warned against the
use of sticky tape to attach your artwork to the paper), so all authors
should read this new version.

\subsection{Language}

All manuscripts must be in English.

\subsection{Dual submission}

Please refer to the author guidelines on the CVPR 2020 web page for a
discussion of the policy on dual submissions.

\subsection{Paper length}
Papers, excluding the references section,
must be no longer than eight pages in length. The references section
will not be included in the page count, and there is no limit on the
length of the references section. For example, a paper of eight pages
with two pages of references would have a total length of 10 pages.
{\bf There will be no extra page charges for CVPR 2020.}

Overlength papers will simply not be reviewed.  This includes papers
where the margins and formatting are deemed to have been significantly
altered from those laid down by this style guide.  Note that this
\LaTeX\ guide already sets figure captions and references in a smaller font.
The reason such papers will not be reviewed is that there is no provision for
supervised revisions of manuscripts.  The reviewing process cannot determine
the suitability of the paper for presentation in eight pages if it is
reviewed in eleven.

\subsection{The ruler}
The \LaTeX\ style defines a printed ruler which should be present in the
version submitted for review.  The ruler is provided in order that
reviewers may comment on particular lines in the paper without
circumlocution.  If you are preparing a document using a non-\LaTeX\
document preparation system, please arrange for an equivalent ruler to
appear on the final output pages.  The presence or absence of the ruler
should not change the appearance of any other content on the page.  The
camera ready copy should not contain a ruler. (\LaTeX\ users may uncomment
the \verb'\cvprfinalcopy' command in the document preamble.)  Reviewers:
note that the ruler measurements do not align well with lines in the paper
--- this turns out to be very difficult to do well when the paper contains
many figures and equations, and, when done, looks ugly.  Just use fractional
references (e.g.\ this line is $095.5$), although in most cases one would
expect that the approximate location will be adequate.

\subsection{Mathematics}

Please number all of your sections and displayed equations.  It is
important for readers to be able to refer to any particular equation.  Just
because you didn't refer to it in the text doesn't mean some future reader
might not need to refer to it.  It is cumbersome to have to use
circumlocutions like ``the equation second from the top of page 3 column
1''.  (Note that the ruler will not be present in the final copy, so is not
an alternative to equation numbers).  All authors will benefit from reading
Mermin's description of how to write mathematics:
\url{http://www.pamitc.org/documents/mermin.pdf}.

\subsection{Blind review}

Many authors misunderstand the concept of anonymizing for blind
review.  Blind review does not mean that one must remove
citations to one's own work---in fact it is often impossible to
review a paper unless the previous citations are known and
available.

Blind review means that you do not use the words ``my'' or ``our''
when citing previous work.  That is all.  (But see below for
techreports.)

Saying ``this builds on the work of Lucy Smith [1]'' does not say
that you are Lucy Smith; it says that you are building on her
work.  If you are Smith and Jones, do not say ``as we show in
[7]'', say ``as Smith and Jones show in [7]'' and at the end of the
paper, include reference 7 as you would any other cited work.

An example of a bad paper just asking to be rejected:
\begin{quote}
\begin{center}
    An analysis of the frobnicatable foo filter.
\end{center}

   In this paper we present a performance analysis of our
   previous paper [1], and show it to be inferior to all
   previously known methods.  Why the previous paper was
   accepted without this analysis is beyond me.

   [1] Removed for blind review
\end{quote}

An example of an acceptable paper:

\begin{quote}
\begin{center}
     An analysis of the frobnicatable foo filter.
\end{center}

   In this paper we present a performance analysis of the
   paper of Smith \etal [1], and show it to be inferior to
   all previously known methods.  Why the previous paper
   was accepted without this analysis is beyond me.

   [1] Smith, L and Jones, C. ``The frobnicatable foo
   filter, a fundamental contribution to human knowledge''.
   Nature 381(12), 1-213.
\end{quote}

If you are making a submission to another conference at the same time,
which covers similar or overlapping material, you may need to refer to that
submission in order to explain the differences, just as you would if you
had previously published related work.  In such cases, include the
anonymized parallel submission~\cite{Authors14} as additional material and
cite it as
\begin{quote}
[1] Authors. ``The frobnicatable foo filter'', F\&G 2014 Submission ID 324,
Supplied as additional material {\tt fg324.pdf}.
\end{quote}

Finally, you may feel you need to tell the reader that more details can be
found elsewhere, and refer them to a technical report.  For conference
submissions, the paper must stand on its own, and not {\em require} the
reviewer to go to a techreport for further details.  Thus, you may say in
the body of the paper ``further details may be found
in~\cite{Authors14b}''.  Then submit the techreport as additional material.
Again, you may not assume the reviewers will read this material.

Sometimes your paper is about a problem which you tested using a tool which
is widely known to be restricted to a single institution.  For example,
let's say it's 1969, you have solved a key problem on the Apollo lander,
and you believe that the CVPR70 audience would like to hear about your
solution.  The work is a development of your celebrated 1968 paper entitled
``Zero-g frobnication: How being the only people in the world with access to
the Apollo lander source code makes us a wow at parties'', by Zeus \etal.

You can handle this paper like any other.  Don't write ``We show how to
improve our previous work [Anonymous, 1968].  This time we tested the
algorithm on a lunar lander [name of lander removed for blind review]''.
That would be silly, and would immediately identify the authors. Instead
write the following:
\begin{quotation}
\noindent
   We describe a system for zero-g frobnication.  This
   system is new because it handles the following cases:
   A, B.  Previous systems [Zeus et al. 1968] didn't
   handle case B properly.  Ours handles it by including
   a foo term in the bar integral.

   ...

   The proposed system was integrated with the Apollo
   lunar lander, and went all the way to the moon, don't
   you know.  It displayed the following behaviours
   which show how well we solved cases A and B: ...
\end{quotation}
As you can see, the above text follows standard scientific convention,
reads better than the first version, and does not explicitly name you as
the authors.  A reviewer might think it likely that the new paper was
written by Zeus \etal, but cannot make any decision based on that guess.
He or she would have to be sure that no other authors could have been
contracted to solve problem B.
\medskip

\noindent
FAQ\medskip\\
{\bf Q:} Are acknowledgements OK?\\
{\bf A:} No.  Leave them for the final copy.\medskip\\
{\bf Q:} How do I cite my results reported in open challenges?
{\bf A:} To conform with the double blind review policy, you can report results of other challenge participants together with your results in your paper. For your results, however, you should not identify yourself and should not mention your participation in the challenge. Instead present your results referring to the method proposed in your paper and draw conclusions based on the experimental comparison to other results.\medskip\\

\begin{figure}[t]
\begin{center}
\fbox{\rule{0pt}{2in} \rule{0.9\linewidth}{0pt}}

\end{center}
   \caption{Example of caption.  It is set in Roman so that mathematics
   (always set in Roman: $B \sin A = A \sin B$) may be included without an
   ugly clash.}
\label{fig:long}
\label{fig:onecol}
\end{figure}

\subsection{Miscellaneous}

\noindent
Compare the following:\\
\begin{tabular}{ll}
 \verb'$conf_a$' &  $conf_a$ \\
 \verb'$\mathit{conf}_a$' & $\mathit{conf}_a$
\end{tabular}\\
See The \TeX book, p165.

The space after \eg, meaning ``for example'', should not be a
sentence-ending space. So \eg is correct, {\em e.g.} is not.  The provided
\verb'\eg' macro takes care of this.

When citing a multi-author paper, you may save space by using ``et alia'',
shortened to ``\etal'' (not ``{\em et.\ al.}'' as ``{\em et}'' is a complete word.)
However, use it only when there are three or more authors.  Thus, the
following is correct: ``
   Frobnication has been trendy lately.
   It was introduced by Alpher~\cite{Alpher02}, and subsequently developed by
   Alpher and Fotheringham-Smythe~\cite{Alpher03}, and Alpher \etal~\cite{Alpher04}.''

This is incorrect: ``... subsequently developed by Alpher \etal~\cite{Alpher03} ...''
because reference~\cite{Alpher03} has just two authors.  If you use the
\verb'\etal' macro provided, then you need not worry about double periods
when used at the end of a sentence as in Alpher \etal.

For this citation style, keep multiple citations in numerical (not
chronological) order, so prefer \cite{Alpher03,Alpher02,Authors14} to
\cite{Alpher02,Alpher03,Authors14}.

\begin{figure*}
\begin{center}
\fbox{\rule{0pt}{2in} \rule{.9\linewidth}{0pt}}
\end{center}
   \caption{Example of a short caption, which should be centered.}
\label{fig:short}
\end{figure*}

\section{Formatting your paper}

All text must be in a two-column format. The total allowable width of the
text area is $6\frac78$ inches (17.5 cm) wide by $8\frac78$ inches (22.54
cm) high. Columns are to be $3\frac14$ inches (8.25 cm) wide, with a
$\frac{5}{16}$ inch (0.8 cm) space between them. The main title (on the
first page) should begin 1.0 inch (2.54 cm) from the top edge of the
page. The second and following pages should begin 1.0 inch (2.54 cm) from
the top edge. On all pages, the bottom margin should be 1-1/8 inches (2.86
cm) from the bottom edge of the page for $8.5 \times 11$-inch paper; for A4
paper, approximately 1-5/8 inches (4.13 cm) from the bottom edge of the
page.

\subsection{Margins and page numbering}

All printed material, including text, illustrations, and charts, must be kept
within a print area 6-7/8 inches (17.5 cm) wide by 8-7/8 inches (22.54 cm)
high.

\subsection{Type-style and fonts}

Wherever Times is specified, Times Roman may also be used. If neither is
available on your word processor, please use the font closest in
appearance to Times to which you have access.

MAIN TITLE. Center the title 1-3/8 inches (3.49 cm) from the top edge of
the first page. The title should be in Times 14-point, boldface type.
Capitalize the first letter of nouns, pronouns, verbs, adjectives, and
adverbs; do not capitalize articles, coordinate conjunctions, or
prepositions (unless the title begins with such a word). Leave two blank
lines after the title.

AUTHOR NAME(s) and AFFILIATION(s) are to be centered beneath the title
and printed in Times 12-point, non-boldface type. This information is to
be followed by two blank lines.

The ABSTRACT and MAIN TEXT are to be in a two-column format.

MAIN TEXT. Type main text in 10-point Times, single-spaced. Do NOT use
double-spacing. All paragraphs should be indented 1 pica (approx. 1/6
inch or 0.422 cm). Make sure your text is fully justified---that is,
flush left and flush right. Please do not place any additional blank
lines between paragraphs.

Figure and table captions should be 9-point Roman type as in
Figures~\ref{fig:onecol} and~\ref{fig:short}.  Short captions should be centred.

\noindent Callouts should be 9-point Helvetica, non-boldface type.
Initially capitalize only the first word of section titles and first-,
second-, and third-order headings.

FIRST-ORDER HEADINGS. (For example, {\large \bf 1. Introduction})
should be Times 12-point boldface, initially capitalized, flush left,
with one blank line before, and one blank line after.

SECOND-ORDER HEADINGS. (For example, { \bf 1.1. Database elements})
should be Times 11-point boldface, initially capitalized, flush left,
with one blank line before, and one after. If you require a third-order
heading (we discourage it), use 10-point Times, boldface, initially
capitalized, flush left, preceded by one blank line, followed by a period
and your text on the same line.

\subsection{Footnotes}

Please use footnotes\footnote {This is what a footnote looks like.  It
often distracts the reader from the main flow of the argument.} sparingly.
Indeed, try to avoid footnotes altogether and include necessary peripheral
observations in
the text (within parentheses, if you prefer, as in this sentence).  If you
wish to use a footnote, place it at the bottom of the column on the page on
which it is referenced. Use Times 8-point type, single-spaced.

\subsection{References}

List and number all bibliographical references in 9-point Times,
single-spaced, at the end of your paper. When referenced in the text,
enclose the citation number in square brackets, for
example~\cite{Authors14}.  Where appropriate, include the name(s) of
editors of referenced books.

\begin{table}
\begin{center}
\begin{tabular}{|l|c|}
\hline
Method & Frobnability \\
\hline\hline
Theirs & Frumpy \\
Yours & Frobbly \\
Ours & Makes one's heart Frob\\
\hline
\end{tabular}
\end{center}
\caption{Results.   Ours is better.}
\end{table}

\subsection{Illustrations, graphs, and photographs}

All graphics should be centered.  Please ensure that any point you wish to
make is resolvable in a printed copy of the paper.  Resize fonts in figures
to match the font in the body text, and choose line widths which render
effectively in print.  Many readers (and reviewers), even of an electronic
copy, will choose to print your paper in order to read it.  You cannot
insist that they do otherwise, and therefore must not assume that they can
zoom in to see tiny details on a graphic.

When placing figures in \LaTeX, it's almost always best to use
\verb+\includegraphics+, and to specify the  figure width as a multiple of
the line width as in the example below
{\small\begin{verbatim}
   \usepackage[dvips]{graphicx} ...
   \includegraphics[width=0.8\linewidth]
                   {myfile.eps}
\end{verbatim}
}

\subsection{Color}

Please refer to the author guidelines on the CVPR 2020 web page for a discussion
of the use of color in your document.

\section{Final copy}

You must include your signed IEEE copyright release form when you submit
your finished paper. We MUST have this form before your paper can be
published in the proceedings.

Please direct any questions to the production editor in charge of these
proceedings at the IEEE Computer Society Press:
\url{https://www.computer.org/about/contact}.

{\small
\bibliographystyle{ieee_fullname}

\section{Datasets}
Waymo Open Dataset (WOD) \cite{sun2019scalability} is a recently public large-scale dataset for autonomous driving research. The dataset provides 1000 scenes for training and validation, and 150 scenes for testing. Each scene contains about 200 frames for each camera and there are 5 high-resolution cameras with resolution 1280$\times$1920 and 886$\times$1920. Overall, the dataset contains about 1.15M images and 9.9M 2D bounding boxes for vehicles, pedestrians and cyclists. Following the official split provided by WOD, we adopt 798 scenes as the training data and 202 scenes for validation in most of cases. All 5 camera images are used for training and evaluation. We do not use other datasets except for ImageNet \cite{deng2009imagenet}.

\section{Final Results}
\begin{table}[]
	\label{table:final}
	\caption{Submission results on WOD testing set. We report the results of different classes at LEVEL 2. The final submission is an ensemble of models trained on the whole dataset and several expert models trained on a subset of the dataset.}\label{table:results}
	\vspace{8pt}
	\centering
	\footnotesize
	\begin{tabular}{c|ccc|c}
		\hline
		\multicolumn{1}{c|}{Methods} & Vehicle & Pedestrian & Cyclist & {ALL\_NS} \\\hline
		Baseline &56.34 &67.63 &36.92 &53.63\\
		Final submission &\textbf{76.63} &\textbf{81.72} & \textbf{64.94} & \textbf{74.43} \\
		\hline
	\end{tabular}
\end{table}
For final submission, we train multiple models on the union of training and validation sets, including ResNet50-v1b, ResNet50-v1d, Res2Net50-v1b, HRNetv2p-W32 and HRNetv2p-W48\footnote{\href{https://github.com/HRNet/HRNet-Image-Classification}{https://github.com/HRNet/HRNet-Image-Classification}} \cite{wang2020deep}. All the models are pretrained on ImageNet. Linear-Reweight strategy is adopted for final ensemble. Results are shown in Table.\ref{table:final}. The final model achieves 74.43 at LEVEL 2 among ALL\_NS in WOD testing set.
\section{Methods}
In this section, we will introduce our method with bells and whistles used in this challenge. We will first analyze the object scale distribution of WOD, and then describe our baseline model setting followed by our improvements. All the experiments are conducted in SimpleDet \cite{chen2019simpledet}.
\subsection{Small objects matters in WOD}
\label{smallobject}
\begin{figure}[t]
	\centering\includegraphics[height=1.4in]{figs/scale_cropped.pdf}
	\caption{Fraction of bounding boxes in the dataset vs scale of bounding boxes relative to the image.}
	\label{fig:scale}
\end{figure}
Following SNIP \cite{singh2018analysis}, we compare the scale distribution of WOD with the most common used 2d object detection dataset, COCO \cite{lin2014microsoft}. As show in Fig. \ref{fig:scale}, the relative scale of objects in WOD in about 3 times smaller than COCO.
According to the definition of small objects (area smaller than 32 $\times$ 32) in COCO, about 70\% of objects\footnote{Assume the resolution of images in COCO is about 400 $\times$ 600. We define small object as the bounding box whose relative scale is smaller than $\sqrt{32*32/400/600}=0.065$} in WOD are small objects. Detecting very small objects is a challenging problem in 2d object detection task. Several methods have been proposed to handle this issue, such as image pyramids \cite{singh2018analysis}, feature pyramids \cite{lin2017feature} and so on. In the following, we will describe how we adopt these strategies to improve the detection performances.
\subsection{Baseline model}
We adopt FPN \cite{lin2017feature} with ResNet50-v1b \cite{he2016deep, he2019bag} as the baseline model. According to the observation in Sec. \ref{smallobject}, we modify the default setting of anchor scale and RoI assignment strategy in FPN. The area of anchors are defined as \{$12^2$, $24^2$, $48^2$, $96^2$, $192^2$\} pixels on \{$P_2$, $P_3$, $P_4$, $P_5$, $P_6$\}. As for the RoI assignment strategy in RCNN, we assign an RoI of width $w$ and height $h$ to the level $P_k$ of the feature pyramid by:
\begin{equation}
k = \floor{k_0 + \log_2(\sqrt{wh} / 76)}.
\end{equation}

The input images are resized to a short side of 640 for both training and testing. Random horizontal flip is adopted during training. By default, models are trained in a batch size of 64 on 4 nodes. Each node contains 8 2080Ti GPUs. We adopt SGD with momentum 0.9 and weight decay 1e-4 for training. The learning rate is set to 0.02 for batch size of 16. The linear scaling rule with warmup scheme \cite{goyal2017accurate} is adopted for larger batch size. Cosine decay \cite{loshchilov2016sgdr} is used to attenuate the learning rate over time. We adopt synchronized Batch Normalization (BN) \cite{peng2018megdet} in the backbone, FPN neck, and heads. Note we only calculate BN statistics across GPUs on the same node. We train the models for 8 epochs\footnote{we train about 12 epochs for the baseline model, and 8 epochs for the remain experiments.}.

For inference, we use the score 0.03 to filter out background bounding boxes and apply the Non-Maximum Suppression (NMS) with the IoU threshold 0.5 per class to get final predictions. The baseline model achieves an mAP of 57.6 at LEVEL 1 and 50.15 at LEVEL 2 among ALL\_NS (the mean over the APs of vehicles, pedestrians and cyclists.)

\subsection{Bells and whistles}
In this section, we show the step-by-step improvements of different components adopted in our methods. Results on WOD validation set can be found in Table. \ref{table:results}. Note that we adopt score first matcher for fast evaluation.

\noindent\textbf{Multi-scale training.} We adopt multi-scale training with random crop. The scale of short side is randomly sampled from \{512, 640, 960, 1280\} and the scale of long edge is fixed as 2000. After rescaling, images are then cropped to a fixed size of 640$\times$960. For image with annotations, we randomly choose an object and jitter the object center as crop center. As for background image, we do random crop.

\noindent\textbf{Large scale testing.} In order to boost the performance of small objects, we adopt a large scale (1280, 1920) for testing. As shown in Table.\ref{table:results}, large scale testing achieves 60.37 mAP at LEVEL 2 among ALL\_NS, which outperforms the small scale results by 6.7.

\noindent\textbf{Cascade RCNN.} WOD adopt different intersection over union (IoU) thresholds to define positives and negatives for different classes, e.g. 0.7 for Vehicles, 0.5 for both Pedestrians and Cyclists. In order to generate high quality detection, we use Cascade RCNN \cite{cai2018cascade}. Following the original implementation, we set the IoU thresholds to 0.5, 0.6 and 0.7 for each RCNN stage respectively. We also try different IoU thresholds, and find that the default setting yields best performance.

\noindent\textbf{Double-Head.} We adopt a Double-Head \cite{wu2019double} method to replace the default fully connected head (\textit{2-fc}) in FPN. For each RCNN stage in Cascade RCNN, a \textit{2-fc} head is used for classification and a convolution head (\textit{conv-head}) with 3 stacked residual blocks is used for bounding boxes regression.

\noindent\textbf{PAFPN.} Following PANet \cite{liu2018path}, we construct PAFPN module by adding bottom-up path aggregation upon FPN. Instead of repeating only once in the original paper, we stack this module 3 times to enhance feature representation.

\noindent\textbf{Training tricks.} (1) We do not append ground truth into proposals in the RCNN stage during training. (2) Following the training hyper-parameters described in MMDetetcion 2.0 \cite{chen2019mmdetection}, we change the number of proposals after nms from 2000 to 1000.

\noindent\textbf{Multi-scale training with more scales.} We adopt more scales for multi-scale training. The scale of short side is randomly sampled from \{640, 960, 1280, 1600, 1920, 2240, 2560\} and the scale of long edge is fixed as 3840.

\begin{table}[]
	\caption{Step-by-Step improvements on WOD validation set. We report the results of different classes at LEVEL 2.}\label{table:results}
	\vspace{8pt}
	\centering
	\footnotesize
	\begin{tabular}{c|ccc|c}
		\hline
		\multicolumn{1}{c|}{Methods} & Vehicle & Pedestrian & Cyclist & {ALL\_NS} \\\hline
		Baseline       & 50.74 & 64.72 & 34.99& 50.15  \\
		+ Multi-scale training       & 51.67 & 67.19 & 42.16 & 53.67 \\
		+ Large scale testing & 57.33 & 72.90 & 50.88 & 60.37 \\
		+ Cascade RCNN    & 61.10 & 73.67  & 52.00  & 62.26 \\
		+ Doubel Head    & 62.79 & 74.97  & 51.96  & 63.24\\
		+ PAFPN & 63.78 & 75.81 & 48.62 & 62.74 \\
		+ Training tricks & 64.21 & 75.56 & 51.51 & 63.76 \\
		+ More scales training & 64.02 & 75.77 & 54.38 & 64.72 \\
		+ Res2Net50-v1b & 64.19 & 76.16 & 54.93 & 65.09 \\
		+ Calibration and flip & 64.98 & 77.40 & 57.98 & 66.79 \\
		+ Scale-aware testing & 66.38 & 78.76 & 61.66 & 68.93 \\
		+ NMS and Voting & 68.99 & 78.72 & 61.20 & 69.64 \\
		+ 3 models ensemble & \textbf{69.81} & \textbf{79.00} & \textbf{62.62} & \textbf{70.48} \\
		\hline
	\end{tabular}
\end{table}

\noindent\textbf{Calibration.} We find that our FPN models tend to predict overconfident classification score on WOD. The scores of many predictions are nearly 1.0. Such score distribution is not suitable for AP metric, since the PR curve may not be smooth at high precision interval in WOD evaluation setting. To optimize for AP, inspired by \cite{guo2017calibration} and \cite{hinton2015distilling}, we calibrate the classification score by adopting a temperature $T=2$ to produce a softer score distribution over classes.

\noindent\textbf{Scale-aware testing.}  Multi-scale testing is a widely adopted trick for 2d object detection. We adopt two scales for testing, (1280, 1920) and (2240, 3360). By analyzing results obtained from different scales, we found that testing large scale image yields higher mAP on small objects\footnote{For vehicles, we define a bounding box whose area smaller than 67 $\times$ 67 as a small object. For pedestrians and cyclists, we set the area threshold to 57 $\times$ 57.} while the small scale one yields higher mAP on large objects. Based on this observation, we degrade the classification score by multiplying a constant value of 0.6 for small scale predictions when testing with (1280, 1920). As for scale (2240, 3360), we decay the score of large scale objects. This simple strategy brings 1.0 improvement than vanilla multi-scale testing.

\noindent\textbf{NMS and Voting.} We set NMS threshold as 0.7,0.5,0.5 for vehicles, pedestrians and cyclists respectively. Bounding box voting \cite{gidaris2015object} is applied after NMS. For simplicity, the threshold of voting is set to the same as NMS threshold. We also try Soft-NMS \cite{bodla2017soft}. Comparing to  valina NMS with bounding box voting, Soft-NMS yields similar results but is inefficient with numerous predictions.

\noindent\textbf{Ensemble.}  We find that the performance between very deep convolutional neural networks, including ResNet101 and ResNet152, and ResNet-50 is not significant. Especially for cyclists, both ResNet101 and ResNet152 yield worse performance than the shallow one. For training efficiency, we adopt three models, Res2Net50-v1b\footnote{\href{https://github.com/Res2Net/Res2Net-PretrainedModels}{https://github.com/Res2Net/Res2Net-PretrainedModels}} \cite{gao2019res2net}, ResNet50-v1d\footnote{\href{https://gluon-cv.mxnet.io/model_zoo/classification.html}{https://gluon-cv.mxnet.io/model\_zoo/classification.html}} \cite{he2019bag} and ResNeXt50\footnote{\href{https://github.com/apache/incubator-mxnet/tree/master/example/image-classification}{https://github.com/apache/incubator-mxnet/tree/master/example/image-classification}} \cite{xie2017aggregated} for ensemble.
We compare the performances of two different ensemble strategies on validation set. The first one is described in PFDet\cite{akiba2018pfdet} and the other one, named Linear-Reweight, is to assign each model a weight of $w_i$ based on its AP rank $k$ on validation set among $n$ candidates:

\begin{equation}
w_i = \theta_0 + (n - k)\cdot\frac{1 - \theta_0}{n-1}.
\end{equation}
Note that $\theta_0$ is set to 0.5 for all classes.
Based on our results, Linear-Reweight is slightly better than the method proposed by PFDet. So we adopt Linear-Reweight as our final ensemble strategy.

\begin{table}[]
	\caption{The performances of expert models and final ensemble model on the WOD validation set. The results are AP at LEVEL 2 for different classes.}\label{table:expert}
	\vspace{8pt}
	\centering
	\footnotesize
	\begin{tabular}{c|ccc|c}
		\hline
		\multicolumn{1}{c|}{Model} & Vehicle & Pedestrian & Cyclist & {ALL\_NS} \\\hline
    	4 expert models ensemble & 68.61 & 78.54 & 62.01 & 69.72 \\
		3 models ensemble & 69.81 & 79.00 & 62.62 & 70.48  \\
		Final ensemble & \textbf{70.13} & \textbf{79.07} & \textbf{63.31} & \textbf{70.90} \\
		\hline
	\end{tabular}
\end{table}
\subsection{Expert model}
We train several expert models for cyclists since the number of cyclist objects in WOD is only 81k, which is much less than vehicles and pedestrians (9.0M and 2.7M). We do class-aware sampling during training. For fast experiments, we randomly sample 120K images from the whole training dataset each epoch until the ratio between different classes is 1:1:1:0.3 for cyclists, pedestrian, vehicle and background images. We train expert models for 20 epoch. Comparing to models trained on the whole training set, expert models use much less training samples while provide competitive performances. The results can be found in Table.\ref{table:expert}. We combine 4 expert models and 3 models trained on the whole dataset. The final ensemble model yields 70.90 mAP on validation set.

\section{Introduction}


\section{Introduction}
\begin{figure}[t]
 \centerline{
 \includegraphics[width=7.2cm]{figs/figure_1.pdf}}
 \caption{The music sketch scenario. The model is designed to fill the missing part based on the known context and user's own specification.}
 \label{fig:sketch_scenario}
 \vspace{-0.4cm}
\end{figure}

As a research area, automatic music generation has a long history of studying and expanding human expression/creativity \cite{Loy-composing}. The use of neural network techniques in automatic music generation tasks has shown promising results in recent years \cite{DLTFMG}. In this paper, we focus on a specific facet of the automatic music generation problem on how to allow users to flexibly and intuitively control the outcome of automatic music generation. Prior work supports various forms of conditional music generation. MuseGan \cite{MuseGan} allows users to condition generated results on full-length multi-track music. DeepBach \cite{DeepBach} provides a constraint mechanism that allows users to limit the generated results to match composer styles. Music Transformer \cite{MusicTransformer} supports a accompaniment arrangement from an existing melody track in classical music. However, all these approaches require the user preference to be defined in terms of complete musical tracks.

Inspired by the sketching and patching work from computer vision \cite{acm-sketch-2d,eccv-sketch-2d,cvpr-sketch-2d,cvpr-sketch-2d-2,sketch-3d}, we propose Music SketchNet\footnote{\href{https://github.com/RetroCirce/Music-SketchNet}{https://github.com/RetroCirce/Music-SketchNet}.}which allows users to specify partial musical ideas in terms of incomplete and distinct pitch and rhythm representations. More specifically, we generalize the concept of sketching and patching -- wherein a user roughly sketches content for a missing portion of an image -- to music, as depicted in Figure \ref{fig:sketch_scenario}. The proposed framework will complete the missing parts given the known context and user input. To the best of our knowledge, there has been limited work on sketching in music generation. Some work \cite{mcmc-music-inpaint,DeepBach} has used Markov Chain Monte Carlo (MCMC) to generate music with given contexts or generate music conditioned on simple starting and ending notes \cite{anticipationrnn}. The most related task is music inpainting: completing a musical piece by generating a sequence of missing measures given the surrounding context, but without conditioning on any form of user preferences. Music InpaintNet,\cite{musicinpaint} completes musical pieces by predicting vector representations for missing measures, then the vector representations are decoded to output symbolic music through the use of a variational autoencoder (VAE) \cite{vae}.

Our proposed music sketching scenario takes music inpainting a step further. We let users specify musical ideas by controlling pitch contours or rhythm patterns, not by complete musical tracks. The user input is optional: users can choose to specify musical ideas, or let the system fill in predictions without conditioning on user preferences.

Music SketchNet consists of three component, as depicted in Figure~\ref{fig:sketchnet_outline}: (1) SketchVAE is a novel variational autoencoder that converts music measures into high dimensional latent variables. By the use of a factorized inference network, SketchVAE decouples latent variables into two parts: pitch contour and rhythm, which serve as the control parameters for users. (2) SketchInpainter contains stacked recurrent networks to handle the element-level inpainting prediction in the latent space. (3) SketchConnector receives users' sketches of pitch, rhythm, or both, combines them with the prediction from SketchInpainter, and finalizes the generation.

In this paper, we show that the proposed SketchVAE is capable of factorizing music input into latent variables meaningfully, and the proposed SketchInpainter/SketchConnector allows users to control the generative process. The novel training and evaluation methodologies of the SketchConnector are also presented.

\begin{figure}
 \centerline{
 \includegraphics[trim=0 10 10 0, clip, width=\columnwidth]{figs/figure_5.pdf}}
 \caption{The Music SketchNet pipeline. The color patterns inside Inpainter and Connector correspond to the latent space transform and completion process in Figure \ref{fig:sketch_scenario}.}
 \label{fig:sketchnet_outline}
\end{figure}

\section{Music Sketching}
We formalize the music sketching task as solving the following three problems: (1) how to represent music ideas or elements, (2) how to generate new materials given the past and future musical context and (3) how to process users' input and integrate it with the system. A visualization of the sketching scenario is depicted in Figure \ref{fig:sketch_scenario}.

We propose three neural network components to tackle the three problems. The SketchVAE encodes/decodes the music between external music measures and the learned factorized latent representations. The SketchInpainter predicts musical ideas in the form of the latent variables given known context. And the SketchConnector combines the predictions from SketchInpainter and users' sketching to generate the final latent variables which are sent into the SketchVAE decoder to generate music output. A diagram showing the proposed pipeline is shown in Figure \ref{fig:sketchnet_outline}.
\subsection{Problem Definition}

More formally, the proposed sketch framework can be described as a joint probability model of the missing musical content, $X^m$, conditioned on the past, future, and user sketching input. The joint probability breaks down into a product of conditional probabilities corresponding to sub-components of the framework:
\begin{align*}
&P_{\phi,\varepsilon,\gamma,\theta,\tau}(X^{m}, Z, S| X^{p}, X^{f}, C) = \\
&\ P_{\phi}(X^{m} | Z^{m}) \tag{SketchVAE Decoder}\\
& * P_{\varepsilon}(Z^{m} | S^{m}, C) \tag{SketchConnector}\\
& * P_{\gamma}(S^{m}_{pitch} | Z^{p}_{pitch},Z^{f}_{pitch}) \tag{SketchInpainter}\\
& * P_{\gamma}(S^{m}_{rhythm} | Z^{p}_{rhythm},Z^{f}_{rhythm}) \tag{SketchInpainter} \\
& * Q_{\theta}(Z^{p}_{pitch}, Z^{f}_{pitch} | X^{p}_{pitch},X^{f}_{pitch})\\
& * Q_{\tau}(Z^{p}_{rhythm}, Z^{f}_{rhythm} | X^{p}_{rhythm},X^{f}_{rhythm}) \tag{SketchVAE Encoders}
\end{align*}
$X$ indicates the input/output music sequence, $Z$ is the sequence for $\{z\}$ the latent variable
, $S$ is the SketchInpainter's predicted sequence, $C$ is users' sketching input. The superscripts, $p$, $m$, $f$ indicate the past, missing and future context. The subscripts, $pitch$ and $rhythm$ indicate the pitch and rhythm latent variables. $Q_{\theta}$, $Q_{\tau}$, $P_{\phi}$ are the SketchVAE pitch/rhythm encoders and decoder parameters, $P_{\gamma}$ represents the SketchInpainter, and $P_{\varepsilon}$ is the SketchConnector.

\subsection{SketchVAE for Representation}

MusicVAE \cite{musicvae} is one of the first works applying the variational auto-encoder \cite{vae} to music. MeasureVAE \cite{musicinpaint} further focuses on representing isolated measures and utilizes a hierarchical decoder to handle ticks and beats. $EC^{2}$-VAE \cite{ec2vae} factorizes music measures with separate vectors representing pitch and rhythm by a single encoder and two decoders. Our proposed SketchVAE aims to factorize representations by introducing a factorized encoder that considers pitch and rhythm information separately in the encoder channels. Different from $EC^{2}$-VAE, it could allow users to enter parts of the information (rhythm and/or pitch) optionally.

SketchVAE aims to represent a single music measure as a latent variable $z$ that encodes rhythm and pitch contour information in separate dimensions ($z_{pitch}, z_{rhythm}$). It contains (1) a pitch encoder $Q_{\theta}(z_{pitch} | x_{pitch})$, (2) a rhythm encoder $Q_{\tau}(z_{rhythm} | x_{rhythm})$, and (3) a hierarchical decoder $P_{\phi}(x | z_{pitch}, z_{rhythm})$ as shown in Figure \ref{fig:sketchvae}.

\subsubsection{Music Score Encoding}
\begin{figure}[h]
\centerline{
 \includegraphics[width=\columnwidth]{figs/figure_6_r.pdf}}
 \caption{An example of the encoding of a monophonic melody.}
 \label{fig:musicencoding}
\end{figure}

Similar to \cite{musicvae}, we encode the monophonic midi melody by using [0, 127] for the note onsets, 128 for holding state, and 129 for the rest state. We cut each measure into 24 frames to correctly quantize eighth-note triplets like \cite{musicinpaint}, and encode the midi as described in the previous sentence.

As Figure \ref{fig:musicencoding} shows, we further process the encoded 24-frame sequence $x$ into $x_{pitch}$ and $x_{rhythm}$, the pitch and rhythm token sequences respectively. The pitch token sequence $x_{pitch}$ is obtained by picking all note onsets in $x$ with padding (shown by "$\bullet$" in Figure \ref{fig:musicencoding}) to fill 24 frames. The rhythm token sequence $x_{rhythm}$ is obtained by replacing all pitch onsets with the same token (shown by "O" and "\_" in Figure \ref{fig:musicencoding}). A similar splitting strategy is also used in \cite{mumeAlex}. Our motivation is to provide users with two intuitive music dimensions to control, and to help enforce better factorization in the latent representation for later prediction and control.

\subsubsection{The Pitch Encoder and Rhythm Encoder}
After pre-processing $x$, $x_{pitch}$ only contains the note value sequence, while $x_{rhythm}$ only has the duration and onset information. $x_{pitch}$ and $x_{rhythm}$ are then fed into two different GRU \cite{GRU} encoders for variational approximation. The outputs of each encoder are concatenated into $z = [z_{pitch}, z_{rhythm}]$.

\subsubsection{The Hierarchical Decoder}
After we obtain the latent variable $z$, we feed it into the hierarchical decoder. This decoder is similar to the decoder used in MeasureVAE \cite{musicinpaint}. As shown in the bottom part in Figure \ref{fig:sketchvae}, it contains an upper "beat" GRU layer and a lower "tick" GRU layer. This division's motivation is to decode $z$ into $n$ beats first and then decode each beat into $t$ ticks. As a result, the note information in each measure will be decoded in a musically intuitive way. For the tick GRU, we use the teacher forcing \cite{teacherforcing-1,teacherforcing-2} and auto-regressive techniques to train the network efficiently. The output is conditioned frame-by-frame not only on the beat token but also on the last tick token.

\begin{figure}[t]
 \centerline{
 \includegraphics[width=\columnwidth]{figs/figure_2.pdf}}
 \caption{SketchVAE structure: pitch encoder, rhythm encoder and hierarchical decoder. Rhythm tokens: the upper dashes denote the onsets of note, and the bottom dashes denote the hold/duration state. We use pitch symbols to represent the tokens numbers for better illustration.}
 \label{fig:sketchvae}
\end{figure}

\begin{figure*}[t]
 \centerline{
 \includegraphics[width=16cm]{figs/figure_3.pdf}}
 \caption{SketchInpainter structure. We feed the music tokens into the SketchVAE and obtain the latent variable sequences. And we feed the sequences into the pitch GRU and the rhythm GRU groups to generate the initial prediction $S$.}
 \label{fig:sketchgen}
\end{figure*}
\subsubsection{Encoding the Past, Missing and Future Musical Context}
The latent variable sequences $Z^p$, $Z^m$, and $Z^f$ are then obtained by processing the music input in measure sequences $X^p$, ${X^m}$, and ${X^f}$. Both $X^m$ and $Z^m$ are masked during training. This encoding part is shown in the left block of Figure \ref{fig:sketchgen}.

\subsection{SketchInpainter for Initial Prediction}
Next, we describe the model component that performs the music inpainting to predict latent representations for the missing measures. The SketchInpainter accepts $Z_{pitch}$ and $Z_{rhythm}$ as two independent inputs from SketchVAE. Then only the past and future $Z_{pitch}$ and $Z_{rhythm}$ are fed into the pitch/rhythm GRU groups respectively. The output from each GRU group is the hidden state $h$, as shown in the middle of Figure \ref{fig:sketchgen}.

Then we combine the past/future hidden states $h$ from both the pitch and rhythm GRU groups and use them as the initial states for the pitch/rhythm generation GRUs. The generation GRUs then predict the missing latent variables by $S^m = (S_{pitch}, S_{rhythm})$, as shown in the green box in Figure \ref{fig:sketchgen}. Each generation GRU is trained with the teach forcing and auto-regressive techniques.

Each output vector $s^m$ from $S^{m}$ has the same dimension as the latent variable $z$ from $Z$. We first build a model with only SketchVAE and SketchInpainter that directly predicts the missing music material, $X^m$. As the right block of Figure \ref{fig:sketchgen} shows, $S^m$ is sent into the SketchVAE decoder and we compute the cross entropy loss between the predicted music output and the ground truth. This is the stage I training in our model, detailed in Section 3.3.

\subsection{SketchConnector for Finalization}
The predicted $S^m$ from SketchInpainter can already serve as a good latent representation for the missing part $X^m$. We continue by devising the SketchConnector, $P_{\varepsilon}(Z^{m} | S^{m}, C)$, to modify the prediction with user control. To make up for the lack of correlation between pitch and rhythm in current predictions, we introduce the SketchConnector as a way to intervene/control the generative process, that also leads to a wider musical expressivity of the proposed system.

\subsubsection{Random Unmasking}
With $S^m$ obtained from SketchInpainter, we concatenate it with $Z^p$ and $Z^f$ again. However, before we feed it back into the network, we randomly unmask some of the missing parts to be the ground-truth (simulating user providing partial musical context). The masked $S^m$ are shown by the red boxes in Figure \ref{fig:sketchconnector}. We replace some $s$ from $S^m$ to be the real answer in $Z^m$, denoted as $C$. We observe that this optimization is very similar to BERT \cite{BERT} training. The difference is that BERT randomly masks the ground truth labels to be unknown, but SketchConnector randomly unmasks the predictions to be truths. The unmasking rate is set to 0.3.

Intuitively, this allows the model to learn a more close relation among current rhythm, pitch tokens, and the nearest neighbour tokens. In the sketch inference scenario, the randomly unmasked measures will be replaced by the user sketching information, which allows a natural transition between the training and testing process.

\subsubsection{Transformer-based Connector}

Then with $S^m$ and the random unmasking data $C$, we feed them into a transformer encoder with absolute positional encoding. In contrast to \cite{MusicTransformer}, we do not use relative positional encoding because our inputs are vectors representing individual measures, whose length is far shorter than midi-event sequences.

\begin{figure}[h]
\centerline{
 \includegraphics[width=6.0cm]{figs/figure_4.pdf}}
 \caption{The SketchConnector: the output of SketchInpainter is randomly unmasked and fed into a transformer encoder to get the final output.}
 \label{fig:sketchconnector}
 \vspace{-0.4cm}
\end{figure}

\begin{table*}[]
\begin{tabular}{@{}c|ccc|ccc|ccc|@{}}
\toprule
& \multicolumn{3}{c}{Irish-Test} & \multicolumn{3}{c}{Irish-Test-R} & \multicolumn{3}{c}{Irish-Test-NR} \\ \midrule
Model               & loss $\downarrow$     & pAcc $\uparrow$     & rAcc $\uparrow$     & loss $\downarrow$      & pAcc $\uparrow$     & rAcc $\uparrow$     & loss $\downarrow$     & pAcc $\uparrow$      & rAcc $\uparrow$      \\ \midrule
Music InpaintNet            & 0.662    & 0.511    & 0.972    & 0.312     & 0.636     & 0.975    & 0.997     & 0.354     & 0.959     \\ \midrule
SketchVAE + InpaintRNN      & 0.714    & 0.510    & 0.975    & 0.473     & 0.619     & 0.981    & 1.075     & 0.374     & 0.964     \\ \midrule
SketchVAE + SketchInpainter & 0.693    & 0.552    & 0.985    & 0.295     & 0.692     & 0.991    & 1.002     & 0.389     & 0.977     \\ \midrule
SketchNet                   & \textbf{0.516}    & \textbf{0.651}    & \textbf{0.985}    & \textbf{0.206}     & \textbf{0.799}     & \textbf{0.991}    & \textbf{0.783}     & \textbf{0.461}     & \textbf{0.977}     \\ \bottomrule
\end{tabular}
\caption{The generation performance of different models in Irish and Scottish  monophonic music dataset. The InpaintRNN is the generative network in Music InpaintNet.}
\label{tab:sketch-result}
\end{table*}

The output of the SketchConnector, $Z^{m}$, will be the final prediction for the missing part. We feed it into the SketchVAE decoder, and compute the cross entropy loss of the output with the ground-truth.

\section{Experiment}
\subsection{Dataset and Baseline}
To evaluate the SketchVAE independently, we compare our model with two related systems: MeasureVAE \cite{musicinpaint} and $EC^2$-VAE \cite{ec2vae}. For SketchNet, we compare our generation results with Music InpaintNet \cite{musicinpaint}, which has shown better results than the earlier baseline \cite{anticipationrnn}. Similar to \cite{musicinpaint}, we use the Irish and Scottish monophonic music Dataset \cite{IrishDataset} and select the melodies with a 4/4 time signature. About 16000 melodies are used for training and 2000 melodies for testing.

\subsection{SketchVAE Measurements}
\subsubsection{Reconstruction}
For SketchVAE, MeasureVAE and $EC^2$-VAE, the dimension of latent variable $|z|$ is set to 256, half for the pitch contour, and the other half for the rhythm. We set the learning rate to 1e-4 and use Adam Optimization with $\beta_1 = 0.9$ and $\beta_2 = 0.998$. Three models achieve the accuracy (the reconstruction rate of melodies) 98.8\%, 98.7\%, 99.0\% respectively. We can clearly conclude that all VAE models are capable of converting melodies to latent variables by achieving the accuracy around 99\%. SketchVAE is capable of encoding/decoding musical materials in SketchNet.

\subsubsection{Comparison with $EC^2$-VAE}
$EC^{2}$-VAE \cite{ec2vae} is also capable of decoupling the latent variable into rhythm and pitch contour dimensions. However, SketchVAE's encoders can accept pitch contour/rhythm inputs separately. Rhythm and pitch controls can be manipulated independently in the sketching scenario where the user might not specify an entire musical measure (e.g., just a rhythm pattern). By contrast, $EC^{2}$-VAE requires a completed measure before encoding. If users want to specify either rhythm or pitch controls, the model must first fill in the other half part before inputting it, which prohibits the possibility of the separate control.

\begin{figure*}[t]
 \centerline{
 \includegraphics[width=\textwidth]{figs/figure_7.pdf}}
 \caption{An example of sketch generation. From top to bottom: original, pitch/rhythm/mixture control. The blue pitch texts denote pitch controls, and the pink segments denote rhythm controls.}
 \label{fig:sketch_example}
\end{figure*}

\subsection{Generation Performance}
\subsubsection{Training Results}
The SketchNet's training is separated into stage I and II. In stage I, after training the SketchVAE, we freeze its parameters and train the SketchInpainter as shown in the right block of Figure \ref{fig:sketchgen}. In stage II, with the trained SketchVAE and SketchInpainter, we freeze both, concatenate $S^m$ with the past/future latent variables, and feed them to the SketchConnector for training.

We compare four models by using 6 measures of past and future contexts to predict 4 measures in the middle (i.e. $n_p = n_f = 6$, and $n_m = 4$ ). Music InpaintNet \cite{musicinpaint} is used as the baseline, along with several variations. Early stopping is used for all systems.

\begin{table}[]
\begin{tabular}{@{}cccc@{}}
\toprule
\multicolumn{1}{l}{Model}         & Complexity$\uparrow$ & Structure$\uparrow$ & Musicality$\uparrow$ \\ \midrule
Original                    & 3.22        & 3.47      & 3.56               \\ \midrule
InpaintNet                     & 2.98        & 3.01      & 3.09               \\ \midrule
SketchNet                         & 3.04        & 3.29      & 3.26               \\ \bottomrule
\end{tabular}
\caption{Results of the subjective listening test.}
\label{tab:listen-result}
\end{table}

We compute three metrics: loss, pitch accuracy, and rhythm accuracy to evaluate the model's performance. The pitch accuracy is calculated by comparing only the pitch tokens between each generation and the ground truth (whether the model generates the correct pitch in the correct position). And the rhythm accuracy is calculated by comparing the duration and onset (regardless of what pitches it generates). The overall accuracy and loss are negatively correlated.

For this part of the experiment, we also use two special test subsets. We compute the similarities between the past and future contexts of each song in the Irish test set, pick the top 10\% similar pairs (past and future contexts are almost the same) and bottom 10\% pairs (almost different), and create the Irish-Test-R (repetition) and Irish-Test-NR (non-repetition) subsets.

From Table \ref{tab:sketch-result}, we can see that SketchNet beats all other models for all test sets. The performance improved more for pitch then for rhythm. The accuracy is almost the same between the 1st and 2nd model. Accuracy is slightly better if we use SketchInpainter to treat rhythm and pitch independently during generation. Lastly, with the power of transformer encoder and random unmasking process done in SketchConnector, we can achieve the best performance by using SketchNet (bottom row in Table \ref{tab:sketch-result}). We further follow \cite{sigtest} to use the Bootstrap significance test to verify the difference between each pair's overall accuracy for models in the whole Irish-Test set (Four models, i.e. six pairs in total). The sample time is set to 10000. After calculation, all p-values except the fist and second model pair (p-value = 0.402) are less than 0.05, which proves that SketchNet is different from the left three models.

In the repetition test subsets, the loss of Music InpaintNet is 0.312, which is lower enough to capture repetitions in the musical context and fill in the missing part by copying. In most cases, copying is the correct behaviour because the original melody has repetitive pattern structures. The loss is a measurement to evaluate if the model can learn the repetitive pattern and copy mechanism from the data. the SketchNet slightly outperforms InpaintNet.

\subsubsection{Subjective Listening Test}
However, the more interesting result is the generation with non-repetition subset. In this case, models cannot merely copy because original melodies do not repeat its content. We see higher losses in all models in this subset compared to the repetition subset. Intuitively, it means that repetitive patterns are essential to the reconstruction task, not necessarily the expressivity of the generated output would be less.

To further evaluate the proposed SketchNet, we conduct an online subjective listening test to let subjects judge the generated melodies from the non-repetition subset. Each subject will listen to three 32-second piano-rendered melodies: the original, the Music InpaintNet's generation, and the SketchNet's generation. Songs are randomly picked from the Irish-Test-NR set. The beginning and ending (past \& future) are the same for the three melodies. Since the subjective feeling of music is complicated to quantify, we chose three criteria: the number of notes (\textbf{complexity}), the repetitiveness between musical structures (\textbf{structure}), and the degree of harmony of the music (\textbf{overall musicality}). In this way, subjects with different levels of music skills can all give reasonable answers.

Before rating the songs, subjects will see three criteria descriptions as we introduced below. The rating is ranged from 1.0 to 5.0 with a 0.5 step. We collected 318 surveyed results from 106 subjects (each subject listens to three groups, nine melodies in total). The average rating of each criteria for all models are shown in Table \ref{tab:listen-result}. The subjective evaluations of all three criteria in SketchNet are better for those of Music InpaintNet. Similar to section 3.3.1, we also conduct a pairwise significance test via Bootstrap in three criteria. All p-values except the <complexity: InpaintNet, SketchNet> (p-value = 0.364) are less than 0.05. It proves that three models (including original songs) are significantly different in structure and overall musicality (subjective feeling to a person). As for the complexity, we believe that the results generated by the two models are similar in terms of the richness of notes, and our model does not significantly increase the number of notes generated.

\begin{table}[]
\centering
\resizebox{5.5cm}{!}{%
\begin{tabular}{cllclllclll}
\hline
\multicolumn{3}{c}{\small{Control Info.}} & \multicolumn{4}{c}{\small{Rhythm}} & \multicolumn{4}{c}{\small{Pitch}} \\ \hline
\multicolumn{3}{c}{\small{Pitch Acc.}} & \multicolumn{4}{c}{\small{0.189}} & \multicolumn{4}{c}{\small{\textbf{0.881}}} \\
\multicolumn{3}{c}{\small{Rhythm Acc.}} & \multicolumn{4}{c}{\small{\textbf{0.973}}} & \multicolumn{4}{c}{\small{0.848}} \\ \hline
\end{tabular}%
}
\caption{The accuracy of the virtual control experiment.}
\label{tab:v-test}
\end{table}

\subsection{Sketch Scenario Usage}
The contribution of Music SketchNet is not only shown in the performance of the generation in section 3.3, but can also be shown in the interactive scenario where users can control the generated output by specifying the rhythm or pitch contour in each measure.

Figure \ref{fig:sketch_example} shows an example of a non-repetition subset melody, where the first and last two measures are given, and the middle parts is generated. The first track is the original melody, the second track is generated with the pitch contour control, the third track is generated with the rhythm control, and the fourth track is controlled with both pitch and rhythm. We can see that each generated melody follows the control from users and develops music phrases accordingly in the missing part. Moreover, each measure is in line with the past and future measures even in the case of scale shift.

We also provide a "virtual control experiment" to statistically show that users' control did influence the model's generation process. We randomly collect 3000 sample pairs (A, B) from the Irish-Test set. And we use the pitch/rhythm of Sample B to be the sketch information in the same missing position of Sample A. Then we let the model make the generation. We then compute the pitch/rhythm accuracy\footnote{The metric to calculate the pitch accuracy is different from section 3.3.1, because the generated pitches in the new song might have different onset positions. We leverage the Longest Common Sequence to calculate the accuracy. The implementation is presented in the code archive.} in the missing position between the generation and Song B. From \ref{tab:v-test} we can see if we sketch song B's rhythm into the model, the generation will follow the rhythm with 97.3\% accuracy but has different (18.9\%) pitches. However, when we sketch pitches, the pitches in the generation will be highly (88.1\%) in line with the sketching. This proves that the user's control has a relatively high guiding effect on the result of the model generated at the specified position.

\section{Conclusion \& Future Work}
In this paper, we propose a new framework to explore decoupling latent variables in music generation. We further convert this decoupling into controllable parameters that can be specified by the user. The proposed Music SketchNet achieves the best results in the objective and subjective evaluations. Practically, we show the framework's application for the music sketching scenario where users can control the pitch contour and/or rhythm of the generated results. There are several possible extensions for this work. Music elements other than pitch and rhythm can be applied into the music sketching scenario by the latent variable decoupling. Also, how to represent a polyphonic music piece in the latent space is another pressing issue. Both are future works that can generalize this model to more applied scenarios.

\section{Acknowledgement}

We would like to thank Cygames for the partial support of this research.