\section{Introduction}

The high energy scale during inflation, presumably well beyond the reach
of the current and future particle accelerator experiments, calls for an
effective theory description of inflation
\cite{Cheung:2007st,Weinberg:2008hq}. This is because by construction
the effective field theory approach is systematic through which we can
account for our ignorance. A key observation in writing the effective
field theory of single-field inflation\footnote{
Extensions to the multi-field case are possible under certain constraints
\cite{multi-eft}.
} is to note that in the time-dependent
background the time translational symmetry is broken, while spatial
diffeomorphism is preserved \cite{Cheung:2007st}. The couplings that
determine the expansion of the effective theory of the Goldstone mode
$\pi$, which realizes the time diffeomorphism, are represented by a set
of mass scales $M_n^4$. In the so-called decoupling regime the Goldstone
$\pi$ could decouple from the metric fluctuations and the effective
action of $\pi$ is dramatically simplified. Especially, the first
expansion parameter $M_2^4$ is manifest in both quadratic and cubic
order of $\pi$: see \eqref{eq:Spi}.

The observation that the coefficient $M_2^4$ is common to the quadratic
and cubic action of $\pi$ indicates that, to leading order in the
decoupling limit, the corresponding correlation functions -- the power
spectrum and bispectrum -- are explicitly correlated. It means that
ideally, given an explicit analytic form of the power spectrum theoretically, we can
find unambiguously the corresponding bispectrum. Or, at the very least
observationally, it remains tantalizing because of the existence of
outliers in the power spectrum of the temperature fluctuations of the
cosmic microwave background~\cite{cmboutliers}. The explicit correlation would make
possible joint analysis using the two- and three-point correlation
functions~\cite{joint}, which can place much stronger constraints on cosmological
parameters. It can also open a compelling way of searching for new physics beyond
the paradigm of standard slow-roll inflation, since any deviations would strongly
signal the typical mass scale associated with new physics~\cite{newscale}.

In this article, we derive a direct and explicit relation between the
power spectrum and bispectrum of the primordial curvature perturbation
using the Goldstone mode $\pi$. Such a correlation was first explicitly
studied in the top-down approach \cite{Achucarro:2012fd} and
expanded into more general context in~\cite{Gong:2014spa}, in which heavy
degrees of freedom are integrated out to lead to an effective single
field description of inflation~\cite{cs-heavy} (see also~\cite{heavyeffects}).
To leading order of the heavy mass scale, the speed of sound $c_s$ uniquely
characterizes the effects of the heavy degrees of freedom
\cite{cs-heavy}, i.e. the coefficients of the effective
theory. Our approach here is conversely bottom-up, complementary to the
previous studies as we will see in the main text.

The article is organized as follows. In the next section, after briefly
reviewing the effective field theory of inflation, we derive the simple
expression of the correction to the power spectrum. By inverting it we
can write the unknown, model-dependent effective theory parameter in
terms of the power spectrum which can be constrained observationally.
In Section~\ref{sec:correlation}, we derive a direct and explicit
relation between the corrections of the power spectrum and bispectrum.
In Section~\ref{sec:squeezed} we discuss the consistency relation of the
squeezed bispectrum \cite{consistency} and the validity of the
correlation we derive. The final section is devoted to summary and
conclusions.

\section{Effective theory and correction to power spectrum}
\label{sec:P}

In this section, after briefly reviewing the effective field theory of inflation,
we give the formula of the correction to the power spectrum due to
the deviation from usual slow-roll phase parametrized by the
expansion coefficient of the effective theory.

\subsection{Brief review of effective field theory of inflation}

We begin with a brief review of the effective field theory of
inflation~\cite{Cheung:2007st}. In unitary gauge,
the information on the primordial curvature perturbation is encoded in
geometrical quantities respecting the time-dependent spatial
diffeomorphism symmetry. Then, the action for the primordial curvature
perturbation is written in general as
\begin{equation}
  S = \int d^4 x \sqrt{-g} F(g^{\mu\nu}, g_{\mu\nu},
  K_{\mu\nu}, R_{\mu\nu\rho\sigma}, \nabla_{\mu}, t) \, ,
\end{equation}
where $K_{\mu\nu}$ is the extrinsic curvature with respect to $t=$ constant
hypersurface. Since the zeroth and first order terms are determined by the
background quantities, the action can be expanded as
\begin{align}
S = \int d^4 x \sqrt{-g} & \left\{ \frac12 m_{\rm Pl}^2 R + m_{\rm Pl}^2 \dot{H}(t) g^{00}
- m_{\rm Pl}^2 \left[ 3H^2(t)+\dot{H}(t) \right]
\right.
\nonumber\\
& + F \left( \delta g^{00},\delta K_{\mu\nu},
             \delta R_{\mu\nu\rho\sigma} ;
             g_{\mu\nu},g^{\mu\nu},\nabla_{\mu},t \right) \bigg\} \, ,
\end{align}
where $F$ represents second and higher order perturbation terms and
is given by
\begin{align}
F & = \frac{1}{2} M_2^4(t) \left( \delta g^{00} \right)^2
                      + \frac{1}{3!} M_3^4(t) \left( \delta g^{00} \right)^3 + \cdots
\nonumber\\
& \quad    - \frac12 \bar{M}_1^3(t) \delta g^{00} \delta K
                      - \frac12 \bar{M}_2^3(t) K \delta K
                      - \frac12 \bar{M}_3^3(t) K^{\mu}{}_{\nu} \delta K^{\nu}{}_{\mu}
                      - \cdots \, ,
\end{align}
with $K \equiv K^\mu{}_\mu$.
It is noticed that time diffeomorphism invariance is broken in this
action. But, it can be recovered by the introduction of the
St\"uckelberg field $\pi(x)$, which corresponds to the Nambu-Goldstone boson
and transforms
under the coordinate transformations  $t \rightarrow \widetilde{t} = t +\xi^0(x)$ and
$\mathbi{x} \rightarrow \widetilde{\mathbi{x}}$ as
\begin{equation}
  \pi(x) \longrightarrow \widetilde{\pi}(\widetilde{x}(x))
  = \pi(x) - \xi^0(x) \, .
\end{equation}
In the decoupling regime $H \gtrsim M_2^2/m_{\rm Pl}$,
the action reduces to
\begin{equation}
\label{eq:Spi}
S_\pi = \int d^4x \sqrt{-g} \left\{ \frac{m_{\rm Pl}^2}{2}R - m_{\rm Pl}^2\dot{H} \left[ \dot\pi^2 - \frac{(\nabla\pi)^2}{a^2} \right]
+ 2M_2^4 \left[ \dot\pi^2 + \dot\pi^3 - \dot\pi\frac{(\nabla\pi)^2}{a^2}
	 \right] - \frac{4}{3}M_3^4\dot\pi^3 + \cdots \right\} \, ,
\end{equation}
where the dots represent the higher derivative terms. The sound velocity
$c_s$ is related to $M_2$ as
\begin{equation}
\label{eq:cs}
c_s^{-2} = 1 - \frac{2M_2^4}{m_{\rm Pl}^2\dot{H}} \, .
\end{equation}
In this article, we further set $M_3(t) = 0$ because $M_3^4 \sim
(1-c_s^{-2}) M_2^4$ on general arguments \cite{Senatore:2009gt}.  $\pi$
and ${\cal R}$ are related to linear order by $\pi = -{\cal R}/H$, so in the
regime \eqref{eq:Spi} is valid we can to first approximation consider
$\dot\pi \approx -\dot{\cal R}/H$.

\subsection{Corrections to the power spectrum}
\label{subsec:power}

We first concentrate on the quadratic part and evaluate the correction
to power spectrum originating from the term with $M_2^4$.
Since the standard slow-roll terms multiplied by $\dot{H}$ in \eqref{eq:Spi}
are dominant as various observations indicate, we treat
the quadratic contribution of $M_2^4$ as perturbation. In terms of the
speed of sound \eqref{eq:cs}, we assume that for a limited duration
$c_s$ deviates from unity, with the deviation being not too far away
from unity.
Neglecting the metric perturbation as we consider the
decoupling regime so that $\sqrt{-g} = a^3$ simply, from \eqref{eq:Spi}
the quadratic part other than the usual slow-roll, which we may call second
order interaction, is
\begin{equation}
S_\text{2,int} = \int d^4x a^3 2M_2^4(t)\dot\pi^2 \, .
\label{eq:2int}
\end{equation}
The interaction Hamiltonian is then\footnote{One should be careful when the
interaction Lagrangian includes derivative terms. Conjugate momentum must
be defined by use of the full Lagrangian rather than the free part.}
\begin{equation}
\label{eq:Hint-2}
H_\text{int} = \int d^3x a^3 (-2) c_s^2 M_2^4(t)\dot\pi^2
             \approx \int d^3x a^3 (-2) M_2^4(t)\dot\pi^2 \, ,
\end{equation}
where we have used the assumption that $c_s$ is not too far away from
unity. This interaction Hamiltonian can be expressed in terms of the
Fourier mode as
\begin{equation}
H_\text{int} = -2aM_2^4 \int \frac{d^3q_1d^3q_2}{(2\pi)^3} \delta^{(3)}(-\mathbi{q}_{12}) \pi_{\mathbi{q}_1}'\pi_{\mathbi{q}_2}' \, ,
\end{equation}
where $\mathbi{q}_{12\cdots n} \equiv \mathbi{q}_1+\mathbi{q}_2+\cdots+\mathbi{q}_n$, a prime
represents a derivative with respect to the conformal time $d\tau =
dt/a$, and
\begin{equation}
   \pi(\tau,\mathbi{x}) = \int \frac{d^3q}{(2\pi)^3} e^{i\mathbi{q}\cdot\mathbi{x}} \pi_{\mathbi{q}}(\tau).
\end{equation}
Now we can compute the corrections using the standard in-in
formalism. We can straightly obtain
\begin{align}
\label{eq:Ppi-corr}
\Delta \left\langle \pi_{\mathbi{k}_1}\pi_{\mathbi{k}_2}(\tau) \right\rangle
& \equiv (2\pi)^3 \delta^{(3)}(\mathbi{k}_{12}) \frac{2\pi^2}{k_1^3} \Delta{\cal P}_\pi
\nonumber\\
& = i \int_{\tau_0\to-\infty}^{\tau\to0} a d\tau' \left\langle 0 \left| \left[ H_\text{int}(\tau'), \pi_{\mathbi{k}_1}\pi_{\mathbi{k}_2}(\tau) \right] \right| 0 \right\rangle
\nonumber\\
& = (2\pi)^3 \delta^{(3)}(\mathbi{k}_{12})
2\Re \left[ 2i \widehat{\pi}_{{k}_1}^*\widehat{\pi}_{{k}_2}^*(0)
\int_{-\infty}^0 d\tau \left( -2a^2M_2^4 \right) \widehat{\pi}_{{k}_1}'\widehat{\pi}_{{k}_2}'(\tau) \right] \, ,
\end{align}
where we have expanded the free field $\pi_\mathbi{k}$ using the creation and annihilation operators as
\begin{equation}
\label{eq:Roperator}
\pi_\mathbi{k} = a_\mathbi{k} \widehat{\pi}_k + a_{-\mathbi{k}}^\dag \widehat{\pi}_k^*
\quad \text{with} \quad
\left[ a_\mathbi{k}, a_\mathbi{q}^\dag \right] = (2\pi)^3 \delta^{(3)}(\mathbi{k}-\mathbi{q}) \, ,
\end{equation}
and $\widehat{\pi}_{k}(\tau)$ is the mode function solution given by
\begin{equation}
\label{eq:pi-sol}
\widehat{\pi}_k(\tau) = -\frac{\widehat{{\cal R}}_k}{H}
= \frac{-i}{\sqrt{4\epsilon k^3}m_{\rm Pl}} (1+ik\tau) e^{-ik\tau} \, .
\end{equation}
Thus, we immediately find the correction to the power spectrum as
\begin{equation}
  \frac{\Delta{\cal P}_\pi}{{\cal P}_\pi}(k) \approx
 \frac{\Delta{\cal P}_{\cal R}}{{\cal P}_{\cal R}}(k)
 \approx \frac{k}{m_{\rm Pl}^2\epsilon H^2}
\int_{-\infty}^0 d\tau \left( -2M_2^4 \right) \sin(2k\tau) \, ,
\label{eq:correction_p}
\end{equation}
where
\begin{equation}
{\cal P}_\pi = \frac{{\cal P}_{\cal R}}{H^2} = \frac{1}{8\pi^2m_{\rm Pl}^2\epsilon}
\end{equation}
is the featureless flat spectrum.

\subsection{Inverting the power spectrum}

For future convenience, let us return to \eqref{eq:Ppi-corr} and write
it in an alternative form. The real part is obtained by
adding the complex conjugate:
\begin{align}
\label{eq:Ppi-corr3}
\frac{2\pi^2}{k_1^3}\Delta{\cal P}_\pi & = 2\Re \left[ 2i \widehat{\pi}_{{k}_1}^*\widehat{\pi}_{{k}_2}^*(0)
\int_{-\infty}^0 d\tau \left( -2a^2M_2^4 \right) \widehat{\pi}_{{k}_1}'\widehat{\pi}_{{k}_2}'(\tau) \right]
\nonumber\\
& = 2i \widehat{\pi}_{{k}_1}^*\widehat{\pi}_{{k}_2}^*(0)
\int_{-\infty}^0 d\tau \left( -2a^2M_2^4 \right) \widehat{\pi}_{{k}_1}'\widehat{\pi}_{{k}_2}'(\tau) + c.c. \, .
\end{align}
By noting from \eqref{eq:pi-sol} that
$\widehat{\pi}_k(-\tau) = -\widehat{\pi}_k^*(\tau)$ and $\widehat{\pi}_k'(-\tau) = {\widehat{\pi}_k}^*(\tau)$,
and by oddly extending $M_2^4$ to {\em define} $\widetilde{M}_2^4$ as
\begin{equation}
\widetilde{M}_2^4(\tau) \equiv \left\{
\begin{array}{ll}
M_2^4(\tau) & \text{ if } \tau < 0
\\
-M_2^4(-\tau) & \text{ if } \tau > 0
\end{array}
\right. \, ,
\end{equation}
\eqref{eq:Ppi-corr3} can be written as\footnote{
Notice that we are at this stage not directly computing the propagator by
adopting the $i\varepsilon$ prescription of the contour, which remains unchanged
though. Our goal is to invert \eqref{eq:correction_p} by incorporating mathematical manipulations in
such a way that the model-dependent parameter $M_2^4$ is given in terms of
$\Delta{\cal P}_\pi$ which can be observationally constrained.
}
\begin{align}
\frac{2\pi^2}{k_1^3}\Delta{\cal P}_\pi & = 2i \widehat{\pi}_{{k}_1}^*\widehat{\pi}_{{k}_2}^*(0)
\int_{-\infty}^\infty d\tau \left( -2a^2\widetilde{M}_2^4 \right) \widehat{\pi}_{{k}_1}'\widehat{\pi}_{{k}_2}'(\tau)
\nonumber\\
& = 2\pi^2{\cal P}_\pi \frac{1}{k_1k_2} \frac{1}{2}
\int_{-\infty}^\infty d\tau \frac{-2\widetilde{M}_2^4}{\epsilonm_{\rm Pl}^2H^2} ie^{-ik_{12}\tau} \, .
\end{align}
Since we have defined $\widetilde{M}_2^4$ oddly, only the odd part of $e^{-ik_{12}\tau}$ survives and finally we have, setting $k_1=k_2=k$,
\begin{equation}
\label{eq:Ppi-corr-final}
\frac{\Delta{\cal P}_\pi}{{\cal P}_\pi} = \frac{k}{2}
\int_{-\infty}^\infty d\tau \frac{-2\widetilde{M}_2^4}{\epsilonm_{\rm Pl}^2H^2} \sin(2k\tau) \, .
\end{equation}

From \eqref{eq:Ppi-corr-final} we can write the coefficient $\widetilde{M}_2^4$, which is essentially $M_2^4$ in the effective action \eqref{eq:Spi},
in terms of $\Delta{\cal P}_\pi/{\cal P}_\pi$ as follows. From
$\sin(2k\tau) = \left( e^{2ik\tau} - e^{-2ik\tau} \right)/(2i)$,
we can multiply $e^{2ik\tau'}$ to both sides of \eqref{eq:Ppi-corr-final}
and integrate over $k$ to obtain
\begin{equation}
\int_{-\infty}^\infty dk e^{2ik\tau'} \frac{2i}{k} \epsilonm_{\rm Pl}^2H^2 \frac{\Delta{\cal P}_{\cal R}}{{\cal P}_{\cal R}}(k)
= \frac{1}{2} \int_{-\infty}^\infty d\tau \left( -2\widetilde{M}_2^4 \right)
\int_{-\infty}^\infty dk \left[ e^{2ik(\tau+\tau')} -
 e^{-2ik(\tau-\tau')} \right]
= 2\pi \widetilde{M}_2^4(\tau') \, .
\end{equation}
Thus,
\begin{equation}
\label{eq:coeff-P}
2\widetilde{M}_2^4(\tau) = i\frac{2\epsilonm_{\rm Pl}^2H^2}{\pi}
\int_{-\infty}^\infty \frac{dk}{k} \frac{\Delta{\cal P}_{\cal R}}{{\cal P}_{\cal R}}(k) e^{2ik\tau} \, .
\end{equation}
This is the inverse formula, in which $M_2^4$ can be expressed in terms
of the correction to power spectrum.

\section{Correlation between power spectrum and bispectrum}
\label{sec:correlation}

In this section, we first give the formula of bispectrum coming from
the cubic action \eqref{eq:Spi},
and then derive the explicit relation between
the correction to the power spectrum and the bispectrum.

\subsection{Bispectrum}

As advertised before, we only consider
the cubic order action with the coefficient $M_2^4$:
\begin{equation}
S_{3} = \int d^4x \sqrt{-g} 2M_2^4 \left[ \dot\pi^3 - \dot\pi\frac{(\nabla\pi)^2}{a^2} \right] \, .
\end{equation}
We can follow the same steps as before: the interaction Hamiltonian is
\begin{align}
H_\text{int} & = -\int d^3x a^3 \cdot 2M_2^4 \left[ \dot\pi^3 - \dot\pi\frac{(\nabla\pi)^2}{a^2} \right]
\nonumber\\
& = -2a^3M_2^4 \int \frac{d^3q_1d^3q_2d^3q_3}{(2\pi)^{3\cdot2}} \delta^{(3)}(-\mathbi{q}_{123})
\left[ \dot\pi_{\mathbi{q}_1}\dot\pi_{\mathbi{q}_2}\dot\pi_{\mathbi{q}_3}
+ \frac{\mathbi{q}_1\cdot\mathbi{q}_2}{3a^2} \pi_{\mathbi{q}_1}\pi_{\mathbi{q}_2}\dot\pi_{\mathbi{q}_3} + \text{2 perm} \right] \, .
\end{align}
Then, the bispectrum of $\pi$ becomes
\begin{align}
\label{eq:B-pi0}
\left\langle \pi_{\mathbi{k}_1}\pi_{\mathbi{k}_2}\pi_{\mathbi{k}_3}(\tau) \right\rangle
& \equiv (2\pi)^3 \delta^{(3)}(\mathbi{k}_{123}) B_\pi(\mathbi{k}_1,\mathbi{k}_2,\mathbi{k}_3)
\nonumber\\
& = (2\pi)^3 \delta^{(3)}(\mathbi{k}_{123}) \left\{
i \widehat{\pi}_{k_1}^*\widehat{\pi}_{k_2}^*\widehat{\pi}_{k_3}^*(0)
\int_{-\infty}^0 d\tau \left( -2aM_2^4 \right)  \right.
\nonumber\\
& \qquad \qquad \qquad  \qquad \times
\Bigl[ 6\widehat{\pi}_{q_1}'\widehat{\pi}_{q_2}'\widehat{\pi}_{q_3}'(\tau)
+ 2(\mathbi{k}_1\cdot\mathbi{k}_2)
\widehat{\pi}_{q_1}\widehat{\pi}_{q_2}\widehat{\pi}_{q_3}'(\tau) +
\text{2 perm} \Bigr] + c.c. \biggr\} \, .
\end{align}
Again, we can find that by extending $M_2^4$ oddly the complex conjugate includes the integral from 0 to $\infty$, so
\begin{equation}
\label{eq:B-pi}
B_\pi(\mathbi{k}_1,\mathbi{k}_2,\mathbi{k}_3) =
i \widehat{\pi}_{k_1}^*\widehat{\pi}_{k_2}^*\widehat{\pi}_{k_3}^*(0) \int_{-\infty}^\infty d\tau \left( -2a\widetilde{M}_2^4 \right)
\left[ 6\widehat{\pi}_{q_1}'\widehat{\pi}_{q_2}'\widehat{\pi}_{q_3}'(\tau)
+ 2(\mathbi{k}_1\cdot\mathbi{k}_2)
\widehat{\pi}_{q_1}\widehat{\pi}_{q_2}\widehat{\pi}_{q_3}'(\tau) +
\text{2 perm} \right] \, .
\end{equation}

\subsection{Bispectrum in terms of the power spectrum}

In this subsection, we can use \eqref{eq:coeff-P} and write the
bispectrum \eqref{eq:B-pi} purely in terms of the power spectrum and its
derivatives. Let us first consider the first term of \eqref{eq:B-pi}. We
can straightforwardly write, with $K \equiv k_{123}$,
\begin{align}
\label{eq:B-pi_term1}
i \widehat{\pi}_{k_1}^*\widehat{\pi}_{k_2}^*\widehat{\pi}_{k_3}^*(0) & \int_{-\infty}^\infty d\tau \left( -2a\widetilde{M}_2^4 \right)
6\widehat{\pi}_{q_1}'\widehat{\pi}_{q_2}'\widehat{\pi}_{q_3}'(\tau)
= (2\pi^2{\cal P}_\pi)^2
\frac{H}{\pi} \frac{3H}{k_1k_2k_3} \int_{-\infty}^\infty \frac{dk}{k} \frac{\Delta{\cal P}_{\cal R}}{{\cal P}_{\cal R}}(k)
\int_{-\infty}^\infty d\tau \tau^2 e^{i(2k-K)\tau}
\nonumber\\
& = (2\pi^2{\cal P}_\pi)^2 \frac{3}{4}H \frac{1}{k_1k_2k_3}
\int_{-\infty}^\infty \frac{dk}{k} \frac{\Delta{\cal P}_{\cal R}}{{\cal P}_{\cal R}}(k) \frac{d^2}{dk^2} \delta\left( k-\frac{K}{2} \right)
\nonumber\\
& = (2\pi^2{\cal P}_\pi)^2 \frac{3}{4}H \frac{1}{k_1k_2k_3}
\frac{d^2}{dk^2} \left. \left[ \frac{1}{k} \frac{\Delta{\cal P}_{\cal R}}{{\cal P}_{\cal R}}(k) \right] \right|_{k=K/2} \, ,
\end{align}
where for the second equality we have replaced $\tau^2$
in the time integral with two derivatives with respect to $k$,
and for the last equality we have iteratively integrated by
parts.

To proceed further, with ${\cal P}_{\cal R}^\text{(total)} = {\cal P}_{\cal R} + \Delta{\cal P}_{\cal R}$, from
\begin{equation}
\log{\cal P}_{\cal R}^\text{(total)} \approx \log{\cal P}_{\cal R} + \frac{\Delta{\cal P}_{\cal R}}{{\cal P}_{\cal R}} \, ,
\end{equation}
with ${\cal P}_{\cal R}$ being flat, we can find the spectral index and the
running respectively as\footnote{In case one takes into
account the slight tilt of ${\cal P}_{\cal R}$, the spectral index and the
running given here represent only the effect of $\Delta{\cal P}_{\cal R}/{\cal P}_{\cal R}$. Since we
assumed that for a limited duration $c_s$ deviates from unity, we can
separate the correction part from the standard slow-roll part, for both
of which, the spectral index and the running can be defined,
respectively.}
\begin{align}
n_{\cal R}-1 & \equiv \frac{d\log{\cal P}_{\cal R}^\text{(total)}}{d\log{k}} = k\frac{d}{dk} \left( \frac{\Delta{\cal P}_{\cal R}}{{\cal P}_{\cal R}} \right) \, ,
\\
\alpha_{\cal R} & \equiv \frac{dn_{\cal R}}{d\log{k}} = k^2\frac{d^2}{dk^2} \left( \frac{\Delta{\cal P}_{\cal R}}{{\cal P}_{\cal R}} \right)
+ k\frac{d}{dk} \left( \frac{\Delta{\cal P}_{\cal R}}{{\cal P}_{\cal R}} \right) \, .
\end{align}
Thus \eqref{eq:B-pi_term1} can be now written as
\begin{align}
& i \widehat{\pi}_{k_1}^*\widehat{\pi}_{k_2}^*\widehat{\pi}_{k_3}^*(0) \int_{-\infty}^\infty d\tau \left( -2a\widetilde{M}_2^4 \right)
6\widehat{\pi}_{q_1}'\widehat{\pi}_{q_2}'\widehat{\pi}_{q_3}'(\tau)
\nonumber\\
& \qquad \qquad  = (2\pi)^4{\cal P}_\pi^2 \frac{3}{2}H \frac{1}{k_1k_2k_3} \frac{1}{K^3}
\left[ \alpha_{\cal R} - 3(n_{\cal R}-1) + 2\frac{\Delta{\cal P}_{\cal R}}{{\cal P}_{\cal R}} \right] \bigg|_{k=K/2} \, .
\end{align}
We can proceed in a similar manner for the second term of \eqref{eq:B-pi}
and find
\begin{align}
& i \widehat{\pi}_{k_1}^*\widehat{\pi}_{k_2}^*\widehat{\pi}_{k_3}^*(0) \int_{-\infty}^\infty d\tau \left( -2a\widetilde{M}_2^4 \right)
2(\mathbi{k}_1\cdot\mathbi{k}_2) \widehat{\pi}_{q_1}\widehat{\pi}_{q_2}\widehat{\pi}_{q_3}'(\tau)
\nonumber\\
& = (2\pi)^4{\cal P}_\pi^2 \frac{1}{H} \frac{-\mathbi{k}_1\cdot\mathbi{k}_2}{(k_1k_2)^3k_3}
\left[ \left( 1 + \frac{k_{12}}{K} + \frac{2k_1k_2}{K^2} \right) \frac{\Delta{\cal P}_{\cal R}}{{\cal P}_{\cal R}}
+ \left( -\frac{k_{12}}{K} - \frac{3k_1k_2}{K^2} \right) (n_{\cal R}-1) +
 \frac{k_1k_2}{K^2} \alpha_{\cal R} \right] \bigg|_{k=K/2} \, .
\end{align}

Thus, the bispectrum can be expressed in terms of the correction to
power spectrum, its first and second derivatives as
\begin{align}
\label{eq:bi-final}
& B_\pi(\mathbi{k}_1,\mathbi{k}_2,\mathbi{k}_3)
\nonumber\\
& = (2\pi)^4{\cal P}_\pi^2 \frac{H}{(k_1k_2k_3)^3}
\left[ {A}(k_1,k_2,k_3) \frac{\Delta{\cal P}_{\cal R}}{{\cal P}_{\cal R}}
+ {B}(k_1,k_2,k_3) (n_{\cal R}-1)
+ {C}(k_1,k_2,k_3) \alpha_{\cal R} \right] \bigg|_{k=K/2} \, ,
\end{align}
where the functions of momenta ${A}$, ${B}$ and ${C}$ are
given by, respectively,
\begin{align}
{A}(k_1,k_2,k_3) &=
                  - \frac{1}{K^2}\sum_{i \ne j} k_i^2 k_j^3
                  + 2 \frac{1}{K}\sum_{i > j} k_i^2 k_j^2
                  - \frac14 \sum_{i} k_i^3 \, ,
\\
{B}(k_1,k_2,k_3) &=
                   2 \frac{1}{K^2}\sum_{i \ne j} k_i^2 k_j^3
                 - 3 \frac{1}{K}\sum_{i > j} k_i^2 k_j^2
                 + \frac14 \sum_{i \ne j} k_i k_j^2
                 - \frac14 k_1 k_2 k_3 \, ,
\\
{C}(k_1,k_2,k_3) &=
                  - \frac{1}{K^2}\sum_{i \ne j} k_i^2 k_j^3
                  + \frac{1}{K}\sum_{i > j} k_i^2 k_j^2
                  - \frac14 k_1 k_2 k_3 \, .
\end{align}
This expression is one of the main results in this article.

\begin{figure}
 \begin{center}
  \includegraphics[width=0.9\textwidth]{fig_eft-corr.pdf}
 \end{center}
 \caption{(Upper panels) power spectrum and (lower panels) the corresponding
 bispectrum for various feature models \eqref{eq:models} discussed in~\cite{Ade:2015lrj}.
 For simplicity, we have set the amplitudes of the features as
 $A_s = A_\text{log} = A_\text{lin} = 1$ and phases as $\varphi_\text{log} = \varphi_\text{in} = 0$.
 For the step model, we have also set the damping scale $x_d = 1$.
 Meanwhile, following~\cite{Ade:2015lrj} we have set $\log_{10}\omega_\text{log} = 1.25$,
 $\log_{10}\omega_\text{lin} = 1.02$ and $n_\text{lin} = 0.66$.
 We show the bispectrum projected onto the equilateral (red dashed), folded
 (blue dotted) and squeezed (black solid) configurations.}
 \label{fig:corr}
\end{figure}

In Figure~\ref{fig:corr}, we show a few examples using the following
parametrized feature models~\cite{Ade:2015lrj}: a localized oscillatory burst
due to e.g. step in the inflaton potential, logarithmic and linear
oscillations and cutoff models given by
\begin{equation}
\label{eq:models}
\frac{\Delta{\cal P}_{\cal R}}{{\cal P}_{\cal R}} = \left\{
\begin{array}{ll}
A_s W_0\left(\dfrac{k}{k_s}\right) D\left(\dfrac{k/k_s}{x_d}\right)
& \text{(step)}
\\
A_\text{log} \cos\left[ \omega_\text{log}\log\left(\dfrac{k}{k_\text{log}}\right) + \varphi_\text{log} \right]
& \text{(logarithmic oscillations)}
\\
A_\text{lin} \left( \dfrac{k}{k_\text{lin}} \right)^{n_\text{lin}} \cos\left( \omega_\text{lin}\dfrac{k}{k_\text{lin}} + \varphi_\text{lin} \right)
& \text{(linear oscillations)}
\\
\log \left( \dfrac{\pi}{16} \dfrac{k}{k_c} |C_c-D_c|^2 \right)
& \text{(cutoff model)}
\end{array}
\right.
\, ,
\end{equation}
where the functions that appear in these parametrized feature models are
\begin{align}
W_0(x) & = \frac{1}{2x^4} \Big[ \left( 18x - 6x^3 \right) \cos(2x) + \left( 15x^2 - 9 \right) \sin(2x) \Big] \, ,
\\
D(x) & = \frac{x}{\sinh{x}} \, ,
\\
C_c & = \exp\left( \frac{-ik}{k_c} \right) \left[ H_0^{(2)}\left( \frac{k}{2k_c} \right) - \left( \frac{1}{k/k_c}+i \right) H_1^{(2)} \left( \frac{k}{2k_c} \right) \right] \, ,
\\
D_c & = \exp\left( \frac{ik}{k_c} \right) \left[ H_0^{(2)}\left( \frac{k}{2k_c} \right) - \left( \frac{1}{k/k_c}-i \right) H_1^{(2)} \left( \frac{k}{2k_c} \right) \right] \, ,
\end{align}
with $H_n^{(2)}$ being the Hankel function of the second kind.
As we can see, each power spectrum gives distinctively different patterns of
the corresponding bispectrum in various configurations.

\section{Squeezed bispectrum and consistency relation}
\label{sec:squeezed}

We can note that \eqref{eq:bi-final} vanishes in the squeezed limit,
say, $k_1 \approx k_2$ and $k_3\to0$.\footnote{It was recently claimed
that for local observers, the squeezed limit vanishes in single-field
inflation
\cite{obs-squeezed}. But in
this article we do not take such effects into account and hence the
consistency relation should hold if we would calculate it adequately.}
This seems to contradict the consistency relation between the power
spectrum and the squeezed limit of the bispectrum
\cite{consistency},
\begin{equation}
B_{\cal R}(k_1,k_2,k_3) \underset{k_3\to0}{\longrightarrow} (1-n_{\cal R})P_{\cal R}(k_1)P_{\cal R}(k_3) \, ,
\end{equation}
because as \eqref{eq:correction_p} shows the power spectrum is well away
from featureless flat one, so the corresponding spectral index is
non-trivial. Indeed, in \cite{Achucarro:2012fd}, the consistency relation
is recovered for features caused by non-trivial speed of sound.

Let us first return to the quadratic action for the {\em curvature perturbation}. Including the speed
of sound, it is written as
\begin{equation}
\label{eq:S2R}
S_2 = \int d^4x a^3m_{\rm Pl}^2\epsilon \left[ \frac{\dot{\cal R}^2}{c_s^2} - \frac{(\nabla{\cal R})^2}{a^2} \right] \, ,
\end{equation}
so there are two possible sources of departure from the usual canonical
slow-roll~\cite{featuresource}: $\epsilon$ and $c_s$. Let us consider these two cases
separately. Our goal here is to see the form of the corrections to the
power spectrum for each case. But this seems unclear, since the form of
the interaction part of the quadratic action -- just $\dot{\cal R}^2$ for
$c_s$, and $\dot{\cal R}^2$ and $(\nabla{\cal R})^2$ for $\epsilon$ -- is different.
Thus naively thinking the resulting correction terms would be of different structure.
We first assume that $c_s$ solely supplies the deviations from the
standard slow-roll in such a way that for a limited duration $c_s$
deviates from unity, with the deviation being not too far away from
unity. We may then write, with the canonical slow-roll part being the
leading, free part,
\begin{equation}
S_2 = \int d^4x a^3m_{\rm Pl}^2\epsilon \left[ \dot{\cal R}^2 - \frac{(\nabla{\cal R})^2}{a^2} \right]
+ \underbrace{ \int d^4x a^3m_{\rm Pl}^2\epsilon \left( \frac{1}{c_s^2} - 1 \right) }_{\equiv
S_\text{2,int}} \, .
\end{equation}
Following the same steps as in Section~\ref{subsec:power}, we find
\begin{equation}
\label{eq:Pcorr-result1}
\frac{\Delta{\cal P}_{\cal R}}{{\cal P}_{\cal R}} = k \int_{-\infty}^0 d\tau \left( c_s^2-1 \right)
\sin(2k\tau) \, ,
\end{equation}
which is of the same structure as \eqref{eq:correction_p}.

For the case in which $\epsilon$ is responsible for the
departure from the standard slow-roll, let us split $\epsilon$
into the slowly varying part $\epsilon_0$ and the rapidly varying but
transient part $\Delta\epsilon$:
\begin{equation}
\epsilon = \epsilon_0 + \Delta\epsilon \, .
\end{equation}
We can rewrite $\Delta\epsilon$ as
\begin{equation}
\Delta\epsilon = \int \dot\epsilon dt
 \approx H\epsilon_0 \int \eta dt
 \approx \epsilon_0\eta H \Delta t\, ,
\end{equation}
where $\Delta t = {\cal O}(1/H)$ is the duration of departure and we have defined
another slow-roll parameter $\eta \equiv \dot\epsilon/(H\epsilon)$.
Then the quadratic action \eqref{eq:S2R}, with $c_s=1$ this time, can be written as
\begin{equation}
\label{eq:S2-epsilon-change}
S_2 = \int d^4x a^3m_{\rm Pl}^2\epsilon_0 \left[ \dot{\cal R}^2 -
				     \frac{(\nabla{\cal R})^2}{a^2} \right]
+ \underbrace{ \int d^4x a^3m_{\rm Pl}^2 \Delta\epsilon \left[ \dot{\cal R}^2 -
				       \frac{(\nabla{\cal R})^2}{a^2}
				      \right] }_{\equiv
S_\text{2,int}} \, ,
\end{equation}
and the corresponding correction to the power spectrum is
\begin{equation}
\label{eq:Pcorr-result2}
\frac{\Delta{\cal P}_{\cal R}}{{\cal P}_{\cal R}} = k \int_{-\infty}^0 d\tau
\left(-2 \frac{\Delta\epsilon}{\epsilon_0} \right) \sin(2k\tau) \, .
\end{equation}
Comparing this with \eqref{eq:Pcorr-result1}, we see
that two sources of the departure from the standard slow-roll
leads to the same structure of the correction as \eqref{eq:correction_p}.
This seems to suggest that indeed $M_2^4$ captures the deviation
from usual slow-roll on general ground.

We now return to our starting equation \eqref{eq:Spi} to clarify this
inconsistency. A key observation is that unlike ${\cal R}$, which is frozen
on super-horizon scales, $\pi$ evolves as
\begin{equation}
\label{eq:pi-R}
\dot\pi = -\frac{\dot{\cal R}}{H} - \epsilon{\cal R} + \frac{\dot{\cal R}^2}{H^2}
+ 3\epsilon \frac{\dot{\cal R}\calR}{H} + \cdots \, ,
\end{equation}
where the non-linear terms follow from the fact that essentially
$\pi$ is the time translation between spatially flat and comoving
hypersurfaces~\cite{Noh:2004bc}. Also we have omitted terms that are further
suppressed in slow-roll parameters. Taking into account the
sub-leading terms in $\dot\pi$, at quadratic order of the curvature
perturbation $M_2^4$ contributes\footnote{
The standard slow-roll terms, multiplied by $\dot{H}$ in \eqref{eq:Spi},
also give rise to additional sub-leading terms, but they are ${\cal O}(\epsilon^2)$
so we do not include them here.
}
\begin{equation}
S_{\cal R} \supset \int d^4 a^3 \left[ \frac{2M_2^4}{H^2}\dot{\cal R}^2
+ 2\epsilon \left( -3M_2^4 + \frac{\dot{M}_2^4}{H} \right) {\cal R}^2 \right] \, .
\end{equation}
Thus, the speed of sound of the curvature perturbation is identical to
that of $\pi$ given by \eqref{eq:cs}. At the same time there do exist
changes in ${\cal R}^2$ terms as \eqref{eq:S2-epsilon-change}, but they are
slow-roll suppressed. Since we have only considered the leading effects
that only capture the speed of sound, the bispectrum \eqref{eq:bi-final}
is enhanced in the equilateral configuration while it is not in the
squeezed limit. Indeed, by considering the sub-leading terms in
\eqref{eq:pi-R}, we have at cubic order new terms
$\dot{\cal R}^2{\cal R}$ and {${\cal R}(\nabla{\cal R})^2$} that lead to
non-vanishing bispectrum in the squeezed
limit~\cite{Cheung:2007sv}. More specifically, the new terms
{$\dot{\cal R}^2{\cal R}$} and {${\cal R}(\nabla{\cal R})^2$} in the cubic
order action {give} up to numerical coefficient
\begin{equation}
\frac{k_1k_2k_3}{k_1^3+k_2^3+k_3^3} \frac{(k_1k_2k_3)^2B_{\cal R}(k_1,k_2,k_3)}{(2\pi)^4{\cal P}_{\cal R}^2}
\underset{k_3\to0}{\longrightarrow} \epsilon \frac{\Delta{\cal P}_{\cal R}}{{\cal P}_{\cal R}} \, ,
\end{equation}
with $k_1 \approx k_2 \equiv k$. Still the consistency relation is not
recovered, but this is because we are not taking into account all the
next-to-leading terms in the decoupling limit, {such as the
modification of the mode functions: terms of ${\cal O}(1/c_s^2)$ and
${\cal O}(\epsilon/c_s^2)$ do not contribute to the squeezed limit while
only terms of ${\cal O}(\epsilon)$ do}~\cite{RenauxPetel:2010ty}. Our
calculation is done only up to the leading order.

\section{Summary}

In this article, we have derived the direct relation between the
corrections of power spectrum and bispectrum of the primordial curvature
perturbation.  Our formula is based on the effective field theory
approach to inflation, which to first approximation captures the effects
of the non-trivial speed of sound. If we would observationally detect
the deviation from the standard slow-roll inflation, it is important to
check the relation derived here, which could prove/disprove whether such
a deviation can be attributed to the variation of sound velocity.

We have also shown that the corrections to the power spectrum from
non-trivial features of sound velocity and expansion rate of the
universe, which characterize the deviation from the standard slow-roll
inflation, have the same form. It is interesting to check whether we can
extend this kind of unified treatment to higher order correlation
functions. We have also discussed the squeezed limit of the bispectrum
and the consistency relation. In the leading order calculations we have
adopted in this article, the squeezed limit vanishes. But, if we take
into account sub-leading orders adequately, the consistency relation
would be recovered.

The next step is to include the sub-leading order effects such as the
terms beyond decoupling limit and the $M_3$ terms. Then, we will have
further (consistency) relation, which is useful to identify new physics
causing such a deviation.

\subsection*{Acknowledgments}

JG thanks Tokyo Institute of Technology for hospitality during ``Workshop on Particle Physics, Cosmology, and Gravitation'' where this work was initiated.
JG acknowledges the support from the Korea Ministry of Education, Science and Technology,
Gyeongsangbuk-Do and Pohang City for Independent Junior Research Groups at the Asia Pacific Center for Theoretical Physics.
JG is also supported in part by a TJ Park Science Fellowship of POSCO TJ Park Foundation
and the Basic Science Research Program through the National Research Foundation of
Korea (NRF) Research Grant NRF-2016R1D1A1B03930408.
MY is supported in part by Japan Society for the Promotion of Science (JSPS)
Grant-in-Aid for Scientific Research Nos. 25287054 and 26610062,
and the Grant-in-Aid for Scientific Research on Innovative Areas
``Cosmic Acceleration'' No. 15H05888.


\section{Introduction}
\label{sec:intro}

Dialog state tracking is one of key sub-tasks of dialog management, whose goal is to transfer human utterances into a slot-value representation (dialog state) that is easy for computer to process and track the information that appeared in the dialog. To provide a common testbed for this task, the series of Dialog State Tracking Challenges (DSTC) was initiated \cite{DSTC5}. This challenge has already been held for four times, during which it provided a very valuable shared recourse for the research in this field and helped to improve the state-of-the-art. Since the forth challenge (DSTC4 2015), the target of dialog state tracking has been shifted from human-machine dialog to human-human dialog, which significantly increases the difficulty of the dialog state tracking task because of the variety and ambiguity in the human-human dialog. One lesson we learned from DSTC4 is the difficulty of building a high performance tracker for human-human dialog with very limited training corpus, no matter whether using machine learning or hand-crafted rule-based approaches \cite{CNN-DSTC4, dernoncourt2016robust}.　This is a very unfavorable situation because building hand-annotated training corpus is very expensive, time-consuming and requires human experts. Not to mention the collection of a new corpus for each language other than English if we need to build trackers for a new language.

The DSTC5 proposed a new challenge based on using the rapidly advancing machine translation (MT) technology, one may be able to adapt the built tracker to a new language with limited training or development corpus in that language. We find this idea very attractive because not only it can reduce the cost of new language adaptation, but also it provides the possibility of building a tracker with cross-language corpus. For example it can be very useful for developing the tourist information systems because one may have corpus collected from different language speakers (i.e. tourists from different countries): for each language, the amount of corpus may be very limited, but together it can be large enough for a good training. On the other hand, although the machine translation technology has achieved great progress recently, the translation quality is still not satisfactory \cite{sutskever2014sequence}. A conventional monolingual tracker trained on the computer-generated translations may lead to an imperfect model, and it can only accept the translations from other languages as input which will also degrade the performance.

To address these problems, we propose a model that can be trained with different languages at the same time, and use both original utterances and their translations as input source for the dialog state tracking. In such way, we can avoid building the tracker only based on computer-generated translations, and maximize the use of all possible input languages to increase the robustness to translation errors. This paper is organized as follows. Sect.\ref{sec:dataset} briefly describes the dataset and the dialog state tracking problem; Sect.\ref{sec:method} presents an overview of our method and explains in detail our multichannel CNN model. Sect.4 presents the evaluation results with analysis and discussion. Sect.5 concludes our work and proposes future improvements.

\section{DATASET AND PROBLEM DESCRIPTION}
\label{sec:dataset}

\begin{table}[t]
	\caption{Example of a transcription and dialog state label for a sub-dialog segment in the topic of `Accommodation'.}
	\label{example-sub-dialog}
	\begin{center}
	\resizebox{8.5cm}{!}{
	\begin{tabular}{p{240pt}|p{75pt}}
		\hline
		\centering{\bf Transcription} & {\bf \ Dialog state label} \\
		\hline
		{\bf Guide:} Let's try this one, okay? & \\
		{\bf Tourist:} Okay. & \\
		{\bf Guide:} It's InnCrowd Backpackers Hostel in Singapore. If you take a dorm bed per person only twenty dollars. If you & \\
		take a room, it's two single beds at fifty nine dollars & {\bf INFO:} Pricerange \\
		{\bf Tourist:} Um. Wow, that's good. & \\
		{\bf Guide:} Yah, the prices are based on per person per bed or dorm. But this one is room. So it should be fifty nine for the two room. So you're actually paying about ten dollars more per person only. & {\bf PLACE: } \hspace{30pt} InnCrowd Backpackers Hostel \\
		{\bf Tourist:} Oh okay. That's- the price is reasonable actually. It's good. & \\
		\hline
	\end{tabular}
	}
	\end{center}
\end{table}

The fifth Dialog State Tracking Challenge (DSTC5) uses the whole dataset (including train/dev/test datasets) from the DSTC4 as the training dataset. This dataset contains 35 dialog sessions on tourist information for Singapore collected from English speakers. Besides the training dataset, a development set which includes 2 dialog sessions collected from Chinese speakers is provided for testing and tuning the trackers' cross-language performance before the final evaluation. Both the training and development sets are labelled with the dialog state tags and come with 5-best hypothesis of English or Chinese translations by a machine translation systems. In the evaluation phase of the challenge, a Test set including 8 unlabeled Chinese dialogs is distributed to each participant, and all prediction results submitted by each participant are evaluated by comparing with the true labels. The test dataset also includes 5-best English translations which are generated by the same machine translation system as the training/development dataset.

The dialog state in this challenge is defined by the same ontology used in DSTC4, which contains 5 topic branches with different slot sets (Table\ref{topic-slot}). These topic-slot combinations indicate the most important information mentioned in that topic, for example the `CUISINE' slot under the topic of `FOOD' refers to the cuisine types, while the `STATION' slot for the topic of `TRANSPORTATION' refers to the train stations. In total there are 30 such topic-slot combinations, and all possible values for each topic-slot are given as a list in the ontology. The main task of DSTC5 is to predict the proper value(s) for each slot given the current utterance and its topic, with all dialog history prior to that turn (e.g. Table\ref{example-sub-dialog}).

\begin{table}[h]
	\caption{List of slots for each topic.}
	\label{topic-slot}
	\begin{center}
	\resizebox{8.5cm}{!}{
	\begin{tabular}{c|p{250pt}}
		\hline
		{\bf Topic} & {\bf \hspace{100pt}SLOT} \\
		\hline
		{\bf Food} & INFO, CUISINE, TYPE\_OF\_PLACE, DRINK, PLACE, MEAL\_TIME, DISH, NEIGHBOURHOOD \\
		\hline
		{\bf Attraction} & INFO, TYPE\_OF\_PLACE, ACTIVITY, PLACE, TIME, NEIGHBOURHOOD \\
		\hline
		{\bf Shopping} & INFO, TYPE\_OF\_PLACE, PLACE, NEIGHBOURHOOD, TIME \\
		\hline
		{\bf Transportation} & INFO, FROM, TO, STATION, LINE, TYPE, TICKET \\
		\hline
		{\bf Accommodation} & INFO, TYPE\_OF\_PLACE, PLACE, NEIGHBOURHOOD \\
		\hline
	\end{tabular}
	}
	\end{center}
\end{table}

\section{METHOD}
\label{sec:method}

In DSTC4 we proposed a method which is based on the convolutional neural networks originally proposed by Kim \cite{kim2014convolutional}. By this method we were able to achieve the best performance for tracking the “INFO” slot. The CNN model we used in this method was modified from the origin by adding a structure of multi-topic convolutional layer, so that it can better handle the information presented in different dialog topics. This model is characterized by its high performance for limited training data, because it can be trained across various topics. More details about this multi-topic model can be found in \cite{CNN-DSTC4}.

In DSTC5 the training data is 75\% more than DSTC4, therefore the situation of limited training data is improved. In order to focus more on the new cross-language problem and keep our method simple, instead of using the more complex multi-topic model we proposed last time, we trained individual CNN model for each slot-topic combination. That is, for example the `INFO' slot in the topic of `FOOD' and the same `INFO' slot in the topic of `SHOPPING' will be trained by two independent models. This is the major difference from the last time, where we trained one single model for all topics. With this new scheme, we can set the hyperparameters in each model for each slot/topic to be exactly the same, so that our method is scalable, universally applicable and easy to tune. Fig \ref{fig:method} is a simple diagram illustrating our method.

\begin{figure}[htb]
\includegraphics[width=8.5cm]{fig-method.eps}
\caption{Overview of our method (for the `FOOD' topic).}
\label{fig:method}
\end{figure}

\subsection{Motivation}
\label{ssec:motivation}

The biggest challenge of DSTC5 is that the training and test corpora are originally collected in different languages. Since both computer-generated Chinese and English translation are provided in the training and test dataset respectively, one straightforward approach is to train a model with English corpus and use it for the English translation in the test data. Alternatively, a model trained on Chinese translations in the training dataset can be used for Chinese utterances in the test data. However, both methods will waste the originally collected utterances either in the training or the test data. In order to fully utilize the corpus resource in both English and Chinese languages, we proposed the following multi-channel model which can be regarded as a combination of both English and Chinese models.

\subsection{Model architecture}
\label{ssec:model-architecture}

Our model is inspired by the multichannel convolutional neural networks commonly used in the image processing \cite{krizhevsky2012imagenet}. Instead of RGB channels used for color images, we apply each input channel to each different language source. In this model, the input of each channel is a two dimensional matrix, each row of which is the embedding vector of the corresponding word:
\begin{equation}
	\label{eq:segment-embedding}
	\mathbf{s} = \left[
	\begin{array}{c}
		\text{---}\ \mathbf{w}_{1}\ \text{---} \\
		\text{---}\ \mathbf{w}_{2}\ \text{---} \\
		\vdots\\
		\text{---}\ \mathbf{w}_{n}\ \text{---}
	\end{array}
	\right],
\end{equation}
where $\mathbf{w}_{i}\in\mathbb{R}^{k}$ is the embedding vector for the $i$-th word in the input text. This 2-dimensional array $\mathbf{s}$ is a matrix representation of the input text. We used three different word embedding in our model --- two for Chinese and one for English. The details of these embedding will be explained later in the Sect. \ref{ssec:embedding-models}. For each channel, a feature map $\mathbf{h}\in\mathbb{R}^{n-d+1}$ is obtained by convolving a trainable filter $\mathbf{m}\in\mathbb{R}^{d\times k}$ with the embedding matrix $\mathbf{s}\in\mathbb{R}^{n\times k}$ using the following equation:
\begin{equation}
	\mathbf{h} = f(\mathbf{m} \ast \mathbf{s} + \mathbf{b}).
\end{equation}
Here $f$ is a non-linear activation function\footnote{We used rectified linear unit (ReLU) for this activation function.}; $*$ is the convolution operator and $\mathbf{b} = (b,\ldots,b) \in\mathbb{R}^n$ is a bias term. The maximum value of this feature map $\hat{h} = \mathrm{max}\{\bm{h}\}$ is then selected by the max-pooling layer. This is the process how one filter extracts one most important feature from the input matrix. In this model, multiple filters are used in each channel to extract multiple features. These features then form the pooling layer which are passed to a fully connected layer for prediction.

The idea of multichannel model is to connect those extracted features from different channels before the final output, so that the model can use richer information obtained from different channels. The fully connect layer in multi-channel model follows the equation:
\begin{equation}
	\mathbf{y} = S\big(\mathbf{w}\cdot (\mathbf{\hat{h}}_{\mathrm{ch}1}\oplus \mathbf{\hat{h}}_{\mathrm{ch}2} \oplus \mathbf{\hat{h}}_{\mathrm{ch}3}) + \mathbf{b}\big),
\end{equation}
where $S$ is the sigmoid function; $\oplus$ is the concatenation operator and $\mathbf{\hat{h}}_{\mathrm{ch}n} = (\hat{h}_1,\ldots,\hat{h}_m)$ is the penultimate layer of the $n$-th channel.

\begin{figure}[!b]
\includegraphics[width=8.5cm]{fig-model-architecture.eps}
\caption{Multichannel CNN model architecture for three input channels.}
\label{fig:model-architecutre}
\vspace{0.5cm}
\includegraphics[width=8.5cm]{fig-model-usage.eps}
\caption{Pre-processing of the input utterances.}
\label{fig:model-usage}
\end{figure}

Notice that in the original paper of Kim, a multi-channel architecture has also been proposed. The main difference between our model and their model is that we use different sets of filters for each channel, while in their model the same filter set are applied to all channels. The reason for this modification is that the word embedding for different languages can vary greatly, for example the same (or nearly the same) embedding vector in different language models may correspond to irrelevant words with very different meanings. Using different sets of filters ensures that proper features can be extracted in each channel no matter how the word embedding varies among different languages.

\subsection{Embedding models}
\label{ssec:embedding-models}

The word2vec \cite{mikolov2013efficient} is one of the most common methods for producing word embeddings. In DSTC5, we applied this method and trained three different models with different training corpus. The details of these models are listed as below:
\begin{enumerate}
 \item English word model: 200-dimension word2vec model trained on English Wikipedia, with all text split by space and all letters lowercased. This model contains 253854 English words.
 \item Chinese word model: 200-dimension word2vec model trained on Chinese Wikipedia, with all text split by word boundary using `jieba' module\footnote{https://github.com/fxsjy/jieba}. This model contains 457806 Chinese words and 53743 English words appeared in the Chinese Wikipedia.
 \item Chinese character model: 200-dimension word2vec model trained on Chinese Wikipedia with all text split into single Chinese character. This model contains 12145 Chinese characters and 53743 English words appeared in the Chinese Wikipedia.
\end{enumerate}

The reason why we trained two models for Chinese language is because identifying word boundaries in Chinese is not a trivial task. For Chinese, the smallest element with meaning (word) varies from one single Chinese character to several concatenated Chinese characters, and the task for Chinese word splitting usually involves parsing the sentence and the state-of-the-art method still cannot achieve perfect accuracy. For this reason the Chinese word model may contain incorrect vocabularies and is not capable of handling unseen Chinese character combinations. On the other hand, the Chinese character model does not rely on word segmentation so that the model is error-free, and also it can easily deal with unseen words. However, since the Chinese character model ignores the word boundaries, the resulting embedding vector may not be able to reflect the precise meaning of each word.

\section{RESULTS}
\label{sec:results}

The results of the proposed method along with the scores of other teams are shown in the Table \ref{tab:results}. Our multichannel CNN model achieves the best score among all 9 teams: the result of entry-3 outperforms the second best team by 50\% (0.0956/0.0635) in Accuracy and 15\% (0.4519/0.3945) in F-measure with the sub-dialog evaluation. Our submitted five entries are the results of 5 different hyperparameters settings which are determined by a rough grid search\footnote{A guide for setting these hyperparameters can be found in \cite{zhang2015sensitivity}}, and those settings are summarized in Table \ref{tab:results-parameters}. Compared these results with each other, one can easily tell that among these hyperparameters the dropout rate is a key factor. The dropout is known as a technique for reducing overfitting in neural networks \cite{hinton2012improving}, and in our case reducing the dropout rate always improves the Precision while degrading the Recall score. One explanation for this is that an over-fitted model only outputs the same labels for the data which are very similar to the training data, and therefore decreases its generalization to unseen data. On the other hand, further decreasing the dropout rate does not improve the overall performance, whose results and parameter settings are also shown in the table as `Additional Expt. \#5\&6'.

\begin{table}[h]
	\caption[Caption for LOF]{Evaluation results (subdialog-level) on DSTC5 test dataset.\setcounter{footnote}{3}\footnotemark}
	\label{tab:results}
	\begin{center}
	\resizebox{8.5cm}{!}{
	\begin{tabular}{c|p{1.5cm}p{1.5cm}p{1.5cm}p{1.5cm}}
		\hline
		{\bf Tracker} & \centering{\bf Accuracy} & \centering{\bf Precision} & \centering{\bf Recall} & \multicolumn{1}{c}{\bf F-measure}\\
		\hline
		Multichannel \#3 & \centering{0.0956} & \centering{\bf 0.5643} & \centering{0.3769} & \multicolumn{1}{c}{\bf 0.4519}\\
		Multichannel \#4 & \centering{0.0872} & \centering{0.5427} & \centering{0.3842} & \multicolumn{1}{c}{0.4499}\\
		Multichannel \#0 & \centering{\bf 0.0964} & \centering{0.5217} & \centering{0.3849} & \multicolumn{1}{c}{0.4430}\\
		Multichannel \#1 & \centering{0.0712} & \centering{0.4340} & \centering{0.4196} & \multicolumn{1}{c}{0.4267}\\
		Multichannel \#2 & \centering{0.0681} & \centering{0.4216} & \centering{\bf 0.4303} & \multicolumn{1}{c}{0.4259}\\
		\hline
		Team4-entry2 & \centering{0.0635} & \centering{0.3768} & \centering{0.4140} & \multicolumn{1}{c}{0.3945}\\
		Team1-entry4 & \centering{0.0612} & \centering{0.3811} & \centering{0.3548} & \multicolumn{1}{c}{0.3675}\\
		Team6-entry1 & \centering{0.0383} & \centering{0.4063} & \centering{0.3124} & \multicolumn{1}{c}{0.3532}\\
		Team5-entry0 & \centering{0.0520} & \centering{0.3637} & \centering{0.3044} & \multicolumn{1}{c}{0.3314}\\
		baseline 2 & \centering{0.0222} & \centering{0.1979} & \centering{0.1774} & \multicolumn{1}{c}{0.1871}\\
		\hline
		\hline
		Additional Expt \#5 & \centering{0.0949} & \centering{0.5786} & \centering{0.3689} & \multicolumn{1}{c}{0.4505}\\
		Additional Expt \#6 & \centering{0.0888} & \centering{0.5677} & \centering{0.3712} & \multicolumn{1}{c}{0.4489}\\
		\hline
	\end{tabular}
	}
	\end{center}
\end{table}
\footnotetext{These are the evaluation results using `Schedule 2' described in the challenge handbook \cite{DSTC5}.}

\begin{table}[h]
	\caption{Main hyperparameter settings for each entry.}
	\label{tab:results-parameters}
	\begin{center}
	\resizebox{8.5cm}{!}{
	\begin{tabular}{m{3.5cm}|ccccccc}
		\hline
		Entry \# in Table\ref{tab:results} & \#3 & \#4 & \#0 & \#1 & \#2 & \#5 & \#6 \\
		\hline\hline
		Dropout rate & 0.4 & 0.5 & 0.6 & 0.8 & 0.8 & 0.2 & 0 \\
		\hline
		Number of filters $\mathbf{m}\in\mathbb{R}^{1\times k}$ for each channel & 1000 & 600 & 1000 & 1400 & 1000 & 1000 & 1000 \\
		\hline
		Number of filters $\mathbf{m}\in\mathbb{R}^{2\times k}$ for each channel & 1000 & 600 & 1000 & 1400 & 1000 & 1000 & 1000 \\
		\hline
		{Learning rate} & \multicolumn{7}{c}{0.001} \\
		\hline
		{Weight decay coefficient for $L2$ regularization} & \multicolumn{7}{c}{0.0005} \\
		\hline
		{Training data} & \multicolumn{7}{c}{Training corpus + 1-best Chinese translation} \\
		& \multicolumn{7}{c}{\& Development corpus + 1-best English translation} \\
		\hline
		{Training epochs} & \multicolumn{7}{c}{100} \\
		\hline
		{Test input} & \multicolumn{7}{c}{Test corpus + 1-best English translation} \\
		\hline
	\end{tabular}
	}
	\end{center}
\end{table}

\begin{table*}[t]
	\caption{Example of predicted labels by different models.}
	\label{tab:compare-example}
	\begin{center}
	\includegraphics[width=17cm]{table-compare-example.eps}
	\end{center}
\end{table*}

\subsection{Multichannel model \& single channel model \& model combination}
\label{ssec:model-compare}

To investigate by how much the proposed multichannel architecture contributes to these results, we compared the performance between the multichannel and ordinary single channel CNN models. For this comparison, we trained three different monolingual single channel CNN models using each of the embedding models mentioned in Section\ref{ssec:embedding-models}. These  models used the same parameter setting as `multichannel \#3' in the Table \ref{tab:results-parameters}, and were trained only on the 1-best machine translation results. Fig.\ref{fig:model-compare} shows the comparing results: the Chinese character model achieves the best overall accuracy among single channel models, while the multichannel model outperforms all three single channel models.

In the earlier DSTC, a simple model combination technique has been used to further improve the predictive performance, where the final output is computed by averaging the scores output by different models \cite{henderson2014word}. We also applied this method to combine the output from all three single channel models, and the result is also shown in the Fig.\ref{fig:model-compare}. This simple model combination method does not perform as good as the multichannel model, but considering its simplicity, we still consider it as a good alternative to improve the performance.

\begin{figure}[h]
\includegraphics[width=8.5cm]{fig-model-compare.eps}
\caption{Overall predictive accuracy of different models.}
\label{fig:model-compare}
\end{figure}

\subsection{Discussion}
\label{ssec:discussion}

We think the above results can be partially explained from the point of view of ensemble learning. In a multichannel model, each channel provides a different view of the data, and an example is described using different feature sets that provide different, complementary information about the instance. The fully connected layer in the multichannel model further provides an optimization to use this information for the prediction, and therefore the resulting model can in principle better deal with the translation errors appeared in different channels.

Table \ref{tab:compare-example} is one of the examples that demonstrate this idea. In this particular sub-dialog segment, none of the 3 single channel models is able to output the correct labels, while the multichannel model gives the correct prediction. As seen in this example, the model combination behaves like a simple voting, which means it only picks up the labels that are supported by majority of the single channel models. The multichannel model, on the other hand, is able to selectively choose which language source to trust more for each particular slot or value. As a result, the label of `Walking' is correctly predicted despite it only appearing in the English model's output, while the `Exhibit' label is correctly rejected even though it is supported by two single channel models out of three.

\begin{figure}[!b]
\includegraphics[width=8.5cm]{fig-model-compare2.eps}
\caption{Comparison of accuracy for each slot in the topic of `Attraction'.}
\label{fig:model-compare2}
\end{figure}

However the real situation is more complex. When we look at the overall predictive accuracy for each slot (Fig.\ref{fig:model-compare2}), we can find that the performance for each model varies on slots. We consider this is due to the ambiguity caused by machine translation which varies on different subjects. For example, as a time expression in English, 96\% of the word ``evening'' and 43\% of the word ``night'' are translated into the same Chinese word ``wan shang''. Although this Chinese word does have both meanings of ``evening'' and ``night'' in Chinese, there are more precise Chinese terms representing each word. This English to Chinese translation ambiguity immediately increases the difficulty of identifying the values of EVENING and NIGHT in Chinese, which leads to the poor performance of the Chinese model in the slot of TIME.

Another problem is that the translation quality often varies by reversing the translation direction, due to the difference in inflections, word order and grammars \cite{vilar2006error}. Since the training corpus only contains one translation direction (English to Chinese), the multichannel model is by no means optimized for the reverse translation direction. This may cause the multichannel model to have bias on certain channels, and it can explain why in certain slots the model combination that treats each channel equally works better. A more sophisticated way to train our multichannel model should be firstly training the model with one translation direction and then fine-tuning the model with the other. Unfortunately this is difficult in DSTC5 because the development dataset that can be used for the fine tuning is too limited.

\section{CONCLUSION}
\label{sec:conclusion}

We proposed a multichannel convolutional neural network, in which we treat multiple languages as different input channels. This multichannel model is found to be robust against the translation errors and outperforms any of the single channel models. Furthermore, our method does not require prior knowledge about new languages, and therefore can be easily applied to available corpus resources of different languages. This not only can reduce the cost for the adaption to a new language, but also offers the possibility to build multilingual dialog state trackers with large-scale cross-language corpora.

In this work we applied three different embedding models, while there is one more we did not try --- the English character model. There are several character-aware language models proposed recently, which are superior in dealing with subword information, rare words and misspelling \cite{kim2015character, zhang2015character}. We believe that integrating them into the multichannel model is a promising research direction.

On the other hand, since our method is purely machine learning based, it cannot handle unseen labels in the test data. This is a very important issue especially for a large ontology, because of the difficulty in obtaining large training corpus that covers all concepts. To overcome this disadvantage, future work should include combining machine learning with other approaches, such as hand-craft rules, data argumentation and so on.

\bibliographystyle{IEEEbib}