\section{Introduction}

Transition metal perovskite oxides (with chemical formula $AB$O$_3$, where $A$ is a rare earth and $B$ is a transition metal) display fascinating physical properties,
such as high-temperature superconductivity in the cuprates or colossal magnetoresistance in the manganites \cite{LeeRMP2006,Tokura2000}. The electronic and structural properties of perovskites and intertwined and lead to complex behaviors due to the interplay of charge, magnetic, structural, and orbital degrees of freedom \cite{Hwang2012}.  Creating heterostructures of transition metal perovskites enlarges the playground for manipulation of the local structure, charge state and electronic properties of these materials.  In this work, we focus on the electronic states of cobalt-containing perovkites.

Bulk LaCoO$_3$ has attracted much attention
because of the complex electronic and magnetic structure of the Co$^{3+}$ cations it contains.  LaCoO$_3$ is a nonmagnetic insulator at the lowest temperatures \cite{BhidePRB1972,ZobelPRB2002} when the
Co$^{3+}$ adopt a low-spin (LS) state ($t^{6}_{2g}$, $S$=0). It becomes a paramagnetic insulator for temperatures 100 K $ < T <$ 500 K and then becomes metallic above 500 K.
However, the spin state of Co in the thermally excited material is still in debate. The magnetic transition near 100 K is  considered as a transition from LS to high-spin (HS)
($t^{4}_{2g}e^{2}_{g}$, $S=2$) \cite{AsaiPRB1994,ItohJPSJ1994}
or to intermediate spin (IS)  ($t^{5}_{2g}e^{1}_{g}$, $S=1$) states
\cite{PotzePRB1995,SaitohPRB1997,AsaiJPSJ1998,YamaguchiPRB1997,ZobelPRB2002},
while IS and HS are mixed as temperature increases \cite{YamaguchiPRB1997,AsaiJPSJ1998}.

Numerous first principles calculations based on DFT+$U$ theory \cite{LDA+U1} for LaCoO$_3$ confirm that the LS state is the ground state
\cite{KorotinPRB1996,KnizekPRB2005,KnizekJPCM2006}, while the IS state is more stable than HS \cite{KorotinPRB1996}.
On the other hand, several theoretical studies suggested that the excited state at 100 K $< T <$ 500 K can be a mixture of HS and LS,  which we deonte as HS/LS below. Using the experimental crystal structures as a function of temperature, GGA+$U$ calculations (with $U=$ 2.7 eV) calculations suggested that HS/LS is stabilized for the structures above 200 K \cite{KnizekJPCM2006}. Unrestricted Hartree-Fock calculations also proposed that the non-magnetic to paramagnetic transition
is due to the LS state to LS/HS state \cite{ZhuangPRB1998},
and  effective Hamiltonian calculations suggested a similar scenario \cite{RopkaPRB2003,BiernackiPRB2006}.
Recent DFT+dynamical mean-field theory calculations showed that LaCoO$_3$ has large charge fluctuations: these calculations show that it is not possible to explain the spin state with a single multiplet at any temperature, although they find that at the onset of the spin-spin transition
HS multiplets are excited with the IS multiplets being excited later around the onset of the metal-insulator transition.

On the other hand, the electronic and magnetic structures of Co$^{2+}$ in bulk CoO does not show such complexity or ambiguity.
At low temperatures, CoO is an insulator with energy gap $E_g =2.4$ eV \cite{ZimmermannJPCM1999}.
The Co$^{2+}$ cations have a HS state ($t^{5}_{2g}e^{2}_{g}$ with $S=3/2$), and  antiferromagnetically  order at $T_N=$  289 K with a magnetic ordering wave vector of $\bm{q_\textnormal{fcc}} = \left(\frac{1}{2},\frac{1}{2},\frac{1}{2}\right)$
\cite{GreenwaldAC1953,TomiyasuPRB2004}.
Unlike LaCoO$_3$, Co$^{2+}$ in CoO does not show LS or IS states nor a spin-state transition.

An intruiguing set of questions arise as to what happens if the Co$^{2+}$ cations are realized in a perovskite heterostructure containing LaCoO$_3$.  Is the difference between the electronic behavior of LaCoO$_3$ and CoO only due to charge state of the Co cations?  To what extent can the Co electronic properties be controlled or manipulated by changing the local lattice structure surrounding the Co cation?  And how can one made strong modifications to the local structure?  One way to control the charge state of transition metal cations and the local lattice structure is via heterostructuring. For example, charge transfer can be triggered from Ti to Ni in LaNiO$_3$+LaTiO$_3$ superlattices~\cite{ChenPRL2013,DisaPRL2015} which results in a  Ni$^{2+}$ charge state with large $e_g$ orbital polarization,
which is in strong contrast to the orbitally unpolarized Ni$^{3+}$ in bulk LaNiO$_3$.  Recent work on LaCoO$_3$+LaTiO$_3$ superlattices show similar charge transfer and orbital polarization as well~\cite{our PRL once it is out soon}, which we examine in great detail here theoretically \cite{JaePRL2019}.

In this work, we study the electronic and magnetic properties of Co in (LaTiO$_3$)$_1$+(LaCoO$_3$)$_1$
and (LaTiO$_3$)$_2$+(LaCoO$_3$)$_2$ superlattices using DFT+$U$ theory.  We explore carefully the effect of the Hubbard $U$ parameter, account for the effects of different exchange-correlation approximations, examine the effect of strain from the substrate, as well as the effects of doping.
We explain the origin of the magnetic stabilities for different spin states and magnetic orderings by using a simple band interaction picture.  Interestingly, from the analysis of the magnetic stabilities, we are able to show that ferromagnetic spin order on for the Co$^{2+}$ cations can be stabilized by the hole doping which is an unexpected and unique finding for Co$^{2+}$ as detailed below (Summary and outlook section).

This paper is organized as follows.
In Sec. \ref{sec:bulk}, we study the $U_{\textnormal{Co}}$-dependent properties of bulk LaCoO$_3$ and CoO,
which have Co$^{3+}$ and Co$^{2+}$ cations, respectively; this allows us to find ranges of  $U_{\textnormal{Co}}$ that describe key electronic and magnetic properties of two materials correctly when compared to experiment.
In Sec. \ref{sec:general_aspects}, we describe the general atomic, electronic, and magnetic properties of
LaTiO$_3$+LaCoO$_3$ superlattices.
In Sec. \ref{sec:LTO1+LCO1}, we discuss the electronic and magnetic properties of
(LaCoO$_3$)$_1$+(LaTiO$_3$)$_1$ superlattices.
In Sec. \ref{sec:spin_states}, we describe the energetics of HS and LS states
as a function of $U_{\textnormal{Co}}$ and examine the origin of the $U_{\textnormal{Co}}$ dependences.
We present the magnetic stabilities of Co as a function of $U_{\textnormal{Co}}$ in Sec.~\ref{sec:mag_stability}
and explain using a simple band interaction model in Sec.~\ref{sec:mag_origin}.
Effects of strain and different exchange-correlation approximations are presented in Sec.~\ref{sec:strain}.
In Sec.~\ref{sec:doping}, we show that the ferromagnetic phase is stabilized by hole-doping in
(LaCoO$_3$)$_1$+(LaTiO$_3$)$_1$, and we explain the nature of its magnetic stability.  In Sec.~\ref{sec:LTO2+LCO2}, we repeat our analysis of electronic and magnetic properties for
(LaCoO$_3$)$_2$+(LaTiO$_3$)$_2$ and examine the importance of out-of-plane Co--Co magnetic interaction.

\section{Computational Details}

We use density functional theory (DFT) with the projector augmented wave (PAW) method \cite{PAW} and the revised version of the generalized gradient
approximation (GGA) proposed by Perdew {\it et al.} (PBEsol) \cite{PBEsol} as implemented in the VASP software \cite{VASP}.
We also have done selective tests using the local density approximation (LDA) \cite{CA,Perdew}.
In all cases, the spin-dependent version of the exchange correlation functional is employed.
A plane wave basis with a kinetic energy cutoff of 500 eV is used.
We use $\Gamma$-centered \textbf{k}-point meshes of size  9$\times$9$\times$7 and 9$\times$9$\times$5
for (LaCoO$_3$)$_1$+(LaTiO$_3$)$_1$ and (LaCoO$_3$)$_2$+(LaTiO$_3$)$_2$, respectively.
The size of the supercells for (LaCoO$_3$)$_1$+(LaTiO$_3$)$_1$ and (LaCoO$_3$)$_2$+(LaTiO$_3$)$_2$
are 20 and 40 atoms, respectively, correspond to the $\sqrt 2 \times \sqrt2$ (i.e., $c(2\times2)$) in-plane unit cells.
These unit cells have two Co atoms in each CoO$_2$ plane, so that the
(LaCoO$_3$)$_1$+(LaTiO$_3$)$_1$ and (LaCoO$_3$)$_2$+(LaTiO$_3$)$_2$ each have 2 and 4 Co atoms in their unit cells, respectively.
Atomic positions within the
unit cells were relaxed until the residual forces were less than 0.01 eV/\AA{} in all axial directions, and the stress is less than 0.5kB along the $z$ axis.

The GGA+$U$ scheme within the rotationally invariant formalism together with the fully localized
limit double-counting formula \cite{LDA+U1} is used to study the effect of electron interactions.
The electronic and structural properties critically depend on
$U_{\textnormal{Co}}$, and  therefore we carefully explore a range of values.
We also explore how the results depend on $U_{\textnormal{Ti}}$,
which plays a secondary role in the physics of these materials
We do not employ an on-site exchange interaction $J$ for any species, as the exchange interaction is already accounted for within the spin-dependent DFT exchange-correlation potential \cite{Hyowon,Chen2015}.    Finally, we used Maximally Localized Wannier Functions (MWLF) as implemented in the Wannier90 software~\cite{wannier90} to obtain the localized tight-binding representation describing Co 3d orbitals and their hybridizaiton.

\subsection{$U$ dependence of LaCoO$_3$ \& CoO}
\label{sec:bulk}

In this section, we aim to obtain optimal $U_{\textnormal{Co}}$ value for Co$^{2+}$ in the LCO+LTO superlattices by studying bulk LaCoO$_3$ (Co$^{3+}$) and CoO (Co$^{2+}$).
Below, we will show  that the experimental nonmagnetic ground state for LaCoO$_3$ is stable with $U_{\textnormal{Co}} \leq 2.5$ eV for LaCoO$_3$, while $U_{\textnormal{Co}} \sim 4.5$ eV is needed to obtain the experimental band gap of bulk CoO (see Figure~\ref{bulk}).
We will conclude that $U_{\textnormal{Co}}=3$ eV is a reasonable value.

\begin{figure}
\begin{center}
\includegraphics[width=0.5\textwidth, angle=0]{bulk.eps}
\caption{Total energies of different magnetic states of bulk LaCoO$_3$ within (a) GGA+$U$ and (b) LDA+$U$, and the
band gap of bulk LaCoO$_3$ within (c) GGA+$U$ and (d) LDA+$U$.
Band gap of bulk CoO within (e) GGA+$U$ and (f) LDA+$U$.  The total energies of the nonmagnetic insulating phase (NM) are set to zero.}
\label{bulk}
\end{center}
\end{figure}

\subsubsection{Bulk LaCoO$_3$ (LCO)}

We consider the energetics of different magnetic configurations of Co$^{3+}$ in LCO.
We begin with bulk LaCoO$_3$ within both GGA+$U$ and LDA+$U$ for $0 \leq U_{\textnormal{Co}} \leq 5$ eV
with the $R\bar{3}c$ crystal structure, with a unit cell containing two Co atoms.
The atomic structures and lattice vectors are  fully relaxed.
As presented in Figs.~\ref{bulk} (a,b) and Table~\ref{lcotable}, we can stabilize five different Co$^{3+}$spin states:
$S$=0, 1/2, 1, 1\#2, and 2.
Interestingly, $S$=1/2, 1, and 1\#2 show charge, spin, and bond length disproportionation (see Table~\ref{lcotable}),
while their  structural symmetry is still $R\bar{3}c$.
For example, for $S$=1, the two Co atoms (Co1 and Co2) have different total numbers of $d$ electrons ($N_d$) and Co magnetic moments ($M$),
in addition to different Co--O bond lengths $d_{\textnormal{Co--O}}$:
$N_{d}({\textnormal{Co1}})=7.349$ with $d_{\textnormal{Co1--O}}= 1.90$ \AA{} while
$N_{d}({\textnormal{Co2}})=7.047$ with $d_{\textnormal{Co2--O}}= 1.98$ \AA.

\begin{table}
\begin{ruledtabular}
\begin{center}
\caption{Magnetic moment $M$ ($\mu_B$) and $N_{d}$  of Co1 and Co2 in bulk LaCoO$_3$.
$U_{\textnormal{Co}}=2.5$ eV is used, except for the case $S=2$ since this state is not even metastable when $U_{\textnormal{Co}} < 3.5 $ eV. $M$ and $N_d$ are computed from the VASP-calculated local 3d occupancies around each atom.}
\label{lcotable}
\renewcommand{\arraystretch}{1.3}
\begin{tabular}{c c c c c}
& \multicolumn{2}{c}{$M$(Co)} ($\mu_B$))	& \multicolumn{2}{c}{$N_d$(Co)} \\
\cline{2-3} \cline{4-5}
& Co1 & Co2 & Co1 & Co2 \\
 \hline
(i) $S$=0 (NM)		& 0		& 0		& 7.332	& 7.332 \\
(ii) $S$=1/2	& 0.344	& 1.795 	& 7.375	& 7.219 \\
(iii) $S$=1		& 0.366	& 2.922	& 7.349	& 7.047 \\
(iv) $S$=1\#2	& 1.545	& 2.322	& 7.222	& 7.196 \\
(v)  $S$=2 	& 3.224	& 3.224	& 6.956	& 6.956 \\
\end{tabular}
\end{center}
\end{ruledtabular}
\end{table}

Within GGA+$U$  (Fig. \ref{bulk}(a)),
the ground state of bulk LaCoO$_3$ within GGA+$U$ is $S=0$ (nonmagnetic state, NM)
for $0 \leq U_{\textnormal{Co}} < 2.5$ eV,
$S=1$ for $2.5 \leq U_{\textnormal{Co}} < 4$ eV,
and $S=1\#2$ for $4 \leq U_{\textnormal{Co}} \leq 5$ eV.
Thus, within GGA+$U$, $0 \leq U_{\textnormal{Co}} < 2.5$ eV is appropriate to reproduce the nonmagnetic low-temperature ground state in the experiment. Within LDA+$U$, the $S=0$ NM state is the ground state for $0 \leq U_{\textnormal{Co}} \leq 4.5$ eV.

In addition to the energetics of the various spin states, we also consider the band gap ($E_g$) of bulk LaCoO$_3$.
As summarized in Fig.~\ref{bulk} (c), $E_g$ within GGA+$U$ with $U=2.5$ eV and $U=3$ eV are 0.51 and 0.67 eV, respectively,
which are both close to the the experimental gap of $\approx0.6$ eV \cite{ChainaniPRB1992}.
Within LDA+$U$, $U=3.5$ eV gives $E_g=0.65$ eV, as shown in Fig.~\ref{bulk} (d).

Considering both the nature of the magnetic ground state and a reasonable value of the energy gap,
we conclude that $U=2.5$ eV within GGA+$U$ and $U=3.5$ eV within LDA+$U$ are  reasonable values for studying bulk LaCoO$_3$.
Our $U$ values are comparable to those from previous DFT+$U$ studies:
$U_{\textnormal{Co}}-J_{\textnormal{Co}}$= 2.7, 3.5, and 4.5 eV~ \cite{KnizekJPCM2006}
or LDA+$U$ values of $U_{\textnormal{Co}}=7.8$ eV and $J_{\textnormal{Co}}= 0.92$ eV~\cite{KorotinPRB1996}.

\subsubsection{Bulk CoO}
In the LaCoO$_3$+LaTiO$_3$ superlattice, the electron transfer from Ti to Co will lead to a 2+ charge state for Co (see the next section).
Therefore, we also calculate the $U_{\textnormal{Co}}$ dependence of the band gap of bulk CoO which also contains Co$^{2+}$.

At low temperature, bulk CoO is antiferromagnetic with wave vector
$\bm{q_\textnormal{fcc}} = \left(\frac{1}{2},\frac{1}{2},\frac{1}{2}\right)$
\cite{GreenwaldAC1953,TomiyasuPRB2004},
and an insulator with energy gap $E_g =2.4$ eV \cite{ZimmermannJPCM1999}.
In Figs. \ref{bulk} (c) and (d), we plot $E_g$ of CoO as a function of $U_{\textnormal{Co}}$.
$U_{\textnormal{Co}}=4$ eV within GGA+$U$ and $U_{\textnormal{Co}}\sim4.8$ eV within LDA+$U$
give the experimental gap.
These values are similar to the previous GGA+$U$ results:
$E_g =2$ eV with $U_{\textnormal{Co}}-J_{\textnormal{Co}}$= 3.3 eV  \cite{WangPRB2006},
$E_g =2.8$ eV with $U_{\textnormal{Co}}=7.1$ eV and $J_{\textnormal{Co}}= 1$ eV \cite{WangPRB2006}.

\subsubsection{Choosing $U_{\textnormal{Co}}$}

Within GGA+$U$, we need $U_{\textnormal{Co}} = 2.5$ eV for bulk LaCoO$_3$ to be described well, and $U_{\textnormal{Co}}=4$ eV for bulk CoO.   Since we expect out superlattice to contain Co$^{2+}$ which is more similar to CoO, we choose $U_{\textnormal{Co}}=3$ eV  unless otherwise specified.
Below, we will explore $0 \leq U_{\textnormal{Co}} \leq 5$ eV to study the $U$-dependence of the electronic and magnetic properties.
We consider $U_{\textnormal{Ti}}$= 0, 3, and 5 eV, and use $U_{\textnormal{Ti}}=3$ eV unless specified otherwise.
This value of $U_{\textnormal{Ti}}$ is obtained by fitting the optical gap of LaTiO$_3$ to be 0.2 eV based on our previous work \cite{LTOLAO}.

\section{ LaCoO$_3$+LaTiO$_3$ superlattices}
\label{sec:LCO+LTO}

\subsection{General aspects}
\label{sec:general_aspects}

In this subsection, we discuss the overall atomic, electronic, and magnetic properties of LaCoO$_3$+LaTiO$_3$ superlattices.
We will show that an electron is transferred from Ti to Co resulting in Ti$^{4+}$ ($d^0$) and Co$^{2+}$ ($d^7$).
This leads to a local electric field between CoO$_2$ and TiO$_2$ layers which results in large shifts of the La$^{3+}$ and O$^{2-}$ ions between them and an elongation of interfacial Co--O bond.  Electronically, the main observation of this section is that the
the interfacial Co$^{2+}$ can have a high-spin (HS, $t^{5}_{2g}e^{2}_{g}$) or low-spin state (LS, $t^{6}_{2g}e^{1}_{g}$).

\begin{figure}
\begin{center}
\includegraphics[width=0.5\textwidth, angle=0]{atm_str.eps}
\caption{Atomic-scale structures of (LaCoO$_3$)$_1$+(LaTiO$_3$)$_1$ and (LaCoO$_3$)$_2$+(LaTiO$_3$)$_2$ superlattices.
(a) Top view and  (b) side view of the (LaCoO$_3$)$_1$+(LaTiO$_3$)$_1$ superlattice.
(c) Side view of the (LaCoO$_3$)$_2$+(LaTiO$_3$)$_2$ superlatttice.
}
\label{atm_str}
\end{center}
\end{figure}

We begin by discussing the structure of the LaCoO$_3$+LaTiO$_3$ (LCO+LTO) superlattices. In addition to the relaxation of the shape of the local oxygen octahedra, resulting in Jahn-Teller-like distortions, tilting of oxygen octahedra is also very important in determining the electronic structure of  transition metal oxides. Unlike bulk oxides, since the tilt pattern of the superlattice is harder to observe (due to thin samples) and is not known {\it a priori}, we consider several tilt patterns and choose the most stable one. Since bulk LCO has the $R\bar{3}c$ crystal structure with $a^-a^-a^-$ octahedral tilts and bulk LaTiO$_3$ has the $Pbnm$ structure with $a^-a^-b^+$ tilts,
we examine three cases: $a^0a^0a^0$, $a^-a^-a^-$ and $a^-a^-b^+$.
We find that $a^-a^-b^+$ is always the most stable, so we only examine this tilt pattern below unless further specified.

The structures are relaxed along the $c$ axis, which is normal to the interface, while in-plane lattice parameters $a$ and $b$ are fixed and set equal $a=b$ to simulate epitaxial strain from a substrate. Three sets of in-plane lattice parameters are used: 3.663 \AA, 3.784 \AA, and 3.905 \AA: 3.784 \AA\ and 3.905 \AA\  are the experimental lattice parameters of LaAlO$_3$ (LAO) and SrTiO$_3$ (STO) substrates, respectively.
The optimized lattice parameter of bulk LCO within GGA+$U$ ($U_{\textnormal{Co}}=3$ eV) is 3.805 \AA ,close to the LAO lattice.
To study the effect of the compressive strain on LCO, we use 3.663 \AA\ which is smaller than LCO lattice.

\begin{figure}
\begin{center}
\includegraphics[width=0.5\textwidth, angle=0]{tdos.eps}
\caption{Total (black) and projected (colors) density of states (DOS) of (LaCoO$_3$)$_1$+(LaTiO$_3$)$_1$ superlattice onto one Co atom.
The Co atoms have high-spin states and AFM spin alignment.
$U_{\textnormal{Co}}=3$ eV and $U_{\textnormal{Ti}}=3$ eV with GGA+$U$ are used, and the in-plane lattice parameter is $a=3.784$ \AA.
Positive and negative DOS describe spin up and spin down electronic states, respectively.
 }
\end{center}
\label{tdos}
\end{figure}

\begin{figure}
\begin{center}
\includegraphics[width=0.5\textwidth, angle=0]{HS_LS_pdos.eps}
\caption{Projected density of states of (LaCoO$_3$)$_1$+(LaTiO$_3$)$_1$ for (a) high-spin and (c) low-spin FM states.
(b) and (d) show schematics of the corresponding atomic-like energy levels.
$U_{\textnormal{Co}}=3$ eV and $U_{\textnormal{Ti}}=3$ eV with GGA+$U$ are used, and the in-plane lattice parameter is $a=3.784$ \AA.
}
\label{HS_LS_pdos}
\end{center}
\end{figure}

Next, we discuss the charge states of transition metals and the electron transfer.
Given the difference in electronegativity between Ti and Co \cite{ALLRED1961215,AllenJACS1989},
 electron transfer is expected from Ti 3$d$ to Co 3$d$ across the interface.
Projected densities of states (Fig.~\ref{tdos}) within GGA+$U$ show that the Ti $d$ bands are completely empty, which corresponds to Ti$^{4+}$, and describe the conduction band minimum.
On the other hand, Co has 2+ charge state with $d^7$ electrons, indicating full charge transfer.

\begin{figure}
\begin{center}
\includegraphics[width=0.45\textwidth, angle=0]{Efield.eps}
\caption{Atomic structure of the (LaCoO$_3$)$_1$+(LaTiO$_3$)$_1$ superlattice with $a^0a^0a^0$ octahderal tilt
and schematic picture of the local electric field.
Distances between the La plane and the TiO$_2$ or CoO$_2$ planeas as well as the shift between La and O in the same plane (prior to displacement) are shown on the figure.  The structure is obtained by using $U_{\textnormal{Co}}=3$ eV, $U_{\textnormal{Ti}}=2.5$ eV, GGA+$U$, and an in-plane lattice parameter of $a=3.784$ \AA.
  }
\label{Efield}
\end{center}
\end{figure}

Due to the charge transfer, we have Ti$^{4+}$ and Co$^{2+}$ and we expect local electric fields going from the TiO$2$ layer to the CoO$_2$ layer.  This can be confirmed by looking at the positions of the positively charged La cations (La$^{2+}$) and negatively charged apical oxygen anions (O$^{2-}$): the displacements are  visible to the eye in Fig.~\ref{atm_str}.  For example, for
 (LCO)$_2$+(LTO)$_2$ on an STO substrate, the out of plane Co--O bond length between the CoO$_2$ planes (denoted as bulk-like Co--O bond in Fig.~\ref{atm_str}(c)) is 2.02 \AA,
while the bond length between the CoO$_2$ and TiO$_2$ planes (denoted as interfacial Co--O bond in Fig.~\ref{atm_str}(c)) is 2.28 \AA, indicating that O$^{2-}$ is shifted to the opposite direction of the local electric field.  A simpler theoretical view unencumbered by octahedral tilting is afforded by considering (LCO)$_1$+(LTO)$_1$ with $a^0a^0a^0$ tilt within GGA+$U$ whose structure is shown in Fig.~\ref{Efield}.  Here, the distance between the La$^{3+}$ plane and the CoO$_2$ plane is 1.75 \AA, while the distance between La$^{3+}$ and TiO$_2$ plane is 2.10 \AA.

We now discuss the Co$^{2+}$ spin states.
The HS state  ($t^{5}_{2g}e^{2}_{g}$) with $S=1$ is depicted in Figs.~\ref{HS_LS_pdos}(a) and (b): the spin-up $d$ bands are fully occupied, while spin-down $d$ bands have two electrons in the $t_{2g}$ channel.
Due to the non-zero $U_{\textnormal{Co}}$, the spin-down $t_{2g}$ splits into two degenerate bands ($d_{xz},d_{yz}$) under the Fermi level
and one empty single band ($d_{xy}$) above the Fermi level, resulting in a Mott-like insulator.
We find that $U_{\textnormal{Co}}>0$ is necessary  for opening a energy gap for the HS state.  In addition, the HS state is not even metastable at $U_{\textnormal{Co}}=0$:
$U_{\textnormal{Co}}>2$ eV is needed (with $U_{\textnormal{Ti}}=3$ eV) to obtain a gapped HS state.  Regarding the orbital polarization,
we find that the degeneracy of the $t_{2g}$ is already removed at $U_{\textnormal{Co}}=0$
because of the broken symmetry due to  interface formation.
This splitting  between $d_{xz}$/$d_{yz}$ and $d_{xy}$ is further enhanced by the imposed epitaxial strain and also enlarged by the non-zero $U_{\textnormal{Co}}$.

The LS state ($t^{6}_{2g}e^{1}_{g}$) has $S=1/2$ and is presented in Figs.~\ref{HS_LS_pdos}(c) and (d):
the $t_{2g}$ bands are fully occupied, while the one remaining electron is in the $e_g$ channel.  The degeneracy of the $e_g$ manifold is already broken at $U_{\textnormal{Co}}=0$ due to the interface formation and epitaxial strain.
Due to the large degeneracy splitting of the $e_g$ bands, only the spin-up $d_{z^2}$ is filled in the LS state while the spin-up $d_{x^2-y^2}$ and spin-down $e_g$ bands are empty.
As a result, LS state is already insulating at $U_{\textnormal{Co}}=0$ and has strong $e_g$ orbital polarization (and $U_{\textnormal{Co}}>0$ enhances the polarization and the energy gap).

\subsection{(LaCoO$_3$)$_1$+(LaTiO$_3$)$_1$}
\label{sec:LTO1+LCO1}

In this subsection, we discuss, in great detail, the electronic and magnetic properties of the (LaCoO$_3$)$_1$+(LaTiO$_3$)$_1$ superlatice ((LTO)$_1$+(LCO)$_1$ below).
In Sec.~\ref{sec:spin_states}, we analyze the energy difference between high-spin and low-spin states
as a function of $U_{\textnormal{Co}}$.
In Sec.~\ref{sec:mag_stability}, we examine the $U_{\textnormal{Co}}$ dependence of the the energy difference
between ferromagnetic and antiferromagnetic spin orders.
We study the effects of strain and different exchange correlation functionals in Sec.~\ref{sec:strain}.

The material system of interest is (LTO)$_1$+(LCO)$_1$ stacked into an infinite superlattice along the [001] direction as depicted in Fig.~\ref{atm_str}(c).
As mentioned above, three in-plane lattice parameters are considered: $a$=3.663, 3.784, and 3.905 \AA.
We first focus on $a=3.784$ \AA, which corresponds to the LAO substrate, and discuss the effect of the strain later.
In (LTO)$_1$+(LCO)$_1$, the CoO$_6$ octahedron has  interfaces on both sides.
Therefore, both out-of-plane Co--O bonds are elongated, due to the local electric field  discussed above.
When $a=3.784$ \AA, the in-plane Co--O bond lengths are 1.947 and 1.950 \AA, whereas the out-of-plane Co--O bond length is 2.205 \AA.

Our supercell of (LTO)$_1$+(LCO)$_1$ has a $c(2\times2)$ in-plane unit cell and therefore two distinct Co atoms, which can be HS or LS separately, and also can be spin-up or spin-down separately.
Therefore, there are 6 different spin configurations:
ferromagnetic with two high-spin Co  (HS FM),
antiferromagnetic with two high-spin Co (HS AFM),
ferromagnetic with one high-spin and one low-spin Co  (HS/LS FM),
ferrimagnetic with one high-spin and one low-spin Co  (HS/LS FIM),
ferromagnetic with two low-spin Co  (LS FM), and
antiferromagnetic with two low-spin Co (LS AFM).
Among them, HS AFM is the most stable over a wide range of parameters, as shown in Fig.~\ref{HS_LS_Udep}(a).
For fixed spin states, HS AFM is more stable than HS FM (Fig. \ref{mag_Udep}(a)),
HS/LS FM is more stable than HS/LS FIM (Fig. \ref{mag_Udep}(c)),
and LS FM is more stable than LS AFM (Fig. \ref{mag_Udep}(e)).

\subsubsection{HS vs LS with fixed magnetic order: $U$ dependence}
\label{sec:spin_states}

\begin{figure}
\begin{center}
\includegraphics[width=0.5\textwidth, angle=0]{HS_LS_Udep.eps}
\caption{ $U_{\textnormal{Co}}$ dependence of the total energies for  AFM states of (LTO)$_1$+(LCO)$_1$ superlattices.
(a) Total energies per Co of different magnetic configurations (the total energy of the LS state is chosen as the zero of energy).  (b) Spectral decomposition of the $+U$ energy contribution (see text).
}
\label{HS_LS_Udep}
\end{center}
\end{figure}

Interestingly, the above energy differences strongly depend on $U_{\textnormal{Co}}$.
Specifically, when the magnetic ordering is fixed to AFM, the energy difference between different spin states of Co
increases monotonically as a function of $U_{\textnormal{Co}}$ as shown in Fig.~\ref{HS_LS_Udep}(a).
To explain this strong dependence, we analyze $E$[HS AFM]$-E$[LS AFM] by rewriting the DFT+$U$ energy as a spectral decomposition \cite{Isaacs2016035120,Isaacs2017045141}
\begin{equation}
\begin{aligned}
E =E_{\textnormal{DFT}+U} &= E_{\textnormal{DFT}}  + E_U - E_{\textnormal{dc}}  \\
                                        &= E_{\textnormal{DFT}} + E_{\textnormal{fill}} + E_{\textnormal{ord}}
\end{aligned}
\label{eq:EDFTUspectral}
\end{equation}
where
\begin{equation}
\begin{aligned}
E_{\textnormal{fill}} &= U(2l+1)\mu(1-\mu), \ \ \ \ E_{\textnormal{ord}}= -U(2l+1) \sigma^2 .
\end{aligned}
\label{eq:EfillEord}
\end{equation}
Here $l=2$ is the angular momentum of the atomic-like states for the DFT+$U$ treatment, and $\mu$ and $\sigma$ are the
mean value and standard deviation of the eigenvalues of the single particle density matrix
of the manifold of atomic-like orbtials (here Co $3d$ states).
Within this spectral decomposition, the combined interaction and double counting
energy simply depends on the two numbers $\mu$ and $\sigma$ extracted from the local single-particle density matrix.

We define $\Delta E_{\textnormal{fill}}^{\rm AFM}$ and $\Delta E_{\textnormal{ord}}^{\rm AFM}$
\begin{equation}
\begin{aligned}
\Delta E_{\textnormal{fill}}^\textnormal{AFM} = E_{\textnormal{fill}}^\textnormal{AFM}[{\rm HS}] - E_{\textnormal{fill}}^\textnormal{AFM}[{\rm LS}]  \\
\Delta E_{\textnormal{ord}}^\textnormal{AFM} = E_{\textnormal{ord}}^\textnormal{AFM}[{\rm HS}] - E_{\textnormal{ord}}^\textnormal{AFM}[{\rm LS}]
\end{aligned}
\end{equation}
and plot them versus $U_{\textnormal{Co}}$ in Fig.~ \ref{HS_LS_Udep}(b).  While the change of $\Delta E_{\textnormal{fill}}^{\rm AFM}$ as a function of $U_{\textnormal{Co}}$ is relatively weak, $\Delta E_{\textnormal{ord}}^{\rm AFM}$ is the dominating contribution, similar to other prior cases \cite{Isaacs2016035120,Isaacs2017045141}.
Therefore, the $U_{\textnormal{Co}}$ behavior is dictated largely by the difference in $\sigma^2$ of the two respective spin configurations.  In other words, the orbital occupancy differentiation (i.e., orbital ordering) is more enhanced in the HS state than in the LS state as $U_{\textnormal{Co}}$ increases, and this enlarges the energy difference between them.

\subsubsection{FM vs AFM for fixed Co spin state: $U_{\textnormal{Co}}$ dependence}
\label{sec:mag_stability}

\begin{figure}
\begin{center}
\includegraphics[width=0.5\textwidth, angle=0]{mag_Udep.eps}
\caption{  $U_{\textnormal{Co}}$ dependence of energies for different magnetic states of (LTO)$_1$+(LCO)$_1$.
(a) energies of HS FM and HS AFM, where the energy of HS FM is set to be zero.
(c) $\Delta E_{\textnormal{fill}}[\textnormal{HS}]$,  $\Delta E_{\textnormal{ord}}[\textnormal{HS}]$, and
$\Delta E_{\textnormal{fill}}[\textnormal{HS}]+\Delta E_{\textnormal{ord}}[\textnormal{HS}]$.
(c) Energies of LS FM and LS AFM, where the energy of LS FM is set to be zero.
(c) $\Delta E_{\textnormal{fill}}[\textnormal{LS}]$,  $\Delta E_{\textnormal{ord}}[\textnormal{LS}]$, and
$\Delta E_{\textnormal{fill}}[\textnormal{LS}]+\Delta E_{\textnormal{ord}}[\textnormal{LS}]$.
(e) Energies of HS/LS FM and HS/LS FIM, where the energy of HS/LS FM is set to be zero.
 (f) $\Delta E_{\textnormal{fill}}[\textnormal{HS/LS}]$,
$\Delta E_{\textnormal{ord}}[\textnormal{HS/LS}]$, and
$\Delta E_{\textnormal{fill}}[\textnormal{HS/LS}]+\Delta E_{\textnormal{ord}}[\textnormal{HS/LS}]$.
The energy of the FM phase is set to  zero.
}
\label{mag_Udep}
\end{center}
\end{figure}

Now we consider the energy difference between two magnetic configurations for fixed Co spin state.
Figs.~\ref{mag_Udep}(a) and (c) show that the energy difference between FM and AFM phases.
While the energy difference between FM and AFM decreases as a function of $U_{\textnormal{Co}}$ for the HS state, this energy difference increases as a function of $U_{\textnormal{Co}}$ for the LS state which is the opposite trend.
To explain these trends, we do a similar $U_{\textnormal{Co}}$-dependent spectral analysis for $\Delta E$[HS]$=E^{\rm AFM}$[HS]$-E^{\rm FM}$[HS] (Figs. \ref{mag_Udep}(a) and (b)),
$\Delta E$[LS]$=E^{\rm AFM}$[LS]$-E^{\rm FM}$[LS] (Figs. \ref{mag_Udep}(c) and (d)),
and $\Delta E$[HS/LS]$=E^{\rm FIM}$[HS/LS]$-E^{\rm FM}$[HS/LS] (Figs. \ref{mag_Udep}(e) and (f)). $\Delta E_{\textnormal{fill}}$ and $\Delta E_{\textnormal{ord}}$ are defined in a similar way as above: e.g.,
\begin{equation}
\begin{aligned}
\Delta E_{\textnormal{fill}}[\textnormal{HS}] = E_{\textnormal{fill}}^{\rm AFM}[\textnormal{HS}] - E_{\textnormal{fill}}^{\rm FM}[\textnormal{HS}]  \\
\Delta E_{\textnormal{ord}}[\textnormal{HS}] = E_{\textnormal{ord}}^{\rm AFM}[\textnormal{HS}]- E_{\textnormal{ord}}^{\rm FM}[\textnormal{HS}]\,.
\end{aligned}
\end{equation}

Looking at the pair of Figs.~\ref{mag_Udep}(a,b) and the pair Figs.~\ref{mag_Udep}(a,b), the situation is very similar to that in the previous subsection: in both cases $\Delta E_{\textnormal{fill}}$[HS] is almost constant versus $U_{\textnormal{Co}}$ and relatively small in size while $\Delta E_{\textnormal{ord}}$[HS]  dominates the behavior in that its increase with increasing $U_{\textnormal{Co}}$ explains the positive slope of $\Delta E$[HS] or $\Delta E$[LS].

We also considered the intermediate spin HS/LS FM and AFM (really ferrimagnetic as the moments are unequal) configurations where the unit cell contains one LS and one HS Co cation.
As presented in Fig.~\ref{mag_Udep}(e), $\Delta E$[HS/LS] varies by about 10 meV as $U_{\textnormal{Co}}$ changes from 2.5 to 5 eV.
$\Delta (E_{\textnormal{fill}}+E_{\textnormal{ord}})$[HS/LS] for the HS Co and the LS Co (see Fig. \ref{mag_Udep}(f)) show opposing behaviors versus  $U_{\textnormal{Co}}$ so their sum is relatively constant .
As a result, the averaged value of $\Delta( E_{\textnormal{fill}}+E_{\textnormal{ord}})$ at $U=2.5$ eV and $U=5$ eV
are only differ by 12 meV.

\subsubsection{Origin of the magnetic stabilities}
\label{sec:mag_origin}

The above numerical observations of the relative magnetic stabilities require  explanations.  Here, we will explain the origin of the magnetic energetics of (LTO)$_1$+(LCO)$_1$ using an energy band picture (i.e., molecular orbital theory) for three different spin configurations:
HS, HS/LS, and LS.

\begin{figure}
\begin{center}
\includegraphics[width=0.5\textwidth, angle=0]{HS-HS_ip.eps}
\caption{  Schematic molecular orbital view of Co 3$d$ energy level diagrams and in-plane magnetic interactions between two neighboring HS Co$^{2+}$ cations. (a) and (b) show two neighboring Co with FM alignment, while (c) and (d) show two neighoring Co with AFM alignment.  Solid horizontal lines indicate energy levels; dashed lines indicate the effect of interactions; arrows indicate electron filling colored by orbital type. }
\label{HS-HS_ip}
\end{center}
\end{figure}

Fig.~\ref{HS-HS_ip} shows the schematic energy diagram of for two in-plane HS Co neighbors with FM and AFM relative spin alignment.  The interaction lines in this diagram do not indicate direct Co 3d-Co 3d hybridization but instead the interaction mediated through the intervening O $2p$ states.  The energetic position of the atomic-like energy levels are drawn based on analysis of projected densities of states diagrams such as Fig.~\ref{HS_LS_pdos}.

The spin-up majority bands of HS FM are fully occupied, thus the Co-Co interactions do not lead to energetic stabilization as per Fig.~\ref{HS-HS_ip}(a) (i.e., both bonding and anti-bonding states are equally occupied).  We note that the our Co cations have strong orbital degeneracy breaking whereby the $t_{2g}$ $d_{yz}$/$d_{zx}$ bands are lower in energy than the $d_{xy}$ band is empty: in the minority-spin channel, the $d_{yz}$/$d_{zx}$ bands are filled while the $d_{xy}$ is empty. As discussed in Sec.~\ref{sec:LTO1+LCO1}, this splitting results a Mott-like insulating state. In the HS FM case, the minority-spin bands also do not lead to energetic stabilization: the $d_{yz}$($d_{zx}$) spin-down on one Co only interact with the  $d_{yz}$($d_{zx}$) spin-down states on the neighboring Co, and as both are filled, there is no net lowering of energy as the bonding and anti-bonding states are again fully occupied (Fig.~\ref{HS-HS_ip}(b)).  Thus the energy lowering due to Co-Co interactions for HS FM is zero, $\Delta E^{\rm FM}[{\rm HS}]=0$.

On the other hand, HS AFM Co-Co superexchange interactions lead to energetic stabilization (Figs.~\ref{HS-HS_ip}(c,d)).
While one Co has 5 spin-up and 2 spin-down electrons, the other Co has 2 spin-up and 5 spin-down electrons:
$d_{z^2}$, $d_{yz}$ and $d_{zx}$  are fully occupied for both Co ions leading to no energy lowering; but the electrons in the majority-spin
$d_{x^2-y^2}$ and $d_{xy}$ states interact with empty counterparts on the other Co which leads to energy lowering (only bonding states are filled) as shown in Figs.~\ref{HS-HS_ip}(c) and (d).
If we define the strength of the Co-Co interaction for orbital $d$ on one Co with the same orbital on the neighbor for the FM case as $-\Delta^{11}_{d}$ (see Fig.~\ref{HS-HS_ip}(b)) and for the AFM case as $-\Delta^{12}_{d}$ (see Fig.~\ref{HS-HS_ip}(c)), respectively,
then the energy lowering of the spin-up channel in the AFM case is
\begin{equation}
\begin{aligned}
\Delta E^{\textnormal{AFM}}_{\uparrow}[{\rm HS}] = - \Delta^{12}_{d_{x^2-y^2}} - \Delta^{12}_{d_{z^2}} - \Delta^{12}_{d_{xy}} .
\end{aligned}
\end{equation}
The spin-down channel has same amount of the energy lowering.
Therefore, within this model, the energy difference between HS FM and HS AFM can be written as
\begin{equation}
\begin{aligned}
\Delta E[\textnormal{HS}]
&= \Delta E^{\textnormal{AFM}}[{\rm HS}] - \Delta E^{\textnormal{FM}}[{\rm HS}] \\
&= -2\left( \Delta^{12}_{d_{x^2-y^2}}+\Delta^{12}_{d_{z^2}}+\Delta^{12}_{d_{xy}} \right)\,,
\end{aligned}
\end{equation}
which is negative as the $\Delta^{ij}_d$ are positive by our convention.  Hence, this analysis provides a simple explanation of the stability of HS AFM compared to HS FM.

\begin{figure}
\begin{center}
\includegraphics[width=0.5\textwidth, angle=0]{HS-LS_ip.eps}
\caption{  Schematic diagram of the in-plane magnetic interaction between HS Co and LS Co.}
\label{HS/LS_ip}
\end{center}
\end{figure}

Figs.~\ref{HS/LS_ip}(a,b) presents the in-plane interactions between a HS Co and its neighboring LS Co for FM and FIM relative spin alignments.
While the majority spin-up $d$ bands of the HS Co are fully occupied, the $d_{x^2-y^2}$ spin-up band of the LS Co is empty
(Fig. \ref{HS/LS_ip}(a)) which leads to an energy lowering in the spin-up channel of  $-\Delta^{11}_{d_{x^2-y^2}}$.
In the minority spin-down channel, the $d_{xy}$ band of the LS Co is occupied while the $d_{xy}$ of the HS Co empty, creating a lowering of energy by $-\Delta^{11}_{d_{xy}}$. Thus, the total energy lowering for FM ordering of HS and LS neighboring Co is
\begin{equation}
\begin{aligned}
\Delta E^{\textnormal{FM}}[{\rm HS/LS}] = - \Delta^{11}_{d_{x^2-y^2}} - \Delta^{11}_{d_{xy}} .
\end{aligned}
\end{equation}

The case of ferrimagnetic (FIM) ordering between a HS Co and LS Co is  shown in Figs.~\ref{HS/LS_ip}(c,d). For the spin-up channel, the HS $d_{x^2-y^2}$ and $d_{z^2}$ are occupied but empty for the LS Co, leading to an energy lowering of $- \Delta^{12}_{d_{x^2-y^2}} - \Delta^{12}_{d_{z^2}}$; a similar lowering happens for the low-spin channel due to $d_{xy}$ and $d_{z^2}$.  The total energetic stabilization for this FIM HS--LS case
\begin{multline}
\Delta E^{\textnormal{FIM}}[{\rm HS/LS}] = -2\Delta^{12}_{d_{z^2}}
- \Delta^{12}_{d_{x^2-y^2}} - \Delta^{12}_{d_{xy}}\,.
\end{multline}

Hence, the energy difference between HS/LS FM and HS/LS FIM is thus
\begin{multline}
\Delta E[\textnormal{HS/LS}]
 = \Delta E^{\textnormal{FM}}[{\rm HS/LS}] - \Delta E^{\textnormal{FIM}}[{\rm HS/LS}] \\
= - \left[ \Delta^{11}_{d_{x^2-y^2}} - \Delta^{12}_{d_{x^2-y^2}} \right] - \left[ \Delta^{11}_{d_{xy}} - \Delta^{12}_{d_{xy}}\right]\\ + 2\Delta^{12}_{d_{z^2}}\,.
\end{multline}
For the $d_{x^2-y^2}$ and $d_{xy}$ contributions, there is some partial cancellation between the $\Delta^{11}$ and $\Delta^{12}$ terms although these contributions should be net negative since FM interactions $\Delta^{11}$ take place between energy degenerate orbitals while the AFM $\Delta^{12}$ are between strongly exchange-split 3d orbitals leading to weaker interactions (i.e., homopolar versus heteropolar bonding in the language of semiconductors).  However, with the addition of the positive contribution $2\Delta^{12}_{d_{z^2}}$, the total is expected to be relatively small in magnitude in agreement the small magnitude of energy differences shown in Fig.~\ref{mag_Udep}(e).  Comparison to data in the figure shows that $\Delta^{11} > \Delta^{12}$ must be true to the extent that it leads to FIM to be higher in energy than FM.

\begin{figure}
\begin{center}
\includegraphics[width=0.5\textwidth, angle=0]{LS-LS_ip.eps}
\caption{  Schematic diagram of the in-plane magnetic interaction between LS Co and LS Co.   }
\label{LS-LS_ip}
\end{center}
\end{figure}

Understanding the stability of the FM state for two LS Co neighbors, however, is more complicated. The simple band interaction picture cannot explain the data in Fig.~\ref{mag_Udep}(e) which show LS FM is more stable than LS AFM: as per Fig.~\ref{LS-LS_ip}, the energy lowering of the FM configuration is zero while the AFM ordering should be lower in energy by $-2\Delta^{12}_{d_{z^2}}$.  Furthermore, standard double exchange for FM configurations is unlikely to be a viable explanation since the system is not metallic (it has an energy gap) due to the large  splitting between $t_{2g}$ and $e_g$ bands.
Nor can Goodenough-Kanamori (GK) type FM superexchange be the origin of the stability,
because GK superexchange is maximum if $\angle$TM--O--TM=90$^\circ$ and cannot be applied for $\angle$TM--O--TM=180$^\circ$
\cite{GoodenoughPRB1955,GoodenoughJPCS1958,KanamoriJPCS1959}.
We find that FM is also stable for the insulating undistorted $a^0a^0a^0$ phase, where Co--O--Co angle is 180$^\circ$.

\begin{figure}
\begin{center}
\includegraphics[width=0.5\textwidth, angle=0]{LS-LS_model.eps}
\caption{  Schematic magnetic interaction model between two neighboring LS Co for (a,b) FM spin order, and (c,d) AFM spin order. Panel (c) shows the  difference in on-site energy $\delta E$ and how the hopping between the two orbitals $t$ leads to an energetic lowering by $t^2/\delta E$ for the bonding state; numerical values are provided in Table~\ref{wannier_parameters}.
}
\label{LS-LS_model}
\end{center}
\end{figure}

In the above analysis, we only considered the interaction between same $d$ orbitals on the neighboring Co, e.g.,
$d_{z^2}$--$d_{z^2}$ or $d_{xy}$--$d_{xy}$, since they are usually the dominant ones.  The model predicted that the LS AFM state is stabilized by $-2\Delta^{12}_{d_{z^2}}$: but will be a very small quantity in this case since the exchange splitting of the $d_{z^2}$ band is larger than 4 eV (see Fig.~\ref{HS_LS_pdos}(b)) and we expect the hopping between the two neighboring $d_{z^2}$ orbitals to be small since they are strongly directional out-of-plane.  Hence, it is likely that interactions between different orbitals on the neighboring Co can play a role in this case.

To capture the essence of the problem, we consider the ideal $a^0a^0a^0$ structure without octrahedral tilts so the Co--O--Co bonding is along a straight line.  In this case, the only symmetry allowed interactions that can lower the energy for the FM order are $d_{z^2}$--$d_{x^2-y2}$ between neighboring atoms for the majority spin-up channel (Figs.~\ref{LS-LS_model}(a,b)).  For AFM order, the relevant hoppings are $d_{z^2}$--$d_{z^2}$ (small and already discussed above) and $d_{z^2}$--$d_{x^2-y^2}$ in both spin channels as per Figs.~\ref{LS-LS_model}(c,d).

\begin{table}
\begin{ruledtabular}
\begin{center}
\caption{Hopping parameters ($t$) and energy splitting of $d_{z^2}$ and $d_{z^2}$ bands ($\delta E$) as obtained from a Maximally Localize Wannier Function analysis.  }
\label{wannier_parameters}
\renewcommand{\arraystretch}{1.3}
\begin{tabular}{c c c c c}
order	& interaction & $\delta E$ (eV)  & $t$ (eV) & $t^2/\delta E$ (eV)\\
 \hline
 FM	& $d_{z^2} - d_{x^2-y2}$	& 2.04	& $-$0.120	& 7.0$\times10^{-3}$ \\
AFM	& $d_{z^2} - d_{z^2}$		& 3.38	& $-$0.010	& 2.9$\times10^{-5}$ \\
AFM	& $d_{z^2} - d_{x^2-y2}$	& 2.98	& $-$0.116 	& 4.5$\times10^{-3}$ \\
\end{tabular}
\end{center}
\end{ruledtabular}
\end{table}

We now create a more quantitative model by extracting on-site energies and hopping parameters for the LS FM and AFM cases by generating Maximally Localize Wannier Functions (MWLF)~\cite{wannier90} that span the energy bands for the $d_{z^2}$ and $d_{x^2-y2}$ character bands near the Fermi level.  These are low-energy or effective Wannier functions that are centered on the Co cations and span the space of the relevant bands (i.e., this is not a p-d model as we don not create any oxygen 2p-like Wannier functions).  Table~\ref{wannier_parameters} displays the key band parameters that describe the hoppings $t$ between neighboring orbitals and their on-site energy differences $\delta E$.  The energy lowering for an interacting pair of neighboring orbitals with a single electron between them is estimated by perturbation theory to be $\Delta=-t^2/\delta E$. Table~\ref{wannier_parameters} shows that (a) the AFM $\Delta^{12}_{d_{z^2}}$  is indeed very weak because the hopping element $t$ is so small, and (b) the FM $d_{z^2}$--$d_{x^2-y^2}$ interaction is more stabilizing due to the smaller energy splitting $\delta E$.  We conclude that the LS FM state is more stable than LS AFM, and the stabilization is driven by more favorable $d_{z^2}$--$d_{x^2-y^2}$ interactions due to the smaller $d_{z^2}$--$d_{x^2-y^2}$ energy splitting in the FM case between Co neighbors.

\subsubsection{Effect of strain and exchange-correlation functional}
\label{sec:strain}

\begin{figure}
\begin{center}
\includegraphics[width=0.5\textwidth, angle=0]{strain_Udep.eps}
\caption{  $U_{\textnormal{Co}}$ dependence of the energies of different magnetic states of (LTO)$_1$+(LCO)$_1$
for different in-plane lattice parameters (3.663 \AA{} for (a,b), 3.784 \AA{} for (c,d), and 3.905 \AA{} for (e,f)) and two different exchange-correlation functionals ((a,c,e) for GGA+$U$ and (b,d,f) for LDA+$U$).  Energies of LS FM phase is set to zero.
}
\label{strain_Udep}
\end{center}
\end{figure}

Given that the spin, orbital, and lattice degrees of freedom strongly interact in a transition metal octahedron,
the electronic and magnetic properties of Co in (LTO)$)1$+(LCO)$_1$  depend on strain strongly.
To study this, we consider the three different in-plane lattice parameters discussed above ($a=3.663$ \AA, 3.784 \AA, and 3.905 \AA).
Within GGA+$U$ for $U_{\textnormal{Co}}=3$ eV, these three lattice parameters
corresponds to 3.7\%  compressive, 0.6\%  compressive, and and 2.6\%  tensile strain.

In Fig. \ref{strain_Udep}, we summarize the strain-dependent stabilities of different spin states and
magnetic orderings of Co within both GGA+$U$ and LDA+$U$.
Interestingly, as shown in Figs.~\ref{strain_Udep}(a,c,e),
the HS phases are stabilized greatly as the lattice parameter increases: the
LS phase is stabilized by compressive strain, whereas the HS phase is stabilized by tensile strain.
In addition, similar to the case of $a=3.784$ \AA{} discussed in detail above in Sec.~\ref{sec:spin_states},
HS FM and HS AFM phases are stabilized as $U_{\textnormal{Co}}$ increases
(see Figs.~\ref{strain_Udep}(a,c,e)).  Various combinations of strain and $U_{\textnormal{Co}}$ can drive magnetic transitions for the Co$^{2+}$: LS FM can be stabilized over the HS AFM for compressive strain and low  $U_{\textnormal{Co}}$ and an intermediate spin (HS/LS combination) state can become the ground state (e.g., $U_{\textnormal{Co}}=2.5$ eV and $a=3.663$ \AA).  However, over broad ranges of parameters, it is the HS AFM that is the ground state.  To the best of our knowledge, HS to LS or FM to AFM transitions have not been suggested or observed for Co$^{2+}$ in bulk CoO, and this makes the LCO+LTO superlattice a potential test bed for modifying the spin state and magnetic interactions of Co$^{2+}$ ions.

We find that the $U_{\textnormal{Co}}$ dependence of the energy difference between HS and LS phases originates from the orbital occupancy differentiation energy $E_{\textnormal{ord}}$ (Eq.~\ref{eq:EfillEord}) much like what was shown in Fig.~\ref{HS_LS_Udep}.  On the other hand, we do not find that the relative stabilization of the LS phase for compressive strain is connected to the atomic-like $+U$ contributions to the total energy $E_{\textnormal{ord}}$ nor $E_{\textnormal{fill}}$ but must instead originate from the underlying DFT energy $E_{DFT}$ that encodes the electronic dispersion and hopping contributions.  The idea that the DFT part of the total energy controls the relative stability of the LS phase means that its stability should strongly depend on the choice of DFT functional. This is borne out by  Fig.~\ref{strain_Udep} which compares the
 GGA and LDA functionals.  The trends versus $U_{\textnormal{Co}}$ and strain are very similar between the two functionals, but the energy of the HS phases is   shifted upwards compared to the LS phases as we switch from GGA to LDA.

\subsubsection{Effect of doping}
\label{sec:doping}

\begin{figure}
\begin{center}
\includegraphics[width=0.48\textwidth, angle=0]{1-1_dop.eps}
\caption{  Magnetic stabilities of electron- and hole-doped (LTO)$_1$+(LCO)$_1$ superlattices,
with (a) $a=3.663$\AA, (b) $a=3.784$ \AA, and (c) $a=3.905$ \AA.
The energy  of the LS FM phase is set to  zero.  }
\label{1-1_dop}
\end{center}
\end{figure}

Electron or hole doping is a general and powerful way to change the magnetic stability of the transition metal oxides such as cuprates \cite{cuprateRMP2006,cuprateRMP2018} and manganites \cite{SalamonRMP2001,HellmanRMP2017}.
By manually changing total number of electrons in the super cell,
we find that both the spin state and the magnetic ordering can  be changed via electron or hole doping in (LTO)+(LCO) superlattices.
In Fig. \ref{1-1_dop}, we present the energies of six different phases as a function of electron/hole doping.
The spin of each Co ion is in fact different since the added electron or hole  resides on one of the Co ions.  Thus, what is referred to as an antiferromagnetic alignment of Co spins is in fact a ferrimagnetic ordering,
but for simplicity we refer to such phases by their parent undoped phase as AFM in Fig. \ref{1-1_dop}.

Overall, the figure shows that relative energetic stabilities are essentially unchanged by electron doping: the HS/LS FM phase is most stable at $a=3.663$ \AA{} (Fig. \ref{1-1_dop}(a)),
and the HS AFM phase is most stable at $a=3.784$ \AA{} and $a=3.905$ \AA{} (Figs. \ref{1-1_dop}(b,c)) for all considered electron doping values.  On the other hand, the relative magnetic stabilities are changed dramatically by  hole doping.
Specifically, the energies of the hole-doped HS phases show large changes
(red and blue curves in Fig. \ref{1-1_dop}).
HS FM phase becomes the ground state if the hole doping per Co is larger than  $\sim$0.2, $\sim$0.13, and $\sim$0.15 for $a=3.663$, 3.784, and 3.905 \AA, respectively.
This result is particularly interesting, because the hole doping via Sr substitution on La sites should be much easier to achieve in LTO-based systems than electron doping.

\begin{figure}
\begin{center}
\includegraphics[width=0.5\textwidth, angle=0]{HS-HS_dop.eps}
\caption{  Schematic diagram of the in-plane magnetic interaction between two neighboring HS Co and HS Co for FM (a,b) and AFM (c,d) configurations in hole-doped (LTO)$_1$+(LCO)$_1$.   }
\label{HS-HS_dop}
\end{center}
\end{figure}

We find that both the electron-doped and hole-doped HS phases become metallic.
Doped electrons partially occupy the minority spin-down $d_{xy}$ states, and doped holes resides on majority spin-up $d_{z^2}$ and $d_{x^2-y^2}$ states.
Given the metallicity of the hole-doped state, the stability of the FM ordering is most likely due to the double exchange mechanism.  We use the schematic band interaction diagrams of in Fig. \ref{HS-HS_dop} to explain this stabilization mechanism. As per Fig. \ref{HS-HS_dop}(a), the energy lowering of FM HS ordering is nonzero because of the holes in majority spin-up $d_{z^2}$ and $d_{x^2-y^2}$ bands. Defining the number of holes as $n^{h}_{d_{z^2}}$ and $n^{h}_{d_{x^2-y^2}}$, the energy lowering of the FM ordering  is
\begin{equation}
\begin{aligned}
\Delta E^{\textnormal{FM}} =
-2n^{h}_{x^2-y^2} \Delta^{11}_{x^2-y^2}  ,
\end{aligned}
\end{equation}
where the factor of two is due to the holes on both neighboring Co migrating to the shared anti-bonding $d_{x^2-y^2}$ state.  The energy lowering of the AFM ordering (strictly speaking, it is ferrimagnetic as discussed above) is
\begin{equation}
\begin{aligned}
\Delta E^{\textnormal{AFM}} =
&-\left( 2- 2n^{h}_{x^2-y^2} \right) \Delta^{12}_{x^2-y^2} \\
&-2 \Delta^{12}_{z^2}
-2\Delta^{12}_{xy} ,
\end{aligned}
\end{equation}

The energy difference between the two phases is thus
\begin{equation}
\begin{aligned}
\Delta E^{\textnormal{FM}} - \Delta E^{\textnormal{AFM}}   &= -2 n^{h}_{x^2-y^2} \left( \Delta^{11}_{x^2-y^2} + \Delta^{12}_{x^2-y^2} \right) \\
&+2 \left(  \Delta^{12}_{xy} +\Delta^{12}_{x^2-y^2} +\Delta^{12}_{z^2} \right)\,.
\end{aligned}
\end{equation}
This energy difference can change sign with increasing hole doping $n^{h}_{d_{x^2-y^2}}$.  Furthermore, since $\Delta^{11}_d$ energies are much larger than $\Delta^{12}_d$ energies, the value of $n^{h}_{d_{x^2-y^2}}$ at which it changes sign should be small in agreement with our above numerical findings.

\subsection{(LaTiO$_3$)$_2$+(LaCoO$_3$)$_2$ superlattice}
\label{sec:LTO2+LCO2}

\begin{figure*}
\begin{center}
\includegraphics[width=0.8\textwidth, angle=0]{2-2_configs.eps}
\caption{  5 possible HS/LS orderings, and 4 possible magnetic orderings; total possible configurations are 5$\times$4=20.
}
\label{2-2_configs}
\end{center}
\end{figure*}

\begin{figure}
\begin{center}
\includegraphics[width=0.49\textwidth, angle=0]{2-2_Udep.eps}
\caption{  $U$ dependence of energies for different spin configurations of (LTO)$_2$+(LCO)$_2$
with different in-plane lattice parameters. Energy of LS FM is set to be zero.
}
\label{2-2_Udep}
\end{center}
\end{figure}

We now turn to describe the electronic properties of (LCO)$_2$+(LTO)$_2$ superlattices.  Our computational supercells contain 4 Co and 4 Ti.  Since there is a great deal of shared physics between the 1+1 and 2+2 superlattices, we will only describe the main physical properties of the 2+2 superlattices while highlighting key differences: our overall approach and methods of analysis are idential for the two superlattices.

The Co$^{2+}$ cations in the (LCO)$_2$+(LTO)$_2$ superlattice can be HS or LS, similar to (LCO)$_1$+(LTO)$_1$, so that there are 5 possible  configurations HS/LS ordering in the supercell as presented in top row of Fig.~\ref{2-2_configs}: all HS order (shortened to HS), A-type HS/LS order with ordering wave vector $\bm{q_\textnormal{sc}} = \left(0,0,\frac{1}{2}\right)$ (A-HS/LS),
C-type HS/LS order with $\bm{q_\textnormal{sc}} = \left(\frac{1}{2},\frac{1}{2},0 \right)$ (C-HS/LS),
G-type HS/LS order with $\bm{q_\textnormal{sc}} = \left(\frac{1}{2},\frac{1}{2},\frac{1}{2}\right)$ (G-HS/LS),
and all LS order (LS).
Separately, there are 4 possible orderings patterns for the Co magnetic moments: FM, A-type AFM, C-type AFM, and G-type AFM (see second row of Fig.~\ref{2-2_configs}).  In total, we investigate all $5\times4=20$ configurations.

We summarize the $U_{\textnormal{Co}}$-dependent properties of the most relevant low-energy configurations in Fig.~\ref{2-2_Udep}.  For most choices of parameters, the full set of $U_{\textnormal{Co}}$-dependent data show that HS G-AFM is the ground state, except for the case of high compressive strain and small $U_{\textnormal{Co}}$ ($a$=3.663 \AA{} and $U_{\textnormal{Co}}$=2.5 eV)
where we find that G-HS/LS A-AFM phase is more stable (to avoid excessive clutter, Fig.~\ref{2-2_Udep}(a) only shows data for G-AFM magnetic ordering).
In general, the energies of the A-HS/LS, C-HS/LS, G-HS/LS phases lie between those of the LS and HS phases.
For a fixed magnetic pattern (Fig.~\ref{2-2_Udep}(a,c,e)),
the energy difference ($\Delta E$) between LS and HS phases is enhanced
as $U_{\textnormal{Co}}$ increases or in-plane lattice parameter $a$ increases which is similar to the behavior of (LTO)$_1$+(LCO)$_1$.
Again, like the behavior of the 1+1 superlattices described in Sec.~\ref{sec:spin_states}, the $U_{\textnormal{Co}}$-dependence of $\Delta E$ for 2+2 superlattices turns out to be explained by the behavior of the ordering energy $\Delta E_{\textnormal{ord}}$ (Eqs.~(\ref{eq:EDFTUspectral},\ref{eq:EfillEord})) between LS and HS phases.

In Figs.~\ref{2-2_Udep}(b,d,f), we compare the energies of different magnetic orderings as a function of $U_{\textnormal{Co}}$ when all then Co are  in the HS state.  The energies in this case obey the relations
\[
E^{\rm FM}[{\rm HS}] > E^\textnormal{A-AFM}[{\rm HS}] > E^\textnormal{C-AFM}[{\rm HS}] > E^\textnormal{G-AFM}[{\rm HS}],
\]
and interestingly this order holds for all $U_{\textnormal{Co}}$ and strain ranges  considered.
\begin{figure}
\begin{center}
\includegraphics[width=0.49\textwidth, angle=0]{HS-HS_oop.eps}
\caption{  Schematic diagram of the out-of-plane magnetic interaction between HS Co and HS Co.   }
\label{HS-HS_oop}
\end{center}
\end{figure}
As described in Fig.~\ref{HS-HS_ip} and its associated discussion, in-plane magnetic interactions stabilizes AFM spin alignment,
thus $E^\textnormal{FM}[\textnormal{HS}]$ and $E^\textnormal{A-AFM}[\textnormal{HS}]$ are above $E^\textnormal{C-AFM}[\textnormal{HS}]$ and $E^\textnormal{G-AFM}[\textnormal{HS}]$.

In order to explain why $E^\textnormal{FM}[\textnormal{HS}]> E^\textnormal{A-AFM}[\textnormal{HS}]$ and $E^\textnormal{C-AFM}[\textnormal{HS}]> E^\textnormal{G-AFM}[\textnormal{HS}]$,
i.e., the stability AFM order along the out-of-plane direction,
we examine the out-of-plane magnetic interaction for the HS phase in Fig.~\ref{HS-HS_oop}.
In the out-of-plane direction, only $d_{z^2}$, $d_{zx}$, and $d_{yz}$ on two neighboring Co will interact significantly.
While the FM alignment has no net energy lowering due to out-of-plane interactions (see Figs.~\ref{HS-HS_oop}(a,b)), the
interactions between the $d_{z^2}$ on the two Co lowers the energy by $-2\Delta^{12}_{d_{z^2}}$ for  AFM alignment (Figs.~\ref{HS-HS_oop}(c,d)).
The energy difference between the FM and AFM configurations due to out-of-plane interactions is
\begin{equation}
\begin{aligned}
\Delta E[\textnormal{HS}]
=\Delta E^\textnormal{AFM}[\textnormal{HS}]-\Delta E^\textnormal{FM}[\textnormal{HS}]  = -2 \Delta^{12}_{d_{z^2}}\,,
\end{aligned}
\end{equation}
so that we expect $E^\textnormal{AFM}[\textnormal{HS}] < E^\textnormal{FM}[\textnormal{HS}]$ and $E^\textnormal{G-AFM}[\textnormal{HS}] < E^\textnormal{C-AFM}[\textnormal{HS}]$.

\section{Summary and outlook}

LCO+LTO superlattices show strong electron transfer, significant structural distortion, and robust orbital polarization.  For a wide range of computational parameters, we predict high-spin Co and a checkerboard antiferromagnetic (AFM) order on the Co superlattice (2D checkerboard for the 1+1 superlattice and 3D checkerboard, i.e., G-type, for the 2+2 superlattice).  A detailed analysis of the electronic structure reveals how superexchange interactions control the stability of the magnetic orders.  Finally, hole doping of the superlattices can lead to stabilization of a ferromagnetic ground state of Co$^{2+}$.  This is unexpected and interesting especially for future experiments.  Generally, materials containing Co$^{2+}$ as the sole magnetic cation, such as CoO or Co$_3$O$_4$ (which also contains non-magnetic Co$^{3+}$ \cite{DuttaJPCM2007}), show antiferromagnetic ordering.  A ferrimagnetic compound such as CoFe$_2$O$_4$ contains Fe$^{3+}$ at the $A$ sites and Fe$^{3+}$/Co$^{2+}$ at $B$ sites which are aligned antiferromagnetically \cite{Stein2018}.  Hence, the ability to create a ferromagnet using high-spin Co$^{2+}$ is a new and surprising prediction.

Experimentally, we hope to see verification of the checkerboard AFM order of these superlattices.  Even more exciting will be the experimental realization of ferromagnetism via hole doping through replacement of La by Sr; by careful control of the doping, the FM and AFM order can be made nearly degenerate in energy, potentially leading to interesting magnetic response and new magnetic phases.

Theoretically, more advanced models such as DFT + dynamical mean field theory can be used to verify the predictions made here using GGA+$U$.    Furthermore, understanding the origin of the orbital polarization in these charge-transfer systems is interesting in terms of basic science as well as electronic engineering.  Finally, investigation of thicker LCO layers in the superlattice, i.e., (LTO)$_2$+(LCO)$_n$ for $n\ge3$, should prove  interesting  as well, since the interaction between Co$^{2+}$ at the interface and Co$^{3+}$ at the bulk-like region may lead to other unexpected physical phenomena.

\section{Acknowledgments}
We thank to Sangjae Lee, F. J. Walker, and Charles H. Ahn for helpful discussions. We thank the Yale Center for Research Computing for guidance and use of the research computing infrastructure.  This work also used the Extreme Science and Engineering Discovery Environment (XSEDE), which is supported by National Science Foundation grant number ACI-1548562, by using computer time on the Comet supercomputer as enabled by XSEDE allocation MCA08X007.


\section{Introduction}

In recent years, the performance of object detection has dramatically improved due to the emergence of object detection networks that utilize the structure of CNNs \cite{Liu18}. Owing to this improvement, CNN-based object detectors have potential practical applications such as video surveillance \cite{Jung17}, autonomous navigation \cite{Wu17}, machine vision \cite{Yao18}, and medical imaging \cite{Jin18}. Several industries have been making efforts to implement these technological advancements in conjunction with industrial applications.

\begin{figure}[t!]
  \centering
  \includegraphics[width=0.8\linewidth]{figs/fig1.pdf}
  \vspace{-2mm}
  \caption{\textbf{Experiment results on MS-COCO dataset with Refinement Detectors.} (a) Speed (fps) versus accuracy (mAP) on MS-COCO. Out models (\textcolor{red}{red color}) have a balanced speed and accuracy compared to the existing real-time object detection network-oriented models (\textcolor{mypurple}{purple color}). Both the speed and accuracy of the proposed model were measured on the NVIDIA Titan XP. Details of the performance measurements are described in Section 4. (b) Speed (fps) variation based on Titan XP and Drive PX2 systems with the different experimental settings. X-axis represents the experiment number as shown in Table 1, and Y-axis represents speed (fps) normalized by the largest fps among 50 experiments for each system. Each experiment showed that the variation in inference speed on the edge board was much more sensitive to the training setup and architecture choices.}
\vspace{-5mm}
\end{figure}

Object detection methods using the CNN structure are broadly classified into two types: The first is a learning method by classifying class information, which locates and classifies objects in an image using a one-stage training method, by including them into one network stream and dividing the encoded features into different depth dimensions. Representative models of this one-stage network are YOLO \cite{Redmon17}, single shot multibox detector (SSD) \cite{Liu17}, SqueezeDet \cite{Wu17}, and RetinaNet \cite{Lin17}. The advantage of these networks is that the encoder of the head part leading to the network output is constructed relatively intuitively, and the inference speed is high, while the accuracy of the bounding box regression, which specifies the exact position of the object, is generally lowered.

The second strategy is a two-stage learning method in which a network that classifies approximate positions and classes of objects is searched to obtain a more precise object location using a structure in which an independent encoder within the head is separated. These two-stage net-works have a region proposal network (RPN) that can more accurately determine the existence of objects in the head of the network. R-CNN \cite{Girshick14} is the first model using this two-stage learning approach and Faster R-CNN \cite{Ren15}, R-FCN \cite{Dai16}, Deformable ConvNets \cite{Dai17}, and Mask-RCNN \cite{He17} are some of the advanced versions. The advantage of the two-stage model is greater accuracy of the bounding box regression compared with the one-stage model. However, owing to its relatively complicated head structure, the inference and training time is longer than the one-stage model.

Therefore, models with modified overall detection network structures have been proposed to compensate for the disadvantages of the one-stage and the two-stage models \cite{Li17,Zhang18}. \cite{Li17} proposed a lightweight head structure to achieve an inference speed close to that of the one-stage models. Conversely, \cite{Zhang18} suggested a model that uses the advantage of the two-stage model while maintaining the overall architecture of the one-stage model. In recent years, due to the increasing research efforts, the performance of object detection using CNN has improved dramatically in terms of accuracy and speed; however, the proposed algorithms still have various practical limitations in real-time applications in terms of accuracy and inference speed, especially in AI edge devices using embedded platforms.

Herein, we present several variations of RefineDet (refinement detector) \cite{Zhang18}, which is one of state-of-the-art object detection networks. To demonstrate how their performances vary according to their parameter settings and backbone architectures, we compare the inference performances between \textit{on high-end GPU} and \textit{on embedded platforms}. Unlike the inference on high-end GPU systems, obtaining the optimal parameters and backbone architectures is considerably important in embedded systems. Figure 1 shows the performance variation according to parameter settings and backbone architectures. Particularly, models tested on Drive PX2 exhibit a higher speed variance than high-end GPU systems, implying that hyper-parameters must be carefully selected when embedding or deploying models using the RefineDet architecture into embedded platforms. The fastest model in high-end GPU systems is not the fastest in embedded platforms.

Meanwhile, to achieve real-time inference speed with limited resources, we examined the limitations of the existing networks and applied various lightweight backbone structures to the head structure of RefineDet \cite{Zhang18} by constructing optimal intermediate layers for an effective training. The utilized backbone CNNs include VGG \cite{Liu16} and shallow ResNet \cite{He16}, which are the most widely used neural networks in image recognition. ResNeXt \cite{Xie17}, Xception \cite{Chollet17}, and MobileNet \cite{Howard17,Sandler18}, which are known for their high computation efficiency compared with layers of the same depth, were used; SENet \cite{Hu18} was applied using the “reweighting by local encoding” structure to generate punchy convolutional feature maps. To confirm the effect of feature generation on the real-time performance of the head structure, extensive comparative experiments were performed by applying head structures with a reduced channel depth for the head structure of RefineDet. Finally, we verified and confirmed the real-time performance on the NVIDIA Drive PX2 and the Jetson Xavier edge platforms using the ResNet18-based RefineDet model of reduced number of parameters and specific NMS parameters (see Table 1).

The major technical contributions of this study can be summarized as follows: 1) We present real-time detection models for AI edge platforms using a modified head structure of RefineDet. 2) The object detection network using the latest lightweight backbone with connections of the RefineDet head structure is extensively compared and analyzed using the MS-COCO 2017 detection dataset \cite{Lin14}. 3) Performance tests and model analysis were performed on NVDIA Drive PX2 and Jetson Xavier to achieve a balanced performance in terms of accuracy and speed for real-time detection on edge platforms. In addition, we introduced some issues related to the object detection models on the embedded GPU and summarized the concerns that must be addressed for achieving high real-time performance. We hope that this study will be useful for the development of real-time applications using diverse CNN-based architectures on AI edge platforms.

\section{Baseline Architecture - RefineDet}
We used the basic structure of a single-shot refinement object detector (RefineDet) \cite{Zhang18} as the baseline architecture, which is derived from a one-stage learning structure of a single-shot multibox detector (SSD) \cite{Liu17}. The architectural weakness of the localization regressor of the one-stage detector was compensated using an anchor refinement module (ARM) branch, which functions similarly as the region proposal network in a two-stage detector. RefineDet minimizes the combined objective with two modules from the modified head structure of the former SSD architecture \cite{Zhang18}

The connection of the head structure of RefineDet to the backbone CNNs is divided into two branches. Among them, the ARM module is a type of region proposal classifier that learns to minimize the binary classification loss of objectness and supports multiclass inference with loss generation by the backpropagation of object existence and location information. The other branch comprises an object detection module (ODM) that is used to deduce the predicted confidence and localization bounding box information for multiclass objects. The entire network is trained such that the balanced loss for both branches is minimized. To utilize the concept of a feature pyramid through information coding between the upper and lower-layer features, \cite{Zhang18} proposed a connection structure of a transfer connection block (TCB) and intermediate layers. To verify the efficiency of the network, the most widely known structure of CNNs, i.e., VGG-16 and ResNet-101, are used as the backbone for the training and evaluation results \cite{Zhang18}.

\section{Combining Light-weight Backbone CNNs}

We combined state-of-the-art convolutional blocks of modified layers and structures with the lightweight backbone of the RefineDet architecture. Furthermore, we applied feature encoding blocks according to each model to the intermediate layer in the ODM branch for an efficient loss propagation between the head structure of RefineDet and the backbones. We applied VGG-16, ResNet-18, ReNeXt-26, ResNeXt-50, SE-ResNeXt-50, Inception-SENet, MobileNetV1, MobileNetV2, and Xception to the learning architecture. Furthermore, we used the feature encoding block of each backbone model as the intermediate layers. All backbone architectures were initialized through pre-training using ImageNet data \cite{Russakovsky15}. To evaluate the tradeoff between inference accuracy and speed enhancement of the detection networks, the capability of the feature pyramid was validated. Hence, we verified the original (256) and reduced (128) channel depths of the TCB and intermediate layers, and we tested additional convolutional layers. Herein, the RefineDet model with reduced channel depth is known as the “reduced RefineDet (rRefineDet).”
\\\\
\noindent \textbf{VGG-16 \cite{Liu16}}. Based on the VGG-16 model proposed in \cite{Liu16}, the $fc6$ and $fc7$ layers were transformed into the convolution layers, $conv\_fc6$ and $conv\_fc7$ respectively, through the subsampling parameters as shown in \cite{Zhang18}. To combine with the head structure of RefineDet, the subsequent layers including the last pooling layer of VGG-16 were removed and convolutions $conv6\_1$ and $conv6\_2$ were added to the top as additional convolution layers. As in \cite{Zhang18}, L2 normalization was used for the intermediate layers and some convolution layers.
\\\\
\noindent \textbf{ResNet-18 \cite{He16}}. The authors of \cite{Zhang18} applied the ResNet-101 architecture as the baseline backbone to improve the accuracy of RefineDet. In our study, RefineDet was learned using ResNet-18 pretrained by ImageNet for real-time inference as the backbone architecture. For the high-level feature encoding of backbone CNNs, the $res6$ block was added after the $res5$ block (similar to VGG-16), and the intermediate layer in the ODM branch used the residual encoding block with a channel depth of 256 in ResNet-18. For other parameter settings such as batch normalization (BN) and activations, learning was performed under the same conditions as those for ResNet-101 in \cite{Zhang18}.
\\\\
\noindent \textbf{ResNeXt-26 and 50 \cite{Xie17}}. ResNeXt is a CNN structure that uses group convolution to improve the efficiency of computation for aggregated residual transformations using identity mapping. According to \cite{Xie17}, its computational efficiency is higher than that of ResNet with the same depth, and it performed better for ImageNet data even though it uses only a few weight parameters. To effectively combine with the RefineDet head for ResNeXt-26, the outputs of $resx4$, $resx6$, and $resx8$ were used as the ARM and ODM outputs among eight $resx$ blocks. Furthermore, $resx9$ was used to process high-level features similar to those of ResNet. We added the features to the upper layer, and it could learn simultaneously by scratching. Regarding ResNeXt-50, $resx7$, $resx13$ and $resx16$ blocks of 16 $resx$ blocks were combined with the head of RefineDet, and $resx17$  was added to the upper layer. Regarding the intermediate layer, each output feature comprised the input for the feature pyramid encoded by one $resx$ block.
\\\\
\noindent \textbf{SE-ResNeXt-50 and Inception \cite{Hu18}.}
SENet is a CNN configured to allow channel reweighting of the convolutional feature by applying a squeeze and excitation (SE) module to the output of the convolutional layer. The SE branch facilitates transformation in the depth dimension through channel-wise 1D encoding of the processed output feature. In our work, ResNeXt-50 and Inception towers with SE module were combined with the head of RefineDet and the performances of these models were verified. For SE-ResNeXt-50, the squeeze and excitation module was applied to all convolutional layers except the $conv1$ layer. To combine the heads, the output features of $conv3\_4$, $conv4\_6$, and $conv5\_3$ layers were used as inputs to the AMR and ODM, and $conv6$ blocks were added to the top layer. The channel depth of the intermediate layer was 256, and that of the final output depth of the $conv6$ block was 256. Inception-SENet comprised 10 inception blocks after the first convolution layer ($conv1$), max pooling, and the second convolution layer ($conv2$). Among the feature outputs obtained when Inception-SENet was used as the backbone CNN structure, $inception\_3b$, $inception\_4d$, $inception\_5b$ and RefineDet head were connected and the $inception\_6$ block was added to the top layer. The final output of the $inception\_6$ block had a channel depth of 256, and the intermediate layer 256, which was the same as that of SE-ResNeXt-50.
\\\\
\noindent \textbf{Xception \cite{Chollet17}.} Xception applies a depth-wise separable convolution to the inception tower to re-evaluate the learning efficiency using CNNs of the same structure as InceptionV3 \cite{Szegedy16}. The authors of \cite{Chollet17} proposed a network structure with a higher accuracy and better inference speed than InceptionV3, based on the fact that the “extreme” structure of Inception is almost equivalent in operation to the depth-wise separable convolution. To combine this Xception structure with the head of RefineDet, we used the outputs of $xception11$, $xception12$, and $conv4\_2$ as inputs to the ARM and ODM branches and added an additional $xception$ block ($xception13$) with a channel output depth of 256 to the top layer. Fur-thermore, we applied an $xception$ block with a channel depth of 256 in the intermediate layer.
\\\\
\noindent \textbf{MobileNetV1 \cite{Howard17} and V2 \cite{Sandler18}.}  In MobileNetV1, the cost of computation is minimized by applying depthwise separable convolutions such as in \cite{Chollet17} to reduce the operation cost from a typical convolution layer structure. In MobileNetV2, the depthwise separable convolution, which affects the computation time, is utilized as is, and the linear bottleneck structure is applied to minimize the burden on performance degradation. For MobileNetV1, we used the outputs of $conv4\_1$, $conv5\_5$, and $conv6$ to connect them to the head of RefineDet and added a convolution block with a channel depth of 512 to the top layer. Considering that the number of parameters of the backbone CNN structure is relatively smaller than that of the other architectures, we attempted to maintain the depth of the intermediate layers at the same feature volume as that of a basic convolutional block. Depth-wise separable convolution was applied to the top convolution block and intermediate layer. Regarding MobileNetV2, the outputs of $conv3\_2$, $conv4\_7$ and $conv6\_4$ were used as the feature outputs for the coupling with head, and the $conv7$ block was added at the top. The output channel depth of $conv7$ was 96 after narrowing down and the intermediate layer was of the same size.

\begin{figure}[t!]
  \centering
  \includegraphics[scale=0.3]{figs/fig3.pdf}
  \caption{\textbf{NVIDIA Drive PX2 (left) and Jetson Xavier (right).} Both AI edge devices support embedded GPU based parallel computing with Pascal (Drive PX2) and Volta (Jetson Xavier) architectures.}
  \vspace{-5mm}
\end{figure}

\section{Evaluation Environments}

\noindent \textbf{Dataset Preparation.} We used the MS-COCO 2017 object detection dataset \cite{Lin14} to evaluate the performances of various RefineDet networks. The dataset comprises 80 object classes, approximately 120,000 training images ($train17$), 5,000 validation images ($val17$), 80,000 test images ($test17$), and 120,000 unlabeled images ($unlabeled17$). For the evaluation of the proposed models, all models were trained with $train17$. Moreover, $val17$ and $test\textnormal{-}dev17$ were used as test sets, separately. For the quantitative evaluations of $val17$ and $test\textnormal{-}dev17$, the mean average precision (mAP) value was used as the evaluation criterion. The final mAP was calculated as the mean value obtained from a range of intersection over union (IoU) with $[0.5: 0.05: 0.95]$.
\\\\
\noindent \textbf{Plaforms.}
We used Titan XP, Drive PX2, and Jetson Xavier as our testing platform. Both platforms were developed for AI edge computing, and specifically for DrivePX2, they are designed for application to autonomous vehicles. We performed training on the Titan XP, Drive PX2, and Jetson Xavier systems. Drive PX2 is based on the Tegra X2 SoC board and contains 12 CPU cores: eight of A57 and four of Denver. The Pascal architecture of the GPU processor is based on the 16FinFET process and supports UART, CAN, LIN, FlexRay, USB, and 1 or 10 Gbit Ethernet communication (see Figure 2). Jetson Xavier is a recently released embedded architecture based on a 512-Core Volta GPU with tensor cores and an 8-core ARM CPU. This architecture has 16 GB 256-bit of LPDDR4x memory and 32 GB of eMMC 5.1 flash storage and supports (2x) NVDLA DL accelerator engines.
\\\\
\noindent \textbf{Training.} For training each model quickly, we used the pretrained weights as the initialization from ImageNet. The top layers of the convolution blocks, intermediate layers, and TCB of the RefineDet structure for the feature pyramid were initialized to a random Gaussian distribution with $\sigma = 0.01$. For a fair performance evaluation of the proposed models, the same parameters related to the detection model (anchor size, IoU with ground truth box, types of training data augmentation, etc.) were used except for the modification of the backbone and head structures. In addition, to confirm the role of the last convolutional block for high-level feature processing and the TBC for the feature pyramid, the channel depth of each feature map was divided into two cases of 128 and 256. The total learning duration was 120 epochs, and stochastic gradient descent was used as an optimizer. The base learning rate was started at 0.001, and a drop rate of 0.1 to 84 and 108 epochs was applied. The weight decay was 0.0005 and the momentum was 0.9. All the learning was performed in the Caffe environment using the Python interface. The depthwise separable convolution utilized the implementation code in the Caffe environment available to the public and the original code for the customizing layer of the SE module. In order to evaluate the performance of the model under various conditions, we trained different models according to the feature depth and input image size for TCB and top layers.
\\\\
\noindent \textbf{Testing.} We performed testing process using $val17$ and $test\textnormal{-}dev17$ datasets for each model. Additionally, we used several non-maxima suppression (NMS) parameters to demonstrate the effectiveness of the NMS according to the system. rRefineDet is a training model that focuses on the inference speed and limits the size of the feature depths of the TCB, intermediate layer, and top layer. As a reference, the evaluation results of MS-COCO $test\textnormal{-}dev14$ data are included to show the results of original four RefineDet models reported in \cite{Zhang18}. Unlike our experiments, the experiments reported in the original RefineDet paper were performed on a Titan X GPU. Therefore, a lower fps was indicated compared with our experiments even when the same model and parameter settings were used. The average for every inference speed was obtained based on 5,000 input images ($10$ for warm-up).

\section{Results and Discussions}

\begin{table*}
  \caption{Performance change in different output feature depths from top layers of the high-level convolutional block, TBC and intermediate layer and difference in inference speed according to the NMS parameter on each platform. rRefineDet is a model that minimizes the feature output depth of the top convolutional blocks, TBC, and intermediate layers. NMS parameters in the NMS column include the maximum number of candidate bounding boxes for the NMS input, the maximum number of predicted bounding boxes according to the NMS output, and the confidence threshold for the final output. Every model was trained using $train17$ and tested on the Drive PX2 embedded platform. $td$ means $test\textnormal{-}dev$. The performance results of the first four models were extracted from the original RefineDet paper \cite{Zhang18}. Red, blue, and green colors represent the 1st, 2nd, and 3rd fastest models among 50 experiments, respectively.}
  \label{tab:freq}
  \resizebox{\textwidth}{!}{
  \begin{tabular}{c|c|c|c|c|c|c|c}

    Exp. No. & Model & Backbone & NMS parameters & mAP ($val17$) & mAP ($td17$) & fps (Titan XP) & fps (Drive PX2) \\ \hline\hline

\cite{Zhang18} & RefineDet320  & VGG-16 & (1000, 500, 0.01) & - & 29.4 ($td14$) & 40.3 (Titan X) & -\\
\cite{Zhang18} & RefineDet320  & VGG-16 & (1000, 500, 0.01) & - & 33 & 24.1 (Titan X) & -\\
\cite{Zhang18} & RefineDet320  & ResNet-101 & (1000, 500, 0.01) & - & 32 & 9.3 (Titan X) & -\\
\cite{Zhang18} & RefineDet320  & ResNet-101 & (1000, 500, 0.01) & - & 36.4 & 5.2 (Titan X) & -\\ \hline
    1 & rRefineDet320 & VGG-16 & (400, 200, 0.1) & 31.8 & 36.1& \textcolor{green}{\textbf{60.2}} &  \textcolor{green}{\textbf{18.8}} \\
    2 & rRefineDet320 & VGG-16 & (1000, 500, 0.01) & 32.7 & 37.2 & 56.5 & 10.9 \\
    3 & RefineDet320 & VGG-16 & (400, 200, 0.1) & 32.2  & 36.4 & \textcolor{blue}{\textbf{61.3}} & 16.7 \\
    4 & RefineDet320 & VGG-16 & (1000, 500, 0.01) & 33.7 & 37.3 & \textcolor{red}{\textbf{62.1}} & 9.7 \\
    5 & rRefineDet512 & VGG-16 & (400, 200, 0.1) & 34.4 & 40.7 & 38.0 & 8.9 \\
    6 & rRefineDet512 & VGG-16 & (1000, 500, 0.01) & 35.5 & 41.7 & 40.3 & 5.8 \\
    7 & RefineDet512 & VGG-16 & (400, 200, 0.1) & 35.4 & 40.9 & 35.5 & 8.2 \\
    8 & RefineDet512 & VGG-16 & (1000, 500, 0.01) & 36.4 & 42.0 & 33.9 & 5.6 \\ \hline
    9 & rRefineDet320 & ResNet-18 & (400, 200, 0.1) & 31.0 & 33.5 & 52.9 & \textcolor{red}{\textbf{21.6}} \\
    10 & rRefineDet320 & ResNet-18 & (1000, 500, 0.01) & 31.7 & 34.3 & 50.3 & 8.9 \\
    11 & RefineDet320 & ResNet-18 & (400, 200, 0.1) & 32.4 & 34.2 & 59.9 & \textcolor{blue}{\textbf{19.6}} \\
    12 & RefineDet320 & ResNet-18 & (1000, 500, 0.01) & 33.6 & 35.1 & 56.2 & 9.5 \\
    13 & rRefineDet512 & ResNet-18 & (400, 200, 0.1) & 35.8 & 40.2 & 44.4 & 13.6 \\
    14 & rRefineDet512 & ResNet-18 & (1000, 500, 0.01) & 36.2 & 40.5 & 40.7 & 6.2 \\
    15 & RefineDet512 & ResNet-18 & (400, 200, 0.1) & 36.6 & 40.3 & 35.7 & 12.2 \\
    16 & RefineDet512 & ResNet-18 & (1000, 500, 0.01) & 37.6 & 41.5 & 39.1 & 6.4 \\ \hline
    17 & rRefineDet320 & MobileNetV1 & (400, 200, 0.1) & 26.2 & 30.0 & 50.8 & 18.7 \\
    18 & rRefineDet320 & MobileNetV1 & (1000, 500, 0.01) & 27.2 & 30.8 & 46.9  & 6.9 \\
    19 & RefineDet320 & MobileNetV1 & (400, 200, 0.1) & 28.2 & 31.3 & 45.7 & 17.0 \\
    20 & RefineDet320 & MobileNetV1 & (1000, 500, 0.01) & 30.0 & 32.0 & 40.5 & 4.5 \\ \hline
    21 & rRefineDet320 & MobileNetV2 & (400, 200, 0.1) & 26.7 & 30.8 & 41.5 & 13.9 \\
    22 & rRefineDet320 & MobileNetV2 & (1000, 500, 0.01) & 27.5 & 31.1 & 38.8 & 7.1 \\
    23 & RefineDet320 & MobileNetV2  & (400, 200, 0.1) & 28.5 & 32.2& 37.2 & 14.7 \\
    24 & RefineDet320 & MobileNetV2  & (1000, 500, 0.01) & 29.2 & 33.0 & 39.7 & 6.7 \\ \hline
    25 & rRefineDet320 & Inception-SENet & (400, 200, 0.1) & 33.1 & 32.5 & 40.3 & 11.5 \\
    26 & rRefineDet320 & Inception-SENet & (1000, 500, 0.01) & 33.2 & 28.3 & 40.3 & 7.3 \\
    27 & RefineDet320 & Inception-SENet & (400, 200, 0.1) & 34.2 & 35.0 & 31.3 & 9.2 \\
    28 & RefineDet320 & Inception-SENet & (1000, 500, 0.01) & 35.2 & 35.8 & 32.1 & 6.1 \\
    29 & rRefineDet512 & Inception-SENet & (400, 200, 0.1) & 37.4 & 41.8 & 21.6 & 4.7 \\
    30 & rRefineDet512 & Inception-SENet & (1000, 500, 0.01) & 37.7 & 42.1 & 21.1 & 3.9 \\
    31 & rRefineDet320 & SEResNeXt-50 & (400, 200, 0.1) & 35.2 & 37.2 & 28.9 & 9.9 \\
    32 & rRefineDet320 & SEResNeXt-50 & (1000, 500, 0.01) & 36.1 & 38.2 & 27.9 & 7.3 \\ \hline
    33 & rRefineDet320 & ResNeXt-26 & (400, 200, 0.1) & 30.5 & 34.9 & 36.0 & 11.8 \\
    34 & rRefineDet320 & ResNeXt-26 & (1000, 500, 0.01) & 31.3 & 35.7 & 29.9 & 6.8 \\
    35 & RefineDet320 & ResNeXt-26 & (400, 200, 0.1) & 32.1 & 35.7 & 32.7 & 11.0 \\
    36 & RefineDet320 & ResNeXt-26 & (1000, 500, 0.01) & 32.8 & 36.6 & 33.8 & 6.9 \\
    37 & rRefineDet512 & ResNeXt-26 & (400, 200, 0.1) & 34.4 & 39.5 & 25.1 & 6.9 \\
    38 & rRefineDet512 & ResNeXt-26 & (1000, 500, 0.01) & 35.4 & 40.5 & 25.2 & 4.6 \\
    39 & RefineDet512 & ResNeXt-26 & (400, 200, 0.1) & 35.2 & 40.3 & 24.8 & 6.4 \\
    40 & RefineDet512 & ResNeXt-26 & (1000, 500, 0.01) & 36.2 & 41.3 & 24.0 & 4.7 \\ \hline
    41 & rRefineDet320 & Xception & (400, 200, 0.1) & 34.6 & 37.2 & 37.9 & 10.9 \\
    42 & rRefineDet320 & Xception & (1000, 500, 0.01) & 35.0 & 38.0 & 36.0 & 6.1 \\
    43 & RefineDet320 & Xception & (400, 200, 0.1) & 34.9 & 37.8 & 35.8 & 10.4 \\
    44 & RefineDet320 & Xception & (1000, 500, 0.01) & 35.7 & 38.9 & 33.8 & 5.2 \\ \hline
    45 & rRefineDet320 & ResNeXt-50 & (400, 200, 0.1) & 35.6 & 37.6 & 32.3 & 10.2 \\
    46 & rRefineDet320 & ResNeXt-50 & (1000, 500, 0.01) & 36.1 & 38.4 & 31.3 & 7.4 \\
    47 & RefineDet320 & ResNeXt-50 & (400, 200, 0.1) & 36.7 & 38.0 & 29.8 & 9.6 \\
    48 & RefineDet320 & ResNeXt-50 & (1000, 500, 0.01) & 37.6 & 38.4 & 28.1 & 7.0 \\
    49 & rRefineDet512 & ResNeXt-50 & (400, 200, 0.1) & 36.9 & 42.5 & 21.1 & 5.4 \\
    50 & rRefineDet512 & ResNeXt-50 & (1000, 500, 0.01) & 37.8 & 43.5 & 20.6 & 4.0 \\
\end{tabular}
}
\end{table*}

\noindent \textbf{Changing Backbone Architecture.}
As shown in Table 1, changing the structure of the feature connection blocks between the head and backbone affected the performance. Models with VGG-16, ResNet-18 as the backbone structure demonstrated superior inference speed compared with the ResNeXt-26, MobileNetV1, and V2 models. Particularly, VGG-16 was the fastest architecture on Titan XP, but ResNet-18 was the fastest on Drive PX2. By combining the head structure of the existing object detection networks and CNN models, a balanced performance can be achieved, which is applicable to various fields. Moreover, the quantitative performance can be enhanced by setting a different input size for the image and applying the deformable operation to a specific convolutional layer according to the available computing resources. In addition, it is clear that using the improved convolutional block such as feature renormalization of the SE module and the inverted residual structure of MobileNetV2 in a specific head and backbone structure helps to prevent speed degradation and improve accuracy.
\\\\
\noindent \textbf{Effectiveness of NMS Parameters.} As shown in Table 1, a slight adjustment of the NMS parameter resulted in a considerable improvement in the inference speed at the expense of a low accuracy. Furthermore, unexpected computational bottlenecks appeared when applying a state-of-the-art algorithm based on CNNs in a Drive PX platform. Layer-wise inference testing was performed on all layers of the RefineDet architecture to analyze the cause. We discovered that the computation times of all layers operating on the embedded GPU increased linearly compared with the number of high-end GPU cores, but the post-processing for the bounding box occupied most of the bottlenecks. Because the model is designed to process bounding box filtering related to the NMS operation of the post-processing layer, the CPU operations on a certain computing platform causes a severe performance degradation compared with high-end GPU systems rich in CPU computing resources. Table 1 shows that performance degradation owing to the CPU operation bottleneck can cause a significant performance degradation not only in embedded environments, but also in general computing resources. Furthermore, Table 1 shows the comparison results of the inference speed on various platforms according to the model structure and NMS parameters. To reduce the burden of the CPU operations, the input bounding box for NMS is set to a maximum of 400, and the confidence threshold for the output to the NMS is increased to 0.1. Although the accuracies are not significantly different, the performance gain in the Drive PX2 environment is extremely high owing to the adjustment of the NMS parameter, which is not significant in the high-end GPU system environment. The tradeoff between speed and performance can be improved significantly by identifying the location of the computational burden in the architecture and adjusting the hyperparameter for the related operation.
\\\\
\noindent \textbf{NVIDIA Drive PX2 vs. Jetson Xavier.}
As our experiments were primarily performed on Drive PX2, we also validated how RefineDet operates on different edge platforms. First, we select two architectures of VGG-16 and ResNet-18 for the comparison, because those backbones are the fastest models on Titan XP and Drive PX2, respectively. Interestingly, ResNet-18, which is the fastest model for Drive PX2, is also the fastest model for Jetson Xavier. This confirms that the fastest network in a high-end system can produce different results on an embedded board. Meanwhile, although the hardware specifications of Drive PX2's GPU are better than those of Jetson Xavier’s GPU, Jetson Xavier shows similar or better performances in some models. We conjecture that this is because Jetson Xavier contains a deep learning accelerator. Finally, the speed variation is more stable in Jetson Xavier, which appears to have a more stable hardware architecture for a balanced computation resource between CPU and GPU cores.
\\\\
\noindent \textbf{mAP with SOTA detectors.} For reference to detection accuracy, Table 3 shows mAPs with state-of-the-art models. We note that depending on the training setup of each model, different accuracy can be achieved despite the same model.

\begin{table}[h]
\vspace{-3mm}
  \caption{Inference speed of NVIDIA Drive PX2 vs. Jetson Xavier (Jetson-X). Index of experiments is the same as that in Table 1.}\vspace{-2mm}
  \label{tab:freq}
  \resizebox{\linewidth}{!}{
  \begin{tabular}{c|c|c|c|c}
\hline
   Exp. No. & Model & Backbone & fps (Drive PX2) & fps (Jetson-X)\\ \hline

    1 & rRefineDet320 & VGG-16 & \textcolor{green}{\textbf{18.8}} & 13.6\\
    2 & rRefineDet320 & VGG-16 & 10.9 & 10.7\\
    3 & RefineDet320 & VGG-16 & 16.7 & 12.5\\
    4 & RefineDet320 & VGG-16 & 9.7 & 11.4\\
    5 & rRefineDet512 & VGG-16 & 8.9 & 6.6\\
    6 & rRefineDet512 & VGG-16 & 5.8 & 6.0\\
    7 & RefineDet512 & VGG-16 & 8.2 & 7.6\\
    8 & RefineDet512 & VGG-16 & 5.6 & 6.9\\ \hline
    9 & rRefineDet320 & ResNet-18 & \textcolor{red}{\textbf{21.6}} & \textcolor{red}{\textbf{22.1}}\\
    10 & rRefineDet320 & ResNet-18 & 8.9 & \textcolor{blue}{\textbf{18.5}}\\
    11 & RefineDet320 & ResNet-18 & \textcolor{blue}{\textbf{19.6}} &  \textcolor{green}{\textbf{17.0}}\\
    12 & RefineDet320 & ResNet-18 & 9.5 & 16.6\\
    13 & rRefineDet512 & ResNet-18 & 13.6 & 11.9\\
    14 & rRefineDet512 & ResNet-18 &6.2 & 11.5\\
    15 & RefineDet512 & ResNet-18 & 12.2 & 11.2\\
    16 & RefineDet512 & ResNet-18 & 6.4 & 11.1\\
\hline
\end{tabular}
}
\vspace{-3mm}
\end{table}
\begin{table}
  \caption{Accuracy of state-of-the-art detectors using the MS-COCO dataset. The order is sorted by accuracy and speed. $td$ means $test\textnormal{-}dev$. We note that even on the same architecture, performance differences may occur due to different settings.}
  \label{tab:freq}
  \resizebox{.49\textwidth}{!}{
  \begin{tabular}{c|c|c|c}
    Model&Backbone& training & mAP (data) \\ \hline
    Fast R-CNN \cite{Girshick15} & VGG-16 & $train14$ & 19.7 ($td14$) \\
    Faster R-CNN \cite{Ren15} & VGG-16 & $trainval14$ & 21.9 ($td14$) \\
    R-FCN \cite{Dai16} & ResNet-101 & $trainval14$ & 29.9 ($td14$) \\
    Def. R-FCN \cite{Dai17} & ResNet-101 & $trainval14$ & 34.5 ($td14$) \\
    Def. R-FCN \cite{Dai17} & Inception-ResNet & $trainval14$ & 37.5 ($td14$) \\
    umd\_det \cite{Bodla17} & ResNet-101 & $trainval14$ & 40.8 ($td14$) \\
    G-RMI \cite{Huang17} & Ensemble & $trainval14$ (32k) & 41.6 ($td14$) \\
    \hline
    SSD321 \cite{Liu17} & ResNet-101 & $trainval14$ (35k) & 28.0 ($td14$) \\
    DSSD321 \cite{Fu17} & ResNet-101 & $trainval14$ (35k) & 28.0 ($td14$) \\
    SSD513 \cite{Liu17} & ResNet-101 & $trainval14$ (35k) & 31.2 ($td14$) \\
    YOLOv2 \cite{Redmon17} & Darknet-19 & $trainval14$ (35k) & 31.6 ($td14$) \\
    RetinaNet500 \cite{Lin17} & ResNet-101 & $trainval14$ (35k) & 32.0 ($td14$) \\
    DSSD513 \cite{Fu17} & ResNet-101 & $trainval14$ (35k) & 33.2 ($td14$) \\
    RetinaNet800 \cite{Lin17} & ResNet-101-FPN & $trainval14$ (35k) & 36.4 ($td14$) \\
    \hline
    RefineDet320 \cite{Zhang18} & VGG-16 & $trainval14$ (35k) & 29.4 ($td14$) \\
    RefineDet320 \cite{Zhang18} & ResNet-101 & $trainval14$ (35k) & 32.0 ($td14$) \\
    RefineDet512 \cite{Zhang18} & VGG-16 & $trainval14$ (35k) & 33 ($td14$) \\
    RefineDet512 \cite{Zhang18} & ResNet-101 & $trainval14$ (35k) & 36.4 ($td14$) \\
    \hline
    RefineDet320 & MobileNetV2 & $train17$ & 33.0 ($td17$) \\
    RefineDet320 & ResNet-18 & $train17$ & 35.1 ($td17$) \\
    RefineDet320 & Inception-SENet & $train17$ & 35.8 ($td17$) \\
    RefineDet320 & ResNeXt-26 & $train17$ & 36.6 ($td17$) \\
    RefineDet320 & VGG-16 & $train17$ & 37.2 ($td17$) \\
    RefineDet320 & Xception & $train17$ & 38.0 ($td17$) \\
    rRefineDet320 & SEResNeXt-50  & $train17$ & 38.2 ($td17$) \\
    RefineDet512 & ResNeXt-26  & $train17$ & 41.3 ($td17$) \\
    RefineDet512 & ResNet-18 & $train17$ & 41.5 ($td17$) \\
    RefineDet512 & VGG-16 & $train17$ & 42.0 ($td17$) \\
    rRefineDet512 & Inception-SENet  & $train17$ & 42.1 ($td17$) \\
    rRefineDet512 & ResNeXt-50 & $train17$ & 43.5 ($td17$) \\
\end{tabular}
}
\end{table}

\section{Conclusions and Future Works}

In this study, we conducted several experiments and discovered that the choice of NMS parameters and the backbone architecture in AI edge hardware were important. Results indicated real-time performances (approximately 20 fps) on embedded platforms such as NVIDIA Drive PX2 and Jetson Xavier boards. For future studies, it is necessary to closely analyze the relationship between the characteristics of the backbone layer of our models and those of the head structure. Hence, it is necessary to perform an ablation study on each backbone structure and TBC layer or intermediate layer to analyze the importance of each relation about connecting to the head. Furthermore, we intend to apply half or mixed precision techniques such as TensorRT to obtain better optimizations on platforms with limited resources.